Rerouting the Stream: How Lyft Moved to the Apache Flink Operator

Summary

Lyft migrated its fleet of hundreds of critical streaming jobs from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes operator. The legacy system imposed maintenance burden, feature gaps, outdated dependencies, fragile savepoint handling, and a rigid memory model that complicated varied workloads, including Apache Beam applications. Lyft used an incremental migration designed to preserve deployment patterns and team workflows while addressing CRD translation, state-machine behavior, and operational compatibility. The new platform added explicit last-state upgrades, autoscaling, automatic memory tuning, and safer upgrade mechanisms, though autotuning required trade-offs with in-place autoscaling and node capacity planning. After the multi-quarter transition, maintenance burden was minimized, the autoscaler right-sized a fleet previously overprovisioned by a few million dollars annually, and Lyft could contribute fixes upstream instead of maintaining its own operator.

Takeaways

1. 1

   The legacy operator used savepoints as the primary state-preservation mechanism, with fragile failure handling; the Apache operator’s explicit last-state upgrade mode uses high-availability metadata or the latest checkpoint by design.

2. 2

   Autotuning and in-place autoscaling required a trade-off: business-critical applications kept in-place autoscaling without autotuning, while less critical applications could use both with restart-based changes.

3. 3

   Autotuning disrupted Lyft’s static node-pool assumptions, prompting adoption of Karpenter to provision EC2 nodes from actual pod requirements and improve utilization.