Apache Flink

Lyft · Aug 31, 2026

Rerouting the Stream: How Lyft Moved to the Apache Flink Operator

Lyft migrated its fleet of hundreds of critical streaming jobs from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes operator. The legacy system imposed maintenance burden, feature gaps, outdated dependencies, fragile savepoint handling, and a rigid memory model that complicated varied workloads, including Apache Beam applications. Lyft used an incremental migration designed to preserve deployment patterns and team workflows while addressing CRD translation, state-machine behavior, and operational compatibility. The new platform added explicit last-state upgrades, autoscaling, automatic memory tuning, and safer upgrade mechanisms, though autotuning required trade-offs with in-place autoscaling and node capacity planning. After the multi-quarter transition, maintenance burden was minimized, the autoscaler right-sized a fleet previously overprovisioned by a few million dollars annually, and Lyft could contribute fixes upstream instead of maintaining its own operator.