---
title: "Rerouting the Stream: How Lyft Moved to the Apache Flink Operator"
description: "Lyft migrated its fleet of hundreds of critical streaming jobs from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes operator. The legacy system imposed maintenance burden, feature gaps, outdated dependencies, fragile savepoint handling, and a rigid memory model that complicated varied workloads, including Apache Beam applications. Lyft used an incremental migration designed to preserve deployment patterns and team workflows while addressing CRD translation, state-machine behavior, and operational compatibility. The new platform added explicit last-state upgrades, autoscaling, automatic memory tuning, and safer upgrade mechanisms, though autotuning required trade-offs with in-place autoscaling and node capacity planning. After the multi-quarter transition, maintenance burden was minimized, the autoscaler right-sized a fleet previously overprovisioned by a few million dollars annually, and Lyft could contribute fixes upstream instead of maintaining its own operator."
---

# Rerouting the Stream: How Lyft Moved to the Apache Flink Operator

[Lyft](https://yomu.fyi/company/lyft) · Maheep Myneni · Aug 31, 2026

**Type:** Problem & solution

## Summary

Lyft migrated its fleet of hundreds of critical streaming jobs from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes operator. The legacy system imposed maintenance burden, feature gaps, outdated dependencies, fragile savepoint handling, and a rigid memory model that complicated varied workloads, including Apache Beam applications. Lyft used an incremental migration designed to preserve deployment patterns and team workflows while addressing CRD translation, state-machine behavior, and operational compatibility. The new platform added explicit last-state upgrades, autoscaling, automatic memory tuning, and safer upgrade mechanisms, though autotuning required trade-offs with in-place autoscaling and node capacity planning. After the multi-quarter transition, maintenance burden was minimized, the autoscaler right-sized a fleet previously overprovisioned by a few million dollars annually, and Lyft could contribute fixes upstream instead of maintaining its own operator.

## Context

Lyft’s internally developed operator accumulated technical debt, outdated dependencies, feature gaps, fragile savepoint handling, and an inflexible memory model as the streaming fleet and workload complexity grew. The migration also had to avoid disrupting existing deployment patterns, team workflows, and production workloads.

## Approach / What changed

Lyft moved the fleet incrementally to the Apache Flink Kubernetes operator, adapting deployment configuration and handling state-machine, CRD, and memory-model differences. It adopted features including last-state upgrades, autoscaling, autotuning, Karpenter, and a two-tier policy that balances autotuning against in-place autoscaling according to application criticality.

## Takeaways

- The legacy operator used savepoints as the primary state-preservation mechanism, with fragile failure handling; the Apache operator’s explicit last-state upgrade mode uses high-availability metadata or the latest checkpoint by design.
- Autotuning and in-place autoscaling required a trade-off: business-critical applications kept in-place autoscaling without autotuning, while less critical applications could use both with restart-based changes.
- Autotuning disrupted Lyft’s static node-pool assumptions, prompting adoption of Karpenter to provision EC2 nodes from actual pod requirements and improve utilization.

**Tags:** [Apache Flink](https://yomu.fyi/topic/apache-flink), [Autoscaling](https://yomu.fyi/topic/autoscaling), [AWS](https://yomu.fyi/topic/aws), [Kubernetes](https://yomu.fyi/topic/kubernetes)

- Source: [Lyft](https://eng.lyft.com/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-36f20246d250)
- Source URL: https://eng.lyft.com/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-36f20246d250
- Ingested by Yomu: 2026-08-31T20:00:48.125Z

[Read original post](https://eng.lyft.com/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-36f20246d250)