A Tale of Two Flink Autoscalers

Summary

Netflix operates over 30,000 Apache Flink jobs across multiple AWS regions, ranging from managed pipelines to complex stateful DAGs. Its initial in-house autoscaler used external telemetry from Mantis and Atlas to scale TaskManagers as a single unit, cutting resource usage by 25–45% but failing on multi-operator stateful topologies. To address these limitations, Netflix adopted the Apache Flink Autoscaler library, which calculates True Processing Rate and per-vertex parallelism using internal metrics. The architecture uses Temporal workflows orchestrated within a Spring Boot application to evaluate jobs individually and prevent cross-job blast radiuses. Implementing the open-source autoscaler reduced annualized compute expenditures by 58% for client telemetry and logging, saving approximately $1.1 million.

Context

Operating over 30,000 Flink jobs as of 2026 across multiple AWS regions required dynamic resource allocation to avoid over-provisioning or lag during traffic surges. Netflix's original 2019 external autoscaler relied on coarse cluster metrics and scaled only total TaskManager counts, making it unable to handle complex multi-operator DAGs or detect degraded states invisible to CPU utilization metrics.

Approach / What changed

Netflix adopted the standalone Apache Flink Autoscaler library to evaluate true processing rates and adjust per-vertex parallelism. They embedded the core logic into a Spring Boot service orchestrated by Temporal workflows running on a per-job basis. The team optimized JobManager metric collection with server-side filtering and caching, adjusted target utilization to 0.45, and integrated scaling decisions with their internal Flink control plane.

Takeaways

• The Apache Flink Autoscaler computes a True Processing Rate per subtask by dividing observed throughput by the busy fraction to adjust per-vertex parallelism.
• Isolating autoscaling evaluations into individual Temporal workflows per job prevented slow or failing metric collections from blocking scaling across the fleet.
• Netflix set its target utilization to 0.45 rather than the open-source default of 0.7 to trade marginal compute efficiency for stability on stateful jobs.

Related reading

• Grab · Oct 30, 2025

  Machine-learning predictive autoscaling for Flink

  Managing resource allocations for expanding Apache Flink stream-processing workloads often leads to over-provisioning or unstable reactive scaling. Reactive autoscaling with Kubernetes Horizontal Pod Autoscaling triggers severe restart spikes because pipelines must reprocess backlogged records from checkpoints, inducing feedback loops and scaling fluctuations. To prevent these spirals, a predictive vertical autoscaling system uses upstream Kafka message throughput as an independent workload metric. The architecture forecasts future workload using time-series models and maps the projected throughput to required TaskManager CPU allocations through a trained regression model. A custom controller vertically scales TaskManager CPU resources before traffic changes arrive, avoiding restart-induced latency spikes.

  Minh Nhat Nguyen

• Netflix · Jul 13, 2026

  Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

  Engineers at Netflix required a unified, real-time view of service dependencies to navigate distributed architecture and improve incident troubleshooting. Traditional batch systems introduced stale data, so the team created a streaming-first platform backed by reactive streams and backpressure handling to ingest flow records from multi-region Kafka streams and Server-Sent Events without data loss. The architecture partitions data into physically separate graph and columnar storage layers covering eBPF network flows, IPC metrics, and distributed traces. Network flow ingestion relies on a three-stage distributed aggregation pipeline using consistent hashing to resolve network intermediaries into logical application connections. The resulting production system serves time-travel and topology queries with sub-second latency while continuously updating dependency views.

  Netflix Technology Blog

• Grab · Oct 13, 2020

  Optimally Scaling Kafka Consumer Applications

  Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.

  Shubham Badkur

• Grab · May 2, 2023

  Safer deployment of streaming applications

  Stateful stream processing frameworks like Apache Flink present unique deployment challenges because conventional canary and blue-green strategies can cause data inaccuracies or state divergence. Grab's real-time data platform team encountered risks of state loss, manual rollback overhead, and absent health checks in their Kubernetes and Spinnaker deployment pipeline. To resolve these operational issues, the team redesigned the deployment workflow around automated Flink savepointing and programmatic health monitoring. The new pipeline halts existing applications after capturing state snapshots and Kafka offsets, monitors target deployments via API health probes, and executes automated rollbacks using versioned ConfigMaps and replica metadata annotations. This automated process ensures state consistency during upgrades and eliminates manual intervention during deployment failures.

  Shi Kai Ng