Kafka vs Temporal: When to Use Each, and When to Use Both
Kafka is a durable event stream; Temporal is a workflow orchestrator. Learn which solves your problem and why many teams use them together.
Loading post…
NetflixNetflix Technology BlogProblem & solution
A Tale of Two Flink Autoscalers
Summary
Netflix operates over 30,000 Apache Flink jobs across multiple AWS regions, ranging from managed pipelines to complex stateful DAGs. Its initial in-house autoscaler used external telemetry from Mantis and Atlas to scale TaskManagers as a single unit, cutting resource usage by 25–45% but failing on multi-operator stateful topologies. To address these limitations, Netflix adopted the Apache Flink Autoscaler library, which calculates True Processing Rate and per-vertex parallelism using internal metrics. The architecture uses Temporal workflows orchestrated within a Spring Boot application to evaluate jobs individually and prevent cross-job blast radiuses. Implementing the open-source autoscaler reduced annualized compute expenditures by 58% for client telemetry and logging, saving approximately $1.1 million.
Context
Operating over 30,000 Flink jobs as of 2026 across multiple AWS regions required dynamic resource allocation to avoid over-provisioning or lag during traffic surges. Netflix's original 2019 external autoscaler relied on coarse cluster metrics and scaled only total TaskManager counts, making it unable to handle complex multi-operator DAGs or detect degraded states invisible to CPU utilization metrics.
Approach / What changed
Netflix adopted the standalone Apache Flink Autoscaler library to evaluate true processing rates and adjust per-vertex parallelism. They embedded the core logic into a Spring Boot service orchestrated by Temporal workflows running on a per-job basis. The team optimized JobManager metric collection with server-side filtering and caching, adjusted target utilization to 0.45, and integrated scaling decisions with their internal Flink control plane.
Takeaways
The Apache Flink Autoscaler computes a True Processing Rate per subtask by dividing observed throughput by the busy fraction to adjust per-vertex parallelism.
Isolating autoscaling evaluations into individual Temporal workflows per job prevented slow or failing metric collections from blocking scaling across the fleet.
Netflix set its target utilization to 0.45 rather than the open-source default of 0.7 to trade marginal compute efficiency for stability on stateful jobs.
Source and freshness
- Source
- netflixtechblog.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.