Loading…
How Netflix Simplified Batch Compute with Kueue
NetflixNetflix Technology Blog
Summary
Netflix transitioned its managed batch compute infrastructure from a homegrown solution called Compute Managed Batch to Kueue on its Titus container platform. CMB previously relied on custom scheduling and admission-only fair sharing without preemption, making feature development cumbersome as the Kubernetes ecosystem evolved. To modernize the platform transparently, Netflix mapped internal tenants to Cohorts and leaf tenants to ClusterQueues and LocalQueues while routing jobs through a custom Kueue router. Kueue operates alongside existing Titus scheduling profiles rather than replacing the kube-scheduler, preserving cluster placement efficiency. The migration was completed in four weeks across millions of batch workloads, significantly increasing average resource utilization through preemption-based fair sharing.
Context
Netflix originally managed batch workloads on its Titus platform using Compute Managed Batch (CMB), a custom solution built in 2018. Over time, maintaining custom scheduling and queueing logic separate from Kubernetes made implementing features like preemption cumbersome. Additionally, CMB lacked preemption capabilities, meaning admitted jobs ran to completion even when fair-share demand shifted across tenants.
Approach / What changed
Netflix modernized its infrastructure with Netflix Batch by replacing CMB queuing and scheduling with Kueue across Titus cells. The team mapped CMB internal tenants to Kueue Cohorts and leaf tenants to LocalQueues and ClusterQueues, converting capacity configurations into resource flavors and nominal quotas. Titus federation routes workloads to Kueue cells using a custom Kueue router while preserving existing API surfaces for end users.
Takeaways
- Kueue was chosen over alternatives like YuniKorn or Volcano because it integrates with existing Titus scheduling profiles without replacing pod scheduling by the kube-scheduler.
- Meeting production throughput requirements required running Kueue with significantly higher QPS, Burst, and groupKindConcurrency settings than the default configurations.
- Migrating the largest and most complex customer first helped build confidence and enabled the team to complete the full production migration within four weeks.
Related reading
Grab ·
How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplication
Grab split a backend service's read and write functionalities into separate services to allow independent scaling. Migrating the write path required transferring processing from 16 source Kafka streams—averaging 20,000 reads per second into DynamoDB tables and output streams—with zero data loss or duplication. Standard feature flags were ruled out because rollout propagation delays could introduce minutes of duplicate or missing data during flag toggling. Instead, engineers extracted processing logic into a shared monorepo commons package that used coordinated timestamps to trigger simultaneous cutovers across both services. Temporary validation sinks verified processing accuracy in production prior to the cutover, completing the stream-by-stream migration across three weeks without downtime.
Md RiyadhNetflix ·
How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC…
Netflix designed a query execution serving layer for its Real-Time Distributed Graph to power sub-100ms responses across diverse graph traversal workloads spanning billions of nodes and edges. To prevent compounding network delays in multi-hop queries, the engine uses a breadth-first traversal model that batches entity lookups across entire frontier levels rather than tracing paths depth-first. The entire serving architecture relies on asynchronous composition across small thread pools of 16 to 24 threads, ensuring no thread blocks while waiting on remote storage or enrichment I/O. Selective caching via EVCache targets stable properties with volatility-matched TTLs, producing 70 to 80 percent cache hit rates and reducing backend storage calls by three to four times. A layered filtering hierarchy pushes depth and edge limits directly to the storage tier, eliminating bespoke code changes while isolating system resources against excessive fan-out.
Netflix Technology BlogNetflix ·
Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned
Engineers at Netflix required a unified, real-time view of service dependencies to navigate distributed architecture and improve incident troubleshooting. Traditional batch systems introduced stale data, so the team created a streaming-first platform backed by reactive streams and backpressure handling to ingest flow records from multi-region Kafka streams and Server-Sent Events without data loss. The architecture partitions data into physically separate graph and columnar storage layers covering eBPF network flows, IPC metrics, and distributed traces. Network flow ingestion relies on a three-stage distributed aggregation pipeline using consistent hashing to resolve network intermediaries into logical application connections. The resulting production system serves time-travel and topology queries with sub-second latency while continuously updating dependency views.
Netflix Technology BlogNetflix ·
In-House LLM Serving at Netflix
Netflix established an in-house serving platform to run large language model inference directly inside existing production environments alongside traditional machine learning models. Built upon NVIDIA Triton Inference Server and vLLM, the unified architecture handles member-scale routing, candidate generation, feature fetching, and model execution over gRPC and OpenAI-compatible HTTP endpoints. The team shifted their primary engine from TensorRT-LLM to vLLM to support custom architectures, simpler debugging, and non-trivial constraint logic. Addressing production obstacles required patching Triton's frontend for guided decoding, pinning dependent library versions, rewriting logits processors in C++ for vLLM V1, and handling state machine resets during engine preemptions. The resulting platform unifies deployment flows while preserving operational stability across evolving model schemas.
Netflix Technology Blog