In-House LLM Serving at Netflix

Summary

Netflix established an in-house serving platform to run large language model inference directly inside existing production environments alongside traditional machine learning models. Built upon NVIDIA Triton Inference Server and vLLM, the unified architecture handles member-scale routing, candidate generation, feature fetching, and model execution over gRPC and OpenAI-compatible HTTP endpoints. The team shifted their primary engine from TensorRT-LLM to vLLM to support custom architectures, simpler debugging, and non-trivial constraint logic. Addressing production obstacles required patching Triton's frontend for guided decoding, pinning dependent library versions, rewriting logits processors in C++ for vLLM V1, and handling state machine resets during engine preemptions. The resulting platform unifies deployment flows while preserving operational stability across evolving model schemas.

Takeaways

1. 1

   Triton's vLLM backend decouples model artifacts from frontend updates using JSON configs, but requires strict platform-level version pinning to prevent backend load failures caused by upstream API drift.

2. 2

   Triton's OpenAI-compatible frontend initially dropped response_format silently, requiring a patch to translate JSON requests into vLLM guided decoding parameters.

3. 3

   Engine preemptions in vLLM V1 can evict KV caches and shrink token history between decode steps, requiring custom state machines to detect non-monotonic history, reset, and reinitialize.