# In-House LLM Serving at Netflix

[Netflix](https://yomu.fyi/company/netflix) · Netflix Technology Blog · Jul 17, 2026

**Type:** Problem & solution

## Summary

Netflix established an in-house serving platform to run large language model inference directly inside existing production environments alongside traditional machine learning models. Built upon NVIDIA Triton Inference Server and vLLM, the unified architecture handles member-scale routing, candidate generation, feature fetching, and model execution over gRPC and OpenAI-compatible HTTP endpoints. The team shifted their primary engine from TensorRT-LLM to vLLM to support custom architectures, simpler debugging, and non-trivial constraint logic. Addressing production obstacles required patching Triton's frontend for guided decoding, pinning dependent library versions, rewriting logits processors in C++ for vLLM V1, and handling state machine resets during engine preemptions. The resulting platform unifies deployment flows while preserving operational stability across evolving model schemas.

## Context

Netflix needed to serve large language models at member scale inside existing production infrastructure without creating an isolated machine learning silo, requiring unified APIs, support for diverse workload types, and low-latency constrained decoding.

## Approach / What changed

Engineers integrated vLLM into NVIDIA Triton Inference Server managed by a Java control plane, exposed both gRPC and OpenAI-compatible HTTP frontends, supported Red-Black and Versioned deployment strategies, and implemented a multi-threaded C++ batch logits processor in vLLM V1.

## Takeaways

- Triton's vLLM backend decouples model artifacts from frontend updates using JSON configs, but requires strict platform-level version pinning to prevent backend load failures caused by upstream API drift.
- Triton's OpenAI-compatible frontend initially dropped response\_format silently, requiring a patch to translate JSON requests into vLLM guided decoding parameters.
- Engine preemptions in vLLM V1 can evict KV caches and shrink token history between decode steps, requiring custom state machines to detect non-monotonic history, reset, and reinitialize.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://netflixtechblog.com/in-house-llm-serving-at-netflix-a5a8e799ea2c)