In-House LLM Serving at Netflix

Summary

Netflix established an in-house serving platform to run large language model inference directly inside existing production environments alongside traditional machine learning models. Built upon NVIDIA Triton Inference Server and vLLM, the unified architecture handles member-scale routing, candidate generation, feature fetching, and model execution over gRPC and OpenAI-compatible HTTP endpoints. The team shifted their primary engine from TensorRT-LLM to vLLM to support custom architectures, simpler debugging, and non-trivial constraint logic. Addressing production obstacles required patching Triton's frontend for guided decoding, pinning dependent library versions, rewriting logits processors in C++ for vLLM V1, and handling state machine resets during engine preemptions. The resulting platform unifies deployment flows while preserving operational stability across evolving model schemas.

Context

Netflix needed to serve large language models at member scale inside existing production infrastructure without creating an isolated machine learning silo, requiring unified APIs, support for diverse workload types, and low-latency constrained decoding.

Approach / What changed

Engineers integrated vLLM into NVIDIA Triton Inference Server managed by a Java control plane, exposed both gRPC and OpenAI-compatible HTTP frontends, supported Red-Black and Versioned deployment strategies, and implemented a multi-threaded C++ batch logits processor in vLLM V1.

Takeaways

• Triton's vLLM backend decouples model artifacts from frontend updates using JSON configs, but requires strict platform-level version pinning to prevent backend load failures caused by upstream API drift.
• Triton's OpenAI-compatible frontend initially dropped response_format silently, requiring a patch to translate JSON requests into vLLM guided decoding parameters.
• Engine preemptions in vLLM V1 can evict KV caches and shrink token history between decode steps, requiring custom state machines to detect non-monotonic history, reset, and reinitialize.

Related reading

• Netflix · Jul 30, 2026

  GenRec: Towards LLM-Native Recommendation at Netflix

  Netflix developed GenRec, an LLM-backed recommendation ranker built by adapting an internal foundation model for large-scale personalization. Traditional recommendation stacks rely on thousands of hand-crafted features and specialized architectures that are costly to maintain across diverse content types. To replace manual feature pipelines, GenRec verbalizes user histories, metadata, and contexts into natural-language prompts and trains with multi-objective losses, including catalog-aware ranking and reward-weighted alignment. At inference time, the model executes in prefill-only mode on vLLM without decoding text. In large-scale online A/B testing against a mature production ranker, GenRec achieved statistically significant improvements in short-term and long-term metrics while using fewer labeled examples.

  Netflix Technology Blog

• Grab · Oct 21, 2025

  Modernising Grab’s model serving platform with NVIDIA Triton Inference Server

  Grab's machine learning serving platform, Catwalk, experienced mounting technical debt, elevated latency, and rising costs from maintaining disparate inference engines for various frameworks. To address these limitations, the team adopted NVIDIA Triton Inference Server to establish a unified engine, starting with an ONNX migration. They built a custom Triton manager component featuring a proxy layer to translate legacy API requests alongside a server manager that handles model downloads, verification, configuration, and health checks. Within ten days, over half of online deployments migrated seamlessly without requiring client code modifications. The transition reduced p90 latency from 120ms to 20ms on large transformer models and cut average infrastructure spend by approximately 20 percent across evaluated services.

  Daniel Tai

• Airbnb · Jul 14, 2026

  From weeks to a day: how we made LLM evaluation fast enough to iterate on

  Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.

  Baharak Saberidokht

• Grab · Jul 15, 2024

  LLM-powered data classification for data entities at scale

  Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.

  Hualin Liu