From weeks to a day: how we made LLM evaluation fast enough to iterate on

Summary

Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.

Takeaways

• Per-sample caching of judge scores and generated references eliminates artificial evaluation noise, ensuring deterministic runs and cross-run metric comparability.
• Targeted hotfixes can be shipped in under an hour using low-rank micro LoRA adapters (rank < 50) managed with strict lifecycle rules around fusion, retraining, and unloading.
• Component-level testing misses cross-boundary regressions at pipeline seams, necessitating fast end-to-end validation across traffic-weighted samples and incident regression cases.

Related reading

• Airbnb · Jul 28, 2026

  Eval-driven development: Lessons from evaluating GenAI at scale

  Airbnb outlines its framework for eval-driven development (EDD), treating GenAI evaluation as a continuous engineering discipline rather than an afterthought. The strategy combines programmatic checks, calibrated LLM-as-a-judge evaluators, and human review to detect subtle quality regressions. By inspecting intermediate agentic execution traces and real-world failure modes, teams ensure generative models remain reliable and aligned with product requirements.

  Rohit Girme

• Airbnb · Aug 25, 2026

  Project Lighthouse — Part 3: Introducing project-lighthouse-anonymize

  Airbnb has open-sourced project-lighthouse-anonymize, a Python library powering their privacy-preserving anonymization used to measure user experience disparities. The release is accompanied by technical papers detailing Core Mondrian, an extended partition-based anonymization algorithm, alongside a framework for measuring anonymized data quality. The library enforces k-anonymity and p-sensitive k-anonymity while offering built-in data quality metrics.

  Adam Bloomston

• Netflix · Jul 17, 2026

  In-House LLM Serving at Netflix

  Netflix established an in-house serving platform to run large language model inference directly inside existing production environments alongside traditional machine learning models. Built upon NVIDIA Triton Inference Server and vLLM, the unified architecture handles member-scale routing, candidate generation, feature fetching, and model execution over gRPC and OpenAI-compatible HTTP endpoints. The team shifted their primary engine from TensorRT-LLM to vLLM to support custom architectures, simpler debugging, and non-trivial constraint logic. Addressing production obstacles required patching Triton's frontend for guided decoding, pinning dependent library versions, rewriting logits processors in C++ for vLLM V1, and handling state machine resets during engine preemptions. The resulting platform unifies deployment flows while preserving operational stability across evolving model schemas.

  Netflix Technology Blog

• Airbnb · May 19, 2026

  Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure

  Airbnb migrated its massive identity graph from a third-party graph database vendor to a unified, in-house knowledge graph platform to solve long-tail latency and scaling bottlenecks. Built on JanusGraph with AWS DynamoDB for persistence and OpenSearch for indexing, the new architecture decouples storage operations from graph traversal logic. The migration improved P99 query latency, eliminated routine instance reboots, and supported a tenfold increase in write throughput.

  Lucen Zhao