Eval-driven development: Lessons from evaluating GenAI at scale

Summary

Airbnb outlines its framework for eval-driven development (EDD), treating GenAI evaluation as a continuous engineering discipline rather than an afterthought. The strategy combines programmatic checks, calibrated LLM-as-a-judge evaluators, and human review to detect subtle quality regressions. By inspecting intermediate agentic execution traces and real-world failure modes, teams ensure generative models remain reliable and aligned with product requirements.

Takeaways

• Adopt a three-layer evaluation strategy: fast programmatic schema checks, targeted single-dimension LLM judges, and expert human review.
• Calibrate LLM-as-a-judge evaluators against a golden dataset (including bad examples) until reaching 80–90%+ agreement with human experts.
• Evaluate full execution trajectories and tool-calling traces in multi-step agentic systems rather than assessing only the final output.

Related reading

• Airbnb · May 19, 2026

  Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure

  Airbnb migrated its massive identity graph from a third-party graph database vendor to a unified, in-house knowledge graph platform to solve long-tail latency and scaling bottlenecks. Built on JanusGraph with AWS DynamoDB for persistence and OpenSearch for indexing, the new architecture decouples storage operations from graph traversal logic. The migration improved P99 query latency, eliminated routine instance reboots, and supported a tenfold increase in write throughput.

  Lucen Zhao

• Airbnb · Jul 14, 2026

  From weeks to a day: how we made LLM evaluation fast enough to iterate on

  Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.

  Baharak Saberidokht

• Airbnb · Jun 9, 2026

  Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world

  To support its expansion from Homes into Experiences and Services, Airbnb evolved its offline data warehouse architecture to balance domain-specific needs with organizational consistency. Rather than enforcing a single global pattern, data teams adopted a framework of foundational principles that paired separate data models for unique product features with monolithic models for cross-cutting domains like payments and messaging.

  Patrick Lam

• Airbnb · Aug 25, 2026

  Project Lighthouse — Part 3: Introducing project-lighthouse-anonymize

  Airbnb has open-sourced project-lighthouse-anonymize, a Python library powering their privacy-preserving anonymization used to measure user experience disparities. The release is accompanied by technical papers detailing Core Mondrian, an extended partition-based anonymization algorithm, alongside a framework for measuring anonymized data quality. The library enforces k-anonymity and p-sensitive k-anonymity while offering built-in data quality metrics.

  Adam Bloomston