Eval-driven development: Lessons from evaluating GenAI at scale

Summary

Airbnb outlines its framework for eval-driven development (EDD), treating GenAI evaluation as a continuous engineering discipline rather than an afterthought. The strategy combines programmatic checks, calibrated LLM-as-a-judge evaluators, and human review to detect subtle quality regressions. By inspecting intermediate agentic execution traces and real-world failure modes, teams ensure generative models remain reliable and aligned with product requirements.

Takeaways

1. 1

   Adopt a three-layer evaluation strategy: fast programmatic schema checks, targeted single-dimension LLM judges, and expert human review.

2. 2

   Calibrate LLM-as-a-judge evaluators against a golden dataset (including bad examples) until reaching 80–90%+ agreement with human experts.

3. 3

   Evaluate full execution trajectories and tool-calling traces in multi-step agentic systems rather than assessing only the final output.