# Eval-driven development: Lessons from evaluating GenAI at scale

[Airbnb](https://yomu.fyi/company/airbnb) · Rohit Girme · Jul 28, 2026

## Summary

Airbnb outlines its framework for eval-driven development (EDD), treating GenAI evaluation as a continuous engineering discipline rather than an afterthought. The strategy combines programmatic checks, calibrated LLM-as-a-judge evaluators, and human review to detect subtle quality regressions. By inspecting intermediate agentic execution traces and real-world failure modes, teams ensure generative models remain reliable and aligned with product requirements.

## Takeaways

- Adopt a three-layer evaluation strategy: fast programmatic schema checks, targeted single-dimension LLM judges, and expert human review.
- Calibrate LLM-as-a-judge evaluators against a golden dataset (including bad examples) until reaching 80–90%+ agreement with human experts.
- Evaluate full execution trajectories and tool-calling traces in multi-step agentic systems rather than assessing only the final output.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Observability](https://yomu.fyi/topic/observability), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://medium.com/airbnb-engineering/eval-driven-development-lessons-from-evaluating-genai-at-scale-e817e5ae5788)