From weeks to a day: how we made LLM evaluation fast enough to iterate on

Summary

Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.

Takeaways

1. 1

   Per-sample caching of judge scores and generated references eliminates artificial evaluation noise, ensuring deterministic runs and cross-run metric comparability.

2. 2

   Targeted hotfixes can be shipped in under an hour using low-rank micro LoRA adapters (rank < 50) managed with strict lifecycle rules around fusion, retraining, and unloading.

3. 3

   Component-level testing misses cross-boundary regressions at pipeline seams, necessitating fast end-to-end validation across traffic-weighted samples and incident regression cases.