# From weeks to a day: how we made LLM evaluation fast enough to iterate on

[Airbnb](https://yomu.fyi/company/airbnb) · Baharak Saberidokht · Jul 14, 2026

## Summary

Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.

## Takeaways

- Per-sample caching of judge scores and generated references eliminates artificial evaluation noise, ensuring deterministic runs and cross-run metric comparability.
- Targeted hotfixes can be shipped in under an hour using low-rank micro LoRA adapters (rank < 50) managed with strict lifecycle rules around fusion, retraining, and unloading.
- Component-level testing misses cross-boundary regressions at pipeline seams, necessitating fast end-to-end validation across traffic-weighted samples and incident regression cases.

**Tags:** [Caching](https://yomu.fyi/topic/caching), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Reliability](https://yomu.fyi/topic/reliability), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://medium.com/airbnb-engineering/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate-on-14e2d35198b4)