How we knew COVID was over (and what our models had to unlearn)

Summary

Airbnb's forecasting team outlines their framework for managing production models that experience drift or external shocks. Rather than treating model updates as a generic retraining task, they distinguish between refitting with new data, respecifying the underlying model structure, and intentionally holding without changes. This discipline prevents teams from chasing temporary noise, carrying obsolete crisis assumptions, or over-engineering solutions during disruptions.

Takeaways

1. 1

   Treat model maintenance as three distinct actions—refitting parameters, respecifying structure/assumptions, or holding—rather than a single generic retraining step.

2. 2

   Persistent one-sided forecast errors indicate a structural misspecification problem that fresh data and simple refits cannot resolve.

3. 3

   Avoid off-cycle refits triggered by sudden surprises or anomalies, as rushing to absorb unverified recent data often causes models to fit temporary noise.