How we knew COVID was over (and what our models had to unlearn)

Summary

Airbnb's forecasting team outlines their framework for managing production models that experience drift or external shocks. Rather than treating model updates as a generic retraining task, they distinguish between refitting with new data, respecifying the underlying model structure, and intentionally holding without changes. This discipline prevents teams from chasing temporary noise, carrying obsolete crisis assumptions, or over-engineering solutions during disruptions.

Takeaways

• Treat model maintenance as three distinct actions—refitting parameters, respecifying structure/assumptions, or holding—rather than a single generic retraining step.
• Persistent one-sided forecast errors indicate a structural misspecification problem that fresh data and simple refits cannot resolve.
• Avoid off-cycle refits triggered by sudden surprises or anomalies, as rushing to absorb unverified recent data often causes models to fit temporary noise.

Related reading

• Airbnb · Jun 2, 2026

  When history fails you, borrow from geography

  When historical travel data became uninformative during the asynchronous post-COVID recovery, Airbnb redesigned its demand forecasting to borrow signals across geographies rather than waiting for local data. Using a hierarchical Bayesian framework, the team propagated updated posterior estimates from early-recovering corridors as informative priors for structurally similar, later-recovering corridors. This allowed accurate corridor-level demand forecasting in near real time during periods of high disruption and uneven market reopening.

  Harrison Katz

• Airbnb · Jun 9, 2026

  Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world

  To support its expansion from Homes into Experiences and Services, Airbnb evolved its offline data warehouse architecture to balance domain-specific needs with organizational consistency. Rather than enforcing a single global pattern, data teams adopted a framework of foundational principles that paired separate data models for unique product features with monolithic models for cross-cutting domains like payments and messaging.

  Patrick Lam

• Salesforce · Aug 24, 2026

  From Prediction to Action: How to Turn AI Outputs Into Decisions

  Salesforce addressed an operational challenge where sellers faced roughly 12,000 dashboards and over 20 applications outputting machine learning predictions without clear next steps. The engineering team reframed machine learning outputs as raw signals rather than standalone answers. To bridge the gap between assessment and action, they built a Next Best Action layer that combines model signals, business logic, and contextual institutional knowledge into actionable recommendations. They integrated this layer with an AI agent using Model Context Protocol (MCP) tool contracts, enabling dynamic discovery and explicit handling of missing data. Finally, rather than introducing a separate dashboard destination, the agent serves on-demand recommendations directly inside Slack where sellers already collaborate.

  Scott Nyberg

• Airbnb · Jul 28, 2026

  Eval-driven development: Lessons from evaluating GenAI at scale

  Airbnb outlines its framework for eval-driven development (EDD), treating GenAI evaluation as a continuous engineering discipline rather than an afterthought. The strategy combines programmatic checks, calibrated LLM-as-a-judge evaluators, and human review to detect subtle quality regressions. By inspecting intermediate agentic execution traces and real-world failure modes, teams ensure generative models remain reliable and aligned with product requirements.

  Rohit Girme