Beyond A/B Testing: Using Surrogacy and Region-Splits to Measure Long-Term Effects in Marketplaces

Summary

Lyft describes a framework for estimating long-term effects of pricing and incentive decisions in its two-sided marketplace, where direct and market-mediated consequences can diverge. The method first uses residualized observational regressions to estimate how policy changes alter distributions of negative experiences such as wait times, surge, cancellations, idleness, and hourly earnings, validating those responses with switch-back experiments. It then applies Augmented Inverse Probability Weighting to estimate how these experiences affect future outcomes, summarizes the relationship with a surrogacy index, and checks calibration using user-split experiments. Finally, Lyft combines mediated and separately estimated direct effects and uses region-split experiments for end-to-end validation, selecting treated and control regions with a forward-selection algorithm inspired by FDiD. The framework is intended to support scenario planning, budget allocation, and continuous calibration of long-run marketplace forecasts.

Takeaways

1. 1

   Residualizing time-of-week patterns and controlling for supply and demand lets policy coefficients represent incremental changes in negative user experiences around normal operating conditions.

2. 2

   AIPW combines exposure-propensity and outcome models to estimate how negative experiences affect future outcomes while controlling for time, location, and user history.

3. 3

   Region-split experiments provide end-to-end validation, but the design requires forward selection to improve pre-intervention fit and expected statistical power.