# Trusting the Untestable: Validation and Diagnostics for the Doubly Robust Models

[Lyft](https://yomu.fyi/company/lyft) · Shima Nassiri · Feb 12, 2026

**Type:** Problem & solution

## Summary

When randomized A/B tests are infeasible for evaluating partnerships or long-term effects, Lyft applies Augmented Inverse Propensity Weighting (AIPW) doubly robust models to estimate treatment effects from observational data. The quasi-experimentation platform mandates rigorous confounder selection prior to treatment exposure and adjusts for downsampling bias via propensity score conversions and outcome reweighting. When comparing AIPW against experimental ground truths, initial estimates understated effects because trimming extreme propensity scores removed key users and unobserved confounders persisted. To evaluate model health and external validity, the platform incorporates diagnostic scorecards featuring common support checks, Marginal Sensitivity Models, and covariate comparisons between trimmed and untrimmed cohorts.

## Context

Lyft frequently encounters business scenarios where randomized A/B testing is impractical, including partnership rollouts, long-term effect tracking, post-launch policy evaluations, and biased experimental data. Because quasi-experiments lack the inherent unbiasedness of randomized trials, unobserved confounding, downsampling, and common support violations can severely distort treatment effect estimates.

## Approach / What changed

Lyft deployed a quasi-experimentation platform using Augmented Inverse Propensity Weighting (AIPW) alongside diagnostic tools. The platform enforces pre-exposure confounder extraction via customizable SQL, applies propensity score corrections and outcome reweighting for downsampled groups, and provides diagnostic scorecards. Diagnostics include common support overlap histograms, Zhao et al.'s Marginal Sensitivity Model to bound unobserved confounding, and covariate comparison plots between trimmed and untrimmed cohorts.

## Takeaways

- AIPW requires fitting an outcome model and a propensity score model, consistently estimating the true Average Treatment Effect as long as at least one of the two models is correctly specified.
- Random downsampling used to balance group sizes introduces bias that requires converting sample propensity scores to true population scores and uniformly rescaling outcome weights.
- Trimming extreme propensity scores to satisfy common support assumptions can exclude distinct user segments, reducing external validity unless trimmed and untrimmed covariates are compared and verified.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://eng.lyft.com/trusting-the-untestable-validation-and-diagnostics-for-the-doubly-robust-models-00853df009df)