Loading post…
EtsyKelly McManusExplainer
Reducing experiment duration with predicted control variates
Summary
Etsy describes how its experimentation platform evolved from CUPED to CUPAC to shorten the time needed to reach statistically powered results. CUPED uses pre-experiment visitor data as covariates in linear regression, while CUPAC first uses a nonlinear model to predict the outcome and then uses that prediction in the variance-reduction correction. Etsy trained and tuned LightGBM models on more than 100 pre-experiment features in Vertex AI, generating predictions through parallel Dataflow jobs and applying the correction with Spark within an Airflow-orchestrated pipeline. CUPED reduced variance by 7% on average, whereas CUPAC achieved 27%, shortening average experiment duration by almost three additional days, although results ranged from 2% to 77% depending on the metric and experiment; Etsy is balancing CUPAC’s impact against its training and maintenance costs.
Context
Etsy’s experiments can take weeks to months because they must collect enough data to achieve statistical significance and power. The company wanted to reduce experiment duration and accelerate learning while accounting for differences in how predictive pre-experiment data is for each metric.
Approach / What changed
Etsy extended its CUPED pipeline with CUPAC, using LightGBM models trained on more than 100 pre-experiment features to generate ML-based covariates. Vertex AI handled model training and tuning, parallel Dataflow jobs produced predictions, Spark performed the variance-reduction regression, and Airflow orchestrated the workflow alongside BigQuery SQL and statistical testing.
Takeaways
CUPAC produced an average 27% variance reduction, compared with 7% for Etsy’s original CUPED implementation.
CUPAC shortened average experiment duration by almost three additional days; a 10-day experiment could reach power in seven days.
Variance reduction ranged from 2% to 77%, and CUPAC models require more training and maintenance than the more scalable CUPED approach.
Source and freshness
- Source
- etsy.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.