---
title: "Reducing experiment duration with predicted control variates"
description: "Etsy describes how its experimentation platform evolved from CUPED to CUPAC to shorten the time needed to reach statistically powered results. CUPED uses pre-experiment visitor data as covariates in linear regression, while CUPAC first uses a nonlinear model to predict the outcome and then uses that prediction in the variance-reduction correction. Etsy trained and tuned LightGBM models on more than 100 pre-experiment features in Vertex AI, generating predictions through parallel Dataflow jobs and applying the correction with Spark within an Airflow-orchestrated pipeline. CUPED reduced variance by 7% on average, whereas CUPAC achieved 27%, shortening average experiment duration by almost three additional days, although results ranged from 2% to 77% depending on the metric and experiment; Etsy is balancing CUPAC’s impact against its training and maintenance costs."
---

# Reducing experiment duration with predicted control variates

[Etsy](https://yomu.fyi/company/etsy) · Kelly McManus · Nov 25, 2025

**Type:** Explainer

## Summary

Etsy describes how its experimentation platform evolved from CUPED to CUPAC to shorten the time needed to reach statistically powered results. CUPED uses pre-experiment visitor data as covariates in linear regression, while CUPAC first uses a nonlinear model to predict the outcome and then uses that prediction in the variance-reduction correction. Etsy trained and tuned LightGBM models on more than 100 pre-experiment features in Vertex AI, generating predictions through parallel Dataflow jobs and applying the correction with Spark within an Airflow-orchestrated pipeline. CUPED reduced variance by 7% on average, whereas CUPAC achieved 27%, shortening average experiment duration by almost three additional days, although results ranged from 2% to 77% depending on the metric and experiment; Etsy is balancing CUPAC’s impact against its training and maintenance costs.

## Context

Etsy’s experiments can take weeks to months because they must collect enough data to achieve statistical significance and power. The company wanted to reduce experiment duration and accelerate learning while accounting for differences in how predictive pre-experiment data is for each metric.

## Approach / What changed

Etsy extended its CUPED pipeline with CUPAC, using LightGBM models trained on more than 100 pre-experiment features to generate ML-based covariates. Vertex AI handled model training and tuning, parallel Dataflow jobs produced predictions, Spark performed the variance-reduction regression, and Airflow orchestrated the workflow alongside BigQuery SQL and statistical testing.

## Takeaways

- CUPAC produced an average 27% variance reduction, compared with 7% for Etsy’s original CUPED implementation.
- CUPAC shortened average experiment duration by almost three additional days; a 10-day experiment could reach power in seven days.
- Variance reduction ranged from 2% to 77%, and CUPAC models require more training and maintenance than the more scalable CUPED approach.

**Tags:** [Airflow](https://yomu.fyi/topic/airflow), [Experimentation](https://yomu.fyi/topic/experimentation), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Spark](https://yomu.fyi/topic/spark)

- Source: [Etsy](https://www.etsy.com/codeascraft/reducing-experiment-duration-with-predicted-control-variates)
- Source URL: https://www.etsy.com/codeascraft/reducing-experiment-duration-with-predicted-control-variates
- Ingested by Yomu: 2026-09-09T16:05:27.496Z

[Read original post](https://www.etsy.com/codeascraft/reducing-experiment-duration-with-predicted-control-variates)