When Can LLMs Replace Humans in A/B Tests?

Summary

Replacing human participants in A/B testing with large language model predictions eliminates causal identification guarantees by design, forcing reliance on surrogate endpoint assumptions. An empirical evaluation using the Upworthy dataset revealed that raw gpt-4o-mini click-through rate predictions exhibited systematic attenuation bias, capturing only 39 percent of observed human treatment effects. To address this discrepancy, researchers formalized surrogacy and comparability conditions, finding that nonlinear machine learning calibrators like random forests and gradient-boosted trees successfully recovered treatment effects where linear ordinary least squares failed. In addition, averaging multiple sample draws per experimental unit mitigated variance and zero-bias caused by model generation temperature. Ultimately, because comparability cannot be verified for novel interventions that depart significantly from historical experiments, calibrated models cannot entirely replace human testing for genuine product innovations.

Takeaways

1. 1

   Raw LLM predictions systematically underestimate treatment effects: prompting gpt-4o-mini on the Upworthy dataset recovered only 39% of the observed human treatment effect due to attenuation bias.

2. 2

   Valid LLM surrogacy requires surrogacy (the LLM captures all treatment aspects affecting human response) and comparability (the calibration function remains stable between historical and new experiments).

3. 3

   Nonlinear models like random forests and gradient-boosted trees successfully calibrated LLM predictions to human benchmarks, whereas linear ordinary least squares calibration failed by 3.8 standard errors.