---
title: "When Can LLMs Replace Humans in A/B Tests?"
description: "Replacing human participants in A/B testing with large language model predictions eliminates causal identification guarantees by design, forcing reliance on surrogate endpoint assumptions. An empirical evaluation using the Upworthy dataset revealed that raw gpt-4o-mini click-through rate predictions exhibited systematic attenuation bias, capturing only 39 percent of observed human treatment effects. To address this discrepancy, researchers formalized surrogacy and comparability conditions, finding that nonlinear machine learning calibrators like random forests and gradient-boosted trees successfully recovered treatment effects where linear ordinary least squares failed. In addition, averaging multiple sample draws per experimental unit mitigated variance and zero-bias caused by model generation temperature. Ultimately, because comparability cannot be verified for novel interventions that depart significantly from historical experiments, calibrated models cannot entirely replace human testing for genuine product innovations."
---

# When Can LLMs Replace Humans in A/B Tests?

[Spotify](https://yomu.fyi/company/spotify) · Spotify Engineering · Aug 13, 2026

**Type:** Explainer

## Summary

Replacing human participants in A/B testing with large language model predictions eliminates causal identification guarantees by design, forcing reliance on surrogate endpoint assumptions. An empirical evaluation using the Upworthy dataset revealed that raw gpt-4o-mini click-through rate predictions exhibited systematic attenuation bias, capturing only 39 percent of observed human treatment effects. To address this discrepancy, researchers formalized surrogacy and comparability conditions, finding that nonlinear machine learning calibrators like random forests and gradient-boosted trees successfully recovered treatment effects where linear ordinary least squares failed. In addition, averaging multiple sample draws per experimental unit mitigated variance and zero-bias caused by model generation temperature. Ultimately, because comparability cannot be verified for novel interventions that depart significantly from historical experiments, calibrated models cannot entirely replace human testing for genuine product innovations.

## Context

Digital product teams seek to replace slow, costly human A/B testing with fast LLM predictions, but substituting models for users removes causal identification guarantees. Furthermore, raw LLM predictions exhibit systematic attenuation bias toward zero, recovering only a fraction of observed human treatment effects.

## Approach / What changed

Using surrogate endpoint theory from biostatistics, the authors formalize surrogacy and comparability conditions to calibrate LLM predictions on historical A/B test data. They evaluate headline click-through rates from the Upworthy dataset using gpt-4o-mini, test linear versus nonlinear machine learning calibration models, and average multiple LLM draws to reduce sampling noise.

## Takeaways

- Raw LLM predictions systematically underestimate treatment effects: prompting gpt-4o-mini on the Upworthy dataset recovered only 39% of the observed human treatment effect due to attenuation bias.
- Valid LLM surrogacy requires surrogacy (the LLM captures all treatment aspects affecting human response) and comparability (the calibration function remains stable between historical and new experiments).
- Nonlinear models like random forests and gradient-boosted trees successfully calibrated LLM predictions to human benchmarks, whereas linear ordinary least squares calibration failed by 3.8 standard errors.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Testing](https://yomu.fyi/topic/testing)

- Source: [Spotify](https://engineering.atspotify.com/2026/8/when-can-llms-replace-humans-in-a-b-tests)
- Source URL: https://engineering.atspotify.com/2026/8/when-can-llms-replace-humans-in-a-b-tests
- Ingested by Yomu: 2026-08-28T01:34:29.389Z

[Read original post](https://engineering.atspotify.com/2026/8/when-can-llms-replace-humans-in-a-b-tests)