Why Spotify Is Not Using Bayesian A/B Testing

Summary

Spotify argues that Bayesian A/B testing is not a single method but a family of configurations defined by a prior, likelihood, and stopping rule. It contrasts flat-prior posterior-probability stopping, which can reproduce frequentist peeking false-positive rates, with Bayes-factor stopping, which controls false positives under optional stopping, and calibrated empirical Bayes priors, which can shrink estimates and control false discovery rates. The post frames experiment design around program goals such as limiting harmful or ineffective launches, achieving estimation precision, and minimizing costs rather than selecting a statistical framework first. It says Bayesian and frequentist approaches overlap substantially, while Bayesian advantages such as richer probability statements require additional prior complexity. Spotify therefore continues using frequentist tooling because supporting two modes would add planning, monitoring, interpretation, and configuration costs without a sufficient benefit.

Takeaways

1. 1

   A flat-prior Bayesian test stopped by a posterior-probability threshold can reproduce the false-positive rate of frequentist peeking.

2. 2

   Bayes-factor stopping provides false-positive-rate control under optional stopping, while calibrated empirical Bayes priors can shrink estimates and control the false discovery rate.

3. 3

   Spotify says the benefits of adding Bayesian inference do not outweigh the costs of different planning, monitoring, interpretation, and outputs across its experimentation program.