---
title: "Why Spotify Is Not Using Bayesian A/B Testing"
description: "Spotify argues that Bayesian A/B testing is not a single method but a family of configurations defined by a prior, likelihood, and stopping rule. It contrasts flat-prior posterior-probability stopping, which can reproduce frequentist peeking false-positive rates, with Bayes-factor stopping, which controls false positives under optional stopping, and calibrated empirical Bayes priors, which can shrink estimates and control false discovery rates. The post frames experiment design around program goals such as limiting harmful or ineffective launches, achieving estimation precision, and minimizing costs rather than selecting a statistical framework first. It says Bayesian and frequentist approaches overlap substantially, while Bayesian advantages such as richer probability statements require additional prior complexity. Spotify therefore continues using frequentist tooling because supporting two modes would add planning, monitoring, interpretation, and configuration costs without a sufficient benefit."
---

# Why Spotify Is Not Using Bayesian A/B Testing

[Spotify](https://yomu.fyi/company/spotify) · Spotify Engineering · Sep 8, 2026

**Type:** Explainer

## Summary

Spotify argues that Bayesian A/B testing is not a single method but a family of configurations defined by a prior, likelihood, and stopping rule. It contrasts flat-prior posterior-probability stopping, which can reproduce frequentist peeking false-positive rates, with Bayes-factor stopping, which controls false positives under optional stopping, and calibrated empirical Bayes priors, which can shrink estimates and control false discovery rates. The post frames experiment design around program goals such as limiting harmful or ineffective launches, achieving estimation precision, and minimizing costs rather than selecting a statistical framework first. It says Bayesian and frequentist approaches overlap substantially, while Bayesian advantages such as richer probability statements require additional prior complexity. Spotify therefore continues using frequentist tooling because supporting two modes would add planning, monitoring, interpretation, and configuration costs without a sufficient benefit.

## Context

Spotify evaluates whether its experimentation program should add Bayesian inference alongside its existing frequentist tooling. Its stated goals include minimizing poor business decisions, making experiment evidence trustworthy and hard to misinterpret, and keeping experiments easy to plan while difficult to misconfigure.

## Approach / What changed

The post classifies Bayesian experimentation by its prior, likelihood, and stopping rule, then compares configurations such as posterior-probability stopping, Bayes-factor stopping, and calibrated empirical Bayes priors. It evaluates them against program goals and concludes that Spotify’s current frequentist setup remains preferable to supporting two inference modes.

## Takeaways

- A flat-prior Bayesian test stopped by a posterior-probability threshold can reproduce the false-positive rate of frequentist peeking.
- Bayes-factor stopping provides false-positive-rate control under optional stopping, while calibrated empirical Bayes priors can shrink estimates and control the false discovery rate.
- Spotify says the benefits of adding Bayesian inference do not outweigh the costs of different planning, monitoring, interpretation, and outputs across its experimentation program.

**Tags:** [Data Science](https://yomu.fyi/topic/data-science), [Decision-Making](https://yomu.fyi/topic/decision-making), [Testing](https://yomu.fyi/topic/testing)

- Source: [Spotify](https://engineering.atspotify.com/2026/9/why-spotify-is-not-using-bayesian-a-b-testing)
- Source URL: https://engineering.atspotify.com/2026/9/why-spotify-is-not-using-bayesian-a-b-testing
- Ingested by Yomu: 2026-09-08T16:00:46.451Z

[Read original post](https://engineering.atspotify.com/2026/9/why-spotify-is-not-using-bayesian-a-b-testing)