Loading post…
SpotifySpotify EngineeringIncident / postmortem
AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity
Summary
Spotify examines how AI-accelerated development and broader industry compute demand challenged quality across content processing, fleet updates, regional failover, and the mobile app. At its scale—777 million monthly active users, about 100 million concurrent clients, 11–12 million backend requests per second, and nearly 3,000 production services—small control gaps can become visible to listeners and creators. A June 24 processing incident combined competing batch work, higher per-episode compute, and a scheduler bug that cut throughput about 10%, leading to hour-long publication delays; Spotify responded with end-to-end monitoring, scheduler fixes, prioritization, and more capacity. Incident reviews found no material direct contribution from AI-authored code, but change volume outpaced some verification controls, prompting stronger review, testing, rollout, observability, rollback, quality signals, and failover practices. Spotify reports no corresponding rise in rework rate, while continuing to watch increasing PR size and complexity.
Context
Spotify describes quality challenges arising as development speed, automated fleet changes, content volume, and external demand for compute all increased. The discussion covers a June 24 content-processing incident, an automated dependency upgrade that failed in production, capacity constraints during regional failover, and mobile regressions that accumulated beyond existing guardrails.
Approach / What changed
Spotify added end-to-end content-processing monitoring, fixed the scheduler, lowered batch-job priority, increased capacity, and reworked service tiering and workload prioritization. It is strengthening safeguards and rollback capacity for automated fleet changes, scheduling them during owning teams’ working hours, broadening mobile quality signals and long-term trend analysis, and improving review, testing, rollout, observability, failover, and quality measurement.
Takeaways
The June 24 publishing delays resulted from competing batch work, increased per-episode compute, and a scheduling bug that reduced throughput by about 10%.
Spotify found no material direct contribution from AI-authored code in reviewed incidents, but change volume grew faster than some verification controls could adapt.
Merged changes more than doubled from roughly 8,100 to 17,000 in August year over year, while quality and optimization work increased from 27% to 31% and rework rate did not rise correspondingly.
FAQ
Did Spotify find that AI-authored code directly caused production incidents?
Across the incidents reviewed so far, Spotify did not identify AI-authored code as a material direct contributor. It did find that the increased volume of change put pressure on review, testing, rollout, and observability controls.
What caused the June 24 content-processing delays?
A scheduled batch job competed with new episodes, a quality improvement increased the compute required per episode, and a scheduling bug reduced throughput by about 10%. Spotify added end-to-end monitoring, fixed the scheduler, increased capacity, lowered batch priority, and changed workload prioritization.
How is Spotify checking whether faster development is harming quality?
Spotify reviews major incidents and tracks metrics including rework rate, code churn, code complexity, PR size, and the mix of merged work. Rework rate has not risen alongside code churn, but PR size and complexity are increasing and remain under observation.
Source and freshness
- Source
- engineering.atspotify.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.