Loading…
Content Ingestion & Podcast Video Incident Report
SpotifySpotify Engineering
Summary
On June 24, video transcoding infrastructure reached maximum capacity, causing hours of publishing delays for Spotify video podcast episodes. Four converging factors drove the disruption: insufficient headroom for content spikes, an active routine batch processing job, higher compute costs from a recent video quality upgrade, and a scheduling bug reducing hardware throughput by roughly 10%. Creators exacerbated the backlog by re-uploading unconfirmed episodes, while four hours elapsed between the earliest alerts and formal incident response. Engineers halted the batch job, patched the scheduling defect, and brought an extra compute cluster online to clear all backlogs overnight. Spotify subsequently increased transcoding capacity by approximately 67% and initiated cross-team investments in workload prioritization, rate limiting, and earlier capacity alerting.
Context
Video transcoding infrastructure hit maximum capacity during a delivery spike on June 24, creating backlogs that delayed video podcast episode publishing for hours. The lack of queue confirmation led creators to re-upload delayed episodes, compounding system load.
Approach / What changed
Engineers stopped a running batch processing job, deployed a fix for a resource scheduling bug that had reduced throughput by 10%, and added an additional processing cluster overnight to clear backlogs. Spotify subsequently expanded transcoding capacity by roughly 67%, improved capacity monitoring alerts, and began implementing stricter prioritization, rate limiting, and creator notification mechanisms.
Takeaways
- A 10% compute underutilization bug combined with higher per-item transcoding processing costs from a quality upgrade left infrastructure with insufficient headroom during a delivery spike.
- A routine background batch job re-processing existing media ran concurrently with normal ingestion, competing for critical transcoding capacity needed by new episode publications.
- Publishing systems lacked upload queue confirmations for creators, causing creators to repeatedly re-upload delayed content and amplify pipeline backlogs.
Related reading
Netflix ·
The Data Canary: How Netflix Validates Catalog Metadata
A manual mitigation action during an incident corrupted a data feed for a subset of titles, causing playback issues and catalog service failures that existing code canary systems failed to catch. To protect streaming reliability, Netflix built an automated data canary system that validates transformed catalog metadata prior to publication. The architecture utilizes a dedicated orchestrator alongside permanent baseline and canary service clusters to coordinate validation using real production traffic. By leveraging custom chaos experiment thresholds, sticky session affinity, and Starts Per Second playback metrics, the system detects regressions in under ten minutes and blocks publication automatically. Controlled failure injection experiments routing approximately 0.2% of global traffic confirmed that issues could be identified in 2.5 to 4 minutes.
Netflix Technology BlogCanva ·
Canva incident report: API Gateway outage