Spotify

Latest articles

• Spotify · Jul 27, 2026

  Indexing the Data Lake for Online Point Queries

  Online services and AI agents require low-latency point queries over massive datasets that are economically impractical to store entirely in key-value systems like Bigtable or DynamoDB. While cloud object storage offers rapid request latency, traditional distributed query engines like Trino and BigQuery introduce seconds of planning and scheduling overhead along with dependent file reads. Random Access Parquet addresses this limitation by using an external index mapping keys directly to exact Parquet files and row locations. Readers execute O(1) index lookups and issue parallel ranged reads to fetch only the required bytes without scanning footers or metadata. By running directly against existing data lake files, this architecture eliminates separate storage copies and allows shared analytical data to serve interactive online lookups.

  Spotify Engineering

• Spotify · Jul 20, 2026

  Content Ingestion & Podcast Video Incident Report

  On June 24, video transcoding infrastructure reached maximum capacity, causing hours of publishing delays for Spotify video podcast episodes. Four converging factors drove the disruption: insufficient headroom for content spikes, an active routine batch processing job, higher compute costs from a recent video quality upgrade, and a scheduling bug reducing hardware throughput by roughly 10%. Creators exacerbated the backlog by re-uploading unconfirmed episodes, while four hours elapsed between the earliest alerts and formal incident response. Engineers halted the batch job, patched the scheduling defect, and brought an extra compute cluster online to clear all backlogs overnight. Spotify subsequently increased transcoding capacity by approximately 67% and initiated cross-team investments in workload prioritization, rate limiting, and earlier capacity alerting.

  Spotify Engineering

• Spotify · Jun 10, 2026

  Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant

  Spotify developed an AI data assistant, Vedder, to scale access to over 70,000 internal datasets without overwhelming domain experts. Because raw warehouse schemas omit critical semantic nuances and overwhelm LLM context windows, engineers implemented a domain-owned context layer organized into clusters. Each cluster contains profiled datasets, expert-curated business documentation, and vetted question-and-SQL pairs that guide a ReAct-based agent. Automated query history extraction proved insufficient on its own, as domain curators accepted only 12.5% of inferred query pairs due to noise. Deployed across Slack, IDEs, and a dedicated web interface, the system maintains reliability through continuous cluster health monitoring that flags schema drift and degraded examples.

  Spotify Engineering

• Spotify · Jun 3, 2026

  Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify

  Spotify experienced rapid codebase growth alongside widespread adoption of AI coding tools, resulting in a 76% increase in pull request frequency. To manage fleet-wide codebase maintenance and complex refactoring, the company combined its Fleetshift automation platform with Honk, a background coding agent powered by Claude. Honk runs inside Kubernetes pods with access to CI build environments and integrates with Backstage via Model Context Protocol tools to inspect component metadata. Standardized service architectures and active lint guardrails enable the agent to maintain consistency and auto-correct invalid design patterns during execution. By automating routine migrations, Spotify reduced fleet-wide Java backend updates from multi-week cross-team efforts down to three days handled by a single engineer.

  Spotify Engineering