Loading…
Spotify
Global music streaming platform developing innovative audio technology and developer tools like Backstage
- 10 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Introducing: Spotify Technology. Proven at Spotify, now yours.
Spotify is launching Spotify Technology, a brand that unites Portal, Confidence, Xirp, and Spotify for Backstage into one system covering the product development lifecycle. These tools were built internally to solve problems at Spotify’s scale, where hundreds of teams ship code daily to 777 million monthly active users, and are now available as open source or commercial products. Portal encodes organizational knowledge so teams and agents build with shared context, Xirp brings that context into agentic development, and Confidence turns experimentation into product intelligence. The tools share a context layer and agent runtime so they compound rather than just coexist, and each earned its place by beating alternatives inside Spotify. Spotify Technology aims to become the connective tissue for R&D organizations, capturing what the organization knows, how it builds, and what it has learned.
AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity
Spotify examines how AI-accelerated development and broader industry compute demand challenged quality across content processing, fleet updates, regional failover, and the mobile app. At its scale—777 million monthly active users, about 100 million concurrent clients, 11–12 million backend requests per second, and nearly 3,000 production services—small control gaps can become visible to listeners and creators. A June 24 processing incident combined competing batch work, higher per-episode compute, and a scheduler bug that cut throughput about 10%, leading to hour-long publication delays; Spotify responded with end-to-end monitoring, scheduler fixes, prioritization, and more capacity. Incident reviews found no material direct contribution from AI-authored code, but change volume outpaced some verification controls, prompting stronger review, testing, rollout, observability, rollback, quality signals, and failover practices. Spotify reports no corresponding rise in rework rate, while continuing to watch increasing PR size and complexity.
Why Spotify Is Not Using Bayesian A/B Testing
Spotify argues that Bayesian A/B testing is not a single method but a family of configurations defined by a prior, likelihood, and stopping rule. It contrasts flat-prior posterior-probability stopping, which can reproduce frequentist peeking false-positive rates, with Bayes-factor stopping, which controls false positives under optional stopping, and calibrated empirical Bayes priors, which can shrink estimates and control false discovery rates. The post frames experiment design around program goals such as limiting harmful or ineffective launches, achieving estimation precision, and minimizing costs rather than selecting a statistical framework first. It says Bayesian and frequentist approaches overlap substantially, while Bayesian advantages such as richer probability statements require additional prior complexity. Spotify therefore continues using frequentist tooling because supporting two modes would add planning, monitoring, interpretation, and configuration costs without a sufficient benefit.
Portal by Spotify cut my Claude Code token usage by 90%
Claude Code often spends frontier-model tokens on file I/O and predictable code generation rather than reasoning, motivating a routing setup built with Portal by Spotify’s AiKA Modes. The implementation defines two public modes—bulk-reader for multi-file analysis and code-writer for pattern-based generation—both shown with Gemini 2.5 Flash, while a Claude Code plugin named shunt enforces delegation through hooks, scripts, and skills. PreToolUse hooks block large Read and shell-based file reads above a configurable threshold, while scripts send files or references to ephemeral worker invocations without exposing the full file corpus or generated code to Claude’s context. In a Java monorepo benchmark, mean bulk-read savings were around 90%, although code-write savings were harder to measure. The approach does not suit editing or reasoning tasks: worker summaries lacked reliable line numbers, missed a subtle thread-safety bug, and added 10–30 seconds of latency, so targeted reads and Claude’s reasoning remain necessary.
Spotify’s Backstage Portal cut my Claude Code token usage by 90%
Claude Code often spends frontier-model tokens on file I/O rather than reasoning, prompting a routing approach that sends predictable work to cheaper models. The post implements this with two public AiKA Modes in Spotify’s Backstage Portal: bulk-reader summarizes large file sets, and code-writer generates patterned files from a specification and reference file, using Gemini 2.5 Flash in the examples. A Claude Code plugin called shunt enforces delegation through PreToolUse hooks, scripts, and skills; reads above a configurable threshold, defaulting to 350 lines, are blocked and redirected. In tests on a Java monorepo, bulk-read saved around 90% of Claude’s tokens, while generated code can go directly to disk. The design does not cover editing or reasoning, and 10–30-second delegation latency makes small tasks counterproductive.
When Can LLMs Replace Humans in A/B Tests?
Replacing human participants in A/B testing with large language model predictions eliminates causal identification guarantees by design, forcing reliance on surrogate endpoint assumptions. An empirical evaluation using the Upworthy dataset revealed that raw gpt-4o-mini click-through rate predictions exhibited systematic attenuation bias, capturing only 39 percent of observed human treatment effects. To address this discrepancy, researchers formalized surrogacy and comparability conditions, finding that nonlinear machine learning calibrators like random forests and gradient-boosted trees successfully recovered treatment effects where linear ordinary least squares failed. In addition, averaging multiple sample draws per experimental unit mitigated variance and zero-bias caused by model generation temperature. Ultimately, because comparability cannot be verified for novel interventions that depart significantly from historical experiments, calibrated models cannot entirely replace human testing for genuine product innovations.
Indexing the Data Lake for Online Point Queries
Online services and AI agents require low-latency point queries over massive datasets that are economically impractical to store entirely in key-value systems like Bigtable or DynamoDB. While cloud object storage offers rapid request latency, traditional distributed query engines like Trino and BigQuery introduce seconds of planning and scheduling overhead along with dependent file reads. Random Access Parquet addresses this limitation by using an external index mapping keys directly to exact Parquet files and row locations. Readers execute O(1) index lookups and issue parallel ranged reads to fetch only the required bytes without scanning footers or metadata. By running directly against existing data lake files, this architecture eliminates separate storage copies and allows shared analytical data to serve interactive online lookups.
Content Ingestion & Podcast Video Incident Report
On June 24, video transcoding infrastructure reached maximum capacity, causing hours of publishing delays for Spotify video podcast episodes. Four converging factors drove the disruption: insufficient headroom for content spikes, an active routine batch processing job, higher compute costs from a recent video quality upgrade, and a scheduling bug reducing hardware throughput by roughly 10%. Creators exacerbated the backlog by re-uploading unconfirmed episodes, while four hours elapsed between the earliest alerts and formal incident response. Engineers halted the batch job, patched the scheduling defect, and brought an extra compute cluster online to clear all backlogs overnight. Spotify subsequently increased transcoding capacity by approximately 67% and initiated cross-team investments in workload prioritization, rate limiting, and earlier capacity alerting.
Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant
Spotify developed an AI data assistant, Vedder, to scale access to over 70,000 internal datasets without overwhelming domain experts. Because raw warehouse schemas omit critical semantic nuances and overwhelm LLM context windows, engineers implemented a domain-owned context layer organized into clusters. Each cluster contains profiled datasets, expert-curated business documentation, and vetted question-and-SQL pairs that guide a ReAct-based agent. Automated query history extraction proved insufficient on its own, as domain curators accepted only 12.5% of inferred query pairs due to noise. Deployed across Slack, IDEs, and a dedicated web interface, the system maintains reliability through continuous cluster health monitoring that flags schema drift and degraded examples.
Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify
Spotify experienced rapid codebase growth alongside widespread adoption of AI coding tools, resulting in a 76% increase in pull request frequency. To manage fleet-wide codebase maintenance and complex refactoring, the company combined its Fleetshift automation platform with Honk, a background coding agent powered by Claude. Honk runs inside Kubernetes pods with access to CI build environments and integrates with Backstage via Model Context Protocol tools to inspect component metadata. Standardized service architectures and active lint guardrails enable the agent to maintain consistency and auto-correct invalid design patterns during execution. By automating routine migrations, Spotify reduced fleet-wide Java backend updates from multi-week cross-team efforts down to three days handled by a single engineer.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.