Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant

Summary

Spotify developed an AI data assistant, Vedder, to scale access to over 70,000 internal datasets without overwhelming domain experts. Because raw warehouse schemas omit critical semantic nuances and overwhelm LLM context windows, engineers implemented a domain-owned context layer organized into clusters. Each cluster contains profiled datasets, expert-curated business documentation, and vetted question-and-SQL pairs that guide a ReAct-based agent. Automated query history extraction proved insufficient on its own, as domain curators accepted only 12.5% of inferred query pairs due to noise. Deployed across Slack, IDEs, and a dedicated web interface, the system maintains reliability through continuous cluster health monitoring that flags schema drift and degraded examples.

Context

Spotify faced scaling bottlenecks as internal demand for insights across 70,000+ datasets and 1.4 trillion daily data points outpaced the capacity of domain experts. Feeding raw schemas into LLMs failed because context windows cannot accommodate the warehouse and schemas lack critical semantics, such as business definitions or legacy test data filtering rules.

Approach / What changed

Spotify created a context layer structured into clusters owned by domain experts. Each cluster bundles profiled datasets, business documentation, and curated question-SQL few-shot pairs. An agent follows a ReAct loop to select context, generate and execute SQL, and expose reasoning across Slack, an IDE MCP server, and a web UI, backed by cluster health scores that track schema drift.

Takeaways

• Relying on raw warehouse query history alone to generate few-shot examples failed because domain curators rejected 87.5% of inferred query pairs due to ad-hoc exploration, debugging noise, and bad patterns.
• Effective text-to-SQL context requires three components: table profiling with value samples, expert-vetted question-and-SQL canonical pairs, and explicit business logic documentation.
• Automated cluster health scores help prevent context decay by alerting domain experts to column renames, invalid curated pairs, poor SQL reproducibility, and lagging question coverage.

Related reading

• Spotify · Jun 3, 2026

  Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify

  Spotify experienced rapid codebase growth alongside widespread adoption of AI coding tools, resulting in a 76% increase in pull request frequency. To manage fleet-wide codebase maintenance and complex refactoring, the company combined its Fleetshift automation platform with Honk, a background coding agent powered by Claude. Honk runs inside Kubernetes pods with access to CI build environments and integrates with Backstage via Model Context Protocol tools to inspect component metadata. Standardized service architectures and active lint guardrails enable the agent to maintain consistency and auto-correct invalid design patterns during execution. By automating routine migrations, Spotify reduced fleet-wide Java backend updates from multi-week cross-team efforts down to three days handled by a single engineer.

  Spotify Engineering

• Spotify · Jul 27, 2026

  Indexing the Data Lake for Online Point Queries