Loading post…
SpotifySpotify EngineeringProblem & solution
Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLM
Summary
Spotify has been applying generative AI to personalize recommendations across surfaces, using LLMs for podcast discovery with beam search and candidate scoring. Their initial inference stack relied on vLLM for beam search, which suffered from CPU-side bottlenecks and KV block recomputation, and SGLang for efficient multi-item scoring, but no single engine supported all required features. To unify these capabilities, Spotify collaborated with NVIDIA to extend TensorRT LLM with efficient beam search implemented in the sampler, multi-item scoring via the Python API, and soft token personalization using out-of-tree model plugins. On a 1.7B Qwen3 model, the resulting implementation delivered up to 54% cost reduction for beam search workloads and 7% for scoring workloads compared to previous setups. Spotify plans to continue optimizing inference with quantization and NVIDIA Blackwell-specific techniques to support increasingly sophisticated personalization at scale.
Context
Spotify’s engineering team describes their collaboration with NVIDIA to extend TensorRT LLM for generative recommendation inference, addressing limitations in existing inference engines like vLLM and SGLang.
Approach / What changed
Spotify and NVIDIA extended TensorRT LLM with efficient beam search implemented in the sampler, multi-item scoring via the Python API, and soft token personalization using out-of-tree model plugins, enabling a unified inference engine for generative recommendations.
Takeaways
vLLM’s beam search implementation introduces CPU-side bottlenecks and requires recomputation of partially filled KV blocks, limiting GPU utilization for generative recommendation inference.
TensorRT LLM’s beam search, implemented in the sampler with overlap scheduling, provides up to 54% cost reduction for beam search workloads on a 1.7B Qwen3 model compared to previous setups.
Spotify uses soft tokens projected from user embeddings into the LLM’s embedding space to personalize recommendations, implemented via a custom Qwen3 extension plugin in TensorRT LLM.
FAQ
Why did Spotify move from vLLM and SGLang to TensorRT LLM?
vLLM's beam search had CPU-side bottlenecks and KV block recomputation, while SGLang lacked beam search support. TensorRT LLM offered better GPU utilization for beam search, and Spotify collaborated with NVIDIA to add missing features like multi-item scoring and soft token support, unifying the inference stack.
How does Spotify personalize LLM recommendations using soft tokens?
Spotify projects existing user embeddings into the LLM's embedding space as a single soft token, which is inserted into the prompt before the forward pass. This approach, based on the Embedding to Prefix method, keeps prompts short and enables personalization without large text histories.
What performance improvements did the collaboration achieve?
On a 1.7B Qwen3 model, the TensorRT LLM implementation delivered up to 54% cost reduction for beam search workloads and up to 7% cost reduction for scoring workloads compared to previous setups.
Source and freshness
- Source
- engineering.atspotify.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.