---
title: "Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLM"
description: "Spotify has been applying generative AI to personalize recommendations across surfaces, using LLMs for podcast discovery with beam search and candidate scoring. Their initial inference stack relied on vLLM for beam search, which suffered from CPU-side bottlenecks and KV block recomputation, and SGLang for efficient multi-item scoring, but no single engine supported all required features. To unify these capabilities, Spotify collaborated with NVIDIA to extend TensorRT LLM with efficient beam search implemented in the sampler, multi-item scoring via the Python API, and soft token personalization using out-of-tree model plugins. On a 1.7B Qwen3 model, the resulting implementation delivered up to 54% cost reduction for beam search workloads and 7% for scoring workloads compared to previous setups. Spotify plans to continue optimizing inference with quantization and NVIDIA Blackwell-specific techniques to support increasingly sophisticated personalization at scale."
---

# Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLM

[Spotify](https://yomu.fyi/company/spotify) · Spotify Engineering · Oct 16, 2026

**Type:** Problem & solution

## Summary

Spotify has been applying generative AI to personalize recommendations across surfaces, using LLMs for podcast discovery with beam search and candidate scoring. Their initial inference stack relied on vLLM for beam search, which suffered from CPU-side bottlenecks and KV block recomputation, and SGLang for efficient multi-item scoring, but no single engine supported all required features. To unify these capabilities, Spotify collaborated with NVIDIA to extend TensorRT LLM with efficient beam search implemented in the sampler, multi-item scoring via the Python API, and soft token personalization using out-of-tree model plugins. On a 1.7B Qwen3 model, the resulting implementation delivered up to 54% cost reduction for beam search workloads and 7% for scoring workloads compared to previous setups. Spotify plans to continue optimizing inference with quantization and NVIDIA Blackwell-specific techniques to support increasingly sophisticated personalization at scale.

## Context

Spotify’s engineering team describes their collaboration with NVIDIA to extend TensorRT LLM for generative recommendation inference, addressing limitations in existing inference engines like vLLM and SGLang.

## Approach / What changed

Spotify and NVIDIA extended TensorRT LLM with efficient beam search implemented in the sampler, multi-item scoring via the Python API, and soft token personalization using out-of-tree model plugins, enabling a unified inference engine for generative recommendations.

## Takeaways

- vLLM’s beam search implementation introduces CPU-side bottlenecks and requires recomputation of partially filled KV blocks, limiting GPU utilization for generative recommendation inference.
- TensorRT LLM’s beam search, implemented in the sampler with overlap scheduling, provides up to 54% cost reduction for beam search workloads on a 1.7B Qwen3 model compared to previous setups.
- Spotify uses soft tokens projected from user embeddings into the LLM’s embedding space to personalize recommendations, implemented via a custom Qwen3 extension plugin in TensorRT LLM.

## FAQ

### Why did Spotify move from vLLM and SGLang to TensorRT LLM?

vLLM's beam search had CPU-side bottlenecks and KV block recomputation, while SGLang lacked beam search support. TensorRT LLM offered better GPU utilization for beam search, and Spotify collaborated with NVIDIA to add missing features like multi-item scoring and soft token support, unifying the inference stack.

### How does Spotify personalize LLM recommendations using soft tokens?

Spotify projects existing user embeddings into the LLM's embedding space as a single soft token, which is inserted into the prompt before the forward pass. This approach, based on the Embedding to Prefix method, keeps prompts short and enables personalization without large text histories.

### What performance improvements did the collaboration achieve?

On a 1.7B Qwen3 model, the TensorRT LLM implementation delivered up to 54% cost reduction for beam search workloads and up to 7% cost reduction for scoring workloads compared to previous setups.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [NVIDIA](https://yomu.fyi/topic/nvidia), [Recommendation Systems](https://yomu.fyi/topic/recommendation-systems), [TensorRT LLM](https://yomu.fyi/topic/tensorrt-llm)

- Source: [Spotify](https://engineering.atspotify.com/2026/10/advancing-generative-recommendation-inference-our-collaboration-with-nvidia-on-tensorrt-llm)
- Source URL: https://engineering.atspotify.com/2026/10/advancing-generative-recommendation-inference-our-collaboration-with-nvidia-on-tensorrt-llm
- Ingested by Yomu: 2026-10-16T15:00:17.527Z

[Read original post](https://engineering.atspotify.com/2026/10/advancing-generative-recommendation-inference-our-collaboration-with-nvidia-on-tensorrt-llm)