---
title: "TensorRT LLM"
description: "2 posts about TensorRT LLM, summarised, each linking to the original."
---

# TensorRT LLM
> 2 posts about TensorRT LLM, summarised, each linking to the original.

## Articles

### [Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLM](https://yomu.fyi/post/advancing-generative-recommendation-inference-our-collaboration-with-n.md)
- Company: [Spotify](https://yomu.fyi/company/spotify.md)
- Author: Spotify Engineering
- Published: Oct 16, 2026
- Type: Problem & solution

Spotify has been applying generative AI to personalize recommendations across surfaces, using LLMs for podcast discovery with beam search and candidate scoring. Their initial inference stack relied on vLLM for beam search, which suffered from CPU-side bottlenecks and KV block recomputation, and SGLang for efficient multi-item scoring, but no single engine supported all required features. To unify these capabilities, Spotify collaborated with NVIDIA to extend TensorRT LLM with efficient beam search implemented in the sampler, multi-item scoring via the Python API, and soft token personalization using out-of-tree model plugins. On a 1.7B Qwen3 model, the resulting implementation delivered up to 54% cost reduction for beam search workloads and 7% for scoring workloads compared to previous setups. Spotify plans to continue optimizing inference with quantization and NVIDIA Blackwell-specific techniques to support increasingly sophisticated personalization at scale.


### [Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing](https://yomu.fyi/post/enabling-private-high-performance-production-ai-inference-with-nvidia.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Sep 22, 2026
- Type: Explainer

NVIDIA Confidential Computing (CC) enables secure AI inference by using memory-encrypted confidential VMs, confidential GPUs, and encrypted NVLink. Running inference frameworks like TensorRT LLM in a CC environment introduces performance overhead because secure execution changes assumptions about memory movement, timing, and multi-GPU communication. To quantify this overhead, the post presents a controlled methodology comparing CC-on and CC-off performance on a workload with long input context, extended output generation, and low concurrency. With CC-aware adaptations—including pageable memory selection, asynchronous readback, GPU %globaltimer for autotuning, and CC-aware multi-GPU communication—TensorRT LLM retained 96.1–98.2% of baseline throughput on eight NVIDIA B200 GPUs while keeping per-token latency overhead below 5%. The post concludes that security configuration and inference optimization should be treated as a single deployment problem for organizations moving private inference into production.