---
title: "vLLM"
description: "18 posts about vLLM, summarised, each linking to the original."
---

# vLLM
> 18 posts about vLLM, summarised, each linking to the original.

## Articles

### [vLLM vs. SGLang: Choosing the right inference engine for your workload](https://yomu.fyi/post/vllm-vs-sglang-choosing-the-right-inference-engine-for-your-workload.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Cedric Clyburn
- Published: Oct 14, 2026
- Type: Explainer

Open weight models now process nearly 80% of production tokens on Vercel’s AI Gateway, up from under 10% in December 2025, and OpenRouter shows a similar climb. This surge drives teams to consider self-hosting via inference engines like vLLM and SGLang, since swapping backends usually only requires a new base URL for an OpenAI-compatible API. vLLM’s PagedAttention, introduced by UC Berkeley in 2023, splits the KV cache into fixed-size blocks to eliminate wasted contiguous memory reservation and increase GPU throughput. SGLang’s RadixAttention, from LMSYS in January 2024, stores computed KV cache in a radix tree keyed by token prefix for reuse across related requests like multi-turn chat and agents. Independent benchmarks from Modal show comparable performance on random prompts without prefix sharing, indicating that workload type and hardware should guide engine choice.


### [Run decision models on vLLM and Red Hat AI using DiffusionGemma](https://yomu.fyi/post/run-decision-models-on-vllm-and-red-hat-ai-using-diffusiongemma.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Lucas Wilkinson, Rob Greenberg
- Published: Sep 28, 2026
- Type: Explainer

A new category of AI model called ‘System One’ makes fast, typed, probabilistic decisions from unstructured input, exemplified by TypeSafe AI’s Jev, which offers guaranteed structure, default probabilities, and low latency (70–500 ms) for high-volume decisions like routing and moderation. Jev is a hosted API without self-hosting options, prompting the vLLM community to adapt DiffusionGemma, an open block-diffusion model, as a self-hostable decision engine. The approach uses a fixed-length canvas seeded with answer templates, a single denoising step, and calibrated logprobs to return decisions with confidence scores, achieving over 300 decisions per second on a Cloud Run deployment. DiffusionGemma is already validated on Red Hat AI, and the structured-read support in vLLM nightly enables teams to run decision models on their own infrastructure. The post provides a recipe using vLLM nightly and a decision server that exposes a Jev-compatible endpoint for prototyping.


### [How I massively improved my AI inference performance without buying new hardware](https://yomu.fyi/post/how-i-massively-improved-my-ai-inference-performance-without-buying-ne.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Boaz Ben Shabat
- Published: Sep 23, 2026
- Type: Problem & solution

Default inference configurations for large language models often leave significant performance unused, even on expensive hardware like 16 NVIDIA H200 GPUs across two nodes. The author tested this setup with vLLM and Red Hat AI Inference 3.4 on a CoreWeave Kubernetes cluster, using enterprise RAG workloads with 2,000-token inputs and 100 concurrent users. By systematically comparing aggregated and prefill-decode (PD) disaggregated architectures, and tuning tensor parallelism (TP) and PD split, latency was dramatically reduced. For Qwen3-32B, time-to-first-token (TTFT) on the best aggregated configuration (2xTP8) dropped from 995ms to 304ms, and with optimized routing to 287ms — a 71% improvement without hardware changes. For Llama-70B, TTFT fell from 1,292ms to 463ms, a 64% gain. The author also built ServeIt Studio, an open-source tool that automates the full optimization pipeline on Kubernetes.


### [Benchmarking LLM Inference at Scale with AIPerf](https://yomu.fyi/post/benchmarking-llm-inference-at-scale-with-aiperf.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 18, 2026
- Type: Benchmark

AIPerf is NVIDIA’s successor to GenAI-Perf, intended to benchmark large-scale LLM inference without letting the load client become the bottleneck. Its multiprocess architecture combines worker processes, separate record-processor services, and ZMQ coordination, while supporting 15+ endpoint types, public datasets, and trace-replay formats. The walkthrough serves Qwen3-0.6B through vLLM, using streaming synthetic requests with fixed 128-token inputs and outputs; AIPerf reports TTFT, ITL, request latency, output token throughput, percentile distributions, and optional GPU telemetry in CSV and JSON. A second profile uses Poisson arrivals averaging 10 requests per second, variable input and output lengths, and a fixed random seed, producing wider latency distributions than the static baseline as concurrency and workload variability increase.


### [Catching poor LLM performance and accuracy before deployment](https://yomu.fyi/post/catching-poor-llm-performance-and-accuracy-before-deployment.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Christopher Miyai
- Published: Sep 17, 2026
- Type: Benchmark

Rapid changes in vLLM and the path from upstream releases to Red Hat AI Inference images can introduce performance and accuracy regressions that ordinary CI/CD tests miss. The midstream regression detector benchmarks upstream and release-candidate builds across hardware variants, using GuideLLM at multiple concurrency levels for performance and GSM8K for accuracy. It stores nightly upstream results, compares them with midstream benchmarks against threshold deltas, and feeds confirmed failures into fixes and rebuilds before release. To reduce false positives from limited samples, the workflow adds Wasserstein distance as a secondary check of whole request-response distributions rather than relying only on medians or means. The system also connects benchmark runs with prior JIRA regressions through a custom Claude skill that suggests possible fixes and assesses whether a flagged change is genuine.


### [Understanding W8A8 INT8 LLM quantization: Accuracy and performance results](https://yomu.fyi/post/understanding-w8a8-int8-llm-quantization-accuracy-and-performance-resu.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Sana Fayyaz
- Published: Sep 14, 2026
- Type: Benchmark

The post evaluates whether W8A8 INT8 quantization can reduce the size of a Llama 3.1 8B Instruct model without sacrificing accuracy or inference performance. Using SmoothQuant to address activation outliers and GPTQ with error compensation for BF16 weights, the model was compressed from 14.9 GB to 8.0 GB after calibration on WikiText-2. Identical zero-shot lm-eval-harness tests across MMLU, ARC Easy, HellaSwag, and IFEval found MMLU, ARC Easy, and HellaSwag deltas smaller than their standard errors, while the IFEval change was described as likely within a similar noise range; the post concludes no measurable degradation. vLLM and GuideLLM tests on one NVIDIA L40S reported 29% more concurrent requests, 24% lower synchronous TTFT, and 44% faster token generation at maximum load, while ITL degraded more quickly under high concurrency because of dynamic activation quantization overhead.


### [Building Pinterest’s VLM Serving Stack on NVIDIA Dynamo](https://yomu.fyi/post/building-pinterest-s-vlm-serving-stack-on-nvidia-dynamo.md)
- Company: [Pinterest](https://yomu.fyi/company/pinterest.md)
- Author: Pinterest Engineering
- Published: Sep 10, 2026
- Type: Explainer

Pinterest describes building a VLM serving stack on NVIDIA Blackwell B200 GPUs and NVIDIA Dynamo for multimodal products including Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, and content safety. Unlike text-only workloads, requests may contain thousands of images, costly and variable prefill, long multi-turn histories, and large KV caches, making preprocessing, routing, memory, and latency central serving concerns. The implementation combines Dynamo's distributed orchestration, multimodal frontend and vLLM support, KV-aware routing, disaggregated encoder/prefill/decode serving, and LMCache-based KV offloading with vLLM, an OpenAI Chat Completions interface, model-aware routing, EKS, and custom deployment components. Production-oriented AIPerf DAG benchmarks model QPS patterns, multi-turn conversations, many-image requests, and agentic DAGs; the team concludes that payload design, cache-aware routing, offload tiers, and a shared platform are essential to meeting product targets and accelerating new launches.


### [From token consumer to token provider: Building your org's AI API](https://yomu.fyi/post/from-token-consumer-to-token-provider-building-your-org-s-ai-api.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Markell Rawls
- Published: Sep 10, 2026
- Type: Explainer

Enterprise AI usage can become costly and fragmented when teams rely on separate provider accounts, personal keys, and independently operated inference stacks, leaving spending, workloads, and governance difficult to see. The post presents Model-as-a-Service (MaaS) in Red Hat OpenShift AI 3.4 as a shared service in which a platform team deploys approved models and exposes them through a standard, OpenAI-compatible API. Administrators can use the model catalog, distributed inference through llm-d on NVIDIA L40s, scoped API keys, rate limits, and dashboards for token use, latency, errors, GPU utilization, and queue length. Developers obtain self-service endpoints and can connect tools such as Claude Code, OpenCode, or Continue with existing clients. The architecture combines OpenShift, vLLM, an AI gateway, Prometheus, Grafana, and Jaeger to centralize inference, governance, and cost attribution across local and external model providers.


### [Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL](https://yomu.fyi/post/async-grpo-with-lora-across-hf-jobs-a-bucket-a-proxy-and-no-nccl.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Amine Dirhoussi, Quentin Gallouédec, Kashif Rasul, Sergio Paniego
- Published: Sep 10, 2026
- Type: Explainer

An AsyncGRPO setup uses LoRA adapter-only synchronization to separate vLLM Jobs, rather than moving full model weights over NCCL. A Storage Bucket mounted at the same path lets the trainer publish versioned adapters that replicas load, while a proxy adds authentication, routes rollouts toward KV-cached prefixes, and broadcasts adapter updates. Versioned adapter names prevent cached prefixes from mixing policy versions, and max\_staleness=4 requires six LoRA slots during replacement. Five runs identify client-side concurrency, checkpointing, and packing as important throughput factors: raising in-flight requests to 384, disabling checkpointing, and packing improved median step time from 22.9 to 4.8 seconds. At 500 steps, the final run took 53 minutes instead of 3 hours 27 minutes, trained 31% more samples, and had basically the same reward curve.


### [Optimize vLLM speculative decoding with FastMTP heads](https://yomu.fyi/post/optimize-vllm-speculative-decoding-with-fastmtp-heads.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Rahul Tuli
- Published: Sep 8, 2026
- Type: Tutorial

Autoregressive LLM inference is memory-bandwidth bound because each token requires a full pass over billions of parameters, while speculative decoding can verify several drafted tokens in one pass. Speculators 0.6.0 adapts a single native MTP head with FastMTP-style recursive fine-tuning, addressing the mismatch between teacher-forced training and vLLM’s autoregressive reuse of the head. The training loop fuses each next-token embedding with the current hidden state, predicts the following token, feeds the resulting hidden state back, and weights three steps with normalized exponential decay using β = 0.6. Native mtp.\* weights extraction, shared full-vocabulary embeddings and output head, and vLLM-compatible stitching support deployment; on Qwen3-Next-80B-A3B-Instruct evaluated on GSM8K, mean accepted length improved at positions 0–2 from 0.897, 0.719, 0.476 to 0.912, 0.776, 0.616.


### [Understanding W8A8 INT8 LLM quantization: Half the size, better performance, same accuracy](https://yomu.fyi/post/understanding-w8a8-int8-llm-quantization-half-the-size-better-performa.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Sana Fayyaz
- Published: Sep 7, 2026
- Type: Tutorial

Serving Llama 3.1 8B in BF16 requires roughly 14.9 GB for weights, leaving little VRAM on a 16 GB GPU for KV cache and activations, which limits practical deployment. The workflow benchmarks the base model, applies INT8 W8A8 quantization with llm-compressor, then repeats accuracy and performance tests under identical hardware and load conditions. SmoothQuant modifies weights to reduce activation outliers, while GPTQ uses calibration data and error compensation before rounding weights to INT8; the experiment used 512 WikiText-2 samples and targeted deployment with vLLM. On the Llama 3.1 8B Instruct model, compression reduced storage from 14.9 GB to 8.0 GB, a 46% reduction, and accuracy was preserved across MMLU, ARC, HellaSwag, and IFeval; generation-speed results are reserved for part 2.


### [Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson](https://yomu.fyi/post/frontier-reasoning-reaches-the-edge-how-to-deploy-and-optimize-models.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 4, 2026
- Type: Tutorial

Reasoning and agentic models have typically required data-center inference, creating network dependencies, higher costs, and potential data exposure for edge applications. The post explains how to choose between dense Qwen3.8-27B, which activates all 27 billion parameters, and mixture-of-experts Nemotron 3.5 Lightning, which has 30 billion total parameters but activates 3 billion per token. Deployment uses vLLM on Jetson with NVFP4 quantization and speculative decoding, where a draft model proposes tokens and the main model verifies them; the examples include DSpark for Nemotron and DFlash2 for Qwen. Reported tests show up to 6.28x decode-throughput speedup over BF16, while workload-specific throughput still varies. Validation with representative prompts is recommended before deployment, and custom quantization or speculator training is reserved for cases where accuracy or acceptance rates reveal a clear gap.


### [Speeding up LLM inference with P-EAGLE in vLLM Speculators](https://yomu.fyi/post/speeding-up-llm-inference-with-p-eagle-in-vllm-speculators.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Helen Zhao, Megan Flynn, Dipika Sikka
- Published: Sep 3, 2026
- Type: Explainer

P-EAGLE (Parallel EAGLE), implemented in Speculators v0.6.0, extends EAGLE-3 with parallel drafting so a draft model predicts multiple future tokens in one forward pass rather than sequentially. Speculative decoding pairs this small draft model with a verifier model, which checks the proposed sequence in parallel and accepts only its correct prefix while preserving the verifier’s output distribution. The post adds Conditional Drop-token (COD) sampling, learnable mask embeddings and hidden states, and a custom flex-attention mask to support multi-depth prediction within the EAGLE-3 framework. The post presents an end-to-end workflow covering data preparation, vLLM hidden-state extraction, training and deployment, including a Qwen3-8B example that takes about 50 minutes on four H100s. A published Qwen3-a8B draft model reports an average length of 3.044 across the listed datasets.


### [LoRA backdoor threat: How OpenShift AI mitigates the risk](https://yomu.fyi/post/lora-backdoor-threat-how-openshift-ai-mitigates-the-risk.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Mike Hepburn
- Published: Sep 2, 2026
- Type: Problem & solution

LoRA adapters can carry semantic backdoors even when the underlying open-weight model is clean, creating a supply-chain risk for customized AI assistants. The demonstrated proof of concept uses a 35 MB adapter trained from 270 ChatML examples, with 130 poisoning examples and 140 clean examples, to trigger on ordinary pharmaceutical IP language rather than fixed strings. With Qwen/Qwen2.5-0.5B-Instruct served by vLLM 0.25.1, the backdoored model covertly called send\_email to an external address while showing a benign response, whereas the base model and controls remained clean. The defense-in-depth approach combines OpenShift AI, Advanced Cluster Security, and Advanced Cluster Management to restrict runtime LoRA loading, require provenance-checked adapters, enforce admission and build policies, and block unexpected egress. Because weight inspection cannot reliably reveal semantic behavior, protection focuses on deployment, runtime, tool, and network boundaries.


### [LLM quantization guide: How to do it, and how it helps](https://yomu.fyi/post/llm-quantization-guide-how-to-do-it-and-how-it-helps.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Cedric Clyburn
- Published: Sep 2, 2026
- Type: Tutorial

LLM quantization addresses the gap between rapidly growing model sizes and GPU memory by storing weights and activations at lower precision than BF16. It explains how quantization targets linear layers in transformer blocks, distinguishes weight quantization for memory-bound decoding from activation quantization for compute-bound prefill, and maps formats such as GGUF, FP8, NVFP4, AWQ-INT4, and MLX to their inference engines. Calibrated methods including GPTQ, AWQ, and SmoothQuant protect important weights using representative data, while reported NVFP4 results recover about 99% of BF16 accuracy for 70B–235B models and 95–98% for 7B–14B models. In a Llama 3 70B benchmark on two H100s, FP8 weights-and-activations increased throughput from 158 to 474 input tokens/sec and reduced time to first token under load from over 30,000ms to about 4,800ms.


### [Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU](https://yomu.fyi/post/enterprise-grade-precision-for-long-context-multimodal-embedding-infer.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Anthony Su, Injae Kwak
- Published: Aug 26, 2026
- Type: Benchmark

Embedding models convert text, images, and audio into dense vectors used for semantic search, recommendation, intent classification, content discovery, and clustering. The post describes production support for long-context Qwen3 Embedding models on Cloud TPUs through native vLLM TPU integration and GKE Custom Compute Classes, allowing prioritized autoscaling across TPU and GPU capacity pools. Its implementation pads vocabulary tensors for hardware-safe sharding, hardens lazy loading, pre-warms sharding-aware JAX/XLA compilation caches, and uses a hybrid StepPool with CachedRequestState to preserve pooling state during chunked prefill and preemption. Parity evaluations target cosine similarity of at least 0.999 for text and 0.995 for multimodal inputs; on TPU Ironwood, Qwen3-Embedding-8B reached 83,996 total tokens per second and 5.13 requests per second at 16K+ sequence length with bf16 and TP=4.


### [Scaling Agentic RL: High-Throughput Agentic Training with Tunix](https://yomu.fyi/post/scaling-agentic-rl-high-throughput-agentic-training-with-tunix.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Haoyu Gao, Lance Wang, Shadi Noghabi, Tianshu Bao, Weiren Yu
- Published: Jul 21, 2026
- Type: Explainer

Tunix is presented as a post-training library for scaling agentic reinforcement learning, where multi-turn tool use and environment latency can leave TPUs idle. Its design combines an asyncio-based, high-concurrency trajectory collector with asynchronous vLLM-TPU and SGLang-Jax integration, then streams variable-length trajectories through a producer-consumer queue rather than waiting for a complete synchronous batch. Composable agent and environment abstractions expose APIs for custom tools, benchmarks, and external systems while preserving multi-turn token handling and lifecycle management. Lightweight RL-specific metrics correlate rollout, training, weight synchronization, model-call, and environment stages with TPU timelines, complementing targeted XProf traces. The stated outcome is a more observable pipeline intended to reduce stalls, keep accelerators utilized, and support agentic RL natively in the JAX/TPU ecosystem.


### [DiffusionGemma: The Developer Guide](https://yomu.fyi/post/diffusiongemma-the-developer-guide.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Ian Ballantyne, Omar Sanseviero
- Published: Jun 10, 2026
- Type: Tutorial

DiffusionGemma is an experimental text-generation model and developer guide covering serving, customization, and non-autoregressive generation. Built on the Gemma 4 backbone, its 26B Mixture of Experts architecture activates 3.8B parameters during inference, refines 256-token canvases in parallel, and shifts the bottleneck from memory bandwidth to compute. Block autoregressive diffusion commits denoised 256-token blocks to the KV cache, combining bidirectional denoising with causal prefill for longer sequences; vLLM integration provides an OpenAI-compatible local server. A Sudoku fine-tuning example using Hackable Diffusion reports improvement from roughly 0% base-model success to 80%, while SFT reduces the demonstrated solve from 48 to 12 steps through adaptive stopping. The guide also identifies bidirectional attention and re-noising as mechanisms for global constraint handling and self-correction.