---
title: "How Blackwell Serving Turns Peak FLOPs Into Cheap Tokens"
description: "Blackwell inference is a pipeline, not a chip upgrade. This explainer walks five stages — fleet validation, batching, KV caching, low-precision kernels, and speculative decoding — that turn peak FLOPs into cheap tokens."
---

# How Blackwell Serving Turns Peak FLOPs Into Cheap Tokens

**Type:** [How it works](https://yomu.fyi/visual-explainers/category/how-it-works.md)

![A left-to-right five-stage flow diagram: validate the GPU fleet, batch sequences into tensor-core math, cache KV state, run NVFP4 and MXFP8 kernels, then draft and verify tokens, ending in a cost-per-token outcome panel.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/dc548d6e-25b1-45dc-bc85-80f3236778c7/image-2e322670-6239-4288-8c33-18471ea24387.webp)

> Blackwell inference is a pipeline, not a chip upgrade. This explainer walks five stages — fleet validation, batching, KV caching, low-precision kernels, and speculative decoding — that turn peak FLOPs into cheap tokens.

## Explanation

A Blackwell GPU only becomes useful inference capacity after several layers of engineering are applied on top of it. The flow starts with validating the GPU fleet against real distributed workloads, then batches many sequences together so matrix math accelerators stay busy instead of waiting on single requests. KV state is cached so shared prefixes are not recomputed, and low-precision formats such as NVFP4 and MXFP8 let the hardware run near peak without changing model behavior. Fused expert kernels and speculative decoding then cut the remaining overhead, which is what turns a ruinously expensive service into a price-competitive one.

## FAQ

### Why do batching and KV caching matter more than peak FLOPs?

A single request leaves the matrix accelerators mostly idle, so serving economics depend on keeping many sequences in flight at once. Caching KV state avoids recomputing shared prefixes, and Modal reported 2.8x per-user and 5.6x across-user gains after combining batching, KV caching, and NVFP4 micro-scaling on B200 GPUs.

### When is speculative decoding worth the extra memory?

A drafter model adds a small memory footprint in exchange for proposing several candidate tokens per step, which the target model then verifies. Liquid AI's 280M-parameter drafter delivered 2.30x to 3.13x faster decoding on an M5 Max without changing output quality, so it pays off when decode latency dominates.

### What should be checked before trusting a Blackwell fleet with real workloads?

Standard health checks can pass while a single degraded GPU or misrouted link still fails under real distributed load. NVIDIA's Cluster Readiness Engine runs topology-aware workloads first and attributes each failure to a specific node and category, such as NCCL bandwidth, before production jobs land.

**Tags:** [Blackwell](https://yomu.fyi/topic/blackwell.md), [GPU](https://yomu.fyi/topic/gpu.md), [Inference](https://yomu.fyi/topic/inference.md), [Mixture of Experts](https://yomu.fyi/topic/mixture-of-experts.md), [Quantization](https://yomu.fyi/topic/quantization.md), [Speculative Decoding](https://yomu.fyi/topic/speculative-decoding.md)

## Related reading

### [Efficient MoE Training for Biological Foundation Models](https://yomu.fyi/post/efficient-moe-training-for-biological-foundation-models.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Sep 24, 2026
- Type: Tutorial

Mixture-of-experts (MoE) architectures scale model capacity more efficiently than dense transformers by activating only a subset of experts per token, but naive implementations suffer from fragmented expert computation, routing overhead, and memory pressure. NVIDIA Transformer Engine (TE) addresses these bottlenecks with optimized primitives including GroupedLinear for batched expert GEMMs, MXFP8 low-precision training, and a fused GroupedMLP kernel that combines quantization, SwiGLU activation, and routing-weight scaling. The BioNeMo recipe implements these techniques for biological foundation models, using TE’s autocast API for MXFP8 precision and the Sequential API to fuse the gate-up, ScaledSwiGLU, and down projections into a single operation. On eight NVIDIA B200 GPUs, the recipe achieved up to 2.21x the training throughput of the Hugging Face baseline for a Mixtral-8x7B model. The tutorial provides prerequisites, code examples, and configuration commands to reproduce the optimizations.


### [Accelerating vision-language models with LFM2.5-VL-DSpark](https://yomu.fyi/post/accelerating-vision-language-models-with-lfm2-5-vl-dspark.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: xx, Yuri Khrustalev, Leonie Monigatti, Viviana Márquez
- Published: Sep 24, 2026
- Type: Announcement

Liquid AI released an experimental DSpark draft model for their vision-language model LFM2.5-VL-3B. The drafter adds a speculative decoding path that trades a minimal increase in memory footprint for a larger speedup without changing output quality. The 280M-parameter drafter, a 4-layer attention-only transformer, captures the target model’s hidden states at fixed tapped layers to draft a block of candidate tokens. On an M5 Max, decoding runs 2.30x to 3.13x faster by task, with end-to-end latency improvements of 1.56x to 2.62x. The drafter ships with day-one support for llama.cpp, MLX-VLM, and SGLang, and is available on Hugging Face in Safetensors and GGUF formats.


### [Validate GPU Cluster Readiness Before AI Workloads Land](https://yomu.fyi/post/validate-gpu-cluster-readiness-before-ai-workloads-land.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Sep 23, 2026
- Type: Explainer

GPU clusters can pass standard health checks yet fail under real AI training workloads due to a single degraded GPU, network link, or misrouted traffic. NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that runs real distributed workloads across topology-aware node groups before production jobs land, measuring results and reporting which nodes failed each test. Its layered API uses Custom Resource Definitions for Certification, Workflow, and Job resources, attributing each failure to a specific node and category such as NCCL bandwidth or training goodput. The WorkloadRun API handles platform detection, NCCL settings, shared-memory volumes, and gang scheduling, so operators do not need to write those manifests by hand. NVCRE integrates with NVSentinel for passive telemetry and is available under Apache 2.0, with a roadmap for inference and automated lifecycle validation.


### [Manage Kubernetes Node Fleets with NodeWright](https://yomu.fyi/post/manage-kubernetes-node-fleets-with-nodewright.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Sep 23, 2026
- Type: Problem & solution

Managing Kubernetes node fleets, especially GPU nodes, is difficult because host-level changes can disrupt long-running training jobs, and traditional tools are not cluster-aware. NodeWright, an open-source Kubernetes-native package manager from NVIDIA, solves this by declaratively configuring and updating node operating systems while respecting PodDisruptionBudgets and non-interruptible workload labels. The operator orchestrates a sequence of cordon, wait, drain, apply, interrupt, and uncordon on each node, using container image packages that carry modifications such as kernel parameters, security agents, and verification scripts. It supports progressive rollouts across a fleet via DeploymentPolicy compartments with configurable disruption budgets and strategies like batch, canary, and rolling. NodeWright integrates with NVIDIA AI Cluster Runtime for validated configurations and is part of the DSX OS open-source portfolio, available under Apache 2.0.


### [How to serve trillions of tokens for trillion-parameter coding agents](https://yomu.fyi/post/how-to-serve-trillions-of-tokens-for-trillion-parameter-coding-agents.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 23, 2026
- Type: Explainer

Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.