Continuous batching from first principles

Hugging FaceRémi Ouazan Reboul, Arthur Zucker, Luc GeorgesNov 25, 2025

Explainer

Summary

Large language model serving requires running expensive next-token generation across multiple concurrent user requests. Traditional batching approaches introduce severe padding inefficiencies when mixing variable-length prompts and different generation phases, especially under static shape constraints like CUDA graphs. Continuous batching resolves these inefficiencies by combining key-value caching, chunked prefill, ragged batching, and dynamic request scheduling. Ragged batching eliminates the traditional batch axis by concatenating token sequences into a single tensor and using boolean attention masks to isolate independent sequences. By dynamically removing completed prompts and packing decoding tokens alongside chunked prefill tokens up to a hardware memory budget, serving systems maintain high hardware utilization and throughput.

Context

Serving large language models concurrently is computationally expensive because generation predicts one token at a time across billions of parameters. Traditional sequence batching introduces severe padding waste when combining varying prompt lengths or mixing decoding and prefill phases, with padding overhead scaling quadratically with batch size and prompt length.

Approach / What changed

The author explains continuous batching by combining key-value caching, chunked prefill, ragged batching, and dynamic scheduling. Ragged batching removes the batch dimension by concatenating sequences and enforcing boundaries via boolean attention masks, while dynamic scheduling immediately replaces completed sequences with chunked prefill tokens to hit GPU token budgets.

Takeaways

Ragged batching removes the batch dimension by concatenating sequences into a single tensor, using boolean attention masks to prevent cross-sequence token interactions and eliminate padding waste.
Introducing an n-token prompt into a batch of B decoding sequences requires (n-1)(B-1) padding tokens under standard batching, causing padding overhead to scale quadratically with batch size and prompt length.
Continuous batching maximizes GPU throughput by allocating decoding tokens first, filling remaining batch token capacity with chunked prefill inputs, and dynamically swapping out finished prompts.

Read original post

Related reading

Hugging Face · Sep 11, 2025
Tricks from OpenAI gpt-oss YOU 🫵 can use with transformers
To support OpenAI's GPT-OSS series of models, the transformers library introduced several performance upgrades that apply across supported architectures. The release integrates zero-build custom kernels downloadable directly from the Hub, reducing external dependency bloat and compilation friction for operations like Liger RMSNorm and MegaBlocks MoE. Native support for MXFP4 quantization groups vector elements into 32-value blocks with shared scales, allowing GPT-OSS 20B to fit in roughly 16 GB of VRAM and GPT-OSS 120B in roughly 80 GB. In addition, transformers incorporates Flash Attention 3 with attention sinks, continuous batching via the generate_batch API for experimentation, and automatic memory pre-allocation to speed up model loading on GPUs.
Aritra Roy Gosthipaty, Sergio Paniego, Vaibhav Srivastav, Pedro Cuenca, Arthur Zucker, Nathan Habib, Cyril Vallez
Amazon · May 15, 2026
Making LLMs faster without sacrificing accuracy
Standard scaling laws optimize parameter count and training data budgets to minimize loss but ignore internal Transformer architecture configurations. To address throughput disparities among equally sized models, researchers developed a scaling framework incorporating hidden size, the multilayer perceptron to attention parameter ratio, and grouped-query attention. The method calibrates a correction surface over standard Chinchilla scaling laws to independently optimize separable architectural parameters for target accuracy. Evaluated across more than 200 models up to three billion parameters, the framework produced the Panda model family for maximum accuracy and the Surefire family for Pareto efficiency. Surefire models matched or exceeded LLaMA-3.2 accuracy while increasing inference throughput by 12% to 47% across vLLM and SGLang serving systems on modern GPUs.
Tao Yu, Youngsuk Park
NVIDIA Developer Blog · Jul 10, 2026
AI Model Co-Design: Hardware-Friendly LLM Design
Efficient large language model serving requires co-designing architectures to balance datacenter token throughput and user interactivity within hardware constraints. Roofline analysis shows that linear layers become memory-bound when hidden or intermediate projection dimensions are small, leaving arithmetic hardware underutilized even at large token counts. For prefill workloads, Chunked Pipeline Parallelism reduces first-token latency on long contexts when models use regular, repeatable layer patterns that split into balanced stages. For latency-critical decoding, decoupling attention and feed-forward network parallelization enables independent optimization, applying expert parallelism for feed-forward networks and Helix parallelism to shard the key-value cache across sequences. Following architectural design rules—such as dimension alignment, width prioritization, and low-precision NVFP4 compatibility—allows models to achieve higher utilization on modern accelerator hardware.
Elizabeth Goodman
Apple · Aug 25, 2026
STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation
Existing multimodal models struggle with structural fragmentation, often sacrificing visual fidelity via discrete tokenization or breaking structural symmetry by pairing causal text generation with diffusion-based denoising. STARFlow2 addresses these limitations by leveraging autoregressive normalizing flows, which share the causal mask, key-value cache mechanism, and left-to-right structure of large language models. Built upon the Pretzel architecture, the system vertically interleaves a frozen pretrained vision-language model stream with a TARFlow stream through residual skip connections under a single causal mask. A deep-shallow flow design combined with a unified FAE latent space allows both text and continuous visual outputs to enter the key-value cache directly without re-encoding. Benchmark experiments validate strong performance across multimodal understanding and continuous image generation tasks.