vLLM vs. SGLang: Choosing the right inference engine for your workload

Summary

Open weight models now process nearly 80% of production tokens on Vercel’s AI Gateway, up from under 10% in December 2025, and OpenRouter shows a similar climb. This surge drives teams to consider self-hosting via inference engines like vLLM and SGLang, since swapping backends usually only requires a new base URL for an OpenAI-compatible API. vLLM’s PagedAttention, introduced by UC Berkeley in 2023, splits the KV cache into fixed-size blocks to eliminate wasted contiguous memory reservation and increase GPU throughput. SGLang’s RadixAttention, from LMSYS in January 2024, stores computed KV cache in a radix tree keyed by token prefix for reuse across related requests like multi-turn chat and agents. Independent benchmarks from Modal show comparable performance on random prompts without prefix sharing, indicating that workload type and hardware should guide engine choice.

Takeaways

1. 1

   vLLM’s PagedAttention reduces memory waste by splitting the KV cache into fixed-size blocks allocated on demand, increasing the number of requests a single GPU can serve simultaneously.

2. 2

   SGLang’s RadixAttention caches computed KV cache in a radix tree keyed by token prefix, enabling reuse across related requests like multi-turn chat and agent loops without recomputation.

3. 3

   Independent benchmarks from Modal, which used random prompts without shared prefixes, found vLLM and SGLang perform comparably; real workload testing is recommended.

FAQ