---
title: "SGLang"
description: "4 posts about SGLang, summarised, each linking to the original."
---

# SGLang
> 4 posts about SGLang, summarised, each linking to the original.

## Articles

### [vLLM vs. SGLang: Choosing the right inference engine for your workload](https://yomu.fyi/post/vllm-vs-sglang-choosing-the-right-inference-engine-for-your-workload.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Cedric Clyburn
- Published: Oct 14, 2026
- Type: Explainer

Open weight models now process nearly 80% of production tokens on Vercel’s AI Gateway, up from under 10% in December 2025, and OpenRouter shows a similar climb. This surge drives teams to consider self-hosting via inference engines like vLLM and SGLang, since swapping backends usually only requires a new base URL for an OpenAI-compatible API. vLLM’s PagedAttention, introduced by UC Berkeley in 2023, splits the KV cache into fixed-size blocks to eliminate wasted contiguous memory reservation and increase GPU throughput. SGLang’s RadixAttention, from LMSYS in January 2024, stores computed KV cache in a radix tree keyed by token prefix for reuse across related requests like multi-turn chat and agents. Independent benchmarks from Modal show comparable performance on random prompts without prefix sharing, indicating that workload type and hardware should guide engine choice.


### [How to serve trillions of tokens for trillion-parameter coding agents](https://yomu.fyi/post/how-to-serve-trillions-of-tokens-for-trillion-parameter-coding-agents.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 23, 2026
- Type: Explainer

Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.


### [Achieve state-of-the-art inference latencies with speculative decoding](https://yomu.fyi/post/achieve-state-of-the-art-inference-latencies-with-speculative-decoding.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 24, 2026
- Type: Problem & solution

Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints.


### [Boosting multimodal inference performance by >10% with a single Python dictionary](https://yomu.fyi/post/boosting-multimodal-inference-performance-by-10-with-a-single-python-d.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 4, 2026
- Type: Problem & solution

Multimodal vision-language models pose new challenges for inference engines like SGLang. Profiling with py-spy revealed that the scheduler’s process\_input\_requests consumed about 13% of CPU time, with a quarter of that spent on repeated calls to torch.UntypedStorage.\_new\_shared\_cuda for CUDA IPC handle book-keeping. The solution was a simple Python dictionary cache for pool handles, eliminating redundant opens. On a Qwen2.5-VL-3B-Instruct benchmark with a single H100, throughput increased 16.2% from 22.2 to 25.7 requests per second, mean time-to-first-token dropped 13.2%, and mean time-per-output-token dropped 17.2%. The improvement is merged in SGLang v0.5.10 and applies to any multimodal model using SGLang’s CUDA IPC transport.