Loading post…
ModalProblem & solution
Boosting multimodal inference performance by >10% with a single Python dictionary
Summary
Multimodal vision-language models pose new challenges for inference engines like SGLang. Profiling with py-spy revealed that the scheduler’s process_input_requests consumed about 13% of CPU time, with a quarter of that spent on repeated calls to torch.UntypedStorage._new_shared_cuda for CUDA IPC handle book-keeping. The solution was a simple Python dictionary cache for pool handles, eliminating redundant opens. On a Qwen2.5-VL-3B-Instruct benchmark with a single H100, throughput increased 16.2% from 22.2 to 25.7 requests per second, mean time-to-first-token dropped 13.2%, and mean time-per-output-token dropped 17.2%. The improvement is merged in SGLang v0.5.10 and applies to any multimodal model using SGLang’s CUDA IPC transport.
Context
SGLang is an open-source inference engine for large language and vision-language models. The scheduler runs on a single host thread and gates GPU work. Multimodal inputs require new code paths for feature hashing and KV cache lookups, which introduced overhead from repeated CUDA IPC handle opens.
Approach / What changed
Implemented a Python dictionary cache for CUDA IPC pool handles in SGLang’s scheduler. The cache stores pool handles after the first open, avoiding repeated calls to torch.UntypedStorage._new_shared_cuda. Cache invalidation is unnecessary because pools are never reallocated. A lock is used only for writes, which are rare.
Takeaways
Profiling with py-spy identified that process_input_requests consumed ~13% of scheduler CPU time, with a quarter of that in _new_shared_cuda calls.
The fix is a simple Python dict cache for CUDA IPC pool handles, eliminating redundant opens and reducing host overhead.
The optimization improved throughput by 16.2% and reduced mean time-per-output-token by 17.2% on a Qwen2.5-VL-3B-Instruct benchmark.
FAQ
Why did decode latency improve even though the fix is in the input path?
The scheduler runs on a single thread, so time saved in input processing allows faster dispatch of decode batches. Additionally, mixed-chunk scheduling can share prefill and decode tokens in a batch.
Does the cache require invalidation?
No, because GPU memory pools are never reallocated, so cache invalidation is unnecessary.
Which models benefit from this optimization?
Any multimodal model using SGLang's CUDA IPC transport, including Qwen2.5-VL-3B-Instruct. Benefits scale with the number of multimodal inputs.
Source and freshness
- Source
- modal.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.