---
title: "Boosting multimodal inference performance by >10% with a single Python dictionary"
description: "Multimodal vision-language models pose new challenges for inference engines like SGLang. Profiling with py-spy revealed that the scheduler’s process_input_requests consumed about 13% of CPU time, with a quarter of that spent on repeated calls to torch.UntypedStorage._new_shared_cuda for CUDA IPC handle book-keeping. The solution was a simple Python dictionary cache for pool handles, eliminating redundant opens. On a Qwen2.5-VL-3B-Instruct benchmark with a single H100, throughput increased 16.2% from 22.2 to 25.7 requests per second, mean time-to-first-token dropped 13.2%, and mean time-per-output-token dropped 17.2%. The improvement is merged in SGLang v0.5.10 and applies to any multimodal model using SGLang’s CUDA IPC transport."
---

# Boosting multimodal inference performance by >10% with a single Python dictionary

[Modal](https://yomu.fyi/company/modal) · May 4, 2026

**Type:** Problem & solution

## Summary

Multimodal vision-language models pose new challenges for inference engines like SGLang. Profiling with py-spy revealed that the scheduler’s process\_input\_requests consumed about 13% of CPU time, with a quarter of that spent on repeated calls to torch.UntypedStorage.\_new\_shared\_cuda for CUDA IPC handle book-keeping. The solution was a simple Python dictionary cache for pool handles, eliminating redundant opens. On a Qwen2.5-VL-3B-Instruct benchmark with a single H100, throughput increased 16.2% from 22.2 to 25.7 requests per second, mean time-to-first-token dropped 13.2%, and mean time-per-output-token dropped 17.2%. The improvement is merged in SGLang v0.5.10 and applies to any multimodal model using SGLang’s CUDA IPC transport.

## Context

SGLang is an open-source inference engine for large language and vision-language models. The scheduler runs on a single host thread and gates GPU work. Multimodal inputs require new code paths for feature hashing and KV cache lookups, which introduced overhead from repeated CUDA IPC handle opens.

## Approach / What changed

Implemented a Python dictionary cache for CUDA IPC pool handles in SGLang’s scheduler. The cache stores pool handles after the first open, avoiding repeated calls to torch.UntypedStorage.\_new\_shared\_cuda. Cache invalidation is unnecessary because pools are never reallocated. A lock is used only for writes, which are rare.

## Takeaways

- Profiling with py-spy identified that process\_input\_requests consumed ~13% of scheduler CPU time, with a quarter of that in \_new\_shared\_cuda calls.
- The fix is a simple Python dict cache for CUDA IPC pool handles, eliminating redundant opens and reducing host overhead.
- The optimization improved throughput by 16.2% and reduced mean time-per-output-token by 17.2% on a Qwen2.5-VL-3B-Instruct benchmark.

## FAQ

### Why did decode latency improve even though the fix is in the input path?

The scheduler runs on a single thread, so time saved in input processing allows faster dispatch of decode batches. Additionally, mixed-chunk scheduling can share prefill and decode tokens in a batch.

### Does the cache require invalidation?

No, because GPU memory pools are never reallocated, so cache invalidation is unnecessary.

### Which models benefit from this optimization?

Any multimodal model using SGLang's CUDA IPC transport, including Qwen2.5-VL-3B-Instruct. Benefits scale with the number of multimodal inputs.

**Tags:** [CUDA IPC](https://yomu.fyi/topic/cuda-ipc), [Inference](https://yomu.fyi/topic/inference), [Multimodal](https://yomu.fyi/topic/multimodal), [Performance](https://yomu.fyi/topic/performance), [SGLang](https://yomu.fyi/topic/sglang)

- Source: [Modal](https://modal.com/blog/boosting-multimodal-inference-performance-by-greater-than-10-with-a-single-python-dictionary)
- Source URL: https://modal.com/blog/boosting-multimodal-inference-performance-by-greater-than-10-with-a-single-python-dictionary
- Ingested by Yomu: 2026-09-20T07:46:47.032Z

[Read original post](https://modal.com/blog/boosting-multimodal-inference-performance-by-greater-than-10-with-a-single-python-dictionary)