How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra

Summary

Production serving for agentic AI must maximize concurrent users on fixed GPU infrastructure while preserving latency, especially with long prompts, reused context, and streamed responses. The article presents NVIDIA NIM as a deployable microservice that packages validated model- and GPU-aware choices, then measures Nemotron 3 Ultra NIM 2.0.12 against an open-source baseline on four B200 GPUs using a 64K/400 workload, 76% KV reuse, and 50 TPS per user (20 ms ITL). The optimized stack combines autotuned MoE and Mamba kernels, tensor and expert-aware parallelism, prefix and Mamba state-cache reuse, scheduling, batching, memory tuning, and MTP speculative decoding. At the stated target, throughput rises from 718 to 1,997 output tokens per second—more than 2.5x—while the article recommends replaying representative traffic with NVIDIA AIPerf to select a Pareto point meeting the application’s latency constraint.

Takeaways

1. 1

   On a four-B200 system at 50 TPS per user, the NIM-on configuration produced 1,997 output tokens per second versus 718 for the NIM-off baseline, exceeding 2.5x the baseline throughput.

2. 2

   The reported gains come from interacting optimization bundles rather than independently additive switches, including partial-prefix matching, Mamba state-cache settings, batching, memory allocation, parallelism, and MTP speculative decoding.

3. 3

   Published curves are a starting point rather than a universal guarantee; representative traffic should be replayed with pinned NIM versions and NVIDIA AIPerf to select a latency-compliant Pareto point.