Serve Qwen3.8-2.4T-A95B, a 2.4T-Parameter Model, with Configurable Reasoning on NVIDIA GB300 NVL72

Summary

Alibaba’s Qwen3.8-2.4T-A95B (Qwen3.8-Max) is an open-weight 2.4T-parameter model with 95B activated per token, designed for reasoning and agentic workloads. Its fine-grained mixture-of-experts architecture combines full and linear attention, supporting context of up to one million tokens and output lengths up to 128K while using bounded recurrent state in linear-attention layers. NVIDIA describes multinode serving on the 72-GPU GB300 NVL72, where a 130 TB/s NVLink domain supports expert communication; in FP8, the model reportedly exceeds 4K tokens per second per GPU and 350 tokens per second per user without additional tuning. Developers can control reasoning depth with low, high, or xhigh modes, choose SGLang, vLLM, NVIDIA Dynamo, or NIM for serving, and use NeMo AutoModel for full SFT or LoRA post-training.

Takeaways

1. 1

   Qwen3.8-2.4T-A95B activates 95B parameters per token from its 2.4T total, using a learned router to select experts and reduce serving cost relative to a comparable dense model.

2. 2

   The hybrid attention design alternates full attention with linear-attention layers that replace the growing KV cache with a bounded recurrent state for long contexts up to one million tokens.

3. 3

   On GB300 NVL72, FP8 serving without additional model tuning exceeds 4K tokens per second per GPU and 350 tokens per second per user; NVFP4 optimizations are expected to improve performance further.