Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer

Summary

Moving large model weights efficiently requires quantization, but mapping weights to NVFP4's eight positive values risks severe precision loss or outlier-induced underflow. To address this, NVIDIA quantized the 550B Nemotron 3 Ultra model using NVIDIA Model Optimizer with a multi-precision strategy rather than quantizing all layers uniformly. The recipe applies four-over-six adaptive block scaling to Mixture of Experts routed weights, while assigning FP8 to shared experts and retaining BF16 for sensitive attention linears and embeddings. This process reduced the checkpoint size from 1,121 GB down to 352.3 GB and enabled single-checkpoint execution across Hopper using W4A16 and Blackwell using native W4A4. As a result, the model achieves up to 5.9x higher inference throughput than the GLM-5.1 754B FP4 model on decode-heavy workloads while matching BF16 benchmark accuracy.

Context

Moving large model weights efficiently becomes critical as context windows expand. Quantizing weights to NVFP4 (a 4-bit floating-point format) presents challenges because it provides only 8 positive representable values, making scale factor selection difficult. Naive absmax scaling allows single weight outliers to compress remaining block weights near zero, while standard mean squared error scaling reduces weight error without reliably translating to downstream model benchmark improvements.

Approach / What changed

NVIDIA used NVIDIA Model Optimizer to quantize the Nemotron 3 Ultra 550B model into a mixed-precision NVFP4 checkpoint. The approach quantizes MoE routed experts to NVFP4 using four-over-six adaptive block scaling, places shared experts and Mamba mixer linears into per-tensor FP8, sets the KV cache to FP8, and retains BF16 precision for embeddings, attention linears, and output classification layers. The checkpoint automatically runs as native W4A4 on Blackwell and W4A16 on Hopper.

Takeaways

• Nemotron 3 Ultra NVFP4 employs a mixed-precision architecture, shrinking the checkpoint from 1,121 GB to 352.3 GB by placing MoE routed experts in NVFP4, shared experts in FP8, and sensitive attention linears in BF16.
• A single exported checkpoint runs across architectures by executing as native W4A4 on NVIDIA Blackwell hardware and switching to W4A16 on Hopper to accommodate Multi-Token Prediction.
• Absmax scaling causes substantial precision loss from weight outliers in NVFP4, prompting the adoption of four-over-six adaptive block scaling to address discretization error between representable grid points.

Related reading