Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer

Summary

NVIDIA’s post presents a workflow for developing an NVFP4 checkpoint of the Nemotron 3.5 Lightning 30B-A3B model, targeting lower memory use and higher inference throughput without sacrificing accuracy. The method starts with post-training quantization (PTQ), producing a W4A16 student from the BF16 teacher, then applies quantization-aware distillation (QAD), where simulated quantization and KL-divergence loss align student logits with the frozen teacher. For the selected four_over_six recipe, 32K calibration performed best, and the recipe using W4A16 Mamba linears offered the best reported trade-off between accuracy degradation and inference performance; the checkpoint is 22 GB versus 66 GB and delivers up to 4x faster throughput. The workflow includes Model Optimizer examples and launchers for Megatron-LM and Megatron-Bridge, covering calibration, distillation, evaluation, and export.

Takeaways

1. 1

   QAD permits more aggressive PTQ settings by targeting 95–99% median accuracy recovery before distillation, rather than the over-99% target commonly used when PTQ is the final step.

2. 2

   Among the evaluated recipes, four_over_six with W4A16 Mamba linear layers provided the best trade-off between accuracy degradation and inference-performance improvement; 32K calibration produced the best PTQ results for four_over_six.

3. 3

   The resulting NVFP4 checkpoint is reported at 22 GB compared with 66 GB for the full-precision checkpoint and provides up to 4x faster throughput while preserving accuracy.