---
title: "Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer"
description: "NVIDIA’s post presents a workflow for developing an NVFP4 checkpoint of the Nemotron 3.5 Lightning 30B-A3B model, targeting lower memory use and higher inference throughput without sacrificing accuracy. The method starts with post-training quantization (PTQ), producing a W4A16 student from the BF16 teacher, then applies quantization-aware distillation (QAD), where simulated quantization and KL-divergence loss align student logits with the frozen teacher. For the selected four_over_six recipe, 32K calibration performed best, and the recipe using W4A16 Mamba linears offered the best reported trade-off between accuracy degradation and inference performance; the checkpoint is 22 GB versus 66 GB and delivers up to 4x faster throughput. The workflow includes Model Optimizer examples and launchers for Megatron-LM and Megatron-Bridge, covering calibration, distillation, evaluation, and export."
---

# Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Tanya Lenz · Aug 17, 2026

**Type:** Tutorial

## Summary

NVIDIA’s post presents a workflow for developing an NVFP4 checkpoint of the Nemotron 3.5 Lightning 30B-A3B model, targeting lower memory use and higher inference throughput without sacrificing accuracy. The method starts with post-training quantization (PTQ), producing a W4A16 student from the BF16 teacher, then applies quantization-aware distillation (QAD), where simulated quantization and KL-divergence loss align student logits with the frozen teacher. For the selected four\_over\_six recipe, 32K calibration performed best, and the recipe using W4A16 Mamba linears offered the best reported trade-off between accuracy degradation and inference performance; the checkpoint is 22 GB versus 66 GB and delivers up to 4x faster throughput. The workflow includes Model Optimizer examples and launchers for Megatron-LM and Megatron-Bridge, covering calibration, distillation, evaluation, and export.

## Context

Teams need model variants that meet latency, throughput, memory, and compute targets. PTQ can cover common NVFP4 compression needs, but tighter memory and higher throughput require more aggressive quantization, which can cause accuracy degradation. The work addresses this trade-off for Nemotron 3.5 Lightning by using QAD to recover accuracy after aggressive quantization.

## Approach / What changed

The workflow uses the BF16 Nemotron 3.5 Lightning model as a frozen teacher and creates a W4A16-NVFP4 student through PTQ. QAD then runs the student through simulated quantization while training it against the teacher with a KL-divergence loss on logits. The recipe evaluates PTQ configurations, selects the four\_over\_six variant with W4A16 Mamba linears and 32K calibration, and supports end-to-end execution and export through NVIDIA Model Optimizer, Megatron-LM, and Megatron-Bridge.

## Takeaways

- QAD permits more aggressive PTQ settings by targeting 95–99% median accuracy recovery before distillation, rather than the over-99% target commonly used when PTQ is the final step.
- Among the evaluated recipes, four\_over\_six with W4A16 Mamba linear layers provided the best trade-off between accuracy degradation and inference-performance improvement; 32K calibration produced the best PTQ results for four\_over\_six.
- The resulting NVFP4 checkpoint is reported at 22 GB compared with 66 GB for the full-precision checkpoint and provides up to 4x faster throughput while preserving accuracy.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

- Source: [NVIDIA](https://developer.nvidia.com/blog/developing-nemotron-3-5-lightning-nvfp4-with-qad-using-nvidia-model-optimizer)
- Source URL: https://developer.nvidia.com/blog/developing-nemotron-3-5-lightning-nvfp4-with-qad-using-nvidia-model-optimizer
- Ingested by Yomu: 2026-08-28T16:00:46.677Z

[Read original post](https://developer.nvidia.com/blog/developing-nemotron-3-5-lightning-nvfp4-with-qad-using-nvidia-model-optimizer)