Loading post…
Red HatEldar Kurtić, Helen Zhao, Charles Hernandez, Dipika SikkaAnnouncement
Optimized support for Open NVIDIA Nemotron 3.5 Super VL
Summary
Red Hat AI released optimized FP8 and DSpark speculative decoding checkpoints for the Nemotron 3.5 Super VL 120B multimodal mixture-of-experts (MoE) model. These checkpoints are designed to double inference throughput on NVIDIA Hopper GPUs without compromising output accuracy. FP8 quantization uses LLM Compressor to reduce model weights and activations from BF16 to 8-bit floating point, cutting memory and storage requirements in half while preserving quality. DSpark speculative decoding proposes 8-16 tokens in a single forward pass and uses a lightweight Markov head to introduce dependencies between speculated tokens. Benchmarks on HumanEval, math_reasoning, and RAG datasets show that each technique delivers meaningful gains individually and combining them outperforms any intermediate configuration at every concurrency level.
Context
Nemotron 3.5 Super VL is a 120B multimodal mixture-of-experts (MoE) model built for long-running agent tasks. Red Hat AI is releasing optimized checkpoints to accelerate inference by combining FP8 quantization and DSpark speculative decoding, targeting NVIDIA Hopper GPUs.
Approach / What changed
Two complementary optimizations: (1) FP8 quantization using LLM Compressor, quantizing attention, expert, and Mamba projection layers of the main model while keeping shared experts, routers, normalization layers, and lm_head in original precision; for multi-token prediction (MTP) layers, quantizing attention, expert, and shared expert layers, leaving normalization layers and eh_proj unquantized. (2) DSpark speculative decoding with a parallel-drafting technique that proposes 8-16 tokens in a single forward pass, and a Markov head to introduce dependencies between speculated tokens. The DSpark speculator was trained using Speculators. Both checkpoints are released as open models.
Takeaways
FP8 quantization reduces model weight memory and storage by half compared to BF16 and leverages native FP8 Tensor Core support on NVIDIA Hopper GPUs for approximately 2x higher theoretical matrix multiplication throughput.
DSpark speculative decoding proposes 8-16 tokens in a single forward pass and uses a lightweight Markov head to maintain token dependencies, producing higher-quality draft sequences compared to independent parallel drafting.
FP8 quantization and DSpark speculative decoding are orthogonal techniques that compose cleanly; benchmarks show combining them outperforms either technique alone at every concurrency level across HumanEval, math_reasoning, and RAG datasets.
FAQ
Does FP8 quantization affect model accuracy?
No. Evaluation across a broad suite of benchmarks, including reasoning-intensive tasks and agentic workflows, shows that the FP8 checkpoint consistently preserves model accuracy compared to the BF16 model under identical conditions.
How does DSpark differ from native MTP layers?
Native MTP layers predict multiple tokens autoregressively. DSpark uses parallel drafting to propose 8-16 tokens in a single forward pass and adds a lightweight Markov head to reintroduce dependencies between speculated tokens, enabling higher-quality candidate sequences and increased inference throughput.
Can I reproduce the FP8 quantization process?
Yes. The quantization process is fully reproducible using the creation script provided in the model card, which uses the LLM Compressor library and its model-free post-training quantization workflow.
Source and freshness
- Source
- developers.redhat.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.