# NVIDIA Blackwell Tops MLPerf Training 6.0 with Industry-Leading Scale and Performance

[NVIDIA Developer Blog](https://yomu.fyi/company/nvidia-developer-blog) · Farshad Ghodsian · Jun 16, 2026

**Type:** Benchmark

## Summary

NVIDIA submitted results across all MLPerf Training v6.0 benchmarks, setting performance records on workloads including the DeepSeek-V3 and GPT-OSS-20B Mixture of Experts models. To scale training up to 8,192 Blackwell GPUs, NVIDIA combined hardware cluster designs with scale-out networking via Spectrum-X Ethernet and Quantum InfiniBand. The engineering team eliminated CPU-GPU synchronization bottlenecks in token-dropless MoEs by implementing full-iteration CUDA graphs with synchronization-free operators and paged stashing. Further software optimizations included CuTe DSL kernel fusions, an MXFP8 attention block in the Transformer Engine, and transitioning router elementwise math to FP32. Across these benchmarks, the GB300 NVL72 platform achieved the fastest time to train at scale and the highest normalized per-accelerator performance.

## Context

MLCommons introduced new pretraining benchmarks in MLPerf Training v6.0 to reflect modern AI architectures, including large Mixture of Experts (MoE) models such as DeepSeek-V3 and GPT-OSS-20B. Training these models across thousands of interconnected processors introduces challenges with CPU-GPU synchronizations, dynamic routing, memory constraints, and bursty network traffic across cluster fabrics.

## Approach / What changed

NVIDIA submitted benchmark results using GB300 NVL72 and GB200 NVL72 systems scaling up to 8,192 GPUs. The deployment incorporated scale-out networking with Spectrum-X Ethernet and Quantum InfiniBand alongside software optimizations, including full-iteration CUDA graphs for token-dropless MoEs, CuTe DSL kernel fusions, MXFP8 attention blocks, and router optimizations in Transformer Engine and cuDNN.

## Takeaways

- Full-iteration CUDA graphs for token-dropless MoEs removed the CPU from the critical execution path by deriving input shapes directly on the GPU and using paged stashing for device memory management.
- Spectrum-X Ethernet uses Advanced Adaptive Routing and real-time congestion control to mitigate packet collisions and incast from bursty, low-entropy MoE traffic flows across large GPU clusters.
- Blackwell Ultra GB300 improves training throughput over GB200 by leveraging expanded memory capacity to run full-iteration CUDA graphs without layer recomputation and reduce cross-GPU communication.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://developer.nvidia.com/blog/nvidia-blackwell-tops-mlperf-training-6-0-with-industry-leading-scale-and-performance)