NVIDIA Blackwell Tops MLPerf Training 6.0 with Industry-Leading Scale and Performance

Summary

NVIDIA submitted results across all MLPerf Training v6.0 benchmarks, setting performance records on workloads including the DeepSeek-V3 and GPT-OSS-20B Mixture of Experts models. To scale training up to 8,192 Blackwell GPUs, NVIDIA combined hardware cluster designs with scale-out networking via Spectrum-X Ethernet and Quantum InfiniBand. The engineering team eliminated CPU-GPU synchronization bottlenecks in token-dropless MoEs by implementing full-iteration CUDA graphs with synchronization-free operators and paged stashing. Further software optimizations included CuTe DSL kernel fusions, an MXFP8 attention block in the Transformer Engine, and transitioning router elementwise math to FP32. Across these benchmarks, the GB300 NVL72 platform achieved the fastest time to train at scale and the highest normalized per-accelerator performance.

Takeaways

1. 1

   Full-iteration CUDA graphs for token-dropless MoEs removed the CPU from the critical execution path by deriving input shapes directly on the GPU and using paged stashing for device memory management.

2. 2

   Spectrum-X Ethernet uses Advanced Adaptive Routing and real-time congestion control to mitigate packet collisions and incast from bursty, low-entropy MoE traffic flows across large GPU clusters.

3. 3

   Blackwell Ultra GB300 improves training throughput over GB200 by leveraging expanded memory capacity to run full-iteration CUDA graphs without layer recomputation and reduce cross-GPU communication.