Loading…
NVIDIA Blackwell Tops MLPerf Training 6.0 with Industry-Leading Scale and Performance
NVIDIA Developer BlogFarshad Ghodsian
Summary
NVIDIA submitted results across all MLPerf Training v6.0 benchmarks, setting performance records on workloads including the DeepSeek-V3 and GPT-OSS-20B Mixture of Experts models. To scale training up to 8,192 Blackwell GPUs, NVIDIA combined hardware cluster designs with scale-out networking via Spectrum-X Ethernet and Quantum InfiniBand. The engineering team eliminated CPU-GPU synchronization bottlenecks in token-dropless MoEs by implementing full-iteration CUDA graphs with synchronization-free operators and paged stashing. Further software optimizations included CuTe DSL kernel fusions, an MXFP8 attention block in the Transformer Engine, and transitioning router elementwise math to FP32. Across these benchmarks, the GB300 NVL72 platform achieved the fastest time to train at scale and the highest normalized per-accelerator performance.
Context
MLCommons introduced new pretraining benchmarks in MLPerf Training v6.0 to reflect modern AI architectures, including large Mixture of Experts (MoE) models such as DeepSeek-V3 and GPT-OSS-20B. Training these models across thousands of interconnected processors introduces challenges with CPU-GPU synchronizations, dynamic routing, memory constraints, and bursty network traffic across cluster fabrics.
Approach / What changed
NVIDIA submitted benchmark results using GB300 NVL72 and GB200 NVL72 systems scaling up to 8,192 GPUs. The deployment incorporated scale-out networking with Spectrum-X Ethernet and Quantum InfiniBand alongside software optimizations, including full-iteration CUDA graphs for token-dropless MoEs, CuTe DSL kernel fusions, MXFP8 attention blocks, and router optimizations in Transformer Engine and cuDNN.
Takeaways
- Full-iteration CUDA graphs for token-dropless MoEs removed the CPU from the critical execution path by deriving input shapes directly on the GPU and using paged stashing for device memory management.
- Spectrum-X Ethernet uses Advanced Adaptive Routing and real-time congestion control to mitigate packet collisions and incast from bursty, low-entropy MoE traffic flows across large GPU clusters.
- Blackwell Ultra GB300 improves training throughput over GB200 by leveraging expanded memory capacity to run full-iteration CUDA graphs without layer recomputation and reduce cross-GPU communication.
Related reading
Enable Real-Time AI for High-Speed Data Acquisition with DAQIRI
Scientific instruments and high-bandwidth sensors generate immense data volumes, but traditional architectures rely on a store-then-analyze model that bottlenecks real-time discovery and forces extensive data discarding. NVIDIA DAQIRI shifts data acquisition to a software-centric pipeline that connects streaming detectors directly to GPU computing. Built as part of the NVIDIA Holoscan Platform, DAQIRI uses DPDK to bypass the Linux kernel and move UDP and RoCE v2 traffic directly from the NIC into GPU DMA buffers at line rates exceeding hundreds of gigabits per second. The system automates packet reordering, flow steering, and on-GPU data conversions such as mapping wire int4 payloads into fp16 tensors using YAML configuration files and simple C++ or Python APIs. This enables real-time edge processing, filtering, and model inference across experimental pipelines like CERN ATLAS.
Cara Laasch