Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72

Summary

Frontier AI pre-training increasingly relies on mixture of experts architectures, which lower compute per token while introducing severe all-to-all communication bottlenecks. To address these scaling limits, the NVIDIA GB300 NVL72 rack-scale system co-designs Blackwell Ultra GPUs, fifth-generation NVLink scale-up fabrics, and 800 Gbps SuperNIC scale-out networking. Testing on DeepSeek-V3 671B across 256 GPUs achieved a pre-training benchmark of 1,648 TFLOPs per GPU using Megatron Core. In addition to hardware co-design, ongoing software optimizations across Megatron Core, TorchTitan, and JAX contributed major throughput gains over time. Scaling from 256 to 1,024 GPUs sustained over 97% per-GPU performance efficiency across all evaluated frameworks by overlapping scale-out gradient traffic behind compute windows.

Takeaways

1. 1

   Pre-training DeepSeek-V3 671B on 256 GB300 NVL72 GPUs achieved 1,648 TFLOPs per GPU using Megatron Core, roughly triple the delivered throughput per GPU of earlier GB200 NVL72 results.

2. 2

   Fifth-generation NVLink provides memory-semantic direct peer HBM access with in-switch reductions, avoiding software send latency for tensor-parallel and MoE all-to-all operations inside the rack.

3. 3

   Scaling DeepSeek-V3 671B from 256 to 1,024 GPUs maintained 98.5% per-GPU throughput in Megatron Core and 97% in TorchTitan and JAX due to 800 Gbps scale-out networking hiding gradient traffic.