Loading…
Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72
NVIDIA Developer BlogKirthi Devleker
Summary
Frontier AI pre-training increasingly relies on mixture of experts architectures, which lower compute per token while introducing severe all-to-all communication bottlenecks. To address these scaling limits, the NVIDIA GB300 NVL72 rack-scale system co-designs Blackwell Ultra GPUs, fifth-generation NVLink scale-up fabrics, and 800 Gbps SuperNIC scale-out networking. Testing on DeepSeek-V3 671B across 256 GPUs achieved a pre-training benchmark of 1,648 TFLOPs per GPU using Megatron Core. In addition to hardware co-design, ongoing software optimizations across Megatron Core, TorchTitan, and JAX contributed major throughput gains over time. Scaling from 256 to 1,024 GPUs sustained over 97% per-GPU performance efficiency across all evaluated frameworks by overlapping scale-out gradient traffic behind compute windows.
Context
Mixture of experts (MoE) models like DeepSeek-V3 reduce compute per token by activating a subset of parameters, but require all-to-all token dispatch and gather communication across GPUs at every layer. This collective communication sits on the critical path, capping scaling efficiency if not hidden behind compute.
Approach / What changed
NVIDIA engineered the GB300 NVL72 rack-scale platform, combining 72 Blackwell Ultra GPUs with fifth-generation memory-semantic NVLink (1.8 TB/s per GPU, 130 TB/s non-blocking rack bandwidth) for scale-up, ConnectX-8 SuperNICs at 800 Gbps for scale-out, BlueField DPUs, and optimized software stacks (Megatron Core, TorchTitan, and JAX).
Takeaways
- Pre-training DeepSeek-V3 671B on 256 GB300 NVL72 GPUs achieved 1,648 TFLOPs per GPU using Megatron Core, roughly triple the delivered throughput per GPU of earlier GB200 NVL72 results.
- Fifth-generation NVLink provides memory-semantic direct peer HBM access with in-switch reductions, avoiding software send latency for tensor-parallel and MoE all-to-all operations inside the rack.
- Scaling DeepSeek-V3 671B from 256 to 1,024 GPUs maintained 98.5% per-GPU throughput in Megatron Core and 97% in TorchTitan and JAX due to 800 Gbps scale-out networking hiding gradient traffic.
Related reading
Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism
Large-scale LLM training jobs spanning thousands of GPUs frequently encounter transient hardware interruptions that bottleneck tightly coupled tensor parallel groups. Nonuniform Tensor Parallelism addresses this bottleneck by dynamically adapting the tensor parallelism degree within affected scale-up domains rather than dropping entire data replicas. When a GPU drops out, the system reconfigures the remaining functional devices to process larger shards and resumes execution from the latest checkpoint. To prevent the downsized replica from stalling global synchronization, advanced rack designs temporarily power-boost the active GPUs to increase clock frequencies and match baseline throughput. Overlapping the necessary tensor resharding with backward computation and parameter synchronization reduces overhead on healthy replicas to under one percent.
Michelle Horton