# Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72

[NVIDIA Developer Blog](https://yomu.fyi/company/nvidia-developer-blog) · Kirthi Devleker · Jul 21, 2026

**Type:** Benchmark

## Summary

Frontier AI pre-training increasingly relies on mixture of experts architectures, which lower compute per token while introducing severe all-to-all communication bottlenecks. To address these scaling limits, the NVIDIA GB300 NVL72 rack-scale system co-designs Blackwell Ultra GPUs, fifth-generation NVLink scale-up fabrics, and 800 Gbps SuperNIC scale-out networking. Testing on DeepSeek-V3 671B across 256 GPUs achieved a pre-training benchmark of 1,648 TFLOPs per GPU using Megatron Core. In addition to hardware co-design, ongoing software optimizations across Megatron Core, TorchTitan, and JAX contributed major throughput gains over time. Scaling from 256 to 1,024 GPUs sustained over 97% per-GPU performance efficiency across all evaluated frameworks by overlapping scale-out gradient traffic behind compute windows.

## Context

Mixture of experts (MoE) models like DeepSeek-V3 reduce compute per token by activating a subset of parameters, but require all-to-all token dispatch and gather communication across GPUs at every layer. This collective communication sits on the critical path, capping scaling efficiency if not hidden behind compute.

## Approach / What changed

NVIDIA engineered the GB300 NVL72 rack-scale platform, combining 72 Blackwell Ultra GPUs with fifth-generation memory-semantic NVLink (1.8 TB/s per GPU, 130 TB/s non-blocking rack bandwidth) for scale-up, ConnectX-8 SuperNICs at 800 Gbps for scale-out, BlueField DPUs, and optimized software stacks (Megatron Core, TorchTitan, and JAX).

## Takeaways

- Pre-training DeepSeek-V3 671B on 256 GB300 NVL72 GPUs achieved 1,648 TFLOPs per GPU using Megatron Core, roughly triple the delivered throughput per GPU of earlier GB200 NVL72 results.
- Fifth-generation NVLink provides memory-semantic direct peer HBM access with in-switch reductions, avoiding software send latency for tensor-parallel and MoE all-to-all operations inside the rack.
- Scaling DeepSeek-V3 671B from 256 to 1,024 GPUs maintained 98.5% per-GPU throughput in Megatron Core and 97% in TorchTitan and JAX due to 800 Gbps scale-out networking hiding gradient traffic.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://developer.nvidia.com/blog/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72)