Loading…
NVIDIA NVLink: The Scale-Up Network for AI Factories
NVIDIA Developer BlogElizabeth Goodman
Summary
Scale-up networking has emerged as a critical architectural component for AI factories supporting trillion-parameter models, mixture-of-experts architectures, and disaggregated serving. NVIDIA presents its sixth-generation NVLink scale-up fabric and NVLink 6 Switch to address latency-sensitive all-to-all GPU communication and offload collective operations. In simulation benchmarks on models like DeepSeek-R1, Qwen 235B, and a simulated 2T parameter model, NVLink achieved up to 2.3X higher decode throughput than off-the-shelf Ethernet in a 72-accelerator domain. The architecture integrates NVLink-C2C to supply 1.8 TB/s of coherent CPU-GPU bandwidth in the Vera Rubin NVL72 platform, as well as NVLink Fusion for integrating semi-custom silicon XPUs. Through co-designed hardware, SHARP in-network compute, and NCCL libraries, the platform optimizes token throughput, factory uptime, and deployment economics.
Context
Modern AI workloads require multiple accelerators functioning as a single compute unit, making GPU-to-GPU all-to-all communication latency and bandwidth major bottlenecks for large-scale training, inference, and mixture-of-experts models.
Approach / What changed
NVIDIA developed its sixth-generation NVLink interconnect and NVLink 6 Switch alongside NVLink-C2C and NVLink Fusion, pairing the hardware with SHARP in-network compute offload and NCCL software to unify GPUs, CPUs, and custom XPUs.
Takeaways
- Sixth-generation NVLink delivers up to 2.3X higher decode throughput compared to off-the-shelf Ethernet in a 72-accelerator scale-up domain based on simulations of DeepSeek-R1, Qwen 235B, and a 2T parameter model.
- NVLink-C2C provides 1.8 TB/s of coherent CPU-to-GPU bandwidth in the Vera Rubin NVL72 platform, representing 7x the bandwidth of PCIe Gen6.
- NVLink Fusion allows third-party custom silicon XPUs to connect directly to the NVLink scale-up stack to reduce development complexity and integrate with GPUs.
Related reading
Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism
Large-scale LLM training jobs spanning thousands of GPUs frequently encounter transient hardware interruptions that bottleneck tightly coupled tensor parallel groups. Nonuniform Tensor Parallelism addresses this bottleneck by dynamically adapting the tensor parallelism degree within affected scale-up domains rather than dropping entire data replicas. When a GPU drops out, the system reconfigures the remaining functional devices to process larger shards and resumes execution from the latest checkpoint. To prevent the downsized replica from stalling global synchronization, advanced rack designs temporarily power-boost the active GPUs to increase clock frequencies and match baseline throughput. Overlapping the necessary tensor resharding with backward computation and parameter synchronization reduces overhead on healthy replicas to under one percent.
Michelle Horton