Loading…
Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism
NVIDIA Developer BlogMichelle Horton
Summary
Large-scale LLM training jobs spanning thousands of GPUs frequently encounter transient hardware interruptions that bottleneck tightly coupled tensor parallel groups. Nonuniform Tensor Parallelism addresses this bottleneck by dynamically adapting the tensor parallelism degree within affected scale-up domains rather than dropping entire data replicas. When a GPU drops out, the system reconfigures the remaining functional devices to process larger shards and resumes execution from the latest checkpoint. To prevent the downsized replica from stalling global synchronization, advanced rack designs temporarily power-boost the active GPUs to increase clock frequencies and match baseline throughput. Overlapping the necessary tensor resharding with backward computation and parameter synchronization reduces overhead on healthy replicas to under one percent.
Context
Large-scale LLM training runs spanning thousands of GPUs face frequent transient interruptions and device unavailability. Because tensor parallel groups are tightly coupled across scale-up domains, a single delayed or failed GPU stalls data-parallel synchronization and degrades overall training Goodput. Existing elastic scaling mechanisms, such as dropping data replicas or swapping hot spares, introduce lost throughput and increased operating costs during degraded states.
Approach / What changed
Nonuniform Tensor Parallelism (NTP) dynamically reconfigures the tensor parallelism degree of an affected group (for example, shifting from eight to seven GPUs) to keep the data-parallel replica productive after resuming from a checkpoint. NTP pairs this software adaptation with hardware power-boosting in the affected scale-up domain to raise GPU clock frequencies and prevent synchronization stalls. Additionally, tensor resharding is overlapped with backward computation and parameter synchronization to keep healthy replica overhead under 1%.
Takeaways
- NTP dynamically reduces the tensor parallelism degree in an affected scale-up domain (such as from 8 to 7 GPUs), allowing a data-parallel replica to remain functional rather than being dropped entirely.
- Dynamic power-boosting temporarily elevates clock frequencies on remaining GPUs within a reduced domain, preventing the smaller replica from causing global synchronization bottlenecks.
- Overlapping tensor and gradient resharding with backward computation and parameter synchronization keeps the adaptation overhead on healthy replicas below 1%.
Related reading
NVIDIA NVLink: The Scale-Up Network for AI Factories
Scale-up networking has emerged as a critical architectural component for AI factories supporting trillion-parameter models, mixture-of-experts architectures, and disaggregated serving. NVIDIA presents its sixth-generation NVLink scale-up fabric and NVLink 6 Switch to address latency-sensitive all-to-all GPU communication and offload collective operations. In simulation benchmarks on models like DeepSeek-R1, Qwen 235B, and a simulated 2T parameter model, NVLink achieved up to 2.3X higher decode throughput than off-the-shelf Ethernet in a 72-accelerator domain. The architecture integrates NVLink-C2C to supply 1.8 TB/s of coherent CPU-GPU bandwidth in the Vera Rubin NVL72 platform, as well as NVLink Fusion for integrating semi-custom silicon XPUs. Through co-designed hardware, SHARP in-network compute, and NCCL libraries, the platform optimizes token throughput, factory uptime, and deployment economics.
Elizabeth Goodman