# Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism

[NVIDIA Developer Blog](https://yomu.fyi/company/nvidia-developer-blog) · Michelle Horton · Jul 6, 2026

**Type:** Problem & solution

## Summary

Large-scale LLM training jobs spanning thousands of GPUs frequently encounter transient hardware interruptions that bottleneck tightly coupled tensor parallel groups. Nonuniform Tensor Parallelism addresses this bottleneck by dynamically adapting the tensor parallelism degree within affected scale-up domains rather than dropping entire data replicas. When a GPU drops out, the system reconfigures the remaining functional devices to process larger shards and resumes execution from the latest checkpoint. To prevent the downsized replica from stalling global synchronization, advanced rack designs temporarily power-boost the active GPUs to increase clock frequencies and match baseline throughput. Overlapping the necessary tensor resharding with backward computation and parameter synchronization reduces overhead on healthy replicas to under one percent.

## Context

Large-scale LLM training runs spanning thousands of GPUs face frequent transient interruptions and device unavailability. Because tensor parallel groups are tightly coupled across scale-up domains, a single delayed or failed GPU stalls data-parallel synchronization and degrades overall training Goodput. Existing elastic scaling mechanisms, such as dropping data replicas or swapping hot spares, introduce lost throughput and increased operating costs during degraded states.

## Approach / What changed

Nonuniform Tensor Parallelism (NTP) dynamically reconfigures the tensor parallelism degree of an affected group (for example, shifting from eight to seven GPUs) to keep the data-parallel replica productive after resuming from a checkpoint. NTP pairs this software adaptation with hardware power-boosting in the affected scale-up domain to raise GPU clock frequencies and prevent synchronization stalls. Additionally, tensor resharding is overlapped with backward computation and parameter synchronization to keep healthy replica overhead under 1%.

## Takeaways

- NTP dynamically reduces the tensor parallelism degree in an affected scale-up domain (such as from 8 to 7 GPUs), allowing a data-parallel replica to remain functional rather than being dropped entirely.
- Dynamic power-boosting temporarily elevates clock frequencies on remaining GPUs within a reduced domain, preventing the smaller replica from causing global synchronization bottlenecks.
- Overlapping tensor and gradient resharding with backward computation and parameter synchronization keeps the adaptation overhead on healthy replicas below 1%.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://developer.nvidia.com/blog/enhancing-goodput-in-large-scale-llm-training-with-nonuniform-tensor-parallelism)