---
title: "Accelerating Reinforcement Learning Weight Refit with the Reshard Collective"
description: "Reinforcement learning post-training uses a disaggregated setup where inference and training run on separate GPU resources, requiring periodic weight transfers called refit. Training and inference instances employ different parallelism configurations, causing weights to be sharded across GPU groups differently. Conventional gather-and-broadcast refit recnstructs the full weight tensor as an intermediate representation, resulting in a logical payload of roughly twice the model’s weight size—for a 2.4 trillion parameter model, about 9.6 TB per refit. The Reshard collective reformulates refit as a DTensor-to-DTensor reshard operation, transferring only the exact shard regions each destiation GPU needs, while NCCL M2N Reshard groups replicas by NVLink domain to maximize bandwith. A NeMo RL implementation achieves 2.7x to 5.7x speedups over gather-and-broadcast refit and is open-sourced in the NVIDIA/nccl-extensions reposity."
---

# Accelerating Reinforcement Learning Weight Refit with the Reshard Collective

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Tanya Lenz · Nov 6, 2026

**Type:** Problem & solution

## Summary

Reinforcement learning post-training uses a disaggregated setup where inference and training run on separate GPU resources, requiring periodic weight transfers called refit. Training and inference instances employ different parallelism configurations, causing weights to be sharded across GPU groups differently. Conventional gather-and-broadcast refit recnstructs the full weight tensor as an intermediate representation, resulting in a logical payload of roughly twice the model’s weight size—for a 2.4 trillion parameter model, about 9.6 TB per refit. The Reshard collective reformulates refit as a DTensor-to-DTensor reshard operation, transferring only the exact shard regions each destiation GPU needs, while NCCL M2N Reshard groups replicas by NVLink domain to maximize bandwith. A NeMo RL implementation achieves 2.7x to 5.7x speedups over gather-and-broadcast refit and is open-sourced in the NVIDIA/nccl-extensions reposity.

## Context

During reinforcement learning post-training, model weights are trained on separate GPU resources while inference runs in parallel on other GPUs. The system must periodically refresh inference weights with the latest parameters, a process called refit. Training and inference instances use different parallelism configurations (data, tensor, expert, pipeline) and GPU counts, leading to different shard boundaries. Existing gather-and-broadcast methods reconstruct the full weight tensor as an intermediate step, causing redundant communication that grows with model size, for example a 2.4 trillion parameter model requires roughly 9.6 TB payload per refit.

## Approach / What changed

The Reshard collective reformulates weight refit as a DTensor-to-DTensor reshard operation. It uses PyTorch’s DTensor abstraction to represent sharded layouts and computes the exact intersections between source and destination shards, transferring only the required data. NCCL M2N Reshard is an open-source NCCL extension implementing this pattern, mapping transfers onto the physical GPU topology and grouping replicas by NVLink domain to maximize bandwidth. In NeMo RL, a hybrid approach uses reshard for dominant feed-forward projection weights (about 95% of parameters) and conventional gather-and-broadcast for smaller weights.

## Takeaways

- Conventional gather-and-broadcast refit construct the full weight tensor as an intermediate step, causing communication payload roughly twice the model’s weight size, which for trillion-parameter models can exceed 10 TB per refit.
- The Reshard collective reduces refit payload by transferring only the exact shard intersections between source and destiation layouts, reformulating the problem as a DTensor-to-DTensor reshard operation.
- NeMo RL’s hybrid implementation achieves 2.7x to5.7x speedup over gather-and-broadcast refit across models up to2.4 trillion parameters, with open-source code available.

## FAQ

### What is weight refit in RL post-training?

Refit transfers the latest model parameters from training GPUs to inference GPUs in a disaggregated RL post-training setup, performed at every training step because inference runs on separate resources and must be updated with the current policy model weights.

### Why is gather-and-broadcast refit inefficient?

It all-gathers the full parameter on training GPUs then broadcast to inference GPUs, so each inference GPU receives the entire tensor even when it needs only a shard, causing redundant communication that scales linearly with model size.

### How does the Reshard collective improve refit throughput?

It reformulates refit as a DTensor-to-DTensor reshard, transferring only the exact shard regions needed. NCL M2N Reshard further optimizes by grouping replica by NLink domain, reducing network transfers. Benchmarks show 2.7x to5.7x speedups.

**Tags:** [Distributed Systems](https://yomu.fyi/topic/distributed-systems), [GPU](https://yomu.fyi/topic/gpu), [NVIDIA](https://yomu.fyi/topic/nvidia), [Reinforcement Learning](https://yomu.fyi/topic/reinforcement-learning)

- Source: [NVIDIA](https://developer.nvidia.com/blog/accelerating-reinforcement-learning-weight-refit-with-the-reshard-collective)
- Source URL: https://developer.nvidia.com/blog/accelerating-reinforcement-learning-weight-refit-with-the-reshard-collective
- Ingested by Yomu: 2026-11-06T21:00:15.702Z

[Read original post](https://developer.nvidia.com/blog/accelerating-reinforcement-learning-weight-refit-with-the-reshard-collective)
