---
title: "Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries"
description: "At-scale reinforcement-learning training is constrained by long, variable data-generation workloads that leave training GPUs idle, especially for reasoning and agentic models. The surveyed architecture separates inference and training onto distinct GPU pools, connects them through a rollout buffer, and synchronizes weights asynchronously so both stages can proceed concurrently. Sixteen open-source libraries are compared across orchestration, buffering, weight synchronization, staleness, partial rollouts, LoRA, and distributed backends; Ray appears in 8 of 16 orchestration implementations, while NCCL broadcast is the default weight-transfer method. The proposed TRL async trainer favors a lightweight bounded queue with per-token model_version metadata, packed NCCL transfers, and experimental support for prefix-resume or abort-and-retry handling of rollouts interrupted by weight updates."
---

# Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

[Hugging Face](https://yomu.fyi/company/hugging-face) · Amine Dirhoussi, Quentin Gallouédec, Kashif Rasul, Lewis Tunstall, Edward Beeching, Albert Villanova del Moral, Nouamane Tazi, Leandro von Werra, Sergio Paniego · Mar 10, 2026

**Type:** Benchmark

## Summary

At-scale reinforcement-learning training is constrained by long, variable data-generation workloads that leave training GPUs idle, especially for reasoning and agentic models. The surveyed architecture separates inference and training onto distinct GPU pools, connects them through a rollout buffer, and synchronizes weights asynchronously so both stages can proceed concurrently. Sixteen open-source libraries are compared across orchestration, buffering, weight synchronization, staleness, partial rollouts, LoRA, and distributed backends; Ray appears in 8 of 16 orchestration implementations, while NCCL broadcast is the default weight-transfer method. The proposed TRL async trainer favors a lightweight bounded queue with per-token model\_version metadata, packed NCCL transfers, and experimental support for prefix-resume or abort-and-retry handling of rollouts interrupted by weight updates.

## Context

Synchronous reinforcement-learning loops leave training GPUs idle while long or variable rollouts are generated. Reasoning models can produce lengthy sequences, GRPO may require multiple rollouts per prompt, and agentic workloads introduce multi-turn trajectories with highly variable latency, making batch and synchronization barriers costly.

## Approach / What changed

The work surveys 16 open-source asynchronous RL libraries across seven axes, then proposes design choices for TRL: lightweight orchestration, a bounded queue with per-token model\_version tags, packed NCCL weight broadcasts, and partial-rollout strategies including prefix-resume and abort-and-retry.

## Takeaways

- Ray is used by 8 of the 16 surveyed distributed-computing libraries for orchestration, and NCCL broadcast is the default model-weight transfer method.
- Staleness handling ranges from dropping outdated samples to importance-sampling correction; the proposed TRL design records model\_version per token to enable token-level admission gating and correction.
- Partial rollouts are addressed experimentally through prefix-resume, which preserves KV-cache work but needs inference-engine support, or abort-and-retry, which is simpler but can waste computation.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/async-rl-training-landscape)
- Source URL: https://huggingface.co/blog/async-rl-training-landscape
- Ingested by Yomu: 2026-08-29T01:46:25.185Z

[Read original post](https://huggingface.co/blog/async-rl-training-landscape)