Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

Summary

At-scale reinforcement-learning training is constrained by long, variable data-generation workloads that leave training GPUs idle, especially for reasoning and agentic models. The surveyed architecture separates inference and training onto distinct GPU pools, connects them through a rollout buffer, and synchronizes weights asynchronously so both stages can proceed concurrently. Sixteen open-source libraries are compared across orchestration, buffering, weight synchronization, staleness, partial rollouts, LoRA, and distributed backends; Ray appears in 8 of 16 orchestration implementations, while NCCL broadcast is the default weight-transfer method. The proposed TRL async trainer favors a lightweight bounded queue with per-token model_version metadata, packed NCCL transfers, and experimental support for prefix-resume or abort-and-retry handling of rollouts interrupted by weight updates.

Takeaways

1. 1

   Ray is used by 8 of the 16 surveyed distributed-computing libraries for orchestration, and NCCL broadcast is the default model-weight transfer method.

2. 2

   Staleness handling ranges from dropping outdated samples to importance-sampling correction; the proposed TRL design records model_version per token to enable token-level admission gating and correction.

3. 3

   Partial rollouts are addressed experimentally through prefix-resume, which preserves KV-cache work but needs inference-engine support, or abort-and-retry, which is simpler but can waste computation.