Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective

Summary

Engineers evaluated the GPT-OSS model for agentic reinforcement learning across multi-step tasks such as ReTool and GSM8K using the verl framework. Initial training runs suffered from exploding gradient norms, increasing KL divergence, and stagnating rewards. Investigation revealed that non-deterministic expert routing in the Mixture of Experts architecture caused log-probability mismatches between forward passes, which falsely triggered Proximal Policy Optimization clipping. The team resolved this by substituting detached current log-probabilities for old log-probabilities during on-policy updates. To address further training instabilities and memory bottlenecks, the implementation integrated an attention-sink-aware backward pass into FlashAttention v3 and applied sequence parallelism using all-to-all communication around attention layers.

Takeaways

1. 1

   MoE gating differences between dual forward passes caused log-probability mismatches, which falsely triggered PPO clipping by pushing the importance sampling ratio away from 1.

2. 2

   Overriding old log-probabilities with detached current log-probabilities when minibatch equals global batch size restores exact on-policy behavior.

3. 3

   Sequence parallelism avoids non-attention layer synchronization and uses all-to-all communication at the attention-head level before and after attention layers to reduce per-GPU memory.