# Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective

[Hugging Face](https://yomu.fyi/company/hugging-face) · Jason Zhu, Hejian Sang, Arup De, Rohit Jain, Yanning Chen · Jan 27, 2026

**Type:** Problem & solution

## Summary

Engineers evaluated the GPT-OSS model for agentic reinforcement learning across multi-step tasks such as ReTool and GSM8K using the verl framework. Initial training runs suffered from exploding gradient norms, increasing KL divergence, and stagnating rewards. Investigation revealed that non-deterministic expert routing in the Mixture of Experts architecture caused log-probability mismatches between forward passes, which falsely triggered Proximal Policy Optimization clipping. The team resolved this by substituting detached current log-probabilities for old log-probabilities during on-policy updates. To address further training instabilities and memory bottlenecks, the implementation integrated an attention-sink-aware backward pass into FlashAttention v3 and applied sequence parallelism using all-to-all communication around attention layers.

## Context

Validating the open-source GPT-OSS model as a backbone for agentic reinforcement learning faced failures, as initial training runs in the verl framework experienced exploding gradient norms, surging KL divergence, and non-improving rewards.

## Approach / What changed

The authors fixed on-policy PPO integrity by overriding old log-probabilities with detached current log-probabilities to avoid MoE routing discrepancies, integrated an attention sink backward pass into FlashAttention v3, and implemented sequence parallelism with all-to-all communication around attention layers.

## Takeaways

- MoE gating differences between dual forward passes caused log-probability mismatches, which falsely triggered PPO clipping by pushing the importance sampling ratio away from 1.
- Overriding old log-probabilities with detached current log-probabilities when minibatch equals global batch size restores exact on-policy behavior.
- Sequence parallelism avoids non-attention layer synchronization and uses all-to-all communication at the attention-head level before and after attention layers to reduce per-GPU memory.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://huggingface.co/blog/LinkedIn/gpt-oss-agentic-rl)