Diverse reasoning traces teach LLMs to make better decisions

Summary

Large language models typically rely on supervised fine-tuning with a single human-verified reasoning trace per example, which limits reasoning breadth and causes mode collapse when attempting parallel reasoning. Researchers presented set-supervised fine tuning (SSFT) and global forking policy optimization (GFPO) to teach models diverse problem-solving strategies. SSFT represents reasoning as a set of complete solution paths paired with global forking tokens, utilizing bipartite matching to achieve order-invariant training and specialized tokens. GFPO applies lightweight reinforcement learning to select the optimal forking token upfront, detaching gradients from generated rollouts to ensure stable optimization. On standard reasoning and coding benchmarks, the combined framework increased single-shot accuracy by 5% to 7% while improving pass@k diversity without degrading pass@1 accuracy.

Context

Large language models are typically trained on a single reasoning trace per example, which causes mode collapse when using naive supervised fine-tuning to elicit diverse parallel reasoning strategies.

Approach / What changed

Researchers introduced set-supervised fine tuning (SSFT), which uses min-cost bipartite matching to assign multiple reasoning traces to global forking tokens, followed by global forking policy optimization (GFPO) to train token selection using reward advantages with detached rollout gradients.

Takeaways

• Set-supervised fine tuning uses bipartite matching between traces and global forking tokens to prevent mode collapse during post-training.
• Global forking policy optimization updates the policy over forking tokens using reward signals while detaching gradients from the generated reasoning traces.
• Combining SSFT and GFPO yields 5% to 7% gains in single-shot accuracy on benchmarks including AIME 2024, AIME 2025, and LiveCodeBench-v5.

Related reading

• Apple · Aug 24, 2026

  Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning

  Visual chain-of-thought methods allow multimodal large language models to reason through intermediate image generation, but synthesizing explicit visual steps creates major inference overhead during proactive video reasoning. To eliminate runtime synthesis, Internalized Visual Thinking introduces a post-training framework that jointly optimizes textual prediction and next-embedding prediction across unlabeled videos. By predicting latent representations of future frames alongside target text answers during training, the model captures motion, object transitions, interactions, and latent intent. At inference time, the model produces answers directly without generating or re-encoding future frames. IVT outperforms text-only post-training across six evaluation settings while matching or exceeding Visual CoT performance and cutting end-to-end latency by more than 5x.

• Red Hat · Aug 26, 2026

  GRPO fine-tuning on Red Hat OpenShift AI: Reinforcement learning from verifiable rewards with Training Hub

  Supervised fine-tuning for structured reasoning tasks like tool calling often demands expensive, labor-intensive dataset curation. Group Relative Policy Optimization (GRPO) resolves this by using programmatic reward functions to score multiple candidate responses without requiring human labels or a dedicated reward model. On Red Hat OpenShift AI, developers can implement GRPO using Training Hub, Kubeflow Trainer, and the Kubeflow SDK to fine-tune models like Qwen3-4B. The setup utilizes vLLM for candidate generation alongside LoRA weight updates, sharing storage via ReadWriteMany persistent volume claims. In a practical tool-calling test, five training iterations raised task accuracy from 33% to 67%, teaching the model to emit parseable function invocations rather than fabricated conversational text.