Diverse reasoning traces teach LLMs to make better decisions

Summary

Large language models typically rely on supervised fine-tuning with a single human-verified reasoning trace per example, which limits reasoning breadth and causes mode collapse when attempting parallel reasoning. Researchers presented set-supervised fine tuning (SSFT) and global forking policy optimization (GFPO) to teach models diverse problem-solving strategies. SSFT represents reasoning as a set of complete solution paths paired with global forking tokens, utilizing bipartite matching to achieve order-invariant training and specialized tokens. GFPO applies lightweight reinforcement learning to select the optimal forking token upfront, detaching gradients from generated rollouts to ensure stable optimization. On standard reasoning and coding benchmarks, the combined framework increased single-shot accuracy by 5% to 7% while improving pass@k diversity without degrading pass@1 accuracy.

Takeaways

1. 1

   Set-supervised fine tuning uses bipartite matching between traces and global forking tokens to prevent mode collapse during post-training.

2. 2

   Global forking policy optimization updates the policy over forking tokens using reward signals while detaching gradients from the generated reasoning traces.

3. 3

   Combining SSFT and GFPO yields 5% to 7% gains in single-shot accuracy on benchmarks including AIME 2024, AIME 2025, and LiveCodeBench-v5.