RLTL;DR: Self-Improvement by Internalizing Self-Generated Feedback

Summary

Standard reinforcement learning with verifiable rewards (RLVR) fails when tasks are so difficult that agents have no chance of success and no teacher models exist. RLTL;DR addresses this by having the policy write its own TL;DR insight after each failed attempt, conditioning the next rollout on all previous insights. Backpropagation on the in-context insights internalizes a direct task-to-insight mapping. On challenging tool-calling and coding datasets filtered to Pass@128=0, standard GRPO training stays flat at 0-1% Pass@1, while RLTL;DR achieves 14-31% Pass@1 with insights in context during training and 12-13% without at eval time. Reducing to SFTL;DR, training on only 4k (task, insight) tuples without rollouts, recovers almost full performance of RLTL;DR and classical SFT, demonstrating a promising compacted training paradigm.

Takeaways

1. 1

   Standard GRPO training on challenging datasets filtered to Pass@128=0 yields 0-1% Pass@1, showing a learning barrier for self-improvement.

2. 2

   RLTL;DR breaks through this barrier, achieving 14-31% Pass@1 with insights in context during training and 12-13% without at eval time.

3. 3

   SFTL;DR, training on only 4k (task, insight) tuples without any rollouts, recovers almost full performance of RLTL;DR and classical SFT on full rollouts.

FAQ