PROOF-Gen: From Optimized Data to Better Distillation

Summary

Standard distillation pipelines for tool-calling agents use generate-and-filter supervised fine-tuning on teacher trajectories, discarding failed trials and missing training signals on hard scenarios. On τ 2-bench, 57% of teacher trials fail, with two-thirds being near-misses undone by a single error. PROOF-Gen (Per-scenario Reflective Optimization to Overcome Failed Generation) addresses this loss by employing a reflector that reviews execution traces and evaluation feedback to craft corrective guidance for the teacher. Once passing trajectories are generated, this scaffold guidance is stripped so students train on clean demonstrations. Per-scenario prompt optimization recovers 93% of failed scenarios on τ 2-bench, boosting Qwen3-4B-Instruct-2507 Passˆ1 from 0.132 to 0.529 and lifting deployed on-device model performance across all locales.

Takeaways

1. 1

   PROOF-Gen recovers 93% of failed scenarios on τ 2-bench via per-scenario prompt optimization.

2. 2

   Fine-tuning on the combined dataset increases Qwen3-4B-Instruct-2507 Passˆ1 from 0.132 to 0.529 and provides a +7.2pp gain for Gemma 4 E4B-it on BFCL v4 multi-turn.

3. 3

   In production pipelines, the approach improves goal completion by +6.3pp and transfers to on-device models with positive gains across every locale.