PROOF-Gen: From Optimized Data to Better Distillation

Summary

Standard distillation pipelines for tool-calling agents use generate-and-filter supervised fine-tuning on teacher trajectories, discarding failed trials and missing training signals on hard scenarios. On τ 2-bench, 57% of teacher trials fail, with two-thirds being near-misses undone by a single error. PROOF-Gen (Per-scenario Reflective Optimization to Overcome Failed Generation) addresses this loss by employing a reflector that reviews execution traces and evaluation feedback to craft corrective guidance for the teacher. Once passing trajectories are generated, this scaffold guidance is stripped so students train on clean demonstrations. Per-scenario prompt optimization recovers 93% of failed scenarios on τ 2-bench, boosting Qwen3-4B-Instruct-2507 Passˆ1 from 0.132 to 0.529 and lifting deployed on-device model performance across all locales.

Context

Distilling tool-calling capabilities into deployable models relies on generate-and-filter supervised fine-tuning that discards failed teacher trajectories. This approach repeatedly incurs teacher costs while failing to learn from hard scenarios, leaving behind near-misses where the teacher committed only one decisive error.

Approach / What changed

PROOF-Gen uses a reflector to evaluate the execution traces and evaluation feedback of failed tasks, generating per-scenario corrective guidance that steers the teacher toward a passing trajectory. The scaffolding guidance is stripped before fine-tuning so student models train on clean demonstrations without task-specific prompts.

Takeaways

• PROOF-Gen recovers 93% of failed scenarios on τ 2-bench via per-scenario prompt optimization.
• Fine-tuning on the combined dataset increases Qwen3-4B-Instruct-2507 Passˆ1 from 0.132 to 0.529 and provides a +7.2pp gain for Gemma 4 E4B-it on BFCL v4 multi-turn.
• In production pipelines, the approach improves goal completion by +6.3pp and transfers to on-device models with positive gains across every locale.

Related reading

• Dropbox · Jun 25, 2026

  How we used DSPy to turn AI evaluations into better responses in Dash chat

  Dropbox improved its Dash chat agent by establishing an automated optimization loop powered by DSPy and LLM-as-judge evaluations. Engineers first calibrated their LLM judges against human-annotated interaction traces, then used those judges to systematically optimize the agent's system prompts via offline counterfactual replay. This automated workflow reduced incomplete responses by 26% while decreasing overall token consumption.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon

• Red Hat · Aug 26, 2026

  GRPO fine-tuning on Red Hat OpenShift AI: Reinforcement learning from verifiable rewards with Training Hub

  Supervised fine-tuning for structured reasoning tasks like tool calling often demands expensive, labor-intensive dataset curation. Group Relative Policy Optimization (GRPO) resolves this by using programmatic reward functions to score multiple candidate responses without requiring human labels or a dedicated reward model. On Red Hat OpenShift AI, developers can implement GRPO using Training Hub, Kubeflow Trainer, and the Kubeflow SDK to fine-tune models like Qwen3-4B. The setup utilizes vLLM for candidate generation alongside LoRA weight updates, sharing storage via ReadWriteMany persistent volume claims. In a practical tool-calling test, five training iterations raised task accuracy from 33% to 67%, teaching the model to emit parseable function invocations rather than fabricated conversational text.