# PROOF-Gen: From Optimized Data to Better Distillation

[Apple](https://yomu.fyi/company/apple) · Aug 26, 2026

**Type:** Problem & solution

## Summary

Standard distillation pipelines for tool-calling agents use generate-and-filter supervised fine-tuning on teacher trajectories, discarding failed trials and missing training signals on hard scenarios. On τ 2-bench, 57% of teacher trials fail, with two-thirds being near-misses undone by a single error. PROOF-Gen (Per-scenario Reflective Optimization to Overcome Failed Generation) addresses this loss by employing a reflector that reviews execution traces and evaluation feedback to craft corrective guidance for the teacher. Once passing trajectories are generated, this scaffold guidance is stripped so students train on clean demonstrations. Per-scenario prompt optimization recovers 93% of failed scenarios on τ 2-bench, boosting Qwen3-4B-Instruct-2507 Passˆ1 from 0.132 to 0.529 and lifting deployed on-device model performance across all locales.

## Context

Distilling tool-calling capabilities into deployable models relies on generate-and-filter supervised fine-tuning that discards failed teacher trajectories. This approach repeatedly incurs teacher costs while failing to learn from hard scenarios, leaving behind near-misses where the teacher committed only one decisive error.

## Approach / What changed

PROOF-Gen uses a reflector to evaluate the execution traces and evaluation feedback of failed tasks, generating per-scenario corrective guidance that steers the teacher toward a passing trajectory. The scaffolding guidance is stripped before fine-tuning so student models train on clean demonstrations without task-specific prompts.

## Takeaways

- PROOF-Gen recovers 93% of failed scenarios on τ 2-bench via per-scenario prompt optimization.
- Fine-tuning on the combined dataset increases Qwen3-4B-Instruct-2507 Passˆ1 from 0.132 to 0.529 and provides a +7.2pp gain for Gemma 4 E4B-it on BFCL v4 multi-turn.
- In production pipelines, the approach improves goal completion by +6.3pp and transfers to on-device models with positive gains across every locale.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning)

[Read original post](https://machinelearning.apple.com/research/proof-gen-optimized-distillation)