Loading post…
Hugging FaceEvelyn Duesterwald, Lilian Ngweta, Vatche Isahagian, Jayaram Radhakrishnan, Vinod Muthusamy, Gaodan Fang, Ashwath Vaithinathan Aravindan, Punleuk Oum, G Thomas, Merve Unuvar, Ayhan Sebin, Michał UlewiczExplainer
Your Agent Aced the Task. Will It Do It Again?
Summary
Agent reliability can be obscured by average benchmark scores: Mean@k reports average success, while Pass^k measures tasks succeeding on every repeated run. On AppWorld, a ReAct agent using GPT-4.1 achieved Mean@5 of 77.4% but Pass^5 of 53.0%, leaving a 24.4-point consistency gap, with temperature set to 0.0. The Consistency Analyzer diagnoses flip-prone decision points by resampling each step in one recorded trajectory with k=5 completions, requiring no ground truth, logits, or end-to-end replay. ALTK-Evolve turns flagged instability into reusable consistency guidelines that are retrieved at inference time. In the reported evaluation, these guidelines raised aggregate Pass^5 to 69.0% from 53.0% and Mean@5 to 81.0% from 77.4%, cutting the gap to 12.0 percentage points; similar-task Pass^5 also improved by 13.0pp.
Context
The post distinguishes average task performance from repeatability. Mean@k averages outcomes across repeated runs, whereas Pass^k counts only tasks that succeed on every run; their difference is the consistency gap. The source argues that this variability is an orthogonal reliability problem, including when an agent uses temperature 0.0.
Approach / What changed
The Consistency Analyzer replays each decision point from one recorded trajectory and requests k completions, with k=5 by default, to identify steps whose outputs vary. Flagged steps are converted into consistency guidelines and passed through ALTK-Evolve's existing storage and retrieval pipeline for inference-time use. Detection is black-box and does not require ground truth, logits, new tool calls, environment interactions, or an end-to-end task rerun.
Takeaways
Mean@5 was 77.4% for the GPT-4.1 ReAct agent on AppWorld, but only 53.0% of tasks passed all five runs, producing a 24.4-point consistency gap.
The Consistency Analyzer needs one recorded trajectory and resamples each decision step with five completions by default; it does not require model internals, ground truth, or live environment replay.
Consistency guidelines raised Pass^5 from 53.0% to 69.0% and Mean@5 from 77.4% to 81.0%, while similar-task Pass^5 improved by 13.0 percentage points.
FAQ
How does Pass^5 differ from Mean@5 and Pass@5?
Mean@5 is the average pass rate across five runs. Pass^5 counts tasks that succeed on all five runs, while Pass@5 counts tasks with at least one successful run; the source states Pass^5 ≤ Mean@5 ≤ Pass@5.
Can the analyzer work without rerunning the task end to end?
Yes. It uses one recorded trajectory, replays the existing context at each decision point, and requests multiple completions in an offline model call. It does not make new tool calls or environment interactions and requires no ground truth.
Do consistency guidelines improve repeatability at the cost of average accuracy?
Not in the reported evaluation. Aggregate Mean@5 increased from 77.4% to 81.0% while Pass^5 rose from 53.0% to 69.0%, and mean accuracy held or improved at every difficulty level.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.