---
title: "Your Agent Aced the Task. Will It Do It Again?"
description: "Agent reliability can be obscured by average benchmark scores: Mean@k reports average success, while Pass^k measures tasks succeeding on every repeated run. On AppWorld, a ReAct agent using GPT-4.1 achieved Mean@5 of 77.4% but Pass^5 of 53.0%, leaving a 24.4-point consistency gap, with temperature set to 0.0. The Consistency Analyzer diagnoses flip-prone decision points by resampling each step in one recorded trajectory with k=5 completions, requiring no ground truth, logits, or end-to-end replay. ALTK-Evolve turns flagged instability into reusable consistency guidelines that are retrieved at inference time. In the reported evaluation, these guidelines raised aggregate Pass^5 to 69.0% from 53.0% and Mean@5 to 81.0% from 77.4%, cutting the gap to 12.0 percentage points; similar-task Pass^5 also improved by 13.0pp."
---

# Your Agent Aced the Task. Will It Do It Again?

[Hugging Face](https://yomu.fyi/company/hugging-face) · Evelyn Duesterwald, Lilian Ngweta, Vatche Isahagian, Jayaram Radhakrishnan, Vinod Muthusamy, Gaodan Fang, Ashwath Vaithinathan Aravindan, Punleuk Oum, G Thomas, Merve Unuvar, Ayhan Sebin, Michał Ulewicz · Sep 15, 2026

**Type:** Explainer

## Summary

Agent reliability can be obscured by average benchmark scores: Mean@k reports average success, while Pass^k measures tasks succeeding on every repeated run. On AppWorld, a ReAct agent using GPT-4.1 achieved Mean@5 of 77.4% but Pass^5 of 53.0%, leaving a 24.4-point consistency gap, with temperature set to 0.0. The Consistency Analyzer diagnoses flip-prone decision points by resampling each step in one recorded trajectory with k=5 completions, requiring no ground truth, logits, or end-to-end replay. ALTK-Evolve turns flagged instability into reusable consistency guidelines that are retrieved at inference time. In the reported evaluation, these guidelines raised aggregate Pass^5 to 69.0% from 53.0% and Mean@5 to 81.0% from 77.4%, cutting the gap to 12.0 percentage points; similar-task Pass^5 also improved by 13.0pp.

## Context

The post distinguishes average task performance from repeatability. Mean@k averages outcomes across repeated runs, whereas Pass^k counts only tasks that succeed on every run; their difference is the consistency gap. The source argues that this variability is an orthogonal reliability problem, including when an agent uses temperature 0.0.

## Approach / What changed

The Consistency Analyzer replays each decision point from one recorded trajectory and requests k completions, with k=5 by default, to identify steps whose outputs vary. Flagged steps are converted into consistency guidelines and passed through ALTK-Evolve's existing storage and retrieval pipeline for inference-time use. Detection is black-box and does not require ground truth, logits, new tool calls, environment interactions, or an end-to-end task rerun.

## Takeaways

- Mean@5 was 77.4% for the GPT-4.1 ReAct agent on AppWorld, but only 53.0% of tasks passed all five runs, producing a 24.4-point consistency gap.
- The Consistency Analyzer needs one recorded trajectory and resamples each decision step with five completions by default; it does not require model internals, ground truth, or live environment replay.
- Consistency guidelines raised Pass^5 from 53.0% to 69.0% and Mean@5 from 77.4% to 81.0%, while similar-task Pass^5 improved by 13.0 percentage points.

## FAQ

### How does Pass^5 differ from Mean@5 and Pass@5?

Mean@5 is the average pass rate across five runs. Pass^5 counts tasks that succeed on all five runs, while Pass@5 counts tasks with at least one successful run; the source states Pass^5 ≤ Mean@5 ≤ Pass@5.

### Can the analyzer work without rerunning the task end to end?

Yes. It uses one recorded trajectory, replays the existing context at each decision point, and requests multiple completions in an offline model call. It does not make new tool calls or environment interactions and requires no ground truth.

### Do consistency guidelines improve repeatability at the cost of average accuracy?

Not in the reported evaluation. Aggregate Mean@5 increased from 77.4% to 81.0% while Pass^5 rose from 53.0% to 69.0%, and mean accuracy held or improved at every difficulty level.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [LLMs](https://yomu.fyi/topic/llm), [Model Evaluation](https://yomu.fyi/topic/model-evaluation), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Hugging Face](https://huggingface.co/blog/ibm-research/altk-evolve-consistency)
- Source URL: https://huggingface.co/blog/ibm-research/altk-evolve-consistency
- Ingested by Yomu: 2026-09-15T16:02:57.979Z

[Read original post](https://huggingface.co/blog/ibm-research/altk-evolve-consistency)