Loading post…
Hugging FaceVatche Isahagian, Gaodan Fang, Jayaram Radhakrishnan, Punleuk Oum, Ashwath Vaithinathan Aravindan, Evelyn Duesterwald, G Thomas, Vinod Muthusamy, Merve Unuvar, Ayhan SebinBenchmark
How Much Memory Does Your Agent Actually Need?
Summary
The post examines how much agentic memory different models can use effectively, evaluating ALTK-Evolve's self-distilled guidelines across eight models on AppWorld's 585 multi-step tasks. Memory is produced by extracting and consolidating lessons from successful and unsuccessful trajectories, then delivering either the full guideline set on every ReAct step or a fixed core plus task-relevant retrieval, without updating model weights. Results identify three patterns: strong models with headroom benefited from the full set, gpt-oss-120b gained 16.1 percentage points in task completion from curated retrieval, and GLM-5 showed no measurable gain; stricter scenario completion sometimes improved more. Curated retrieval gave gpt-oss-120b that gain with only 5% more tokens versus 51% for its full set, while prompt caching is presented as a way to reduce recurring input cost, supporting calibration to model capability rather than simply accumulating memory.
Context
The work addresses the observation that adding more agentic memory does not consistently improve performance. Across models, the useful amount and delivery strategy vary with factors including capability, benchmark headroom, context-window size, architecture, guideline quality, and task distribution. The evaluation uses AppWorld's 585 multi-step tasks across nine simulated apps, with Task Goal Completion and the stricter Scenario Goal Completion as metrics.
Approach / What changed
ALTK-Evolve extracts behavioral guidelines from successful and unsuccessful agent trajectories, consolidates them into a reusable set, and injects them at inference time without changing model weights. The evaluation compares a no-memory baseline, the full guideline set injected on every ReAct step, and curated retrieval consisting of a fixed high-confidence core plus task-relevant guidelines. Guidelines are mined only from the AppWorld training split, and prompt caching is proposed for reducing the cost of repeated static context.
Takeaways
Strong models with remaining benchmark headroom generally benefited from the full guideline set; DeepSeek-V3.2 gained 9.5 percentage points in TGC and 16.1 points in SGC.
gpt-oss-120b performed best with curated retrieval, gaining 16.1 percentage points in TGC while increasing tokens per task by only 5%, compared with a 51% increase for its full guideline set.
GLM-5 showed no measurable gain from the tested memory configurations, illustrating that additional context did not improve every evaluated model.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.