Loading post…
Hugging FaceLysandre, Nathan Habib, Pedro CuencaBenchmark
Is it agentic enough? Benchmarking open models on your own tooling
Summary
Final-answer checks can show whether an agent completed a task, but they miss the turns, tokens, seconds, errors, and command path required. The proposed tool-specific benchmark measures that work on transformers, comparing models, revisions, tasks, and three access tiers: bare installation, source clone, and a packaged Skill containing CLI documentation and examples. It uses deterministic tasks with exact matching, launches one Hugging Face Job per model–revision–task combination on identical hardware, and records traces in a Hugging Face Bucket. Reports include match rate, median new, cached, and generated tokens, time, error rate, and marker adoption, while traces expose the agent's commands. On transformers, the CLI plus Skill helped large open models but caused Qwen3-14B to misinterpret documentation as an executable tool, dropping classify-sentiment match from 100% to 0% in the cited comparison.
Context
Final-answer evaluation does not reveal how much work an agent required, including its turns, token use, latency, errors, and path. The benchmark was created to measure whether library changes improve agent performance and to detect cases where an affordance helps stronger models but harms smaller ones.
Approach / What changed
The harness evaluates command-line-operable tools across models, revisions, and deterministic tasks. For transformers, it compares bare installation, a source checkout, and a packaged Skill with CLI documentation and examples. Each model–revision–task run executes as a separate Hugging Face Job on identical hardware; exact-match results, timing, token counts, errors, marker adoption, and native agent traces are collected into reports. The runs are driven by the pi coding agent using open models.
Takeaways
The three evaluation tiers are not nested: a Skill provides curated documentation rather than the source tree, and a model can perform better with the clone than with the Skill.
The harness reports match percentage, median time, new/cached/generated token counts, error percentage, and marker adoption, while guarding against silent runs with no output, tool calls, or answer.
For Qwen3-14B, the Skill caused classify-sentiment match to fall from 100% on the clone variant to 0% at the CLI plus Skill revision because the model treated the documented CLI as an unregistered callable tool.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.