---
title: "Is it agentic enough? Benchmarking open models on your own tooling"
description: "Final-answer checks can show whether an agent completed a task, but they miss the turns, tokens, seconds, errors, and command path required. The proposed tool-specific benchmark measures that work on transformers, comparing models, revisions, tasks, and three access tiers: bare installation, source clone, and a packaged Skill containing CLI documentation and examples. It uses deterministic tasks with exact matching, launches one Hugging Face Job per model–revision–task combination on identical hardware, and records traces in a Hugging Face Bucket. Reports include match rate, median new, cached, and generated tokens, time, error rate, and marker adoption, while traces expose the agent's commands. On transformers, the CLI plus Skill helped large open models but caused Qwen3-14B to misinterpret documentation as an executable tool, dropping classify-sentiment match from 100% to 0% in the cited comparison."
---

# Is it agentic enough? Benchmarking open models on your own tooling

[Hugging Face](https://yomu.fyi/company/hugging-face) · Lysandre, Nathan Habib, Pedro Cuenca · Jun 18, 2026

**Type:** Benchmark

## Summary

Final-answer checks can show whether an agent completed a task, but they miss the turns, tokens, seconds, errors, and command path required. The proposed tool-specific benchmark measures that work on transformers, comparing models, revisions, tasks, and three access tiers: bare installation, source clone, and a packaged Skill containing CLI documentation and examples. It uses deterministic tasks with exact matching, launches one Hugging Face Job per model–revision–task combination on identical hardware, and records traces in a Hugging Face Bucket. Reports include match rate, median new, cached, and generated tokens, time, error rate, and marker adoption, while traces expose the agent's commands. On transformers, the CLI plus Skill helped large open models but caused Qwen3-14B to misinterpret documentation as an executable tool, dropping classify-sentiment match from 100% to 0% in the cited comparison.

## Context

Final-answer evaluation does not reveal how much work an agent required, including its turns, token use, latency, errors, and path. The benchmark was created to measure whether library changes improve agent performance and to detect cases where an affordance helps stronger models but harms smaller ones.

## Approach / What changed

The harness evaluates command-line-operable tools across models, revisions, and deterministic tasks. For transformers, it compares bare installation, a source checkout, and a packaged Skill with CLI documentation and examples. Each model–revision–task run executes as a separate Hugging Face Job on identical hardware; exact-match results, timing, token counts, errors, marker adoption, and native agent traces are collected into reports. The runs are driven by the pi coding agent using open models.

## Takeaways

- The three evaluation tiers are not nested: a Skill provides curated documentation rather than the source tree, and a model can perform better with the clone than with the Skill.
- The harness reports match percentage, median time, new/cached/generated token counts, error percentage, and marker adoption, while guarding against silent runs with no output, tool calls, or answer.
- For Qwen3-14B, the Skill caused classify-sentiment match to fall from 100% on the clone variant to 0% at the CLI plus Skill revision because the model treated the documented CLI as an unregistered callable tool.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Testing](https://yomu.fyi/topic/testing)

- Source: [Hugging Face](https://huggingface.co/blog/is-it-agentic-enough)
- Source URL: https://huggingface.co/blog/is-it-agentic-enough
- Ingested by Yomu: 2026-08-29T01:50:39.692Z

[Read original post](https://huggingface.co/blog/is-it-agentic-enough)