Loading post…
NVIDIAMichelle HortonBenchmark
Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator
Summary
NVIDIA SkillEvaluator measures whether packaged agent skills improve agent performance by comparing task runs with and without a skill installed. It combines three evaluation tiers: static safety and structure checks, embedding-based distinctiveness analysis, and live evaluations in isolated Harbor sandboxes. In the August 12, 2026 benchmark snapshot, more than 300 verified skills spanning over 30 NVIDIA products were tested across Claude Code and Codex, with macro-averaged Skill Lift of +31 points across all dimensions and +39 excluding Security. Correctness rose from 46 to 87 and Effectiveness from 39 to 78, while Security moved from 97 to 98; token and runtime effects varied by skill, with one reducing both and another increasing them.
Context
AI agents may spend extra steps finding tools, waste tokens on dead ends, or struggle with specialized NVIDIA tasks even when capable models and documentation are available. The work measures whether skills improve agent trajectories and outputs, while checking that skills are safe, distinct, and correctly activated.
Approach / What changed
SkillEvaluator evaluates skills in three tiers: static validation and security checks, embedding similarity analysis for duplication and overlap, and live comparisons using Harbor. Each case runs in isolated sandboxes with the same prompt, model, inputs, and grading criteria, once with the skill and once without it. Results are compared across Claude Code and Codex using Skill Lift, defined as the with-skill score minus the without-skill score.
Takeaways
The evaluation pipeline checks schema and frontmatter, quality, prompt injection, data exfiltration, secrets, PII, licenses, and script linting before a skill is published.
Skill Lift varied more across NVIDIA products, from roughly +2 to +46, than across harnesses; Claude Code and Codex differed by about 5 points on average.
Token savings were not automatic: jetson-optimize-memory reduced tokens by 76.9% and execution time by 53.7%, while cuopt-install increased tokens by 120.3% and execution time by 20.8%.
Source and freshness
- Source
- developer.nvidia.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.