Red HatDejan BosanacExplainer
Skills in AI agent projects silently misfire when their trigger descriptions are passive or too general, causing the agent to skip using them. The authoring tool skill-creator, from Anthropic’s skills collection, provides an eval loop that generates positive and negative test prompts and iterates on description accuracy using a training and held-out test set to avoid overfitting. These tests are ephemeral, so the open-source framework promptfoo extends them into permanent configuration files that can be committed alongside skills. Promptfoo’s agent SDK providers test through the actual agent runtime and include a skill-used assertion to confirm proper routing, and its YAML-based configuration fits naturally into version control. The final step is wiring promptfoo evals into CI, using GitHub Actions with path filters to run on pull requests that modify skills, catching regressions before they ship and costing roughly $0.10-0.15 per eval.
Skills in AI agent projects can silently misfire when trigger descriptions are weak, and there is no built-in mechanism to re-run tests after edits, leading to regressions that go unnoticed. The post addresses how to move from ephemeral authoring evals to permanent, CI-integrated test suites.
Use skill-creator’s eval loop during authoring to generate and iterate on test prompts, then migrate those test prompts to promptfoo configuration files that commit alongside skills, and finally wire promptfoo evals into CI via GitHub Actions to run on any change to skills or evals.
Silent misfire is the most common skill failure mode: the agent does not invoke the skill because the trigger description is passive or too general, and there is no error to indicate the problem.
skill-creator’s description optimizer uses a training/held-out test split to automatically improve trigger descriptions and avoid overfitting to the eval prompts.
Promptfoo’s skill-used assertion and agent SDK providers let you test through the actual agent runtime, ensuring the skill is invoked via proper routing, not just read from the filesystem.
Silent misfire occurs when the agent does not invoke a skill because the trigger description uses passive or general language, making the task seem achievable from the agent's own context. The skill sits unused without any error or warning.
Use skill-creator's eval loop during authoring to generate positive and negative test prompts and evaluate description accuracy. Then migrate those test prompts to promptfoo configuration files to create permanent, version-controlled tests that can run in CI.
Because testing by browsing the source repo allows the agent to read the skill description directly, bypassing the routing process. You need to test through a fixture directory with the appropriate working_dir and agent runtime to ensure the skill is discovered and triggered through the proper mechanism.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A five-step workflow for turning agent output into versioned artifacts: scope a repo per agent, branch instead of copying, run in an isolated sandbox, gate changes with push-triggered evals, then deploy or roll back.
Agentic development turns AI assistance into delegated work: a harness runs the agents, skills supply domain knowledge, a sandbox bounds execution, and evals verify the result.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
