An Engineer’s Guide to Better AI Skills: Implementing a Testing Process to Optimize Agent…

Summary

Software engineers deploying AI agents often encounter unreliable skill invocation for repository-specific architectures, such as Pinterest's iOS framework. To quantify and resolve this issue, a Bash test harness piped positive and negative prompts into Pin-agent and Claude Code while parsing streamed JSON logs for execution signatures. Initial vanilla benchmarks yielded baseline accuracies of 73% for GPT 5.2-codex and 62% for Opus 4.5 Claude. Optimizations such as expanding YAML frontmatter descriptions with architectural context, utilizing capitalized imperative directives, and adding reference tables in AGENTS.md raised invocation rates. Both agents reliably loaded skills when prompts contained explicit invocations, demonstrating that empirical testing paired with descriptive prompting stabilizes agent skill adoption.

Context

AI agents failed to consistently load a domain-specific knowledge skill (rx-mvvm) for Pinterest's iOS architecture, particularly during architectural reviews when engineers supplied terse or ambiguous prompts.

Approach / What changed

Built a Bash test harness that pipes 15 positive and 5 negative prompt cases across multiple runs into Pin-agent and Claude Code, parses JSON logs for skill invocation patterns, and measures accuracy across optimizations like frontmatter descriptions, aggressive language, and AGENTS.md tables.

Takeaways

• Baseline skill invocation accuracy was 73% on Pin-agent (GPT 5.2-codex) and 62% on Claude Code (Opus 4.5) when subjected to terse test prompts.
• Adding contextual architectural information to skill YAML frontmatter improved skill invocation rates across both tested agents.
• Every test run on both agents successfully loaded the skill when the prompt explicitly instructed the agent to load it.

Related reading

• Grab · Jun 22, 2026

  Scaling out Distroless adoption With AI

  Grab is transitioning its microservices to Distroless base images to eliminate unnecessary binaries and reduce vulnerability risks, but the migration risks runtime failures from missing shared objects and system utilities. To safely validate container execution in continuous integration without staging dependencies, the team relied on medium tests that run containerized services alongside internal dependencies managed by Testcontainers. Because hundreds of services lacked this test harness, Grab implemented an agentic workflow using Claude Code and Model Context Protocol integrations to inspect repositories, generate test boilerplate, and resolve configuration errors. Once test baselines are established, an automated patch-test-compare pipeline updates Dockerfiles, constructs multi-stage builds for necessary dynamic libraries, and creates draft merge requests for human approval.

  Jia Yee Chong

• Dropbox · May 28, 2026

  Beyond code generation: rethinking engineering productivity in the age of AI agents