Agent Seer: Synthesizing Scenarios from Specification Understanding

Summary

Agent Seer addresses the problem of creating realistic, scalable test scenarios for AI agents that use external tools, where hand-built cases require domain expertise and static benchmarks cannot track changing APIs. It synthesizes scenarios from a single Model Context Protocol (MCP) specification containing function names, natural-language descriptions, and typed parameter schemas, without examples, live tool access, or domain-specific tuning. The pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock-data-grounded, multi-turn dialogues, then evaluates tool-calling correctness and conversational coherence across seven MCP specifications spanning domains and tool-suite sizes. It achieves strong quality across all domains and complete tool coverage on small and medium specifications; parameter schema complexity correlates most strongly with quality variation, while tool-suite size has a smaller orthogonal role. Among imperfect scenarios, argument value accuracy is the dominant failure mode, which coarse-grained name-match metrics do not reveal.

Takeaways

1. 1

   Agent Seer generates evaluation scenarios from MCP function names, natural-language descriptions, and typed parameter schemas without examples or live tool access.

2. 2

   Parameter schema complexity is the strongest correlate of quality variation, while tool-suite size has a smaller, orthogonal role.

3. 3

   Argument value accuracy is the dominant failure mode in imperfect scenarios and is not visible through coarse-grained name-match metrics.