---
title: "Agent Seer: Synthesizing Scenarios from Specification Understanding"
description: "Agent Seer addresses the problem of creating realistic, scalable test scenarios for AI agents that use external tools, where hand-built cases require domain expertise and static benchmarks cannot track changing APIs. It synthesizes scenarios from a single Model Context Protocol (MCP) specification containing function names, natural-language descriptions, and typed parameter schemas, without examples, live tool access, or domain-specific tuning. The pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock-data-grounded, multi-turn dialogues, then evaluates tool-calling correctness and conversational coherence across seven MCP specifications spanning domains and tool-suite sizes. It achieves strong quality across all domains and complete tool coverage on small and medium specifications; parameter schema complexity correlates most strongly with quality variation, while tool-suite size has a smaller orthogonal role. Among imperfect scenarios, argument value accuracy is the dominant failure mode, which coarse-grained name-match metrics do not reveal."
---

# Agent Seer: Synthesizing Scenarios from Specification Understanding

[Apple](https://yomu.fyi/company/apple) · Aug 28, 2026

**Type:** Benchmark

## Summary

Agent Seer addresses the problem of creating realistic, scalable test scenarios for AI agents that use external tools, where hand-built cases require domain expertise and static benchmarks cannot track changing APIs. It synthesizes scenarios from a single Model Context Protocol (MCP) specification containing function names, natural-language descriptions, and typed parameter schemas, without examples, live tool access, or domain-specific tuning. The pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock-data-grounded, multi-turn dialogues, then evaluates tool-calling correctness and conversational coherence across seven MCP specifications spanning domains and tool-suite sizes. It achieves strong quality across all domains and complete tool coverage on small and medium specifications; parameter schema complexity correlates most strongly with quality variation, while tool-suite size has a smaller orthogonal role. Among imperfect scenarios, argument value accuracy is the dominant failure mode, which coarse-grained name-match metrics do not reveal.

## Context

Creating realistic evaluation scenarios for tool-using AI agents requires manual domain expertise, does not scale across tool ecosystems, and produces static benchmarks that cannot track evolving APIs.

## Approach / What changed

Agent Seer uses a single Model Context Protocol specification to enrich raw tool schemas, generate graded scenarios with synthetic tool outputs, and expand them into mock-data-grounded multi-turn dialogues. It operates without examples, live tool access, or domain-specific tuning, and evaluates the resulting scenarios across seven MCP specifications using tool-calling correctness and conversational coherence.

## Takeaways

- Agent Seer generates evaluation scenarios from MCP function names, natural-language descriptions, and typed parameter schemas without examples or live tool access.
- Parameter schema complexity is the strongest correlate of quality variation, while tool-suite size has a smaller, orthogonal role.
- Argument value accuracy is the dominant failure mode in imperfect scenarios and is not visible through coarse-grained name-match metrics.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Testing](https://yomu.fyi/topic/testing)

- Source: [Apple](https://machinelearning.apple.com/research/agent-seer-synthesizing-scenarios)
- Source URL: https://machinelearning.apple.com/research/agent-seer-synthesizing-scenarios
- Ingested by Yomu: 2026-08-28T16:00:44.297Z

[Read original post](https://machinelearning.apple.com/research/agent-seer-synthesizing-scenarios)