Loading…
SOP-Bench: A new benchmark for evaluating AI agents on real business procedures
AmazonRohith Nama, Nandi Subhrangshu
Summary
AI agents often struggle to execute standard operating procedures because real-world business workflows contain implicit domain knowledge and procedural ambiguity that standard benchmarks overlook. To evaluate agent reliability, researchers introduced SOP-Bench, an open framework featuring more than 2,000 runnable tasks across 12 industrial domains paired with functioning tool interfaces and ground-truth answers. Domain experts authored the workflows, while an Anthropic Claude 3.5 Sonnet v2 model generated mock APIs, tool code, and edge-case datasets before expert verification. Experiments evaluating function-calling and reasoning agents across 11 frontier models revealed that newer models did not consistently outperform older versions, and providing 20 extraneous tools nearly halved task success. The benchmark, baseline agents, and evaluation code are released on GitHub and HuggingFace for reproducible testing.
Context
AI agents struggle with standard operating procedures because real business workflows rely on unstated domain knowledge, ambiguous instructions, and dynamic judgment calls. Existing agent benchmarks isolate single capabilities in clean, machine-formatted environments, failing to test multi-step tool coordination, context tracking, and error recovery against ground truth across realistic enterprise tasks.
Approach / What changed
Authors built SOP-Bench, an open evaluation framework comprising over 2,000 executable tasks across 12 business domains. Constructed by pairing domain experts with an Anthropic Claude 3.5 Sonnet v2 model to generate schemas, mock APIs, and test cases, the benchmark evaluates agents through reproducible tool interactions scored directly against ground-truth outcomes rather than model-based grading.
Takeaways
- Upgrading model generations does not guarantee better performance; for example, the newer Claude 4.5 family scored lower than the older Claude 4 family when using a reasoning-style agent.
- Adding 20 extraneous, plausible tools to an agent's toolkit nearly halved its success rate, demonstrating that superfluous capabilities can degrade agent execution.
- Agent performance varied dramatically across procedural domains, achieving approximately 90% accuracy on email triage tasks but dropping to around 25% on driving video object annotation.
Related reading
AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality
AssetOpsBench is an evaluation framework designed to benchmark agentic AI systems within industrial Asset Lifecycle Management operations, such as chillers and air handling units. The benchmark incorporates 2.3 million sensor telemetry points, 4,200 work orders, 53 structured failure modes, and over 140 curated scenarios spanning four agent roles. Systems are assessed across six qualitative criteria, including task completion, retrieval accuracy, result verification, sequence correctness, clarity and justification, and hallucination rate. Evaluation traces are analyzed using a trajectory-level diagnostic pipeline called TrajFM, which extracts and clusters failure patterns. In community tests covering over 300 agents, no evaluated model family reached the 85-point readiness threshold.
Dhaval Patel, James Rayfield, Saumya Ahuja, Chathurangi Shyalika, Shuxin Lin, Zhou, Ayhan Sebin