# SOP-Bench: A new benchmark for evaluating AI agents on real business procedures

[Amazon](https://yomu.fyi/company/amazon) · Rohith Nama, Nandi Subhrangshu · Aug 21, 2026

**Type:** Benchmark

## Summary

AI agents often struggle to execute standard operating procedures because real-world business workflows contain implicit domain knowledge and procedural ambiguity that standard benchmarks overlook. To evaluate agent reliability, researchers introduced SOP-Bench, an open framework featuring more than 2,000 runnable tasks across 12 industrial domains paired with functioning tool interfaces and ground-truth answers. Domain experts authored the workflows, while an Anthropic Claude 3.5 Sonnet v2 model generated mock APIs, tool code, and edge-case datasets before expert verification. Experiments evaluating function-calling and reasoning agents across 11 frontier models revealed that newer models did not consistently outperform older versions, and providing 20 extraneous tools nearly halved task success. The benchmark, baseline agents, and evaluation code are released on GitHub and HuggingFace for reproducible testing.

## Context

AI agents struggle with standard operating procedures because real business workflows rely on unstated domain knowledge, ambiguous instructions, and dynamic judgment calls. Existing agent benchmarks isolate single capabilities in clean, machine-formatted environments, failing to test multi-step tool coordination, context tracking, and error recovery against ground truth across realistic enterprise tasks.

## Approach / What changed

Authors built SOP-Bench, an open evaluation framework comprising over 2,000 executable tasks across 12 business domains. Constructed by pairing domain experts with an Anthropic Claude 3.5 Sonnet v2 model to generate schemas, mock APIs, and test cases, the benchmark evaluates agents through reproducible tool interactions scored directly against ground-truth outcomes rather than model-based grading.

## Takeaways

- Upgrading model generations does not guarantee better performance; for example, the newer Claude 4.5 family scored lower than the older Claude 4 family when using a reasoning-style agent.
- Adding 20 extraneous, plausible tools to an agent's toolkit nearly halved its success rate, demonstrating that superfluous capabilities can degrade agent execution.
- Agent performance varied dramatically across procedural domains, achieving approximately 90% accuracy on email triage tasks but dropping to around 25% on driving video object annotation.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Reliability](https://yomu.fyi/topic/reliability), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://www.amazon.science/blog/sop-bench-a-new-benchmark-for-evaluating-ai-agents-on-real-business-procedures)