Evaluating AI Agents Live at the Grounded Reasoning Cup

Summary

The Grounded Reasoning Cup evaluated whether AI agents could generalize grounded reasoning improvements from OfficeQA to a newly released, enterprise-style document benchmark. Eleven academic teams had about two months to build agents with a partner lab’s model family, then applied them live to OfficeQA Pro V2, based on U.S. Treasury Accounts of Receipts and Expenditures, after a 36-hour release window. Stanford won with 63.3% accuracy, while the top systems combined document preprocessing, targeted or hybrid retrieval, specialized tools, parallelism, and answer verification; 18.8% of questions remained unsolved by every team. The results show that benchmark gains did not reliably transfer, and that parsing, retrieval, tool use, verification, and operational infrastructure can matter as much as model choice in end-to-end enterprise reasoning.

Takeaways

1. 1

   Improvements developed on OfficeQA did not always transfer reliably to OfficeQA Pro V2, supporting the use of representative held-out evaluations for testing generalization.

2. 2

   The average score gap between the highest- and lowest-scoring teams using the same model was 30.4 points, indicating that parsing, retrieval, tools, verification, and infrastructure materially affected results.

3. 3

   Strong systems combined pre-parsed documents, lexical and dense retrieval, specialized tools, explicit verification, and operational features such as preprocessing, retries, parallelism, and submission scaffolding.