Loading…
Evaluating AI Agents Live at the Grounded Reasoning Cup
Databricks AI Research Team
- Source
- Databricks
- Published
- Added to Yomu
Summary
The Grounded Reasoning Cup evaluated whether AI agents could generalize grounded reasoning improvements from OfficeQA to a newly released, enterprise-style document benchmark. Eleven academic teams had about two months to build agents with a partner lab’s model family, then applied them live to OfficeQA Pro V2, based on U.S. Treasury Accounts of Receipts and Expenditures, after a 36-hour release window. Stanford won with 63.3% accuracy, while the top systems combined document preprocessing, targeted or hybrid retrieval, specialized tools, parallelism, and answer verification; 18.8% of questions remained unsolved by every team. The results show that benchmark gains did not reliably transfer, and that parsing, retrieval, tool use, verification, and operational infrastructure can matter as much as model choice in end-to-end enterprise reasoning.
Context
The competition addressed whether improvements developed on the OfficeQA grounded-reasoning benchmark would generalize to similar real-world tasks and a newly released document corpus. It also examined how well AI agents could answer complex questions over enterprise-style collections under live, time-constrained conditions.
Approach / What changed
Teams built agents over approximately two months using their assigned industry partner’s model family, evaluated techniques on OfficeQA, and then applied those systems to OfficeQA Pro V2 during six live rounds. Teams could choose their agent framework, corpus versions, retrieval strategy, tools, and human-in-the-loop workflow, subject to the model-family constraint.
Takeaways
- Improvements developed on OfficeQA did not always transfer reliably to OfficeQA Pro V2, supporting the use of representative held-out evaluations for testing generalization.
- The average score gap between the highest- and lowest-scoring teams using the same model was 30.4 points, indicating that parsing, retrieval, tools, verification, and infrastructure materially affected results.
- Strong systems combined pre-parsed documents, lexical and dense retrieval, specialized tools, explicit verification, and operational features such as preprocessing, retries, parallelism, and submission scaffolding.