Loading…
Agentic reasoning in practice: Making sense of structured and unstructured data
The Databricks AI Research Team
- Source
- Databricks
- Published
- Added to Yomu
Summary
Enterprise questions that connect structured records with unstructured text are difficult for single-turn retrieval systems to answer reliably. The post evaluates Databricks Agent Bricks Supervisor Agent (SA), a declarative agent builder built on the internal aroll framework, across STaRK’s Amazon, MAG, and Prime domains and six KARLBench tasks. SA decomposes questions, routes sub-questions to Genie or Knowledge Assistant, combines results, and can reformulate queries or use SQL JOINs when initial retrieval is incomplete. It outperforms rerun SoTA baselines by 4% on STaRK-Amazon, 21% on MAG, and 38% on Prime, while delivering a 23% improvement on FinanceBench and consistent gains across KARLBench; the authors attribute the results to multi-step tool selection, reflection, and precise instructions rather than custom code.
Context
Enterprise questions may require reasoning across structured data, such as product attributes, citation networks, biomedical entities, and financial filings, together with unstructured reviews, papers, literature, or documentation. The post compares multi-step agentic reasoning with single-turn retrieval baselines on STaRK and KARLBench.
Approach / What changed
Supervisor Agent uses declarative instructions and configured tools to decompose questions, route sub-questions to Genie or Knowledge Assistant, synthesize results, detect incomplete retrieval, reformulate searches, and issue SQL JOINs when needed. The evaluation uses the three STaRK domains and six KARLBench datasets without custom code for individual tasks.
Takeaways
- Supervisor Agent improved Hit@1 over rerun SoTA baselines by 4% on STaRK-Amazon, 21% on STaRK-MAG, and 38% on STaRK-Prime.
- For a STaRK-MAG query, the agent combined Genie author counts with Knowledge Assistant paper retrieval, then used a SQL JOIN and verification steps after finding no initial overlap.
- Supervisor Agent achieved a 23% improvement on FinanceBench and a 78% relative improvement on BrowseComp+ by reformulating incomplete searches and decomposing multi-constraint questions.