Databricks developed a new enterprise data-agent benchmark in collaboration with Turing to evaluate Genie against two frontier agents on business questions in a live Databricks workspace. The benchmark uses a fictional UK energy company, Calderwick Energy, with hundreds of tables, notebooks, dashboards, saved SQL queries, and documents that capture institutional knowledge. Genie achieved a 91% pass rate, compared to 45% and 58% for the other agents, while using fewer than half the tokens and completing tasks 3-4x faster. Genie Ontology automatically extracts and organizes enterprise context from these assets, surfacing relevant information that table schemas alone cannot provide. The results demonstrate that context retrieval and harness quality, not just token-generating intelligence, determine agent performance on real-world enterprise tasks.
Context
Existing data agent benchmarks focus mainly on tables and pipeline code, but real enterprise tasks require agents to discover and reason about business knowledge distributed across notebooks, dashboards, saved SQL queries, and documents. Genie Ontology provides a shared enterprise context layer that gives AI agents a common understanding of business definitions, calculation rules, and institutional knowledge.
Approach / What changed
Databricks developed a new enterprise data-agent benchmark in collaboration with Turing, using a fictional company Calderwick Energy with hundreds of tables, notebooks, dashboards, saved SQL queries, and documents. Genie uses Genie Ontology to automatically extract and organize enterprise context from these assets, surfacing relevant information for answering business questions. The evaluation compared Genie against two frontier agents (Agent A and Agent B) on accuracy, token usage, and wall time across multiple runs per task in a live Databricks workspace.
Takeaways
1
Genie achieved a 91% pass rate on the enterprise benchmark, compared to 45% and 58% for two frontier agents, while using fewer than half the tokens and completing tasks 3-4x faster.
2
Genie Ontology automatically extracts and organizes business knowledge from notebooks, dashboards, saved SQL queries, and documents, providing context that table schemas alone cannot establish.
3
The benchmark uses a fictional company Calderwick Energy with hundreds of tables (largest ~120 million rows) and real-world messiness like copied notebooks, abandoned analyses, and frozen reporting tables.
FAQ
What is Genie Ontology?
Genie Ontology is an enterprise context layer that automatically extracts and organizes business knowledge from notebooks, dashboards, saved SQL queries, documents, and other analytical assets. It surfaces relevant context for Genie to use when answering questions, giving every AI agent a shared understanding of the business.
What does the benchmark dataset contain?
The benchmark uses a fictional UK energy company called Calderwick Energy with hundreds of tables (largest ~120 million rows), plus notebooks, dashboards, saved SQL queries, code, metrics, documents, and Genie Agents. It includes real-world messiness like copied notebooks, abandoned analyses, old documents, and frozen reporting tables.
How did Genie compare to other agents?
Genie achieved a 91% pass rate, compared to 45% for Agent A and 58% for Agent B. Genie was 3-4x faster and used fewer than half the tokens on average (31,829 vs 889,019 and 1,051,070). The other agents did not get the correct historical rate in the example walkthrough.
Atlas Search finds matching document IDs fast, but materializing each hit from the collection is where the latency and CPU go. This trade-off map shows the benefit, the cost, and the condition for accepting that second phase.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.