Loading…
The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents
- Source
- Clickhouse
- Published
- Added to Yomu
Summary
The post presents data-agent-mnist, an open harness for measuring analytical agents on correctness, cost, and speed against a warehouse-shaped workload rather than a single text-to-SQL query. The authors extracted 201 production analytics questions from DWAINE, curated and anonymized them, reconstructed a deterministic synthetic warehouse with 18 tables and 865 columns, and evaluated 28 proprietary and open-weight models. The benchmark preserves schema discovery, multi-turn querying, planning, and multi-hop joins across curated marts and a dimensional CRM layer, with provider-diverse ground truth and judging described as part of the workflow. On the authors’ warehouse, Claude Fable 5.1 ranked first for correctness at 76.6%, while the full run cost $1 with DeepSeek V4 Flash versus $52 with Fable 5.1, an 11-percentage-point correctness difference. The harness is intended to identify the best production fit for a specific warehouse, not a universal winner.
Context
Traditional text-to-SQL benchmarks assume that the schema is provided, the model gets one attempt, and answers are judged against a single gold query. Production analytical agents instead discover schemas, run multiple queries, and work across complex warehouse structures, so model suitability depends on the organization’s questions and data.
Approach / What changed
The authors built a reusable harness from 201 questions logged by DWAINE, applying curation, deterministic anonymization, synthetic warehouse reconstruction, and planted entity references. Evaluations replay against an identical warehouse state, recover ground truth with a provider-diverse committee, and use a judge panel that does not grade its own model family. The harness can be adapted to another warehouse and traffic set.
Takeaways
- The benchmark’s synthetic warehouse preserves real schema complexity: 18 tables and 865 columns across curated marts and a dimensional CRM layer, with relationships and scales kept internally consistent while names and figures are anonymized.
- Correctness rankings were workload-dependent: Claude Fable 5.1 scored 76.6% on the authors’ 201 questions, while frontier-level models occupied all top 10 positions and five of those models were open-weight.
- The complete benchmark run cost $1 with DeepSeek V4 Flash and $52 with Fable 5.1, with the cheaper option sacrificing 11 percentage points of correctness.