Loading post…
ClickhouseBenchmark
The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents
Summary
The post presents data-agent-mnist, an open harness for measuring analytical agents on correctness, cost, and speed against a warehouse-shaped workload rather than a single text-to-SQL query. The authors extracted 201 production analytics questions from DWAINE, curated and anonymized them, reconstructed a deterministic synthetic warehouse with 18 tables and 865 columns, and evaluated 28 proprietary and open-weight models. The benchmark preserves schema discovery, multi-turn querying, planning, and multi-hop joins across curated marts and a dimensional CRM layer, with provider-diverse ground truth and judging described as part of the workflow. On the authors’ warehouse, Claude Fable 5.1 ranked first for correctness at 76.6%, while the full run cost $1 with DeepSeek V4 Flash versus $52 with Fable 5.1, an 11-percentage-point correctness difference. The harness is intended to identify the best production fit for a specific warehouse, not a universal winner.
Context
Traditional text-to-SQL benchmarks assume that the schema is provided, the model gets one attempt, and answers are judged against a single gold query. Production analytical agents instead discover schemas, run multiple queries, and work across complex warehouse structures, so model suitability depends on the organization’s questions and data.
Approach / What changed
The authors built a reusable harness from 201 questions logged by DWAINE, applying curation, deterministic anonymization, synthetic warehouse reconstruction, and planted entity references. Evaluations replay against an identical warehouse state, recover ground truth with a provider-diverse committee, and use a judge panel that does not grade its own model family. The harness can be adapted to another warehouse and traffic set.
Takeaways
The benchmark’s synthetic warehouse preserves real schema complexity: 18 tables and 865 columns across curated marts and a dimensional CRM layer, with relationships and scales kept internally consistent while names and figures are anonymized.
Correctness rankings were workload-dependent: Claude Fable 5.1 scored 76.6% on the authors’ 201 questions, while frontier-level models occupied all top 10 positions and five of those models were open-weight.
The complete benchmark run cost $1 with DeepSeek V4 Flash and $52 with Fable 5.1, with the cheaper option sacrificing 11 percentage points of correctness.
Source and freshness
- Source
- clickhouse.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.