---
title: "The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents"
description: "The post presents data-agent-mnist, an open harness for measuring analytical agents on correctness, cost, and speed against a warehouse-shaped workload rather than a single text-to-SQL query. The authors extracted 201 production analytics questions from DWAINE, curated and anonymized them, reconstructed a deterministic synthetic warehouse with 18 tables and 865 columns, and evaluated 28 proprietary and open-weight models. The benchmark preserves schema discovery, multi-turn querying, planning, and multi-hop joins across curated marts and a dimensional CRM layer, with provider-diverse ground truth and judging described as part of the workflow. On the authors’ warehouse, Claude Fable 5.1 ranked first for correctness at 76.6%, while the full run cost $1 with DeepSeek V4 Flash versus $52 with Fable 5.1, an 11-percentage-point correctness difference. The harness is intended to identify the best production fit for a specific warehouse, not a universal winner."
---

# The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents

[Clickhouse](https://yomu.fyi/company/clickhouse) · Sep 1, 2026

**Type:** Benchmark

## Summary

The post presents data-agent-mnist, an open harness for measuring analytical agents on correctness, cost, and speed against a warehouse-shaped workload rather than a single text-to-SQL query. The authors extracted 201 production analytics questions from DWAINE, curated and anonymized them, reconstructed a deterministic synthetic warehouse with 18 tables and 865 columns, and evaluated 28 proprietary and open-weight models. The benchmark preserves schema discovery, multi-turn querying, planning, and multi-hop joins across curated marts and a dimensional CRM layer, with provider-diverse ground truth and judging described as part of the workflow. On the authors’ warehouse, Claude Fable 5.1 ranked first for correctness at 76.6%, while the full run cost $1 with DeepSeek V4 Flash versus $52 with Fable 5.1, an 11-percentage-point correctness difference. The harness is intended to identify the best production fit for a specific warehouse, not a universal winner.

## Context

Traditional text-to-SQL benchmarks assume that the schema is provided, the model gets one attempt, and answers are judged against a single gold query. Production analytical agents instead discover schemas, run multiple queries, and work across complex warehouse structures, so model suitability depends on the organization’s questions and data.

## Approach / What changed

The authors built a reusable harness from 201 questions logged by DWAINE, applying curation, deterministic anonymization, synthetic warehouse reconstruction, and planted entity references. Evaluations replay against an identical warehouse state, recover ground truth with a provider-diverse committee, and use a judge panel that does not grade its own model family. The harness can be adapted to another warehouse and traffic set.

## Takeaways

- The benchmark’s synthetic warehouse preserves real schema complexity: 18 tables and 865 columns across curated marts and a dimensional CRM layer, with relationships and scales kept internally consistent while names and figures are anonymized.
- Correctness rankings were workload-dependent: Claude Fable 5.1 scored 76.6% on the authors’ 201 questions, while frontier-level models occupied all top 10 positions and five of those models were open-weight.
- The complete benchmark run cost $1 with DeepSeek V4 Flash and $52 with Fable 5.1, with the cheaper option sacrificing 11 percentage points of correctness.

**Tags:** [Data Warehousing](https://yomu.fyi/topic/data-warehousing), [LLMs](https://yomu.fyi/topic/llm), [Open Source](https://yomu.fyi/topic/open-source), [Testing](https://yomu.fyi/topic/testing)

- Source: [Clickhouse](https://clickhouse.com/blog/agentic-analytics-benchmark-data-agent-mnist)
- Source URL: https://clickhouse.com/blog/agentic-analytics-benchmark-data-agent-mnist
- Ingested by Yomu: 2026-09-01T20:00:51.511Z

[Read original post](https://clickhouse.com/blog/agentic-analytics-benchmark-data-agent-mnist)