---
title: "The Agentic Analytics Benchmark, round two: GPT-6, Claude Opus 5.5, Claude Sonnet 5.5"
description: "ClickHouse added five models to its Agentic Analytics Benchmark—GPT-6 Astra, Luna, Sol, Claude Opus 5.5, and Claude Sonnet 5.5—using the same 201 analytics questions, synthetic warehouse, and judge panel. All five scored between 68.7% and 69.7%, statistically tied and mid-board, while Claude Fable 5.1 retains the top spot at 76.6%. Claude Sonnet 5.5 delivered the largest improvement, raising accuracy from 62.2% to 68.7% and eliminating budget-limit failures while reducing median turns from 9.1 to 3.5 and cost to about five cents per question. The gain concentrated on the 83 questions requiring the dimensional CRM layer, where Sonnet 5.5 scored 73.5% versus 62.7% for Sonnet 5, and the harness was updated with engine pinning and credential preflight. Contamination probes found no memorization of anonymized entity names, and the board now ranks 34 models with a frozen question set."
---

# The Agentic Analytics Benchmark, round two: GPT-6, Claude Opus 5.5, Claude Sonnet 5.5

[Clickhouse](https://yomu.fyi/company/clickhouse) · Oct 20, 2026

**Type:** Benchmark

## Summary

ClickHouse added five models to its Agentic Analytics Benchmark—GPT-6 Astra, Luna, Sol, Claude Opus 5.5, and Claude Sonnet 5.5—using the same 201 analytics questions, synthetic warehouse, and judge panel. All five scored between 68.7% and 69.7%, statistically tied and mid-board, while Claude Fable 5.1 retains the top spot at 76.6%. Claude Sonnet 5.5 delivered the largest improvement, raising accuracy from 62.2% to 68.7% and eliminating budget-limit failures while reducing median turns from 9.1 to 3.5 and cost to about five cents per question. The gain concentrated on the 83 questions requiring the dimensional CRM layer, where Sonnet 5.5 scored 73.5% versus 62.7% for Sonnet 5, and the harness was updated with engine pinning and credential preflight. Contamination probes found no memorization of anonymized entity names, and the board now ranks 34 models with a frozen question set.

## Context

The Agentic Analytics Benchmark evaluates how well LLMs answer analytics questions by writing SQL in an agentic loop over a synthetic ClickHouse data warehouse. This round adds five new models to the frozen board of 34 models, with harness improvements to ensure reproducibility and robustness.

## Approach / What changed

The benchmark uses an agentic loop where each model writes SQL, reads results, and decides when to answer, with a 60-turn budget. Answers are judged by a panel of three models from different providers, with no model grading its own. The harness now includes engine and timezone pinning, credential preflight checks, configurable numeric tolerance, a linker fix for wide result sets, and a Rasch model for judge uncertainty.

## Takeaways

- Claude Sonnet 5.5 improved accuracy from 62.2% to 68.7% and eliminated budget-limit failures, reducing median turns from 9.1 to 3.5 and cost to about five cents per question.
- The GPT-6 family (Astra, Luna, Sol) scored between 68.7% and 69.7% at default reasoning effort, not surpassing GPT-5.5, and all three are within one point of each other.
- The benchmark harness now pins the query engine and session timezone, preventing silent drift that previously flipped a ground truth answer, and includes a contamination probe that found no memorization in the new models.

## FAQ

### How does Claude Sonnet 5.5 compare to Sonnet 5 in efficiency?

Sonnet 5.5 reduced median turns per question from 9.1 to 3.5, eliminated budget-limit failures (zero vs. 16), and lowered cost from about seven cents to about five cents per question.

### Did any GPT-6 variant outperform GPT-5.5?

No, all three GPT-6 variants (Astra, Luna, Sol) scored below GPT-5.5 at default reasoning effort, landing in the 68.7-69.7% band.

### What harness changes were made to ensure reproducibility?

The harness now pins the query engine release and session timezone, refuses runs on different versions, and includes credential preflight checks to prevent dead judge credentials from degrading scoring.

**Tags:** [Agents](https://yomu.fyi/topic/agents), [Benchmark](https://yomu.fyi/topic/benchmark), [ClickHouse](https://yomu.fyi/topic/clickhouse), [LLMs](https://yomu.fyi/topic/llm), [SQL](https://yomu.fyi/topic/sql)

- Source: [Clickhouse](https://clickhouse.com/blog/agentic-analytics-benchmark-round-two)
- Source URL: https://clickhouse.com/blog/agentic-analytics-benchmark-round-two
- Ingested by Yomu: 2026-10-20T09:00:11.673Z

[Read original post](https://clickhouse.com/blog/agentic-analytics-benchmark-round-two)