ClickhouseBenchmark
ClickHouse added five models to its Agentic Analytics Benchmark—GPT-6 Astra, Luna, Sol, Claude Opus 5.5, and Claude Sonnet 5.5—using the same 201 analytics questions, synthetic warehouse, and judge panel. All five scored between 68.7% and 69.7%, statistically tied and mid-board, while Claude Fable 5.1 retains the top spot at 76.6%. Claude Sonnet 5.5 delivered the largest improvement, raising accuracy from 62.2% to 68.7% and eliminating budget-limit failures while reducing median turns from 9.1 to 3.5 and cost to about five cents per question. The gain concentrated on the 83 questions requiring the dimensional CRM layer, where Sonnet 5.5 scored 73.5% versus 62.7% for Sonnet 5, and the harness was updated with engine pinning and credential preflight. Contamination probes found no memorization of anonymized entity names, and the board now ranks 34 models with a frozen question set.
The Agentic Analytics Benchmark evaluates how well LLMs answer analytics questions by writing SQL in an agentic loop over a synthetic ClickHouse data warehouse. This round adds five new models to the frozen board of 34 models, with harness improvements to ensure reproducibility and robustness.
The benchmark uses an agentic loop where each model writes SQL, reads results, and decides when to answer, with a 60-turn budget. Answers are judged by a panel of three models from different providers, with no model grading its own. The harness now includes engine and timezone pinning, credential preflight checks, configurable numeric tolerance, a linker fix for wide result sets, and a Rasch model for judge uncertainty.
Claude Sonnet 5.5 improved accuracy from 62.2% to 68.7% and eliminated budget-limit failures, reducing median turns from 9.1 to 3.5 and cost to about five cents per question.
The GPT-6 family (Astra, Luna, Sol) scored between 68.7% and 69.7% at default reasoning effort, not surpassing GPT-5.5, and all three are within one point of each other.
The benchmark harness now pins the query engine and session timezone, preventing silent drift that previously flipped a ground truth answer, and includes a contamination probe that found no memorization in the new models.
Sonnet 5.5 reduced median turns per question from 9.1 to 3.5, eliminated budget-limit failures (zero vs. 16), and lowered cost from about seven cents to about five cents per question.
No, all three GPT-6 variants (Astra, Luna, Sol) scored below GPT-5.5 at default reasoning effort, landing in the 68.7-69.7% band.
The harness now pins the query engine release and session timezone, refuses runs on different versions, and includes credential preflight checks to prevent dead judge credentials from degrading scoring.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
An adversarial prefix is a challenge prepended to a prompt to see whether a model holds its original decision. This diagram maps the layers that turn that pressure test into a measurable, reviewable signal.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
