Introducing CostBench: an open benchmark for data warehouse cost-performance

Tom Schreiber, Lionel PalacinMay 27, 2026

Source: Clickhouse
Published: May 27, 2026
Added to Yomu: Aug 28, 2026

Summary

Traditional data warehouse benchmarks evaluate raw execution speed without accounting for vendor-specific compute pricing. CostBench addresses this operational limitation by introducing an open framework designed to measure performance-per-dollar across cloud platforms. The initial benchmark release assesses read-side query efficiency across ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Amazon Redshift using 43 production-derived analytical queries executed against an anonymized dataset. To normalize comparisons across varied billing metrics like credits, DBUs, slot-seconds, compute units, and RPUs, CostBench maps runtime execution directly to actual compute billing models. In this evaluation, ClickHouse Cloud remains in the low-cost, high-performance quadrant as data scales, outperforming its nearest competitor by 23x in cost-performance.

Context

Comparing data warehouse performance is challenging because platforms expose costs through disparate billing metrics such as credits, DBUs, slot-seconds, compute units, and RPUs. Furthermore, agentic analytics and continuous data ingestion place severe cost and latency pressures on both write-side preparation and read-side query workloads.

Approach / What changed

CostBench establishes an open benchmark that normalizes performance-per-dollar across ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Redshift. It evaluates systems using 43 production-derived analytical queries over an anonymized dataset, mapping execution times to vendor billing models while open-sourcing all scripts, configurations, and raw results.

Takeaways

Speed comparisons alone are incomplete for cloud data warehouses because distinct billing units mask the actual compute costs required to complete workloads.
CostBench evaluates cost-performance across two dimensions: read-side query execution and write-side ingestion efficiency.
In the read-side evaluation across five cloud data platforms, ClickHouse Cloud outpaced its nearest competitor by 23x in cost-performance as data scaled.

Open Source Performance Scalability

Read original post

Introducing CostBench: an open benchmark for data warehouse cost-performance

Tom Schreiber, Lionel PalacinMay 27, 2026

Source: Clickhouse
Published: May 27, 2026
Added to Yomu: Aug 28, 2026

Summary

Context

Approach / What changed

Takeaways

Speed comparisons alone are incomplete for cloud data warehouses because distinct billing units mask the actual compute costs required to complete workloads.
CostBench evaluates cost-performance across two dimensions: read-side query execution and write-side ingestion efficiency.
In the read-side evaluation across five cloud data platforms, ClickHouse Cloud outpaced its nearest competitor by 23x in cost-performance as data scaled.

Open Source Performance Scalability

Read original post

Related reading

Clickhouse · May 6, 2026
Agentic analytics starts with query-ready data: the write-side cost of Snowflake vs. ClickHouse
Agentic analytics workflows introduce bursts of concurrent, low-latency analytical queries over continuously ingested data. Analytical query efficiency relies on columnar storage and chunk-level pruning, which require contiguous, sorted data to avoid full-table scans. Rather than evaluating warehouses solely at query runtime, this benchmark compares the write-side cost of maintaining query-ready data in ClickHouse and Snowflake during continuous ingestion at approximately one million rows per second. Snowflake relies on post-write clustering to organize data, whereas ClickHouse constructs sorted layouts on the write path and refines them via background merges. Across 100B, 200B, and 300B row checkpoints on the ClickBench dataset, ClickHouse achieved a query-ready layout at 22 times lower cost and delivered 28 times better write-side cost-performance.
Tom Schreiber, Lionel Palacin
Clickhouse · Apr 1, 2026
Agentic coding at ClickHouse
Industry perspectives on AI coding agents remain divided between dismissal and excessive automation claims. ClickHouse integrated agentic coding across internal workflows, progressing from simple boilerplate scripts to active backend C++ development. Engineers primarily use command-line agents such as Claude Code with Claude Opus 4.5 and Codex CLI alongside observability platforms like Langfuse and Librechat. While agents effectively investigate continuous integration logs, generate internal dashboards, and assist with small C++ tasks, they lack architectural competence and require rigorous code review. ClickHouse treats agents as pair programmers for contained tasks, retaining human ownership over product strategy, attention to detail, and core code quality.
Alexey Milovidov
Clickhouse · Apr 2, 2026
PostgresBench: A Reproducible Benchmark for Postgres Services
PostgresBench is a public, reproducible benchmarking suite created to compare transactional performance across managed PostgreSQL providers. The benchmark uses the standard pgbench tool with a TPC-B-like write-intensive workload executing across 256 clients and 16 threads for ten-minute durations. Evaluators tested five services—including Postgres managed by ClickHouse, AWS Aurora, AWS RDS, Neon, and Crunchy Bridge—under 100 GB and 500 GB scale factors using default configurations in the us-east-2 region. Across tested configurations, Postgres managed by ClickHouse delivered higher transactions per second and lower P99 latencies due to physically co-located NVMe primary storage. These results demonstrate that remote network storage layers, such as EBS volumes, introduce significant latency overhead during sustained write and fsync operations.
Lionel Palacin
Clickhouse · Mar 17, 2026
AI is redrawing the database market
Emerging artificial intelligence workloads are fundamentally transforming data platform requirements across real-time analytics, data warehousing, and observability. Conversational analytics and autonomous agents replace periodic batch queries with rapid bursts of concurrent, low-latency requests, exposing the architectural limits of traditional cloud data warehouses and standalone transactional databases. To resolve these constraints, architectures are converging on engine-level integrations between Postgres for transactions and ClickHouse for analytical scale, while unifying operational observability and business event storage. ClickHouse also incorporates turnkey agent interfaces through its acquisition of LibreChat alongside dedicated execution tracing from tools like Langfuse. These combined capabilities ensure organizations can reliably support interactive agentic workflows without incurring severe latency penalties or compounding infrastructure costs.
Tanya Bragin