Introducing CostBench: an open benchmark for data warehouse cost-performance

Summary

Traditional data warehouse benchmarks measure query execution speed without accounting for the varying compute costs across platforms. CostBench introduces an open benchmarking framework designed to evaluate data warehouses on a performance-per-dollar metric across both read and write workloads. Its initial release evaluates ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Redshift using 43 production-derived analytical queries on an anonymized dataset combined with each vendor's compute billing model. In these read-side tests, ClickHouse Cloud maintained high speed at lower costs as data scaled, outperforming the nearest competitor by 23 times in cost-performance. The benchmark repository openly publishes its scripts, configurations, pricing assumptions, and raw JSON results to ensure all comparative metrics remain fully inspectable and reproducible.

Takeaways

1. 1

   CostBench standardizes comparisons across disparate vendor billing units such as credits, DBUs, slot-seconds, compute units, and RPUs to measure actual compute expenditure.

2. 2

   In read-side tests across 43 analytical queries, ClickHouse Cloud stayed in the fast and low-cost zone as data scaled, while the nearest competitor scored 23x worse in cost-performance.

3. 3

   The benchmark repository is open source, publishing all query workloads, scripts, configurations, pricing assumptions, methodology, and raw JSON results for public reproducibility.