Measuring real-time performance per dollar under continuous load: CostBench’s first end-to-end results

Summary

CostBench measures real-time performance per dollar by evaluating the full path from continuously arriving rows to query-ready data and query results. Its end-to-end workload streams more than 100 billion stock-market quotes at a target rate of 1 million rows per second while aggregate and drill-down queries run, comparing ClickHouse Cloud, Snowflake, BigQuery, and Redshift Serverless through their recommended push-based ingestion paths. The benchmark accounts for fresh-data-path work—columnar ingestion, pruning-friendly layouts, and current pre-aggregations—alongside query-serving cost and accumulated runtime. ClickHouse Cloud recorded the lowest fresh-data-path cost, query-serving cost, and accumulated runtime, while the alternatives showed 412–1,996× worse end-to-end performance per dollar; the source says it was the only tested system keeping both data representations current during ingestion. The post introduces follow-up architecture and billing analyses.

Takeaways

1. 1

   Query-ready analytical data uses columnar storage, pruning-friendly physical layouts, and current pre-aggregations to reduce data read and repeated query-time work.

2. 2

   The benchmark’s end-to-end comparison expands the isolated query-side gap of 32–101× worse cost-performance to 412–1,996× when continuous ingestion and query-readiness work are included.

3. 3

   ClickHouse Cloud had the lowest fresh-data-path cost, query-serving cost, and accumulated query runtime, and was the only tested system reported to keep both event-level and pre-aggregated data current during ingestion.