Benchmarks and Obscurantism: A “red” line that should not be crossed

Summary

Melvyn Peignon examines a Databricks keynote benchmark in which ClickHouse was shown “crashing” under workloads associated with Reyden, Databricks’ gated low-latency compute product. The post argues that the presentation omitted hardware, cost, configuration, cache settings, deployment details, and sufficient methodology for independent validation. Using the disclosed TPC-H SF1 dataset and Q6 query, ClickHouse was tested on a 30 vCPU, 120 GB node without tuning; it did not crash and sustained about 420 QPS at sub-second P90 latency. The post reports that reaching 15,000 QPS required straightforward sizing to roughly 30–40 untuned nodes, while further gains could come from settings such as thread counts, query cache, and projections. It concludes that benchmarks should be open, reproducible, and detailed enough to inspect rather than accepted as keynote claims.

Takeaways

1. 1

   A 30 vCPU ClickHouse node sustained about 420 QPS at sub-second P90 latency for the TPC-H SF1 Q6 workload without tuning, and it did not crash.

2. 2

   The post reports that 15,000 QPS could be sustained through straightforward sizing at roughly 30–40 untuned ClickHouse nodes.

3. 3

   The Databricks benchmark did not disclose enough hardware, cost, configuration, cache, or methodology information to make its result independently reproducible or actionable.