How the D. E. Shaw group powers high-cardinality observability at scale with ClickHouse

Summary

Inside D. E. Shaw group, a global investment and technology development firm, millions of compute workloads run monthly across its high-performance computing infrastructure. To support capacity planning and resource allocation, it collects task-level metrics for requested and utilized compute, producing millions of distinct time series that its previous observability platform could not accommodate. Engineers benchmarked databases on identical hardware and datasets; ClickHouse ingested approximately 3.5 million samples per second on one node versus roughly 480,000 for a competing solution, while similar queries completed in seconds instead of timing out. They retained InfluxDB line protocol, batched messages into ClickHouse, and built disk-based tooling to transform and stream historical data; the resulting four-server NVMe cluster stores about 68 TB compressed and sustains 530k records per second on average. Materialized views support capacity-planning dashboards, while OpenTelemetry tracing extends the platform to other observability workloads, with SQL migration and data modeling requiring engineering effort.

Takeaways

1. 1

   In the benchmark, a single ClickHouse node ingested approximately 3.5 million samples per second, compared with roughly 480,000 samples per second for a competing solution.

2. 2

   Historical backfill through the previous platform’s query API was impractical, so engineers built tooling to read data files directly, transform them, and stream the results into ClickHouse.

3. 3

   Unique task IDs reduce compression efficiency for some fields, but preserve workload-level visibility; the deployment stores about 68 TB of compressed metrics data and achieves up to 5x compression for those fields.