Shopify powers observability for global-scale commerce with ClickHouse

Summary

Shopify rebuilt its observability infrastructure after metrics, logs, and traces became fragmented across vendors, with unpredictable costs and limited cross-signal analysis. It created Observe, a unified platform that runs metrics, logs, traces, profiles, and exceptions on self-hosted ClickHouse, treating them as structured, time-ordered, high-dimensional events and serving roughly 50 million events per second normally and 100 million at BFCM peak, or about 110 GB/s. Kafka buffers telemetry for large synchronous durable writes, while materialized views support flexible schemas, autocomplete, typed hot keys, and identifier-based correlation; the latter reportedly makes targeted searches 10x faster. Shopify reports a 16x query improvement out of the box, exceeding 30x at peak performance, with data queryable in under a minute, and says it would likely choose ClickHouse Cloud if starting again because managed operations could reduce the self-hosting burden.

Takeaways

1. 1

   Shopify reports a 16x query improvement from ClickHouse out of the box, with performance exceeding 30x under peak conditions.

2. 2

   Kafka buffering lets Shopify combine large ClickHouse writes with durability: batches are acknowledged only after ClickHouse has stored them.

3. 3

   Materialized views make flexible-schema telemetry faster to query and provide a correlation lookup that reportedly turns fleet-wide searches into targeted scans, delivering a 10x speedup.