A Quadrillion Rows across three Clouds: scaling LogHouse

Summary

LogHouse now stores 431 PiB of uncompressed data across 1.59 quadrillion rows, spans more than 30 regions across three cloud providers, and serves typical queries in a few hundred milliseconds. The platform grew 23x from 19 PiB in two years, while peak ingestion reached 80 GiB/s and 190 million rows per second. Its write architecture uses local geosharding, multiple cells in busy regions, and ClickHouse Async Inserts to combine small incoming writes into larger parts without overwhelming merges. On the read side, a single Distributed table hides the cell topology, while unavailable shards remain visible by default rather than producing silently incomplete results. Current work includes reducing index and metadata memory, improving async-insert durability, collecting system data from object storage, and expanding OpenTelemetry traces and metrics.

Takeaways

1. 1

   Geosharding keeps writes local, avoids cross-region egress costs, and lets regions scale independently; additional cells also limit the blast radius of operational failures.

2. 2

   Async Inserts coalesced roughly 24 otel_logs client inserts into each part and reduced part creation from 290,284 to 12,082 per hour during the stated peak-hour comparison.

3. 3

   The read layer favors complete results by leaving skip_unavailable_shards disabled by default, accepting query failure when a cell is unavailable instead of silently returning incomplete data.