Loading post…
ClickhouseProblem & solution
How LinkedIn extended ClickHouse from distributed tracing to metric discovery and analytics
Summary
LinkedIn’s observability team first deployed ClickHouse for distributed tracing, where even at 1% head sampling the system generates 0.8 trillion spans and 200 TB of uncompressed data per day across three data centers. They then extended ClickHouse to consolidate a legacy metric metadata system built on RRDtool semantics, which had grown into three separate services that had reached their scaling limits. The new single-shard ClickHouse index serves over 150,000 queries per minute with an average latency of 68 milliseconds across 13 billion-plus metrics with 30% daily turnover. Migrating off the legacy stack reduced memory use to roughly one-fifth and compute to about two-thirds, with enough headroom to double the metric volume. By exposing individual dimensions as columns, the index enables engineers to replace regex-heavy queries with efficient native ones, supporting the larger migration away from RRD semantics.
Context
LinkedIn’s observability team manages the full observability stack for over a billion members and thousands of services. They first adopted ClickHouse for distributed tracing, handling 0.8 trillion spans/day. They then faced a legacy metric metadata system based on RRDtool semantics, with 13B+ metrics and 30% daily turnover, served by three separate services (custom in-memory search index, Elasticsearch, custom KV store) that had reached scaling limits and could not answer all analytical queries.
Approach / What changed
Built a single-shard ClickHouse index with a daily rebuild pipeline that builds a fresh table from the offline time-series database, catches up with change-capture data, and uses an alias swap for zero-downtime traffic cutover. Projections handle heavy aggregations like service-to-metric counts. The schema exposes individual dimensions as columns, replacing regex-over-RRD-string queries with column filters.
Takeaways
The consolidated ClickHouse index serves 150k+ queries/min at 68ms average latency across 13B+ metrics with 30% daily turnover.
Migrating from the three-system legacy stack reduced memory use to roughly one-fifth and compute to about two-thirds, with headroom to double metric volume.
Exposing individual dimensions as columns enables engineers to rewrite regex-heavy queries into efficient native column filters, potentially allowing the cluster to scale down over time.
FAQ
How does the daily rebuild pipeline work?
The pipeline builds a fresh daily table from the offline time-series database, catches it up with change-capture data, then uses an alias to swap the table pointer and cut traffic over before deleting the old table.
What were the limitations of the legacy metric metadata system?
The legacy system consisted of three separate services (custom in-memory search index, Elasticsearch, custom KV store) that had reached scaling limits, couldn't handle larger spanning aggregations, and required serializing large result sets in memory.
How does the new index handle heavy aggregations?
The index uses ClickHouse projections to handle heavy aggregations such as service-to-metric counts, service-to-unique-RRD counts, and datacenter-to-host listings, which the old systems could not serve.
Source and freshness
- Source
- clickhouse.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.