---
title: "How LinkedIn extended ClickHouse from distributed tracing to metric discovery and analytics"
description: "LinkedIn’s observability team first deployed ClickHouse for distributed tracing, where even at 1% head sampling the system generates 0.8 trillion spans and 200 TB of uncompressed data per day across three data centers. They then extended ClickHouse to consolidate a legacy metric metadata system built on RRDtool semantics, which had grown into three separate services that had reached their scaling limits. The new single-shard ClickHouse index serves over 150,000 queries per minute with an average latency of 68 milliseconds across 13 billion-plus metrics with 30% daily turnover. Migrating off the legacy stack reduced memory use to roughly one-fifth and compute to about two-thirds, with enough headroom to double the metric volume. By exposing individual dimensions as columns, the index enables engineers to replace regex-heavy queries with efficient native ones, supporting the larger migration away from RRD semantics."
---

# How LinkedIn extended ClickHouse from distributed tracing to metric discovery and analytics

[Clickhouse](https://yomu.fyi/company/clickhouse) · Oct 1, 2026

**Type:** Problem & solution

## Summary

LinkedIn’s observability team first deployed ClickHouse for distributed tracing, where even at 1% head sampling the system generates 0.8 trillion spans and 200 TB of uncompressed data per day across three data centers. They then extended ClickHouse to consolidate a legacy metric metadata system built on RRDtool semantics, which had grown into three separate services that had reached their scaling limits. The new single-shard ClickHouse index serves over 150,000 queries per minute with an average latency of 68 milliseconds across 13 billion-plus metrics with 30% daily turnover. Migrating off the legacy stack reduced memory use to roughly one-fifth and compute to about two-thirds, with enough headroom to double the metric volume. By exposing individual dimensions as columns, the index enables engineers to replace regex-heavy queries with efficient native ones, supporting the larger migration away from RRD semantics.

## Context

LinkedIn’s observability team manages the full observability stack for over a billion members and thousands of services. They first adopted ClickHouse for distributed tracing, handling 0.8 trillion spans/day. They then faced a legacy metric metadata system based on RRDtool semantics, with 13B+ metrics and 30% daily turnover, served by three separate services (custom in-memory search index, Elasticsearch, custom KV store) that had reached scaling limits and could not answer all analytical queries.

## Approach / What changed

Built a single-shard ClickHouse index with a daily rebuild pipeline that builds a fresh table from the offline time-series database, catches up with change-capture data, and uses an alias swap for zero-downtime traffic cutover. Projections handle heavy aggregations like service-to-metric counts. The schema exposes individual dimensions as columns, replacing regex-over-RRD-string queries with column filters.

## Takeaways

- The consolidated ClickHouse index serves 150k+ queries/min at 68ms average latency across 13B+ metrics with 30% daily turnover.
- Migrating from the three-system legacy stack reduced memory use to roughly one-fifth and compute to about two-thirds, with headroom to double metric volume.
- Exposing individual dimensions as columns enables engineers to rewrite regex-heavy queries into efficient native column filters, potentially allowing the cluster to scale down over time.

## FAQ

### How does the daily rebuild pipeline work?

The pipeline builds a fresh daily table from the offline time-series database, catches it up with change-capture data, then uses an alias to swap the table pointer and cut traffic over before deleting the old table.

### What were the limitations of the legacy metric metadata system?

The legacy system consisted of three separate services (custom in-memory search index, Elasticsearch, custom KV store) that had reached scaling limits, couldn't handle larger spanning aggregations, and required serializing large result sets in memory.

### How does the new index handle heavy aggregations?

The index uses ClickHouse projections to handle heavy aggregations such as service-to-metric counts, service-to-unique-RRD counts, and datacenter-to-host listings, which the old systems could not serve.

**Tags:** [ClickHouse](https://yomu.fyi/topic/clickhouse), [Observability](https://yomu.fyi/topic/observability)

- Source: [Clickhouse](https://clickhouse.com/blog/linkedin-observability-at-scale)
- Source URL: https://clickhouse.com/blog/linkedin-observability-at-scale
- Ingested by Yomu: 2026-10-01T12:00:09.288Z

[Read original post](https://clickhouse.com/blog/linkedin-observability-at-scale)