ClickHouse vs Prometheus for High Cardinality, Part 1: Understanding the Problem

Summary

Observability systems frequently encounter high cardinality when metrics contain numerous unique label combinations across multidimensional cloud infrastructure. In Prometheus, every unique label combination instantiates an independent time series that incurs metadata overhead, requires in-memory memSeries registration, and continually expands inverted index posting lists. The ingestion engine stores samples in two-hour Head blocks using compressed 120-sample chunks with XOR-based encodings, but rapid series creation and elevated scrape frequencies multiply chunk allocations and memory consumption. When executing read operations, broad aggregations and regular expressions require intersecting expansive posting lists and decoding entire chunks sequentially because the engine cannot push value predicates directly into compressed storage blocks. Consequently, managing high-cardinality workloads in series-oriented architectures forces operational trade-offs between system stability and granular visibility into ephemeral components.

Takeaways

1. 1

   Each unique label combination in Prometheus creates an independent memSeries structure with approximately 200 bytes of base metadata plus chunk and label overhead.

2. 2

   Inverted indexes store posting lists referencing all series for each label pair, expanding significantly in memory and write overhead as cardinality increases.

3. 3

   Prometheus cannot push arbitrary value predicates down into compressed chunk storage, requiring full chunk decoding for every selected series during read operations.