How ClickStack makes ClickHouse faster for observability

Summary

ClickHouse delivers high performance for telemetry data, but unoptimized queries on high-cardinality observability workloads can bypass pruning, inflate intermediate state, and exhaust cluster resources. ClickStack addresses this challenge by embedding engine-specific optimization best practices directly into how queries are generated, structured, and executed. For search workflows, the platform executes progressive queries across bounded time windows using optimize_read_in_order to return initial rows rapidly without scanning entire historical ranges. ClickStack also segments large chart aggregations into parallel bucket-aligned queries, automatically redirects map attribute filters to pre-materialized columns, and extends lazy materialization thresholds to minimize runtime input-output overhead. These query formulation strategies are being packaged into specialized analytical endpoints to power notebooks and external tools without requiring manual SQL tuning or exposing raw database interfaces.

Context

Observability workloads on ClickHouse often involve high cardinality and semi-structured telemetry data. Naive or poorly shaped SQL queries can bypass partition pruning, trigger expensive runtime map extractions, inflate intermediate state, and waste cluster CPU and memory during large-scale scans and aggregations.

Approach / What changed

ClickStack integrates query construction directly with ClickHouse internals to automatically optimize execution patterns. It implements progressive time-window searching aligned with table ordering keys, executes granularity-aligned parallel chunked queries for charts, transparently rewrites map filters to ingest-time materialized columns, and raises the query_plan_max_limit_for_lazy_materialization threshold to defer column loading across larger result sets.

Takeaways

• Progressive searching across expanding time windows combined with optimize_read_in_order delivers fast initial search results without requiring full table scans or global sorts over large date ranges.
• Chunking long-range chart queries into granularity-aligned time windows enables parallel execution and progressive client rendering while preventing cluster resource saturation and timeouts.
• Extracting semi-structured Map attributes into materialized columns at ingest time allows ClickStack to transparently rewrite filters to physical columns, improving compression and pruning.

Related reading