Loading post…
ClickhouseProblem & solution
How we made ClickStack 5x faster for ClickHouse observability
Summary
ClickStack’s inherited OpenTelemetry Collector schema was struggling with slow searches, Top N queries, and aggregations that could time out on large observability workloads. The team prioritized common log and trace access patterns, including exact-attribute searches, body-text queries, histograms, individual log lookups, and distinct-key or value introspection. They used fixed production-like workloads and iterative testing with ClickCannon, Kubernetes-managed environments, ephemeral ClickHouse Cloud instances, and controlled schema, query, and version comparisons. The resulting redesign combined primary-key changes, text indexes, query rewrites, alias columns, materialized views, and newer ClickHouse features, reducing representative query latency by more than 5× while balancing ingestion, storage, and operational efficiency.
Context
The default schema came from the OpenTelemetry Collector’s ClickHouse exporter and had not been fully adapted to newer ClickHouse capabilities or rigorously benchmarked across workloads and versions. At large customer scales, slow searches, Top N queries, and aggregations increasingly affected the interactive observability experience.
Approach / What changed
The team identified the query patterns most important to ClickStack users, fixed a representative workload configuration, and iteratively evaluated schema changes with ClickCannon and a Kubernetes-based test system managing ephemeral ClickHouse Cloud instances. The redesign used primary-key changes, text indexes, query rewrites, alias columns, materialized views, and newer ClickHouse features, validating both query performance and infrastructure impact.
Takeaways
The prioritized workload included exact-attribute and body-text searches, histogram aggregations, individual log lookups, and distinct-key or value introspection.
Tests held schema, query workload, and ClickHouse version as the variables while using 640 MiB/s throughput, three replicas with 24 CPU cores each, and timestamp-ordered representative data.
The optimized schemas reduced representative query latency by more than 5× while maintaining a balanced profile for ingestion, storage, and operational efficiency.
Source and freshness
- Source
- clickhouse.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.