Do you still need Elasticsearch for log analytics? ClickHouse says no.

Summary

ClickHouse introduced a redesigned full-text search powered by inverted indexes that feeds matching documents directly into its vectorized analytical engine for observability and log analytics workloads. To evaluate this capability, the open-source versions of ClickHouse and Elasticsearch were benchmarked on single-node AWS EC2 instances across synthetic OpenTelemetry log datasets scaling up to 50 billion rows. The benchmark evaluated nine representative query patterns combining text search with filtering, grouping, counting, and time-series aggregations under both cold and hot cache conditions. ClickHouse achieved a roughly fivefold smaller storage footprint on disk while running cold queries four to six times faster and hot queries approximately 1.7 to 2.6 times faster than Elasticsearch. Furthermore, ClickHouse ingested the 50-billion-row dataset in under four hours out of the box, whereas the tuned Elasticsearch setup required approximately five days.

Context

Observability and log analytics workloads typically require finding text patterns followed immediately by filtering, counting, grouping by attributes, or aggregating over time intervals. Historically, organizations maintained Elasticsearch for text search and ClickHouse for analytics, but maintaining separate engines complicates architectures when log events serve primarily as analytical data that contains text.

Approach / What changed

The benchmark compared open-source ClickHouse v26.3 and Elasticsearch v9.3.2 on identical single-node AWS m6i.8xlarge EC2 instances using 1 billion, 10 billion, and 50 billion row OpenTelemetry log datasets stored in Parquet format. ClickHouse used a MergeTree table sorted by ServiceName and Timestamp with a full-text inverted index on the Body column, while Elasticsearch used matching index sort settings and standard analyzers with best_compression. Query performance was evaluated across nine query patterns covering retrieval, counting, grouping, and time-bucket aggregations under cold and hot cache conditions.

Takeaways

• ClickHouse executed full-text analytical log workloads 2x to 6x faster overall than Elasticsearch across scales up to 50 billion rows, with cold queries running 4x to 6x faster.
• ClickHouse compressed the OpenTelemetry log dataset to roughly one-fifth the disk storage footprint required by Elasticsearch configured with best_compression.
• Loading 50 billion log rows from Parquet into single-node ClickHouse took under 4 hours natively, compared to roughly 5 days for Elasticsearch via NDJSON streaming and the Bulk API.

Related reading