Loading…
ClickHouse is data lake ready
ClickhouseKarolina Ruiz Rogelj, Melvyn Peignon
Summary
ClickHouse announced expanded data lake capabilities, enabling direct querying, accelerated ingestion into MergeTree, and writes across Apache Iceberg and Delta Lake tables. Open lake architectures provide storage portability across cloud providers and catalogs, but standard query engines often struggle with latency for real-time workloads. Over two years of development, ClickHouse engineered a native Parquet reader with page-level parallelism, integrated the Delta Rust Kernel, and introduced support for schema evolution, time travel, and DML operations. The platform functions across AWS Glue, Unity Catalog, Polaris, and Iceberg REST Catalog on storage systems including Amazon S3, Google Cloud Storage, and Azure Blob Storage. These enhancements allow organizations to execute federated, high-concurrency SQL queries across multiple catalogs or write processed analytical results back to their lakes without vendor lock-in.
Context
Organizations adopt open data lake table formats like Apache Iceberg and Delta Lake to avoid vendor lock-in and query shared data across multiple engines. However, because these lake formats were designed for open storage and interoperability rather than raw speed, queries at scale lack the specialized indexes, caching, and engine optimizations required for real-time analytics workloads.
Approach / What changed
ClickHouse introduced native support for Iceberg, Delta Lake via the Delta Rust Kernel, and optimized Parquet reads using page-level parallelism without an intermediary Arrow layer. It integrated with catalogs such as AWS Glue, Unity Catalog, Polaris, and Iceberg REST Catalog across S3, GCS, and Azure Blob Storage, while supporting DML operations, schema evolution, time travel, and writing results back to open lake formats.
Takeaways
- ClickHouse release 25.8 introduced a native Parquet reader featuring page-level parallelism and removing the Arrow layer, delivering an average 1.8x read performance speedup across ClickBench.
- Integrating the community Delta Rust Kernel enabled Delta Lake reads, writes, change data feeds, schema evolution, time travel, partition pruning, and statistic-based pruning.
- Users can query Iceberg and Delta Lake tables in place across S3, GCS, and Azure Blob Storage or write aggregated results back out from ClickHouse to keep data open for engines like Spark and Trino.