Announcing chDB 4: write Pandas, run ClickHouse, now on Hex

Summary

chDB 4 introduces Data Store, a native component that allows users to write Pandas-like code that executes on the ClickHouse OLAP SQL engine. Standard Pandas workflows suffer from eager execution, excessive memory materialization, and lack of automatic query optimizations. To solve this, Data Store implements a four-layer architecture that lazily records operations as a chain of LazyOp objects and compiles them into optimized ClickHouse SQL plans upon evaluation triggers. When encountering operations unsupported by ClickHouse, the QueryPlanner splits the pipeline into segments, routing tasks between ClickHouse and Pandas while using Python's memoryview to minimize data copying overhead. This release also integrates natively into Hex notebooks.

Takeaways

1. 1

   Data Store records operations lazily and triggers execution only on operations like print(), len(), or .columns, compiling queries with filter pushdown, column pruning, and limit propagation.

2. 2

   chDB avoids full data copies for numeric and fixed-width columns by exposing ClickHouse internal memory buffers directly to Pandas without serialization cycles.

3. 3

   When a Pandas segment sits in the middle of a pipeline, downstream segments operate on intermediate in-memory data and cannot benefit from source-level optimizations like Parquet row-group skipping.