Loading post…
ClickhouseAnnouncement
chDB Durable Layer for agent memory
Summary
Agent memory built on chDB, an embedded SQL OLAP engine, accumulated state such as project rules, user preferences, and recall traces on a single laptop. Moving that memory to CI, sandboxes, or another laptop broke the local-disk assumption because the MergeTree directory existed on only one disk. Adding a ClickHouse server backend fixed portability but turned every recall into a network round trip. The chDB Durable Layer fills the gap by keeping the local analytical hot path while publishing authoritative state to object storage (S3, GCS, or Azure Blob). Applications write to a local working copy and use flush() to publish writes and checkpoint() to compact the log, avoiding a sidecar or database server. The contract supports Python, Node, Go, and Rust bindings with cross-binding recoverability.
Context
The problem arose when agent memory built on chDB needed to move across laptops, CI jobs, and short-lived sandboxes, but the state was tied to a single local disk. Existing options such as SQLite, SQLite plus Litestream, server databases, Cloudflare Durable Objects, or plain DuckDB/chDB each had tradeoffs: either no analytical capability, network latency on every recall, vendor coupling, or state that still lived on one disk without recoverability.
Approach / What changed
The chDB Durable Layer wraps an embedded chDB database with a small durability layer that keeps the authoritative copy in object storage (S3, GCS, or Azure Blob), while queries run against a local MergeTree working copy. Applications publish state on demand via flush() and compact logs via checkpoint(), and head.json with conditional writes ensures single-writer consistency.
Takeaways
The chDB Durable Layer keeps queries local to an embedded MergeTree working copy while the authoritative copy lives in object storage, eliminating network latency on recall.
flush() and checkpoint() give the application explicit control over durability boundaries: flush() publishes writes to storage, checkpoint() compacts the WAL into a new base snapshot.
The durable V1 contract is shared across Python, Node, Go, and Rust bindings, enabling cross-binding recoverability of objects written by one binding and restored by another.
FAQ
When does data actually become durable?
When flush() returns, the writes it covers have reached object storage. The application chooses that boundary, which matters for small continuous writes so that not every write immediately triggers an expensive network call.
What is the role of checkpoint()?
Between checkpoints, durable state consists of a base snapshot plus a WAL. checkpoint() writes a new base snapshot, so the next open does not have to replay a long log.
Which bindings support the Durable Layer?
The durable V1 contract is shared across Python, Node, Go, and Rust bindings, all building on chdb-core 26.7.3. An object written by one binding is meant to be recoverable by another binding that speaks the same contract.
Source and freshness
- Source
- clickhouse.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.