Introducing WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

Summary

ClickHouse announces WalShadow, an open-source engine that replicates Postgres data to ClickHouse directly from physical WAL, targeting sub-second analytical availability without logical replication. It consumes the physical WAL stream outside the source database, tracks schema in a shadow Postgres instance, decodes heap records through parallel Rust-based decoders, builds ClickHouse-native blocks, and inserts them concurrently. Because parallel blocks can arrive out of order, each row carries the source WAL position (_lsn), while barriers preserve ordering for schema changes and truncations. In benchmarks using modest c8i.2xlarge instances, WalShadow reached approximately 200 ms commit-to-visible latency and 289,000 rows per second, compared with PeerDB’s roughly 10 seconds and 120,000 rows per second; it supports initial load, continuous replication, schema evolution, restart recovery, and planned switchovers.

Takeaways

1. 1

   WalShadow avoids logical replication slots, Kafka, JSON serialization, and a separate normalization step; the source only ships physical WAL, with a load profile described as similar to a physical standby.

2. 2

   The replication lifecycle includes initial load, continuous replication, schema evolution, restart recovery, and planned source switchovers. Supported schema changes include ADD COLUMN, RENAME COLUMN, DROP COLUMN, and CREATE TABLE.

3. 3

   In the stated benchmark, WalShadow delivered approximately 200 ms commit-to-visible latency and 289,000 rows per second, versus PeerDB’s roughly 10 seconds and 120,000 rows per second.