Why we rewrote WAL-G for Postgres backups in Rust: Meet WAL-RUS

Summary

ClickHouse Cloud built WAL-RUS, an open-source Rust implementation of PostgreSQL backup and WAL archival tooling, after encountering difficulty predicting WAL-G’s memory usage in resource-constrained environments. The project preserves WAL-G’s core functionality and compatibility while using bounded worker pools, controlled concurrency, persistent object-storage connections, and a streaming pipeline that limits buffering and data copies. It uses the same WALG_ configuration variables, and WAL-G and WAL-RUS can read each other’s archives. In a sustained WAL-heavy benchmark with four archival workers, WAL-RUS stayed below 1 GB of peak virtual memory versus nearly 2.8 GB for WAL-G, while both maintained minimal backlog and comparable CPU usage. The authors plan to make WAL-RUS the default backup and archival mechanism for ClickHouse Cloud’s managed Postgres offering and are working to upstream its Postgres 17 wal summaries support to WAL-G.

Takeaways

1. 1

   In the benchmark, WAL-RUS remained below 1 GB of peak virtual memory, while WAL-G reached nearly 2.8 GB, reducing peak virtual memory by more than 70%.

2. 2

   WAL-RUS and WAL-G both maintained minimal archival backlog under the sustained WAL-heavy workload; pgBackRest accumulated a larger backlog during periods of intense WAL activity.

3. 3

   WAL-G can read archives generated by WAL-RUS and WAL-RUS can read WAL-G archives, while both tools use the same WALG_ configuration variables.