What is WAL backpressure, and why does ClickHouse Managed Postgres need it?

Summary

Postgres writes every change to the write-ahead log (WAL), and Managed Postgres uploads completed segments to object storage for point-in-time recovery; if uploads lag, retained WAL can fill the disk and cause a PANIC shutdown. ClickHouse Managed Postgres counters this with a data-plane systemd timer that checks pending segments every 15 seconds and uses cgroup v2 I/O controls to throttle client write bandwidth at 80%, 50%, or 20% of provisioned throughput as backlog thresholds rise. The drain path—including the archiver, checkpointer, and related processes—is placed in an immune cgroup, while client backends are throttled and reads remain uncapped. In a test with a 500 MB/s gp3 volume and an archive limited to 4 MB/s, the backlog reached 3,433 segments and 53 GiB before recovery. Once the limit was removed, the archiver cleared the queue, the throttle released automatically, and the disk stayed below 33% full.

Takeaways

1. 1

   The write caps are 80%, 50%, and 20% of provisioned disk throughput at 100, 500, and 1,000 pending segments respectively; on a 500 MB/s disk, those limits are 400, 250, and 100 MB/s.

2. 2

   In the test, a 4 MB/s archive limit let the backlog reach 3,433 segments and 53 GiB of retained WAL. After recovery, the immune archiver drained about 850 segments per minute while the workload continued.

3. 3

   The throttle released automatically once the backlog fell below 100 segments, the count reached zero shortly afterward, and the data disk never exceeded 33% full.