Autoscaling Lakebase Postgres

Summary

Lakebase Postgres addresses the waste and uncertainty of choosing a fixed database instance size before workload demand is known. Its decoupled architecture separates stateless compute from durable storage, allowing compute to start, stop, move, or resize without moving the database. The autoscaling algorithm combines CPU load, memory pressure, and a time-aware estimate of the Postgres working set, using modified HyperLogLog registers that store timestamps and estimate windows from one to 60 minutes. Scaling occurs live through autoscaler-agent, vm-monitor, Kubernetes scheduling, and NeonVM, with checks preventing unsafe downscales and scheduler approval for upscales. The system scales in both directions without dropping connections and operates across 100-millisecond, five-second, and 20-second timescales, including more than 32,000 resizes per month.

Takeaways

1. 1

   Compute nodes own no durable state, so they can be resized or live-migrated without moving the underlying database or dropping existing connections.

2. 2

   The working-set estimator modifies HyperLogLog by storing update timestamps, allowing distinct-page estimates for windows from one minute through 60 minutes.

3. 3

   Upscaling requires Kubernetes scheduler approval, while vm-monitor validates downscales to ensure Postgres and the guest still have enough memory.