Fixed cadence to seconds: making ClickHouse Cloud autoscaling more reactive

Summary

ClickHouse Cloud’s vertical autoscaling recommender originally ran on a fixed timer, creating latency when workloads suddenly increased and services needed resources before the next scheduled pass. The redesign keeps the periodic sweep as a safety net while adding a reactive fast path, using controller-runtime as an event-processing layer with sources feeding a shared rate-limited work queue and reconciler. The queue provides deduplication, exponential backoff, bounded concurrency, leader election, and metrics, while ClickHouse stores near-real-time signals that a polling source reads; the excerpt notes polling sets a floor on reaction speed. The reactive path is rolling out cautiously in observe-only mode, with periodic reconciliation remaining the main driver, and the stated goal is to trigger scale-ups for critical events such as out-of-memory conditions in seconds rather than waiting for the next scheduled run.

Takeaways

1. 1

   The shared controller-runtime queue separates trigger logic from job-processing concerns, so periodic and reactive sources do not need to coordinate deduplication, retries, or concurrency themselves.

2. 2

   The signal source polls a table rather than consuming a push stream, making its polling interval the lower bound on reaction speed; a restart does not permanently lose an event because later queries can enqueue the service again.

3. 3

   The design favors idempotent, inexpensive recomputation and periodic self-correction because signal volume is modest; guaranteed delivery, strict ordering, or stateful-window correlation would likely require a dedicated streaming platform.