MongoDB Predictive Auto-Scaling: An Experiment

Summary

MongoDB Atlas initially relied on a reactive auto-scaler that took minutes to adjust instance sizes and only transitioned between adjacent tiers. Because these delays caused transient server over-utilization or unnecessary customer costs, researchers prototyped predictive auto-scaling to forecast demand. The experimental architecture used short-term and long-term forecasters for customer-driven metrics, an estimator using boosted decision trees to project CPU utilization, and a planner to pick the most cost-effective tier. In testing, the prototype kept replica sets closer to target CPU ranges while saving an average of nine cents per replica set hourly. MongoDB subsequently implemented a production version that scales replica sets up proactively before expected spikes while retaining reactive downscaling.

Takeaways

1. 1

   Forecasting models targeted exogenous customer-driven metrics such as queries per second and scanned-objects rate to avoid circular feedback loops caused by scaling CPU capacity.

2. 2

   The experimental long-term forecaster included a self-censoring confidence mechanism to disregard its predictions for non-seasonal replica sets when recent errors grew large.

3. 3

   The production rollout in MongoDB Atlas operates conservatively by predicting scale-up events while relying on the reactive auto-scaler to downscale instances after load drops.