Autoscaling Is a Control Loop, Not a Button
Autoscaling trades responsiveness, stability, and cost: this explainer shows how signal choice, scaling direction, and capacity limits shape the right control loop for a workload.
Loading…
Topic
Autoscaling
22 posts about Autoscaling. Every summary links to the original.
- 22 posts
- 9 companies
- 1 explainer
See the idea
Autoscaling explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How Intelligent Load Shedding Prevents Cascading Failures in Tier-0 Systems
Salesforce’s Cloud Atlas team operates a globally distributed identity data store that must maintain five nines of availability, roughly five minutes of downtime per year. The team redesigned service protection after per-instance rate limiting proved insufficient under autoscaling and increasingly bursty, unpredictable agent-driven traffic. The new approach combines intelligent load shedding, which monitors queue time as an early indicator of system pressure before error rates spike, with coordinator-free global quota management. Servers independently measure and share customer usage to throttle noisy tenants without a centralized coordinator, preventing one customer’s surge from affecting others. This design prevents cascading failures by shedding lower-priority work before overload spreads across dependent services, ensuring graceful degradation rather than widespread outage.
Comparison of Cluster Autoscaler and KEDA MachineSet Autoscaling on Red Hat OpenShift
This article compares two autoscaling approaches for Red Hat OpenShift compute infrastructure: the built-in Cluster Autoscaler and the KEDA-driven MachineSet Autoscaler. The key difference is the trigger mechanism: Cluster Autoscaler reacts to Pending pods, while KEDA scales based on external metrics like PromQL queries. KEDA triggers faster, scaling within about 15 minutes versus 24 minutes for Cluster Autoscaler, because it acts on metric thresholds before pods become unschedulable. Cluster Autoscaler offers built-in scale-down safety and zero configuration, while KEDA provides fine-grained, metric-driven control but requires the Custom Metrics Autoscaler operator. The choice depends on workload predictability, with Cluster Autoscaler suited for general-purpose workloads and KEDA for dedicated pools with predictable metric patterns. The two autoscalers cannot coexist on the same MachineSet, as they would fight over replica counts.
How agentic software development will change databases
AI agents are changing software development from a slow, linear process into rapid evolutionary experimentation, creating many small, ephemeral applications and databases. The post identifies three resulting database requirements: near-zero marginal cost with elastic growth, native branching for rapid variants, and open ecosystems that agents can use reliably. Lakebase addresses branching through O(1) metadata copy-on-write at the storage layer, so agents can branch production or test state without physical data copying and pay only for experiment compute. Its decoupled, serverless compute can scale in sub-second intervals and down to zero, while standard open Postgres page formats in cloud object storage keep data accessible to external engines and tools. Lakebase telemetry reports about 10 branches per project on average, some nested beyond 500 iterations, and roughly four times as many databases created by agents as by humans.
Red Hat OpenShift autoscaling using MachineSet autoscaler with KEDA
The demonstration compares OpenShift MachineSet Autoscaler with KEDA against the previously used Cluster Autoscaler, using metrics to scale a MachineSet before pods remain pending. After installing Red Hat’s Custom Metrics Autoscaler operator (KEDA 2.19), granting permissions for MachineSet scaling and Thanos/Prometheus access, it creates a ScaledObject with a Prometheus trigger targeting demo-p4p95-worker-eastus3. Because a MachineSet has no pods, the HPA path cannot use metricType: Value, so AverageValue is paired with a PromQL query that multiplies CPU request utilization by the current node count. With nine one-CPU test pods, the MachineSet scaled from one to two to three nodes as the metric crossed 75%, and all pods reached Running; two additional Standard_D4s_v3 VMs were provisioned in zone 3.
Rerouting the Stream: How Lyft Moved to the Apache Flink Operator
Lyft migrated its fleet of hundreds of critical streaming jobs from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes operator. The legacy system imposed maintenance burden, feature gaps, outdated dependencies, fragile savepoint handling, and a rigid memory model that complicated varied workloads, including Apache Beam applications. Lyft used an incremental migration designed to preserve deployment patterns and team workflows while addressing CRD translation, state-machine behavior, and operational compatibility. The new platform added explicit last-state upgrades, autoscaling, automatic memory tuning, and safer upgrade mechanisms, though autotuning required trade-offs with in-place autoscaling and node capacity planning. After the multi-quarter transition, maintenance burden was minimized, the autoscaler right-sized a fleet previously overprovisioned by a few million dollars annually, and Lyft could contribute fixes upstream instead of maintaining its own operator.
Autoscaling Lakebase Postgres
Lakebase Postgres addresses the waste and uncertainty of choosing a fixed database instance size before workload demand is known. Its decoupled architecture separates stateless compute from durable storage, allowing compute to start, stop, move, or resize without moving the database. The autoscaling algorithm combines CPU load, memory pressure, and a time-aware estimate of the Postgres working set, using modified HyperLogLog registers that store timestamps and estimate windows from one to 60 minutes. Scaling occurs live through autoscaler-agent, vm-monitor, Kubernetes scheduling, and NeonVM, with checks preventing unsafe downscales and scheduler approval for upscales. The system scales in both directions without dropping connections and operates across 100-millisecond, five-second, and 20-second timescales, including more than 32,000 resizes per month.
Autoscaling endpoints for LLM inference
Dedicated LLM inference deployments face a trade-off between paying for idle GPUs and suffering sharp p95 latency when replicas hit batching limits. The autoscaling loop converts observed load into desired replicas with ceil(N × observed/target), then applies scale-up and scale-down windows, replica bounds, and GPU placement. It presents inflight_requests as a leading, engine-agnostic default, while TTFT and end-to-end latency target responsiveness but trail demand, and GPU or token utilization prioritize efficiency with less latency headroom. Cold starts can take several minutes because replicas require GPU placement, weight downloads, VRAM loading, and warm-up, so policies should scale early; streaming is required for TTFT, decoding speed, and throughput-per-replica metrics, whereas end-to-end latency supports both streaming and non-streaming workloads.
Configuring Dedicated Model Inference
Dedicated Model Inference is organized around three parts: an endpoint, deployments that run a model revision with replicas, and immutable configs that define how it runs. A deployment binds one model revision to one config and an autoscaling policy, while the endpoint provides a stable API-facing name and traffic split. Routing uses each deployment’s weight multiplied by its ready-replica count, so equal weights produce a 25%/75% split when deployments have one and three ready replicas; unavailable replicas contribute no capacity. The configuration workflow uses certified model-and-config profiles, selectable by accelerator, GPU count, optimization target, and topology, while new config revisions preserve tested rollback targets. This capacity-aware design makes scaling, rollouts, A/B tests, shadow traffic, and zero-downtime changes possible, but raw API or SDK deployments must be added to the traffic split before they receive requests.
The production platform for open-weight AI inference
The update presents a production inference platform for open-weight, licensed closed-weight, and fine-tuned models, aimed at giving teams control over performance, cost, quality, and functionality without building an inference stack from scratch. It supports uploading weights or adapters from Hugging Face, S3, or local machines, uses tested deployment profiles for hardware, quantization, tensor parallelism, speculative decoding, and serving engine choices, and lets operators tune placement and autoscaling signals. Endpoints can keep a stable interface while multiple deployments receive traffic through canary, blue-green, rolling, A/B, or shadow testing, with threshold-based automatic rollback. The rebuilt caching and distribution layer proactively warms shared weights; internal testing reports roughly 4× faster warm starts across several frontier models, while a closed beta adds full-weight and LoRA reinforcement learning and supervised fine-tuning with deployable checkpoints.
Introducing Modal Auto Endpoints: Optimized inference you actually own
Modal Auto Endpoints is a self-serve service for production-grade LLM inference that gives teams ownership of the entire serving stack, not just an API. Unlike proprietary providers, it exposes the inference code, engine flags, and GPU selection, and provides detailed metrics like speculative decoding acceptance length and per-replica token latency quantiles in a dashboard. Built on Modal’s infrastructure, it uses autoscaling to handle traffic spikes by spinning up additional replicas automatically, global GPU capacity without capacity management, and Modal Servers for ultra-low-latency routing with only 5ms overhead. For each supported model, it offers a pre-configured deployment using speculative decoding with the DFlash block-diffusion drafter from Z Lab, and teams can benchmark latency and throughput tradeoffs with a click after deployment. The platform is designed with a declarative interface based on workloads and SLOs, with future automation levels including autoinference, autospec, autodistill, and autoresearch for continuous improvement of inference performance.
AI Serving Platform That Adapts to Your Model
Databricks Custom Model Serving addresses the operational burden of serving custom models, whose resource profiles, traffic patterns, and latency requirements vary widely from small CPU classifiers to large GPU-backed language models. Its fully managed platform packages MLflow models and uses isolated Kubernetes deployments, model-appropriate runtimes, and a short request path to limit interference and per-request overhead. At the center, the AutoPilot Pod Autoscaler combines active concurrency and queue signals for horizontal scaling with CPU, GPU, and memory measurements for model-aware target-concurrency adjustment, allowing one controller to adapt across workloads. Warm pools, provisioned concurrency, and zero-downtime updates address startup and deployment concerns, while reported production results include 90%+ cost savings for some customers, up to 2x improvement in p99 and p50 latency, 100K+ QPS, and 99.99% availability.
Reliable LLM Inference at Scale
Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP_NUM_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.
Introducing Always-On pricing: automatic savings for Databricks Lakebase
Databricks introduces Always-On pricing for Lakebase, offering a lower rate for baseline compute while retaining elastic autoscaling. On any Lakebase Postgres Autoscaling project, operators turn off scale-to-zero and set an autoscaling range; the minimum capacity becomes the baseline, which receives the lower rate after 24 hours of continuous use. Capacity above the minimum continues to autoscale up to the configured maximum and is billed at standard Autoscaling rates, while users can re-enable scale-to-zero later. The pricing targets established workloads with a consistent activity floor; intermittent or new workloads can retain scale-to-zero to avoid paying for idle hours and uncertain baseline estimates. The announcement says the baseline price is 25% lower, and an additional 50% promotional discount runs through January 31, 2027.
Rethinking Distributed Systems for Serverless Performance and Reliability
Building serverless compute for Apache Spark requires more than warm machine pools or basic autoscaling because traditional deployments couple user applications to infrastructure, allowing contention and inefficiencies to undermine performance and reliability. The proposed architecture separates these concerns through Spark Connect’s client-server model over gRPC, a gateway that routes workloads using query size, cluster utilization, and latency profile, and an adaptive autoscaler that adjusts capacity horizontally and vertically. Spark Connect isolates user applications from drivers, while the gateway continually re-evaluates placement to reduce interference between workloads. The autoscaler offers Standard and Performance-Optimized modes and can respond to out-of-memory errors by restarting tasks on larger VMs without manual intervention. Reported outcomes include a 99.998% upgrade success rate across more than 4.5 billion workloads, 2–5x faster Unilever pipelines, and operational-cost reductions of 25%.
Machine-learning predictive autoscaling for Flink
Grab’s Flink platform saw application count grow 2.5 times, making cost-effective, self-service CPU provisioning difficult for users who often relied on trial and error and over-provisioned resources. Reactive autoscaling with Adaptive Scheduler, Kubernetes HPA, and metrics such as CPU usage or consumer latency caused restart spikes, scaling spirals, fluctuations, and additional constraints when connectors such as Kafka reached fixed partition parallelism. Predictive Resource Advisor instead uses vertical scaling and predictive, deterministic CPU decisions, automatically collecting data, training models, and applying configuration changes. The source says applications using it experienced fewer autoscaling-related disruptions, adapted to workload growth and surges, and averaged more than 35% cloud infrastructure cost savings. It also reports simpler default-configuration deployment, while planned work includes memory tuning, improved burst handling, and more efficient model training.
Rethinking Serverless with FLAME
Rethinking Serverless with FLAME presents a pattern for running selected application functions on short-lived copies of the full app instead of moving code into bespoke FaaS services. FLAME, short for Fleeting Lambda Application for Modular Execution, uses FLAME.call to send a function and closed-over values to a runner pool, while retaining access to the application’s database connections and other code. The Elixir flame library includes LocalBackend and FlyBackend adapters, and Fly.io can boot a same-image Machine in the parent’s region in about three seconds. Runners may stay warm, scale to zero, or grow elastically based on configuration. The post also describes process-based streaming with LiveView and supervision and monitoring mechanisms for transient runners.
Logbook - 2022-07-18
This Logbook entry records Fly.io product, platform, documentation, and interface changes on 2022-07-18. Highlights include $99 in usage credits for the Pro Plan, SOC2 compliance, persistent-volume expansion through fly volumes extend, SFTP access to apps over 6PN and WireGuard, and routing changes intended to improve latency. flyctl gained debug JSON formatting, expanded machine-app deployment and status commands, preserved mounted volumes during deployments, and added rolling deploys that wait for each VM to start. The entry also describes fixes for autoscaling, the Fly Redis image, GraphQL validation, Phoenix launches, and dashboard presentation, alongside updated Laravel and persistent-volume documentation and improved Heroku token-failure emails.
The January 2021 Fly ChangeLog
The January 2021 Fly ChangeLog covers platform and flyctl changes from December 2020 through January 2021, focusing on private networking, VPN access, scaling, and infrastructure work. 6PN networking adds private communication and DNS-based app discovery over Fly's WireGuard mesh, while IPv6 WireGuard Peering provides tunnels into an organization; the new wireguard command creates, lists, and removes tunnels. Fly also made instance-count scaling the default model, separating it from autoscaling through scale and autoscale commands. Backend updates switched image pulling to containerd, improving subsequent boot times for larger images, added app lifecycle events to application logs, and made volume initialization lazy with encryption enabled by default. The listed flyctl releases also add volume, memory-scaling, region, completion, and WireGuard updates, alongside bug fixes.
How Shopify Reduced Storefront Response Times with a Rewrite
Shopify rewrote the server-side Storefront Renderer, which loads Liquid themes and storefront data before returning HTML, because the legacy Rails-monolith implementation had developed stricter performance demands and rising time-to-first-byte as traffic grew. The new single-purpose application separates storefront traffic from checkout, admin, and API traffic, uses active-active replication with dedicated read replicas, and adds mechanisms for high-load resilience. During migration, a Ruby verifier compares status codes, headers, and bodies from both implementations, while a custom Lua module on OpenResty samples production traffic and routes requests based on verification results. The rollout had reached more than 90% feature parity, and the new implementation averaged 4x faster server response times, with ongoing work aimed at full parity and retiring the legacy system.
Fly Changelog May 15th 2020
Fly’s May 15, 2020 changelog records changes to the Fly platform, flyctl, and related websites since the previous update. The main release was a new scaling system based on region pools, accompanied by new scaling commands in flyctl 0.0.122; the earlier autoscale command from version 0.0.121 was removed. The platform also rolled out an HTTP2-only backhaul for traffic between the Fly edge and datacenters, with the changelog reporting improved performance and stability for applications using the http handler. Other entries document new autoscale and regions commands in 0.0.121, consistent naming for flyctl binary artifacts, removal of references to 32-bit Windows and Linux builds during deprecation, and a fix for custom Dockerfile mapping.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.