Loading…
Autoscaling
8 posts about Autoscaling. Every summary links to the original.
Red Hat ·
Red Hat OpenShift autoscaling using MachineSet autoscaler with KEDA
The demonstration compares OpenShift MachineSet Autoscaler with KEDA against the previously used Cluster Autoscaler, using metrics to scale a MachineSet before pods remain pending. After installing Red Hat’s Custom Metrics Autoscaler operator (KEDA 2.19), granting permissions for MachineSet scaling and Thanos/Prometheus access, it creates a ScaledObject with a Prometheus trigger targeting demo-p4p95-worker-eastus3. Because a MachineSet has no pods, the HPA path cannot use metricType: Value, so AverageValue is paired with a PromQL query that multiplies CPU request utilization by the current node count. With nine one-CPU test pods, the MachineSet scaled from one to two to three nodes as the metric crossed 75%, and all pods reached Running; two additional Standard_D4s_v3 VMs were provisioned in zone 3.
Ramon Gordillo Gutierrez, Jose Ortiz PadillaLyft ·
Rerouting the Stream: How Lyft Moved to the Apache Flink Operator
Lyft migrated its fleet of hundreds of critical streaming jobs from an internally developed Flink Kubernetes operator to the open-source Apache Flink Kubernetes operator. The legacy system imposed maintenance burden, feature gaps, outdated dependencies, fragile savepoint handling, and a rigid memory model that complicated varied workloads, including Apache Beam applications. Lyft used an incremental migration designed to preserve deployment patterns and team workflows while addressing CRD translation, state-machine behavior, and operational compatibility. The new platform added explicit last-state upgrades, autoscaling, automatic memory tuning, and safer upgrade mechanisms, though autotuning required trade-offs with in-place autoscaling and node capacity planning. After the multi-quarter transition, maintenance burden was minimized, the autoscaler right-sized a fleet previously overprovisioned by a few million dollars annually, and Lyft could contribute fixes upstream instead of maintaining its own operator.
Maheep MyneniAutoscaling Lakebase Postgres
Lakebase Postgres addresses the waste and uncertainty of choosing a fixed database instance size before workload demand is known. Its decoupled architecture separates stateless compute from durable storage, allowing compute to start, stop, move, or resize without moving the database. The autoscaling algorithm combines CPU load, memory pressure, and a time-aware estimate of the Postgres working set, using modified HyperLogLog registers that store timestamps and estimate windows from one to 60 minutes. Scaling occurs live through autoscaler-agent, vm-monitor, Kubernetes scheduling, and NeonVM, with checks preventing unsafe downscales and scheduler approval for upscales. The system scales in both directions without dropping connections and operates across 100-millisecond, five-second, and 20-second timescales, including more than 32,000 resizes per month.
Carlota SotoAI Serving Platform That Adapts to Your Model
Databricks Custom Model Serving addresses the operational burden of serving custom models, whose resource profiles, traffic patterns, and latency requirements vary widely from small CPU classifiers to large GPU-backed language models. Its fully managed platform packages MLflow models and uses isolated Kubernetes deployments, model-appropriate runtimes, and a short request path to limit interference and per-request overhead. At the center, the AutoPilot Pod Autoscaler combines active concurrency and queue signals for horizontal scaling with CPU, GPU, and memory measurements for model-aware target-concurrency adjustment, allowing one controller to adapt across workloads. Warm pools, provisioned concurrency, and zero-downtime updates address startup and deployment concerns, while reported production results include 90%+ cost savings for some customers, up to 2x improvement in p99 and p50 latency, 100K+ QPS, and 99.99% availability.
Anshul GuptaReliable LLM Inference at Scale
Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP_NUM_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.
Ying Chen, Wendy Hu, Ankit Mathur, Mike Eastham, Pei-Lun Liao, Wai Wu, Arjun DCunhaIntroducing Always-On pricing: automatic savings for Databricks Lakebase
Databricks introduces Always-On pricing for Lakebase, offering a lower rate for baseline compute while retaining elastic autoscaling. On any Lakebase Postgres Autoscaling project, operators turn off scale-to-zero and set an autoscaling range; the minimum capacity becomes the baseline, which receives the lower rate after 24 hours of continuous use. Capacity above the minimum continues to autoscale up to the configured maximum and is billed at standard Autoscaling rates, while users can re-enable scale-to-zero later. The pricing targets established workloads with a consistent activity floor; intermittent or new workloads can retain scale-to-zero to avoid paying for idle hours and uncertain baseline estimates. The announcement says the baseline price is 25% lower, and an additional 50% promotional discount runs through January 31, 2027.
Kunal Kande, Mike JeromeRethinking Distributed Systems for Serverless Performance and Reliability
Building serverless compute for Apache Spark requires more than warm machine pools or basic autoscaling because traditional deployments couple user applications to infrastructure, allowing contention and inefficiencies to undermine performance and reliability. The proposed architecture separates these concerns through Spark Connect’s client-server model over gRPC, a gateway that routes workloads using query size, cluster utilization, and latency profile, and an adaptive autoscaler that adjusts capacity horizontally and vertically. Spark Connect isolates user applications from drivers, while the gateway continually re-evaluates placement to reduce interference between workloads. The autoscaler offers Standard and Performance-Optimized modes and can respond to out-of-memory errors by restarting tasks on larger VMs without manual intervention. Reported outcomes include a 99.998% upgrade success rate across more than 4.5 billion workloads, 2–5x faster Unilever pipelines, and operational-cost reductions of 25%.
Aaron Davidson, Roland Fäustlin, Zach WilliamsShopify ·
How Shopify Reduced Storefront Response Times with a Rewrite
Shopify rewrote the server-side Storefront Renderer, which loads Liquid themes and storefront data before returning HTML, because the legacy Rails-monolith implementation had developed stricter performance demands and rising time-to-first-byte as traffic grew. The new single-purpose application separates storefront traffic from checkout, admin, and API traffic, uses active-active replication with dedicated read replicas, and adds mechanisms for high-load resilience. During migration, a Ruby verifier compares status codes, headers, and bodies from both implementations, while a custom Lua module on OpenResty samples production traffic and routes requests based on verification results. The rollout had reached more than 90% feature parity, and the new implementation averaged 4x faster server response times, with ongoing work aimed at full parity and retiring the legacy system.
2023-10-18
