Loading post…
Together AIExplainer
Autoscaling endpoints for LLM inference
Summary
Dedicated LLM inference deployments face a trade-off between paying for idle GPUs and suffering sharp p95 latency when replicas hit batching limits. The autoscaling loop converts observed load into desired replicas with ceil(N × observed/target), then applies scale-up and scale-down windows, replica bounds, and GPU placement. It presents inflight_requests as a leading, engine-agnostic default, while TTFT and end-to-end latency target responsiveness but trail demand, and GPU or token utilization prioritize efficiency with less latency headroom. Cold starts can take several minutes because replicas require GPU placement, weight downloads, VRAM loading, and warm-up, so policies should scale early; streaming is required for TTFT, decoding speed, and throughput-per-replica metrics, whereas end-to-end latency supports both streaming and non-streaming workloads.
Context
The subject is autoscaling dedicated LLM inference endpoints under variable traffic. The central concern is balancing per-replica cost, queueing, latency SLOs, and the delay involved in starting additional GPU-backed replicas.
Approach / What changed
Configure replica bounds, one or more inference-native scaling metrics with targets, and separate scale-up and scale-down windows. The control loop calculates desired replicas proportionally, dampens changes through the timing windows, clamps the result to the configured bounds, and routes traffic according to capacity.
Takeaways
inflight_requests is the stated safe default because it is a leading, engine-agnostic signal that maps to concurrent work and does not require streaming or percentile selection.
Scale-up windows should generally be short, while scale-down windows should remain longer than the traffic’s natural rhythm; false scale-downs can trigger costly cold starts before the next peak.
Setting min_replicas equal to max_replicas disables autoscaling, while setting both to zero stops the deployment and stops billing. Bounds without a metric use inflight_requests with a target of 8.
FAQ
Why can GPU utilization look healthy while inference latency worsens?
GPU utilization measures arithmetic activity rather than queue pressure, so a GPU can show 60% utilization while requests are already backing up. Scaling on that signal may respond to a number that does not describe the latency problem.
Why must scale-up and scale-down windows be tuned differently?
A short scale-up window helps respond before a traffic peak causes user-facing latency, and a false scale-up costs only a few replica-minutes. A longer scale-down window avoids removing capacity just before the next peak, when restarting a replica would add a cold start.
Which autoscaling metrics require streaming traffic?
TTFT, decoding_speed, and throughput_per_replica are emitted only on the streaming path, so they provide no data for non-streaming clients. e2e_latency is the exception and is measured for both streaming and non-streaming workloads.
Source and freshness
- Source
- together.ai
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.