---
title: "Autoscaling endpoints for LLM inference"
description: "Dedicated LLM inference deployments face a trade-off between paying for idle GPUs and suffering sharp p95 latency when replicas hit batching limits. The autoscaling loop converts observed load into desired replicas with ceil(N × observed/target), then applies scale-up and scale-down windows, replica bounds, and GPU placement. It presents inflight_requests as a leading, engine-agnostic default, while TTFT and end-to-end latency target responsiveness but trail demand, and GPU or token utilization prioritize efficiency with less latency headroom. Cold starts can take several minutes because replicas require GPU placement, weight downloads, VRAM loading, and warm-up, so policies should scale early; streaming is required for TTFT, decoding speed, and throughput-per-replica metrics, whereas end-to-end latency supports both streaming and non-streaming workloads."
---

# Autoscaling endpoints for LLM inference

[Together AI](https://yomu.fyi/company/together-ai) · Jul 31, 2026

**Type:** Explainer

## Summary

Dedicated LLM inference deployments face a trade-off between paying for idle GPUs and suffering sharp p95 latency when replicas hit batching limits. The autoscaling loop converts observed load into desired replicas with ceil(N × observed/target), then applies scale-up and scale-down windows, replica bounds, and GPU placement. It presents inflight\_requests as a leading, engine-agnostic default, while TTFT and end-to-end latency target responsiveness but trail demand, and GPU or token utilization prioritize efficiency with less latency headroom. Cold starts can take several minutes because replicas require GPU placement, weight downloads, VRAM loading, and warm-up, so policies should scale early; streaming is required for TTFT, decoding speed, and throughput-per-replica metrics, whereas end-to-end latency supports both streaming and non-streaming workloads.

## Context

The subject is autoscaling dedicated LLM inference endpoints under variable traffic. The central concern is balancing per-replica cost, queueing, latency SLOs, and the delay involved in starting additional GPU-backed replicas.

## Approach / What changed

Configure replica bounds, one or more inference-native scaling metrics with targets, and separate scale-up and scale-down windows. The control loop calculates desired replicas proportionally, dampens changes through the timing windows, clamps the result to the configured bounds, and routes traffic according to capacity.

## Takeaways

- inflight\_requests is the stated safe default because it is a leading, engine-agnostic signal that maps to concurrent work and does not require streaming or percentile selection.
- Scale-up windows should generally be short, while scale-down windows should remain longer than the traffic’s natural rhythm; false scale-downs can trigger costly cold starts before the next peak.
- Setting min\_replicas equal to max\_replicas disables autoscaling, while setting both to zero stops the deployment and stops billing. Bounds without a metric use inflight\_requests with a target of 8.

## FAQ

### Why can GPU utilization look healthy while inference latency worsens?

GPU utilization measures arithmetic activity rather than queue pressure, so a GPU can show 60% utilization while requests are already backing up. Scaling on that signal may respond to a number that does not describe the latency problem.

### Why must scale-up and scale-down windows be tuned differently?

A short scale-up window helps respond before a traffic peak causes user-facing latency, and a false scale-up costs only a few replica-minutes. A longer scale-down window avoids removing capacity just before the next peak, when restarting a replica would add a cold start.

### Which autoscaling metrics require streaming traffic?

TTFT, decoding\_speed, and throughput\_per\_replica are emitted only on the streaming path, so they provide no data for non-streaming clients. e2e\_latency is the exception and is measured for both streaming and non-streaming workloads.

**Tags:** [Autoscaling](https://yomu.fyi/topic/autoscaling), [GPU](https://yomu.fyi/topic/gpu), [LLMs](https://yomu.fyi/topic/llm), [Model Serving](https://yomu.fyi/topic/model-serving)

- Source: [Together AI](https://www.together.ai/blog/autoscaling-endpoints-for-llm-inference)
- Source URL: https://www.together.ai/blog/autoscaling-endpoints-for-llm-inference
- Ingested by Yomu: 2026-09-19T10:13:44.170Z

[Read original post](https://www.together.ai/blog/autoscaling-endpoints-for-llm-inference)