Configuring Dedicated Model Inference

Summary

Dedicated Model Inference is organized around three parts: an endpoint, deployments that run a model revision with replicas, and immutable configs that define how it runs. A deployment binds one model revision to one config and an autoscaling policy, while the endpoint provides a stable API-facing name and traffic split. Routing uses each deployment’s weight multiplied by its ready-replica count, so equal weights produce a 25%/75% split when deployments have one and three ready replicas; unavailable replicas contribute no capacity. The configuration workflow uses certified model-and-config profiles, selectable by accelerator, GPU count, optimization target, and topology, while new config revisions preserve tested rollback targets. This capacity-aware design makes scaling, rollouts, A/B tests, shadow traffic, and zero-downtime changes possible, but raw API or SDK deployments must be added to the traffic split before they receive requests.

Takeaways

1. 1

   Equal traffic weights do not guarantee equal traffic shares: routing is proportional to each deployment’s weight multiplied by its ready-replica count.

2. 2

   Latency, throughput, and balanced configs encode different serving trade-offs; latency favors time-to-first-token, while throughput favors aggregate tokens per GPU-hour.

3. 3

   A deployment with no ready replicas contributes zero capacity, and a raw API or SDK deployment receives no traffic until it is explicitly included in the endpoint’s traffic split.

FAQ