llm-d flow control: Priority queuing for shared GPU inference

Summary

Shared GPU model pools allow platform teams to maximize utilization, but mixed workloads often contend for serving capacity with differing service requirements. Generally available in Red Hat AI Inference 3.5, llm-d flow control introduces explicit admission control prior to backend routing. An Endpoint Picker sits in the request path and evaluates active priority bands from highest to lowest integer values whenever pool saturation is detected. Saturation metrics combine per-replica queue depth and key-value cache pressure to determine when dispatch gates should engage. Once a priority band is admitted, tenant fairness policies distribute dispatch turns, and ordering rules select the next request to forward to a vLLM replica via Envoy.

Takeaways

1. 1

   Flow control decouples admission, which decides when a request can advance, from scheduling, which selects the vLLM backend pod.

2. 2

   The utilization detector aggregates per-replica queue pressure and KV cache pressure into a single normalized pool-saturation value.

3. 3

   Requests are categorized into priority bands by integer value, defaulting missing objectives to priority 0 and marking negative priorities as sheddable during overload.