---
title: "llm-d flow control: Priority queuing for shared GPU inference"
description: "Shared GPU model pools allow platform teams to maximize utilization, but mixed workloads often contend for serving capacity with differing service requirements. Generally available in Red Hat AI Inference 3.5, llm-d flow control introduces explicit admission control prior to backend routing. An Endpoint Picker sits in the request path and evaluates active priority bands from highest to lowest integer values whenever pool saturation is detected. Saturation metrics combine per-replica queue depth and key-value cache pressure to determine when dispatch gates should engage. Once a priority band is admitted, tenant fairness policies distribute dispatch turns, and ordering rules select the next request to forward to a vLLM replica via Envoy."
---

# llm-d flow control: Priority queuing for shared GPU inference

[Red Hat](https://yomu.fyi/company/red-hat) · Alexa Griffith, Rishabh Saini · Aug 27, 2026

**Type:** Announcement

## Summary

Shared GPU model pools allow platform teams to maximize utilization, but mixed workloads often contend for serving capacity with differing service requirements. Generally available in Red Hat AI Inference 3.5, llm-d flow control introduces explicit admission control prior to backend routing. An Endpoint Picker sits in the request path and evaluates active priority bands from highest to lowest integer values whenever pool saturation is detected. Saturation metrics combine per-replica queue depth and key-value cache pressure to determine when dispatch gates should engage. Once a priority band is admitted, tenant fairness policies distribute dispatch turns, and ordering rules select the next request to forward to a vLLM replica via Envoy.

## Context

Dedicated GPU pools provisioned for peak traffic result in expensive idle capacity, while sharing a single model pool across diverse tenants leads to resource contention and unmanaged queue delays across different service guarantees.

## Approach / What changed

llm-d flow control introduces an admission gate in the Endpoint Picker prior to scheduling, using pool-wide saturation metrics combining queue and key-value cache pressure alongside priority bands, tenant fairness policies, and ordering rules to govern request dispatch.

## Takeaways

- Flow control decouples admission, which decides when a request can advance, from scheduling, which selects the vLLM backend pod.
- The utilization detector aggregates per-replica queue pressure and KV cache pressure into a single normalized pool-saturation value.
- Requests are categorized into priority bands by integer value, defaulting missing objectives to priority 0 and marking negative priorities as sheddable during overload.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/08/27/llm-d-flow-control-priority-queuing-for-shared-gpu-inference)
- Source URL: https://developers.redhat.com/articles/2026/08/27/llm-d-flow-control-priority-queuing-for-shared-gpu-inference
- Ingested by Yomu: 2026-08-27T14:55:39.039Z

[Read original post](https://developers.redhat.com/articles/2026/08/27/llm-d-flow-control-priority-queuing-for-shared-gpu-inference)