Red HatHarshal PatilTutorial
A day in the life of an NVIDIA H100 on Red Hat OpenShift 5
Summary
Dynamic resource allocation (DRA) in Red Hat OpenShift 5 changes how Kubernetes schedules GPUs by publishing each device as an API object with attributes and capacity. The scheduler, not a node-local plugin, matches workloads to devices using ResourceClaims. Six DRA features are demonstrated on a single NVIDIA H100 80 GB card: extended resources, partitionable devices, shared counters, resource health, admin partitions, and binding conditions. Existing manifests requesting nvidia.com/gpu: 1 continue to work unchanged because the DeviceClass maps the extended resource automatically. The DRA driver for NVIDIA GPUs v0.5.1 does not yet support resource health or binding conditions, but the platform side is ready in Red Hat OpenShift 5. Platform teams can support more AI use cases with existing GPU capacity without rewriting manifests or rebooting nodes.
Context
The post illustrates six DRA features in Red Hat OpenShift 5 through a fictional regional insurer’s day with one NVIDIA H100 80 GB card. Three teams (policy assistant, compliance, data science) use the card for different AI workloads. The setup runs on AWS p5.4xlarge with Red Hat OpenShift 5, the NVIDIA GPU operator (device plugin off), DRA driver for NVIDIA GPUs v0.5.1, and vLLM v0.32.0. Two features (resource health, binding conditions) are demonstrated with the upstream reference driver because the NVIDIA driver does not yet support them.
Approach / What changed
Red Hat OpenShift 5 ships six dynamic resource allocation (DRA) features on Kubernetes 1.36. The DRA driver for NVIDIA GPUs publishes every device as an API object with attributes and capacity. Workloads describe their needs in a ResourceClaim, and the scheduler matches them. The driver supports extended resources (nvidia.com/gpu: 1 works unchanged), partitionable devices (MIG profiles requested via CEL selectors), shared counters (memory and compute capacity tracked as a pool), resource health (driver reports device faults), admin partitions (drain a device without rebooting the node), and binding conditions (scheduler waits for fabric-attached devices to be ready).
Takeaways
DRA in Red Hat OpenShift 5 lets the scheduler reason about GPU attributes (memory, multiprocessors, MIG profiles) instead of just a count, enabling finer-grained allocation.
Existing nvidia.com/gpu: 1 manifests work unchanged because the DeviceClass maps the extended resource and the scheduler creates the ResourceClaim automatically.
The DRA driver for NVIDIA GPUs v0.5.1 does not yet support resource health or binding conditions; those features are demonstrated with the upstream reference driver.
FAQ
Do existing GPU manifests need rewriting for DRA?
Yes. When a DeviceClass maps nvidia.com/gpu as an extended resource, the scheduler automatically creates a ResourceClaim and allocates the device through DRA. Existing manifests that request nvidia.com/gpu: 1 continue to work without changes.
Which DRA features are not yet in the NVIDIA driver?
The DRA driver for NVIDIA GPUs v0.5.1 does not yet report resource health or support binding conditions. The post used the upstream reference driver (dra-example-driver) for those two scenes. The platform side of both features is already in Red Hat OpenShift 5, so workloads will get them when the NVIDIA driver catches up.
How does the scheduler prevent overcommitting GPU resources?
The scheduler sees a shared counter set (memory, multiprocessors, copy engines, etc.) for the whole card. It knows that a whole-GPU claim and a 3g.40gb MIG slice cannot both be on the same card because they draw from the same pool. This prevents overcommitment.
Source and freshness
- Source
- developers.redhat.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.