Loading…
Topic
NVIDIA H100
2 posts about NVIDIA H100. Every summary links to the original.
- 2 posts
- 1 company
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
A day in the life of an NVIDIA H100 on Red Hat OpenShift 5
Dynamic resource allocation (DRA) in Red Hat OpenShift 5 changes how Kubernetes schedules GPUs by publishing each device as an API object with attributes and capacity. The scheduler, not a node-local plugin, matches workloads to devices using ResourceClaims. Six DRA features are demonstrated on a single NVIDIA H100 80 GB card: extended resources, partitionable devices, shared counters, resource health, admin partitions, and binding conditions. Existing manifests requesting nvidia.com/gpu: 1 continue to work unchanged because the DeviceClass maps the extended resource automatically. The DRA driver for NVIDIA GPUs v0.5.1 does not yet support resource health or binding conditions, but the platform side is ready in Red Hat OpenShift 5. Platform teams can support more AI use cases with existing GPU capacity without rewriting manifests or rebooting nodes.
Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5
Tiered KV cache offloading in Red Hat OpenShift AI 3.5 moves KV cache blocks from GPU HBM to CPU DRAM, local NVMe, or CephFS to avoid recomputation in long-context workloads. On agentic workloads averaging 50K input tokens, offloading improved throughput by up to 5.8x and reduced mean time to first token from 60 s to 10 s for Nemotron-Ultra-253B on H100 GPUs. CPU offload alone delivered 3.3x throughput improvement, while adding NVMe or CephFS nearly doubled that to 5.8x. The benefit depends on whether the KV working set exceeds the CPU tier capacity; when it fits, CPU offload captures most of the gain. The post recommends workload-driven evaluation: characterize storage latency, calculate the restore-versus-recompute crossover, and validate with representative inference traces before production deployment.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.