---
title: "Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5"
description: "Tiered KV cache offloading in Red Hat OpenShift AI 3.5 moves KV cache blocks from GPU HBM to CPU DRAM, local NVMe, or CephFS to avoid recomputation in long-context workloads. On agentic workloads averaging 50K input tokens, offloading improved throughput by up to 5.8x and reduced mean time to first token from 60 s to 10 s for Nemotron-Ultra-253B on H100 GPUs. CPU offload alone delivered 3.3x throughput improvement, while adding NVMe or CephFS nearly doubled that to 5.8x. The benefit depends on whether the KV working set exceeds the CPU tier capacity; when it fits, CPU offload captures most of the gain. The post recommends workload-driven evaluation: characterize storage latency, calculate the restore-versus-recompute crossover, and validate with representative inference traces before production deployment."
---

# Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5

[Red Hat](https://yomu.fyi/company/red-hat) · Ramesh Doddaiah, Alberto Perdomo, Nathan Scott, Ashish Kamra, Naina Singh · Nov 2, 2026

**Type:** Benchmark

## Summary

Tiered KV cache offloading in Red Hat OpenShift AI 3.5 moves KV cache blocks from GPU HBM to CPU DRAM, local NVMe, or CephFS to avoid recomputation in long-context workloads. On agentic workloads averaging 50K input tokens, offloading improved throughput by up to 5.8x and reduced mean time to first token from 60 s to 10 s for Nemotron-Ultra-253B on H100 GPUs. CPU offload alone delivered 3.3x throughput improvement, while adding NVMe or CephFS nearly doubled that to 5.8x. The benefit depends on whether the KV working set exceeds the CPU tier capacity; when it fits, CPU offload captures most of the gain. The post recommends workload-driven evaluation: characterize storage latency, calculate the restore-versus-recompute crossover, and validate with representative inference traces before production deployment.

## Context

Long-context workloads in enterprise AI, such as agentic applications, RAG, and multi-turn assistants, repeatedly process the same prompt prefixes. The KV cache, which enables reuse, resides in GPU memory, which is limited. When GPU memory is exhausted, the KV cache is discarded and recomputed, increasing latency and reducing throughput. Tiered offloading moves KV cache blocks from GPU HBM to cheaper, higher-capacity storage tiers to avoid recomputation.

## Approach / What changed

The post evaluates tiered KV cache offloading in Red Hat OpenShift AI 3.5 using vLLM’s file system offload connector. It tests three tiers: CPU DRAM, local NVMe, and CephFS. The evaluation starts with raw storage latency characterization using FIO, translates latency into a break-even context length, and then measures throughput and latency on representative agentic workloads with three models on NVIDIA H100 GPUs.

## Takeaways

- Tiered KV cache offloading improved throughput by up to 5.8x and reduced mean TTFT from 60 s to 10 s for Nemotron-Ultra-253B on H100 GPUs with 50K-token agentic workloads.
- CPU offload alone delivered 3.3x throughput improvement; adding NVMe or CephFS nearly doubled that to 5.8x when the KV working set exceeded CPU tier capacity.
- CephFS, despite higher per-block latency, sustained cache reuse as concurrency increased in multi-replica tests, while CPU-only configurations increasingly fell back to recomputation.

## FAQ

### How is the break-even context length determined?

The break-even context length is the point at which restoring a KV block from a storage tier is faster than recomputing that block's prefix on the GPU. It depends on storage tail latency, model block size, and concurrency. The post recommends calculating this crossover point to determine if offloading will reduce latency.

### What are the trade-offs between CPU, NVMe, and CephFS tiers?

CPU DRAM provides the first and most efficient extension beyond GPU HBM. Local NVMe delivers the lowest secondary-tier latency and suits single-node deployments. CephFS, a shared distributed file system, introduces higher per-block latency but enables cross-replica cache reuse, which becomes valuable in multi-node deployments and under high concurrency.

### How does the CPU tier interact with secondary tiers?

The CPU tier is privileged and acts as a mandatory staging layer between GPU and secondary tiers. Secondary tiers do not access GPU memory directly. On store, a chunk is first copied from GPU to CPU, then cascaded to secondary tiers. On load, a secondary-tier hit is promoted to CPU before being copied to GPU blocks.

**Tags:** [CephFS](https://yomu.fyi/topic/cephfs), [KV Cache](https://yomu.fyi/topic/kv-cache), [NVIDIA H100](https://yomu.fyi/topic/nvidia-h100), [OpenShift AI](https://yomu.fyi/topic/openshift-ai), [vLLM](https://yomu.fyi/topic/vllm)

- Source: [Red Hat](https://developers.redhat.com/articles/2026/11/02/benchmarking-tiered-kv-cache-offloading-h100-gpus-openshift-ai-3-5)
- Source URL: https://developers.redhat.com/articles/2026/11/02/benchmarking-tiered-kv-cache-offloading-h100-gpus-openshift-ai-3-5
- Ingested by Yomu: 2026-11-02T15:00:17.262Z

[Read original post](https://developers.redhat.com/articles/2026/11/02/benchmarking-tiered-kv-cache-offloading-h100-gpus-openshift-ai-3-5)
