---
title: "KV Cache"
description: "3 posts about KV Cache, summarised, each linking to the original."
---

# KV Cache
> 3 posts about KV Cache, summarised, each linking to the original.

## Articles

### [Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory](https://yomu.fyi/post/five-ways-nvidia-co-design-maximizes-the-value-of-every-byte-in-the-ai.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Shara Tibken
- Published: Nov 5, 2026
- Type: Explainer

The post explains how NVIDIA co-designs GPU architecture, interconnect, serving software, CPU, and storage to maximize useful work per byte of HBM in an AI factory. NVFP4 4-bit quantization reduces weight and KV cache footprint by up to 3.8x relative to FP16, freeing memory for larger batches and higher concurrency. Wide expert parallelism (Wide-EP) spreads MoE experts across many GPUs via NVLink, reducing redundant weight copies and increasing arithmetic intensity. NVIDIA Dynamo treats KV cache as a fleet-wide resource through KV-aware routing, disaggregated serving, and a tiered KV Cache Runner (KVCR) that preserves context across prefill/decode boundaries. The Vera CPU offloads KV-cache management during tool-call idle gaps, and CMX provides a pod-level flash tier for shared KV cache reuse, together delivering up to 30x throughput per megawatt on Vera Rubin NVL72 versus GB300 NVL72. The five levers combine to measure memory by useful tokens per byte rather than capacity alone.


### [Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5](https://yomu.fyi/post/benchmarking-tiered-kv-cache-offloading-on-h100-gpus-in-openshift-ai-3.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Ramesh Doddaiah, Alberto Perdomo, Nathan Scott, Ashish Kamra, Naina Singh
- Published: Nov 2, 2026
- Type: Benchmark

Tiered KV cache offloading in Red Hat OpenShift AI 3.5 moves KV cache blocks from GPU HBM to CPU DRAM, local NVMe, or CephFS to avoid recomputation in long-context workloads. On agentic workloads averaging 50K input tokens, offloading improved throughput by up to 5.8x and reduced mean time to first token from 60 s to 10 s for Nemotron-Ultra-253B on H100 GPUs. CPU offload alone delivered 3.3x throughput improvement, while adding NVMe or CephFS nearly doubled that to 5.8x. The benefit depends on whether the KV working set exceeds the CPU tier capacity; when it fits, CPU offload captures most of the gain. The post recommends workload-driven evaluation: characterize storage latency, calculate the restore-versus-recompute crossover, and validate with representative inference traces before production deployment.


### [Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang](https://yomu.fyi/post/cross-request-kv-cache-exposure-in-the-glm-5-3-workers-ai-model-and-ou.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Michelle Chen
- Published: Oct 26, 2026
- Type: Incident / postmortem

On September 8, 2026, security researcher Gadi Evron reported a tenant isolation issue in the GLM-5.3 Workers AI model via HackerOne. Between August 27 and September 9, 155 responses to prompts longer than 681,280 tokens contained data from other customers’ requests; no other models were affected. The root cause was a configuration mismatch where the KV cache pool held only 681,280 tokens while the model’s context window was 1,048,576 tokens, triggering a bug in SGLang’s disaggregated prefill/decode mode. This bug left stale cache pages from rejected prompts that decode nodes used for generation. Cloudflare remediated by resizing the pool to hold about 2.04 million tokens and patching the engine to reject over-length requests before any KV transfer, with upstream fixes merged, and they plan to apply formal verification to the KV cache transfer protocol.
