CephFS as a KV Cache Tier: When Restoring Beats Recomputation
A tiered KV cache flow that moves blocks from GPU HBM to DRAM, local NVMe, and CephFS—and shows how to decide when restoring a block beats recomputing it.
Loading…
Topic
1 post about CephFS. Every summary links to the original.
See the idea
A tiered KV cache flow that moves blocks from GPU HBM to DRAM, local NVMe, and CephFS—and shows how to decide when restoring a block beats recomputing it.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Tiered KV cache offloading in Red Hat OpenShift AI 3.5 moves KV cache blocks from GPU HBM to CPU DRAM, local NVMe, or CephFS to avoid recomputation in long-context workloads. On agentic workloads averaging 50K input tokens, offloading improved throughput by up to 5.8x and reduced mean time to first token from 60 s to 10 s for Nemotron-Ultra-253B on H100 GPUs. CPU offload alone delivered 3.3x throughput improvement, while adding NVMe or CephFS nearly doubled that to 5.8x. The benefit depends on whether the KV working set exceeds the CPU tier capacity; when it fits, CPU offload captures most of the gain. The post recommends workload-driven evaluation: characterize storage latency, calculate the restore-versus-recompute crossover, and validate with representative inference traces before production deployment.
The best engineering reads, once a week.