---
title: "CephFS as a KV Cache Tier: When Restoring Beats Recomputation"
description: "A tiered KV cache flow that moves blocks from GPU HBM to DRAM, local NVMe, and CephFS—and shows how to decide when restoring a block beats recomputing it."
---

# CephFS as a KV Cache Tier: When Restoring Beats Recomputation

**Type:** [How it works](https://yomu.fyi/visual-explainers/category/how-it-works.md)

![Numbered left-to-right diagram of a four-tier KV cache ladder: GPU HBM, CPU DRAM, local NVMe, and CephFS, with offload and restore arrows, plus a decision panel comparing CPU-only offload with adding NVMe or CephFS.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/5a5f6729-aa4b-4118-a01f-ff4cc90ce70f/image-0853c241-a714-4363-be55-cf76f3622ca9.webp)

> A tiered KV cache flow that moves blocks from GPU HBM to DRAM, local NVMe, and CephFS—and shows how to decide when restoring a block beats recomputing it.

## Explanation

A KV cache tier ladder extends GPU memory with CPU DRAM, local NVMe, and shared CephFS so long-context requests do not recompute attention blocks they already produced. Each tier adds capacity but also adds restore latency, so the useful question is where the restore-versus-recompute crossover sits for a given workload. On agentic workloads averaging 50K input tokens, CPU offload alone raised throughput 3.3x, and adding NVMe or CephFS nearly doubled that again while cutting mean time to first token from 60 seconds to 10. The explainer walks the flow and the checks—storage latency, working-set size, and representative traces—that decide which tiers earn their place.

## FAQ

### Why add NVMe or CephFS if CPU DRAM offload already delivers a gain?

CPU offload alone gave 3.3x throughput on agentic workloads averaging 50K input tokens, but it only helps while the KV working set fits in DRAM. Adding NVMe or CephFS nearly doubled throughput to 5.8x by holding blocks that would otherwise be recomputed. The benefit depends on the working set exceeding the CPU tier capacity.

### How do I decide between restoring a cached block and recomputing it?

Characterize storage latency, then calculate the restore-versus-recompute crossover for your workload and validate it with representative inference traces. The source material recommends this workload-driven evaluation before production deployment rather than assuming offload always wins.

### Does the path from GPU to CephFS need special validation?

Moving data between GPUs and storage is not the same as an ordinary file read; GPUDirect Storage exists to give a direct data path that reduces latency and CPU overhead, and the gds-diag tool audits a platform's configuration before you depend on it. Without that check, a slow tier can cost more than the recomputation it replaces.

**Tags:** [CephFS](https://yomu.fyi/topic/cephfs.md), [Filesystems](https://yomu.fyi/topic/filesystems.md), [KV Cache](https://yomu.fyi/topic/kv-cache.md), [Latency](https://yomu.fyi/topic/latency.md), [NVMe](https://yomu.fyi/topic/nvme.md), [Storage](https://yomu.fyi/topic/storage.md)

## Related reading

### [Lakebase Postgres is now generally available on Google Cloud](https://yomu.fyi/post/lakebase-postgres-is-now-generally-available-on-google-cloud.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Cassie Murray
- Published: Nov 2, 2026
- Type: Announcement

Lakebase Postgres is now generally available on Google Cloud, giving Databricks users a fully managed, serverless Postgres option for production and agentic workloads. Its architecture separates compute from storage and shares the same storage layer as analytical data, which removes pipeline maintenance and data duplication. The service offers serverless autoscaling with scale to zero, instant zero-copy branching, point-in-time recovery, standard Postgres tooling, pgvector-compatible search, and governance through Unity Catalog. On Google Cloud it provides multi-zone high availability with automatic failover and a 99.99% SLA, single-zone endpoints with a 99.9% SLA, beta private connectivity through Private Service Connect, and Databricks OAuth or native Postgres credentials. Lakebase Search is described as delivering twice the throughput of pgvector at four times lower cost, while direct writes can load 1TB in under five minutes; GA regions include asia-southeast1, europe-west3, us-central1, us-east4, and us-west1.


### [Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5](https://yomu.fyi/post/benchmarking-tiered-kv-cache-offloading-on-h100-gpus-in-openshift-ai-3.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Ramesh Doddaiah, Alberto Perdomo, Nathan Scott, Ashish Kamra, Naina Singh
- Published: Nov 2, 2026
- Type: Benchmark

Tiered KV cache offloading in Red Hat OpenShift AI 3.5 moves KV cache blocks from GPU HBM to CPU DRAM, local NVMe, or CephFS to avoid recomputation in long-context workloads. On agentic workloads averaging 50K input tokens, offloading improved throughput by up to 5.8x and reduced mean time to first token from 60 s to 10 s for Nemotron-Ultra-253B on H100 GPUs. CPU offload alone delivered 3.3x throughput improvement, while adding NVMe or CephFS nearly doubled that to 5.8x. The benefit depends on whether the KV working set exceeds the CPU tier capacity; when it fits, CPU offload captures most of the gain. The post recommends workload-driven evaluation: characterize storage latency, calculate the restore-versus-recompute crossover, and validate with representative inference traces before production deployment.


### [Lakebase branching for CI/CD: lessons from 100 PR databases](https://yomu.fyi/post/lakebase-branching-for-ci-cd-lessons-from-100-pr-databases.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Yasamin Tari, Kartik Mathur
- Published: Nov 1, 2026
- Type: Problem & solution

Snorkel AI needed each pull request in its CI pipeline to have an isolated copy of production data for realistic testing, but restoring a full database copy took about 20 minutes per environment. They adopted Databricks’ Lakebase Postgres, whose copy-on-write branching creates writable branches in seconds regardless of database size because no data is copied at branch creation. AWS DMS continuously replicates the production Aurora cluster into Lakebase’s main branch, keeping it within about two minutes of the source. For every PR, the CI pipeline creates a branch via API, runs tests, and either deletes the branch or relies on automatic TTL expiration to clean up. Provisioning time dropped from over 30 minutes to under 60 seconds at p95, supporting about 100 concurrent PR branches on a 2.5 TB dataset with storage growing only for changes written.


### [Build a Validated Kubernetes Runtime for NVIDIA Dynamo](https://yomu.fyi/post/build-a-validated-kubernetes-runtime-for-nvidia-dynamo.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Stefan Schimanski
- Published: Oct 30, 2026
- Type: Tutorial

Deploying NVIDIA Dynamo on Kubernetes needs a validated runtime layer including GPU drivers, scheduling, networking, and monitoring. NVIDIA AI Cluster Runtime (AICR) packages Dynamo and its infrastructure into validated recipes with pinned component versions and compatibility constraints. AICR supports two routing paths: the Dynamo Frontend router for direct worker selection based on KV-cache overlap and load, and the Gateway API Inference Extension with Dynamo Endpoint Picker for Kubernetes-native ingress. The walkthrough details deploying and validating the runtime using AICR’s Helmfile or GitOps bundles, with validation checks covering cluster baseline, platform health, accelerator access, scheduling, observability, and ingress. AICR ensures a repeatable foundation for serving models, with validation phases that must pass before application workloads are installed.


### [How to Check and Configure NVIDIA GPUDirect Storage for Your Platform](https://yomu.fyi/post/how-to-check-and-configure-nvidia-gpudirect-storage-for-your-platform.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Phil Carns
- Published: Oct 30, 2026
- Type: Tutorial

NVIDIA GPUDirect Storage (GDS) accelerates GPU storage access by providing a direct data path between GPUs and storage, reducing latency and CPU overhead. The gds-diag tool, an open-source command-line utility distributed via the NVIDIA Magnum IO GitHub repository, simplifies configuring and optimizing GDS deployments by assessing environments, identifying problems, and suggesting improvements. It includes an optional AI agent skill that adds a natural-language interface and orchestration layer for multi-step diagnostics, working with coding assistants like Claude and Codex. The tool supports bare-metal, virtual machine, and container environments, and its subcommands include pre-install, post-install, config-audit, mount-check, and container-check, with each finding having a status badge (FAIL, WARN, INFO, PASS) and a mitigation plan. It requires Python 3.8 or newer, does not modify the system, and can output findings as structured JSON, while the support-matrix subcommand shows supported platform combinations with color-coded compatibility.
