CephFS as a KV Cache Tier: When Restoring Beats Recomputation
A tiered KV cache flow that moves blocks from GPU HBM to DRAM, local NVMe, and CephFS—and shows how to decide when restoring a block beats recomputing it.
NVIDIAStefan SchimanskiTutorial
Deploying NVIDIA Dynamo on Kubernetes needs a validated runtime layer including GPU drivers, scheduling, networking, and monitoring. NVIDIA AI Cluster Runtime (AICR) packages Dynamo and its infrastructure into validated recipes with pinned component versions and compatibility constraints. AICR supports two routing paths: the Dynamo Frontend router for direct worker selection based on KV-cache overlap and load, and the Gateway API Inference Extension with Dynamo Endpoint Picker for Kubernetes-native ingress. The walkthrough details deploying and validating the runtime using AICR’s Helmfile or GitOps bundles, with validation checks covering cluster baseline, platform health, accelerator access, scheduling, observability, and ingress. AICR ensures a repeatable foundation for serving models, with validation phases that must pass before application workloads are installed.
NVIDIA Dynamo is a distributed inference framework for coordinating model serving across GPU workers. Deploying it on Kubernetes requires a validated runtime layer including GPU drivers, scheduling, networking, and monitoring. NVIDIA AI Cluster Runtime (AICR) packages Dynamo and its supporting infrastructure into validated recipes with pinned component versions and compatibility constraints.
AICR provides validated recipes that package NVIDIA Dynamo and its supporting infrastructure components (GPU Operator, Node Feature Discovery, DRA Driver, KAI Scheduler, Grove, NVSentinel, Prometheus, agentgateway) with pinned versions and compatibility constraints. Users generate deployment bundles via Helmfile or GitOps (Argo CD, Flux) and run validation phases (deployment, conformance, performance) before installing application workloads.
AICR packages NVIDIA Dynamo and runtime components into validated recipes with pinned versions and compatibility constraints.
AICR supports two routing paths: Dynamo Frontend router for direct worker selection and Gateway API Inference Extension with Dynamo Endpoint Picker for Kubernetes-native ingress.
AICR validation includes deployment and conformance checks covering cluster baseline, platform health, accelerator access, scheduling, observability, and ingress, and must pass before application workloads are installed.
Dynamo supports two routing paths: the Dynamo Frontend router for direct worker selection based on KV-cache overlap and load, and the Gateway API Inference Extension with Dynamo Endpoint Picker for Kubernetes-native ingress.
AICR validation checks cluster baseline, platform health, accelerator access, scheduling and scaling, observability, and ingress readiness. It runs deployment and conformance phases before application workloads are installed.
AICR encodes component dependencies in a generated Helmfile bundle. Helmfile processes dependency tiers in order, applying independent components concurrently. Rerunning apply reconciles installed releases with generated configuration.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A tiered KV cache flow that moves blocks from GPU HBM to DRAM, local NVMe, and CephFS—and shows how to decide when restoring a block beats recomputing it.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.