---
title: "Checkpoint/Restore: Four Layers Between a Snapshot and a Running Process"
description: "Checkpoint/Restore captures live state so work resumes instead of restarting. This architecture view maps the state surface, the capture mechanism, the store tier, and the restore path."
---

# Checkpoint/Restore: Four Layers Between a Snapshot and a Running Process

**Type:** [Architecture](https://yomu.fyi/visual-explainers/category/architecture.md)

![A four-layer architecture diagram showing checkpoint state surfaces at the top, capture mechanisms, storage tiers, and restore paths below, connected by downward arrows with a decision cue comparing restore cost against recompute cost.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/a4d85411-30e8-4108-be9f-3c16e685b83c/image-1d833c00-1c86-47ef-a284-cb48db1ec758.webp)

> Checkpoint/Restore captures live state so work resumes instead of restarting. This architecture view maps the state surface, the capture mechanism, the store tier, and the restore path.

## Explanation

Checkpoint/Restore is an architecture, not a single command. It has four layers: the state surface you choose to capture, the mechanism that captures it, the tier where it lands, and the path that brings it back. The mechanism determines what is cheap — a CUDA context restore is fast but narrow, a copy-on-write branch is instant but only covers storage, and continuous replication keeps a copy warm at the cost of ongoing traffic. The restore path is where the real decision lives: restoring is only worth it when it costs less than recomputing the work from scratch.

## FAQ

### What does a checkpoint actually capture at the process and GPU level?

It can span more than application data. Modal reduced replica startup latency by using CPU and CUDA checkpoint/restore to fast-forward through initialization, directly restoring processes and GPU contexts into memory rather than replaying startup. That is narrower than a full machine snapshot but enough to skip expensive initialization.

**Tags:** [Autonomous Recovery](https://yomu.fyi/topic/autonomous-recovery.md), [Checkpoint/Restore](https://yomu.fyi/topic/checkpoint-restore.md), [Copy-on-Write](https://yomu.fyi/topic/copy-on-write.md), [Disaster Recovery](https://yomu.fyi/topic/disaster-recovery.md), [KV Cache](https://yomu.fyi/topic/kv-cache.md)

## Related reading

### [Keep Distributed Training Running with Automated Workload Recovery](https://yomu.fyi/post/keep-distributed-training-running-with-automated-workload-recovery.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Nov 2, 2026
- Type: Announcement

NVIDIA Recovery is an open-source, workload-level service in NVIDIA DSX OS that automates recovery across a cluster’s distributed training jobs. It detects interruptions from application logs, GPU metrics, and scheduler signals, attributes failures to implicated resources, isolates those resources, and coordinates checkpoint restarts on healthy capacity. The recovery loop has four stages: detect crashes, hangs, and stragglers; attribute the interruption; isolate implicated nodes or stop unproductive retries; and restart or stop through scheduler integration. NVIDIA Recovery integrates with Slurm, including Slurm on Kubernetes through slurm-operator, and can optionally use NVIDIA Attribution for deeper diagnosis. In production monitoring from July 9 to October 9, one training cluster covered nearly 200 jobs per day and recorded about 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds.


### [Automating the full AWS DRS agent lifecycle: Install, heal, and offboard](https://yomu.fyi/post/automating-the-full-aws-drs-agent-lifecycle-install-heal-and-offboard.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Manisha Yadav
- Published: Nov 2, 2026
- Type: Problem & solution

An enterprise customer with more than 300 Linux and Windows instances in AWS Elastic Disaster Recovery scope had agents stall after network changes and descoped servers keep replicating and adding cost. The solution automates onboarding, healing, and offboarding through AWS Systems Manager Automation documents started by EventBridge or a schedule. A single EC2 DR tag with values yes, pause, or no selects the phase; unrecognized values are normalized, skipped, and logged as SKIPPED\_MISCONFIGURED. Onboarding runs every 30 minutes through State Manager, healing restarts or reinstalls the agent for agent-side stalls and escalates infrastructure stalls through Amazon SNS, and offboarding checks safeguards before stopping replication, deleting the source server, uninstalling the agent, and removing tags. Offboarding records whether replication billing stopped, while the documents emit SNS notifications, per-event JSON audit reports to Amazon S3, and CloudWatch Outcome metrics.


### [Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5](https://yomu.fyi/post/benchmarking-tiered-kv-cache-offloading-on-h100-gpus-in-openshift-ai-3.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Ramesh Doddaiah, Alberto Perdomo, Nathan Scott, Ashish Kamra, Naina Singh
- Published: Nov 2, 2026
- Type: Benchmark

Tiered KV cache offloading in Red Hat OpenShift AI 3.5 moves KV cache blocks from GPU HBM to CPU DRAM, local NVMe, or CephFS to avoid recomputation in long-context workloads. On agentic workloads averaging 50K input tokens, offloading improved throughput by up to 5.8x and reduced mean time to first token from 60 s to 10 s for Nemotron-Ultra-253B on H100 GPUs. CPU offload alone delivered 3.3x throughput improvement, while adding NVMe or CephFS nearly doubled that to 5.8x. The benefit depends on whether the KV working set exceeds the CPU tier capacity; when it fits, CPU offload captures most of the gain. The post recommends workload-driven evaluation: characterize storage latency, calculate the restore-versus-recompute crossover, and validate with representative inference traces before production deployment.


### [Lakebase branching for CI/CD: lessons from 100 PR databases](https://yomu.fyi/post/lakebase-branching-for-ci-cd-lessons-from-100-pr-databases.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Yasamin Tari, Kartik Mathur
- Published: Nov 1, 2026
- Type: Problem & solution

Snorkel AI needed each pull request in its CI pipeline to have an isolated copy of production data for realistic testing, but restoring a full database copy took about 20 minutes per environment. They adopted Databricks’ Lakebase Postgres, whose copy-on-write branching creates writable branches in seconds regardless of database size because no data is copied at branch creation. AWS DMS continuously replicates the production Aurora cluster into Lakebase’s main branch, keeping it within about two minutes of the source. For every PR, the CI pipeline creates a branch via API, runs tests, and either deletes the branch or relies on automatic TTL expiration to clean up. Provisioning time dropped from over 30 minutes to under 60 seconds at p95, supporting about 100 concurrent PR branches on a 2.5 TB dataset with storage growing only for changes written.


### [How we achieved truly serverless GPUs](https://yomu.fyi/post/how-we-achieved-truly-serverless-gpus.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: May 12, 2026
- Type: Problem & solution

GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s.
