Topic

Checkpoint/Restore

1 post about Checkpoint/Restore. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Modal
How we achieved truly serverless GPUs
GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s.

The best engineering reads, once a week.

Topic

1 post about Checkpoint/Restore. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Modal
How we achieved truly serverless GPUs
GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s.

The best engineering reads, once a week.