Loading post…
ModalCharles FryeProblem & solution
How we achieved truly serverless GPUs
Summary
GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s.
Context
Inference workloads have variable demand, causing low GPU Allocation Utilization (often 10-20%) when capacity is fixed. Naive auto-scaling takes tens of minutes to spin up new replicas, degrading QoS. Modal’s system solves this by reducing startup latency to tens of seconds.
Approach / What changed
Four key ingredients: (1) Cloud buffers: maintain a small buffer of idle GPUs shared across applications, taking instance allocation out of the hot path. (2) Custom filesystem: serve container images lazily from a content-addressed, multi-tier cloud-native cache. (3) CPU checkpoint/restore: fast-forward through CPU-side initialization by directly restoring processes into memory using Linux CRIU. (4) CUDA checkpoint/restore: fast-forward through GPU-side initialization by directly restoring CUDA contexts into memory, including GPU memory and driver state.
Takeaways
Modal reduced inference server replica startup from over 2 kiloseconds to about 50 seconds using four optimizations spanning cloud buffers, custom filesystem, and CPU/GPU checkpoint/restore.
GPU Allocation Utilization for inference is often 10-20% due to spiky demand; serverless GPUs with fast scaling can improve this by tightly matching supply to demand.
CUDA checkpoint/restore allows restoring GPU memory and driver state in seconds, enabling fast cold starts for models up to tens of gigabytes, popular for audio/voice use cases.
FAQ
What is GPU Allocation Utilization?
It is the ratio of GPU-seconds running application code to GPU-seconds paid for. For inference workloads with spiky demand, it is often low (10-20%) because capacity must be over-provisioned to handle peaks.
How does Modal reduce GPU startup latency?
Modal uses four techniques: a buffer of idle GPUs, a custom lazy-loading filesystem, CPU checkpoint/restore via CRIU, and CUDA checkpoint/restore to directly restore GPU contexts. Together they cut startup from kiloseconds to tens of seconds.
What are the limitations of GPU snapshots?
GPU snapshots are currently restricted to a single GPU, so they are most commonly used for models in the few to few tens of gigabytes range, including smaller LLMs and audio/voice models.
Source and freshness
- Source
- modal.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.