---
title: "How we achieved truly serverless GPUs"
description: "GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s."
---

# How we achieved truly serverless GPUs

[Modal](https://yomu.fyi/company/modal) · Charles Frye · May 12, 2026

**Type:** Problem & solution

## Summary

GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s.

## Context

Inference workloads have variable demand, causing low GPU Allocation Utilization (often 10-20%) when capacity is fixed. Naive auto-scaling takes tens of minutes to spin up new replicas, degrading QoS. Modal’s system solves this by reducing startup latency to tens of seconds.

## Approach / What changed

Four key ingredients: (1) Cloud buffers: maintain a small buffer of idle GPUs shared across applications, taking instance allocation out of the hot path. (2) Custom filesystem: serve container images lazily from a content-addressed, multi-tier cloud-native cache. (3) CPU checkpoint/restore: fast-forward through CPU-side initialization by directly restoring processes into memory using Linux CRIU. (4) CUDA checkpoint/restore: fast-forward through GPU-side initialization by directly restoring CUDA contexts into memory, including GPU memory and driver state.

## Takeaways

- Modal reduced inference server replica startup from over 2 kiloseconds to about 50 seconds using four optimizations spanning cloud buffers, custom filesystem, and CPU/GPU checkpoint/restore.
- GPU Allocation Utilization for inference is often 10-20% due to spiky demand; serverless GPUs with fast scaling can improve this by tightly matching supply to demand.
- CUDA checkpoint/restore allows restoring GPU memory and driver state in seconds, enabling fast cold starts for models up to tens of gigabytes, popular for audio/voice use cases.

## FAQ

### What is GPU Allocation Utilization?

It is the ratio of GPU-seconds running application code to GPU-seconds paid for. For inference workloads with spiky demand, it is often low (10-20%) because capacity must be over-provisioned to handle peaks.

### How does Modal reduce GPU startup latency?

Modal uses four techniques: a buffer of idle GPUs, a custom lazy-loading filesystem, CPU checkpoint/restore via CRIU, and CUDA checkpoint/restore to directly restore GPU contexts. Together they cut startup from kiloseconds to tens of seconds.

### What are the limitations of GPU snapshots?

GPU snapshots are currently restricted to a single GPU, so they are most commonly used for models in the few to few tens of gigabytes range, including smaller LLMs and audio/voice models.

**Tags:** [Checkpoint/Restore](https://yomu.fyi/topic/checkpoint-restore), [CUDA](https://yomu.fyi/topic/cuda), [GPU](https://yomu.fyi/topic/gpu), [Inference](https://yomu.fyi/topic/inference), [Serverless](https://yomu.fyi/topic/serverless)

- Source: [Modal](https://modal.com/blog/truly-serverless-gpus)
- Source URL: https://modal.com/blog/truly-serverless-gpus
- Ingested by Yomu: 2026-09-20T07:46:48.641Z

[Read original post](https://modal.com/blog/truly-serverless-gpus)