---
title: "Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo"
description: "LLM engine process failures typically trigger cold restarts that reload weights into HBM, compile kernels, and recapture CUDA graphs, leaving surviving workers to absorb displaced traffic for minutes. Shadow engine recovery, a preview feature in NVIDIA Dynamo, keeps a fully initialized standby on the same GPUs as the active engine. GPU Memory Service (GMS) preserves one physical weight copy across engine processes, while the parked shadow precomputes its CUDA context, graphs, communicators, and mappings but defers KV cache materialization. In a GLM-5.2 test, the shadow resumed service in 7.3 seconds after a worker failure, compared with 283 seconds for a cold restart, reducing post-fault p50 TTFT from 23,815 ms to 1,311 ms and improving decode rate from 12 to 46 tok/s/user. The preview does not cover hardware, node, or multi-node failures and currently starts promoted shadows with empty KV caches."
---

# Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Michelle Horton · Aug 25, 2026

**Type:** Problem & solution

## Summary

LLM engine process failures typically trigger cold restarts that reload weights into HBM, compile kernels, and recapture CUDA graphs, leaving surviving workers to absorb displaced traffic for minutes. Shadow engine recovery, a preview feature in NVIDIA Dynamo, keeps a fully initialized standby on the same GPUs as the active engine. GPU Memory Service (GMS) preserves one physical weight copy across engine processes, while the parked shadow precomputes its CUDA context, graphs, communicators, and mappings but defers KV cache materialization. In a GLM-5.2 test, the shadow resumed service in 7.3 seconds after a worker failure, compared with 283 seconds for a cold restart, reducing post-fault p50 TTFT from 23,815 ms to 1,311 ms and improving decode rate from 12 to 46 tok/s/user. The preview does not cover hardware, node, or multi-node failures and currently starts promoted shadows with empty KV caches.

## Context

Recoverable LLM engine faults can lose only the process and its corrupted state while leaving the hardware, drivers, and node healthy, yet replacement engines still require lengthy initialization. Weights are tied to the engine process, and non-transferable NCCL, torch.distributed, and CUDA graph state must otherwise be recreated after every restart.

## Approach / What changed

Shadow engine recovery combines a GMS sidecar, a preinitialized standby engine, and worker-level coordination. GMS keeps weights resident in persistent physical GPU memory and lets multiple engine processes map the same bytes. The shadow initializes communicators, CUDA graphs, mappings, and warm-up work before parking; on promotion, it materializes its KV cache and takes the worker lock.

## Takeaways

- GMS keeps a single physical copy of weights in HBM available across engine-process failures, while each engine maps those pages into its own CUDA address space.
- In the GLM-5.2 two-worker test, shadow recovery restored a second serving worker in 7.3 seconds versus 283 seconds for a cold restart.
- Promoted shadows begin with empty KV caches, causing a slight post-cutover TTFT bump; cache state transfer is still under development.

**Tags:** [Kubernetes](https://yomu.fyi/topic/kubernetes), [LLMs](https://yomu.fyi/topic/llm), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [NVIDIA](https://developer.nvidia.com/blog/restore-llm-inference-capacity-in-seconds-with-shadow-engine-recovery-in-nvidia-dynamo)
- Source URL: https://developer.nvidia.com/blog/restore-llm-inference-capacity-in-seconds-with-shadow-engine-recovery-in-nvidia-dynamo
- Ingested by Yomu: 2026-08-29T01:39:03.290Z

[Read original post](https://developer.nvidia.com/blog/restore-llm-inference-capacity-in-seconds-with-shadow-engine-recovery-in-nvidia-dynamo)