Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo

Summary

LLM engine process failures typically trigger cold restarts that reload weights into HBM, compile kernels, and recapture CUDA graphs, leaving surviving workers to absorb displaced traffic for minutes. Shadow engine recovery, a preview feature in NVIDIA Dynamo, keeps a fully initialized standby on the same GPUs as the active engine. GPU Memory Service (GMS) preserves one physical weight copy across engine processes, while the parked shadow precomputes its CUDA context, graphs, communicators, and mappings but defers KV cache materialization. In a GLM-5.2 test, the shadow resumed service in 7.3 seconds after a worker failure, compared with 283 seconds for a cold restart, reducing post-fault p50 TTFT from 23,815 ms to 1,311 ms and improving decode rate from 12 to 46 tok/s/user. The preview does not cover hardware, node, or multi-node failures and currently starts promoted shadows with empty KV caches.

Takeaways

1. 1

   GMS keeps a single physical copy of weights in HBM available across engine-process failures, while each engine maps those pages into its own CUDA address space.

2. 2

   In the GLM-5.2 two-worker test, shadow recovery restored a second serving worker in 7.3 seconds versus 283 seconds for a cold restart.

3. 3

   Promoted shadows begin with empty KV caches, causing a slight post-cutover TTFT bump; cache state transfer is still under development.