NVIDIA Recovery is an open-source, workload-level service in NVIDIA DSX OS that automates recovery across a cluster’s distributed training jobs. It detects interruptions from application logs, GPU metrics, and scheduler signals, attributes failures to implicated resources, isolates those resources, and coordinates checkpoint restarts on healthy capacity. The recovery loop has four stages: detect crashes, hangs, and stragglers; attribute the interruption; isolate implicated nodes or stop unproductive retries; and restart or stop through scheduler integration. NVIDIA Recovery integrates with Slurm, including Slurm on Kubernetes through slurm-operator, and can optionally use NVIDIA Attribution for deeper diagnosis. In production monitoring from July 9 to October 9, one training cluster covered nearly 200 jobs per day and recorded about 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds.
Context
Distributed training at cluster scale is disrupted by stalled ranks, unresponsive GPU nodes, and stragglers. Manual recovery requires log analysis, fault isolation, and checkpoint restarts, leaving affected GPUs idle. The post frames workload recovery as a distinct coordination problem beyond node break-fix automation, in-job libraries such as NVRx, and workload managers such as Slurm. It cites an 11-month study of 24,000 A100 GPUs at Meta in which hardware failures interrupted fewer than 1% of jobs but those jobs accounted for 19% of GPU runtime, showing that large, long-running jobs are most exposed.
Approach / What changed
NVIDIA Recovery runs a cluster-level control loop with four stages. The Anomaly Detector combines application logs and scheduler signals to identify crashes, hangs, and stragglers, recognizing PyTorch, NCCL, CUDA, and interconnect failure patterns. The attribute stage classifies the interruption and can use NVIDIA Attribution for deeper diagnosis to inform node exclusion. The Anomaly Remediator excludes implicated nodes or stops unproductive retries, with a watchdog that lets normal scheduling proceed if a recovery decision stalls or fails. Scheduler integration acts through the Advisor API to restart from the last checkpoint or stop the job. Workloads opt in through an sbatch comment, starting in observe mode before action mode.
Takeaways
1
A job spanning 10,000 GPUs has only about a 0.005% chance of completing a day uninterrupted if each GPU has a 99.9% daily success probability, illustrating why rare per-GPU failures compound at cluster scale.
2
NVIDIA Recovery’s four-stage loop detects crashes, hangs, and stragglers; attributes failures; isolates implicated nodes or stops unproductive retries; and restarts from checkpoint or stops through scheduler integration.
3
The service integrates with Slurm, including Slurm on Kubernetes through slurm-operator, and can optionally use NVIDIA Attribution for deeper diagnosis while remaining independently adoptable.
FAQ
How does NVIDIA Recovery decide whether to restart a workload or stop it?
It uses scheduler signals and application logs to classify the interruption and guide recovery actions. The Anomaly Remediator excludes implicated nodes from subsequent restart attempts or stops the workload when further retries would be unproductive. Optional NVIDIA Attribution adds deeper diagnosis to distinguish hardware faults, system-software issues, and application bugs.
What happens if a recovery decision takes too long or fails?
A watchdog service acts as a safety net. It allows normal scheduling to proceed if a recovery decision takes too long or outright fails, preventing workloads from waiting indefinitely.
How can a Slurm workload be opted in without autonomous recovery actions?
Add an sbatch comment directive with the nvidia-recovery environment and mode set to observe. In monitor mode, NVIDIA Recovery publishes recovery recommendations without taking action. After validating them in the Recovery Console, set mode to action and acknowledgeSchedulerActions to true in the Helm values to enable autonomous recovery.
Checkpoint/Restore captures live state so work resumes instead of restarting. This architecture view maps the state surface, the capture mechanism, the store tier, and the restore path.
A side-by-side comparison of ad-hoc, human-run remediation and tag-driven AWS Systems Manager Automation, showing what each costs in audit trail, drift, and time to recover.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.