---
title: "How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes"
description: "NVIDIA Mission Control 2.5 extends its autonomous recovery engine, previously limited to Slurm-based HPC, to Kubernetes workloads managed via NVIDIA Run:ai for AI factories, coordinating three separate services: the scheduler for workload placement, the job recovery service for failure-context decisions, and the hardware recovery service for node isolation and diagnostics. When a hardware fault occurs, job recovery restarts the workload on healthy nodes while hardware recovery concurrently isolates the suspect GPU. For transient software errors, only job restart occurs without hardware quarantine. Early degradation signals cause proactive node removal before job scheduling. This consistent recovery model across Kubernetes and Slurm reduces MTTR and improves GPU utilization by shifting from reactive manual intervention to proactive self-healing, benefiting both platform teams and AI practitioners."
---

# How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Elizabeth Goodman · Oct 26, 2026

**Type:** Announcement

## Summary

NVIDIA Mission Control 2.5 extends its autonomous recovery engine, previously limited to Slurm-based HPC, to Kubernetes workloads managed via NVIDIA Run:ai for AI factories, coordinating three separate services: the scheduler for workload placement, the job recovery service for failure-context decisions, and the hardware recovery service for node isolation and diagnostics. When a hardware fault occurs, job recovery restarts the workload on healthy nodes while hardware recovery concurrently isolates the suspect GPU. For transient software errors, only job restart occurs without hardware quarantine. Early degradation signals cause proactive node removal before job scheduling. This consistent recovery model across Kubernetes and Slurm reduces MTTR and improves GPU utilization by shifting from reactive manual intervention to proactive self-healing, benefiting both platform teams and AI practitioners.

## Context

Large-scale AI training on thousands of GPUs over days or weeks demands proactive monitoring and automation. Traditional reactive approaches struggle with hardware anomalies, software errors, and resource contention at that scale. Previously, NVIDIA Mission Control’s autonomous recovery engine only supported Slurm-based HPC. Mission Control 2.5 extends this capability to Kubernetes workloads orchestrated by NVIDIA Run:ai, providing a self-healing framework for AI factories.

## Approach / What changed

The engine integrates three coordinated components: the Kubernetes scheduler (NVIDIA Run:ai) for workload placement, the autonomous job recovery service for context-aware restart decisions, and the autonomous hardware recovery service for node isolation and diagnostics. It handles three failure paths: hardware faults trigger parallel job restart and hardware isolation; transient software errors retry without hardware action; early degradation proactively removes nodes before job placement.

## Takeaways

- Mission Control 2.5 extends autonomous recovery to Kubernetes via NVIDIA Run:ai, supporting both Kubernetes and Slurm with a consistent recovery model.
- Separating job recovery from hardware recovery allows workloads to restart without waiting for root-cause analysis, reducing downtime.
- The system distinguishes hardware faults, transient software errors, and early degradation, applying appropriate recovery to minimize unnecessary hardware quarantine.

## FAQ

### Which schedulers does Mission Control 2.5 autonomous recovery support?

It supports Slurm-based HPC and Kubernetes workloads via NVIDIA Run:ai. The engine works with both schedulers using the same three-layer control loop, providing a consistent recovery model across environments.

### How does the system handle a GPU hardware fault during training?

Job recovery restarts the workload on healthy nodes while hardware recovery concurrently isolates and diagnoses the suspect GPU. This parallel approach reduces downtime compared to waiting for root-cause analysis.

### What happens for transient software errors in Mission Control 2.5?

Only job restart occurs without hardware quarantine. The hardware recovery service stays in background monitoring unless new evidence points to a real hardware issue, preserving scheduling pool size.

**Tags:** [AI Factories](https://yomu.fyi/topic/ai-factories), [Autonomous Recovery](https://yomu.fyi/topic/autonomous-recovery), [Kubernetes](https://yomu.fyi/topic/kubernetes), [NVIDIA](https://yomu.fyi/topic/nvidia)

- Source: [NVIDIA](https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes)
- Source URL: https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes
- Ingested by Yomu: 2026-10-26T18:00:15.852Z

[Read original post](https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes)
