---
title: "How Autonomous Recovery Works: One Loop, Two Recovery Tracks"
description: "Autonomous Recovery is a closed loop that senses degradation, classifies a fault as transient or hardware, recovers workloads and quarantines hardware in parallel, then verifies against an RTO."
---

# How Autonomous Recovery Works: One Loop, Two Recovery Tracks

**Type:** [How it works](https://yomu.fyi/visual-explainers/category/how-it-works.md)

![A five-stage left-to-right diagram of autonomous recovery: detect and correlate, classify the fault, two parallel recovery tracks for workload and hardware, verify against an RTO, and harden the root cause.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/10886285-23c0-43c8-9080-5df26a30d899/image-bd259908-b5d7-4def-a4dd-adc37c9adcb9.webp)

> Autonomous Recovery is a closed loop that senses degradation, classifies a fault as transient or hardware, recovers workloads and quarantines hardware in parallel, then verifies against an RTO.

## Explanation

Autonomous recovery replaces manual intervention with a loop: telemetry and topology are correlated into one incident, the fault is classified as transient or hardware, and recovery runs on two tracks at once. The workload track restarts or reschedules the job on healthy nodes while the hardware track isolates and diagnoses the suspect device; a transient software error triggers only the restart. Verification scores each recovery against success criteria and a recovery time objective, and hardening fixes the root cause so the failure class cannot recur. The payoff shows up as lower mean time to recovery and higher GPU utilization, not as clever automation.

## FAQ

### Why restart the job and quarantine the GPU at the same time instead of in sequence?

The two actions answer different questions. Job recovery decides where the workload should run next, while hardware recovery isolates and diagnoses the suspect device. Running them concurrently is what shortens mean time to recovery, and for a transient software error only the restart is needed, so the node is never quarantined unnecessarily.

### How do you know an automated recovery actually worked?

You score it. Recovery runs should be evaluated against explicit success criteria and the recovery time objective in a resilience policy, producing a pass or fail outcome plus a point-in-time snapshot of resources. Fault injection can generate those experiments automatically instead of leaving test design to a human.

### What breaks autonomous recovery at scale?

Noise and misclassification. Static thresholds miss gray failures such as a GPU running hotter than its peers, and uncorrelated events bury operators, so a correlation layer that collapses related events into one incident matters. Misclassified dependencies and configuration mismatches are the other trap, which is why hardening the root cause closes the loop.

**Tags:** [Autonomous Recovery](https://yomu.fyi/topic/autonomous-recovery.md), [Disaster Recovery](https://yomu.fyi/topic/disaster-recovery.md), [Incident Response](https://yomu.fyi/topic/incident-response.md), [Monitoring](https://yomu.fyi/topic/monitoring.md), [Reliability](https://yomu.fyi/topic/reliability.md), [Social Listening](https://yomu.fyi/topic/social-listening.md)

## Related reading

### [Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang](https://yomu.fyi/post/cross-request-kv-cache-exposure-in-the-glm-5-3-workers-ai-model-and-ou.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Michelle Chen
- Published: Oct 26, 2026
- Type: Incident / postmortem

On September 8, 2026, security researcher Gadi Evron reported a tenant isolation issue in the GLM-5.3 Workers AI model via HackerOne. Between August 27 and September 9, 155 responses to prompts longer than 681,280 tokens contained data from other customers’ requests; no other models were affected. The root cause was a configuration mismatch where the KV cache pool held only 681,280 tokens while the model’s context window was 1,048,576 tokens, triggering a bug in SGLang’s disaggregated prefill/decode mode. This bug left stale cache pages from rejected prompts that decode nodes used for generation. Cloudflare remediated by resizing the pool to hold about 2.04 million tokens and patching the engine to reject over-length requests before any KV transfer, with upstream fixes merged, and they plan to apply formal verification to the KV cache transfer protocol.


### [Recommended resilience testing in the next generation of AWS Resilience Hub](https://yomu.fyi/post/recommended-resilience-testing-in-the-next-generation-of-aws-resilienc.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Jennifer Moran
- Published: Oct 26, 2026
- Type: Announcement

AWS Resilience Hub introduced recommended resilience testing, which uses AWS Fault Injection Service to automatically create fault injection experiments from existing policies and resource knowledge. The feature offers four pre-configured tests: Availability Zone recovery, dependency validation, Multi-Region isolation, and Multi-Region recovery. Dependency validation blocks connectivity to selected dependencies using DNS query log analysis with a 35-day look-back window, and validates whether the application remains healthy when they are unavailable, surfacing misclassified dependencies. Each test run produces a pass or fail outcome evaluated against success criteria and the recovery time objective defined in your resilience policy, with an auto-generated report capturing a point-in-time snapshot of resources. This capability removes manual experiment design, helps organizations prove their architecture holds under failure, and feeds results into compliance and reporting workflows.


### [How NVIDIA Minerva Turns Telemetry Chaos Into Peak GPU Yield](https://yomu.fyi/post/how-nvidia-minerva-turns-telemetry-chaos-into-peak-gpu-yield.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Oct 26, 2026
- Type: Problem & solution

NVIDIA Minerva is an AI-native AIOps platform built for the scale and complexity of AI factories. It ingests metrics, logs, and topology from GPUs, Spectrum-X switches, NVLink fabrics, storage, and schedulers, building a live knowledge graph that ties infrastructure to active jobs. Purpose-built supervised and unsupervised models detect gray failures—such as a GPU running hotter than peers or an NVMe device failing under load—that static thresholds would miss. The correlation engine uses the graph to collapse related events into a single incident; one case reduced 81 alerting events to a single notification, saving 54 hours of cluster time. An AI agent named Sherlock lets operators ask natural-language questions about job slowdowns or component changes, shortening triage from cross-team handoffs to guided investigation.


### [Expanding Customer Notifications for ClickHouse Managed Postgres](https://yomu.fyi/post/expanding-customer-notifications-for-clickhouse-managed-postgres.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Oct 26, 2026
- Type: Announcement

ClickHouse Managed Postgres launched storage alerts in August, warning at 85% disk usage, and those alerts have already helped customers identify issues like unused replication slots. The service now adds five targeted notifications: inactive replication slots, planned switchovers, unplanned failovers, long-running transactions, and connection-usage warnings. The replication-slot notification triggers when a logical slot retains over 10 GiB of unconfirmed WAL data for two hours, excluding managed CDC slots, and includes slot names. Planned-switchover and unplanned-failover notifications each specify the affected service and completion time, distinguishing scaling events from failure recovery. Long-running transactions open for at least two hours and connection usage at 95% of the limit trigger warnings with transaction age, helping operators check idle sessions, locks, or pool settings before impact, with delivery via console, email, or Slack.


### [How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes](https://yomu.fyi/post/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernete.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Oct 26, 2026
- Type: Announcement

NVIDIA Mission Control 2.5 extends its autonomous recovery engine, previously limited to Slurm-based HPC, to Kubernetes workloads managed via NVIDIA Run:ai for AI factories, coordinating three separate services: the scheduler for workload placement, the job recovery service for failure-context decisions, and the hardware recovery service for node isolation and diagnostics. When a hardware fault occurs, job recovery restarts the workload on healthy nodes while hardware recovery concurrently isolates the suspect GPU. For transient software errors, only job restart occurs without hardware quarantine. Early degradation signals cause proactive node removal before job scheduling. This consistent recovery model across Kubernetes and Slurm reduces MTTR and improves GPU utilization by shifting from reactive manual intervention to proactive self-healing, benefiting both platform teams and AI practitioners.


### [Feng Shui for Robots: The Fly MCP Server](https://yomu.fyi/post/feng-shui-for-robots-the-fly-mcp-server.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Oct 23, 2026
- Type: Announcement

In October 2023 AI agents created 0.1% of databases on Neon; by October 2025 they created 80%, and at Fly.io agents have overtaken humans as the most common customer species. Fly.io built mcp.fly.io, an MCP server that connects agents to Sprites for sandboxed development and to Fly Machines for production, providing a path from prototype to scalable deployment. Sprites are real Linux machines with persistent disk, service management, HTTPS URLs, and filesystem checkpoints that allow agents to install packages, run servers, and roll back after failures, and they sleep when idle. When a project needs always-on availability, the agent can migrate it from a Sprite to a Fly Machine using the same MCP server, creating the Machine, volume, and secrets in the same conversation. The MCP server is authorized per app and organization with no API token, and Fly.io is investigating adding MPG skills for database management.
