NVIDIA Minerva is an AI-native AIOps platform built for the scale and complexity of AI factories. It ingests metrics, logs, and topology from GPUs, Spectrum-X switches, NVLink fabrics, storage, and schedulers, building a live knowledge graph that ties infrastructure to active jobs. Purpose-built supervised and unsupervised models detect gray failures—such as a GPU running hotter than peers or an NVMe device failing under load—that static thresholds would miss. The correlation engine uses the graph to collapse related events into a single incident; one case reduced 81 alerting events to a single notification, saving 54 hours of cluster time. An AI agent named Sherlock lets operators ask natural-language questions about job slowdowns or component changes, shortening triage from cross-team handoffs to guided investigation.
Context
AI factories combine hundreds of thousands of GPUs with complex networking, storage, and schedulers. In synchronous training, a single degraded component creates stragglers that waste compute across healthy resources. Gray failures—subtle degradations that stay within specification—are missed by static thresholds. Diagnostic evidence is split across network, compute, and ML teams, leading to lengthy cross-team handoffs.
Approach / What changed
NVIDIA Minerva collects telemetry across the full stack and builds a dynamic knowledge graph that maps infrastructure to jobs. It applies supervised and unsupervised machine learning models that compare component behavior to peers to detect anomalies. A topology-aware correlation engine collapses related events into single incidents. An AI agent, Sherlock, allows natural-language querying of live evidence for root-cause analysis and remediation guidance.
Takeaways
1
NVIDIA Minerva’s knowledge graph maps infrastructure, scheduler, and workload elements, enabling navigation from a failing interface to the affected job and vice versa, providing a shared causal map across teams.
2
Purpose-built models detect gray failures by comparing each component’s telemetry to its peers; for example, a GPU running hotter than siblings under the same load is flagged even if below absolute temperature limits.
3
Topology-aware correlation reduced a pause-frame storm that produced 81 separate events into a single incident, saving an estimated 54 hours of cluster uptime by focusing operators on the root cause.
FAQ
What is the dynamic knowledge graph in NVIDIA Minerva?
The knowledge graph is built through auto-discovery that maps relationships between GPUs, switches, storage, schedulers, and jobs. It updates as the environment changes, allowing operators to navigate from any component to affected workloads or from a job ID to its underlying infrastructure.
How does Minerva detect gray failures?
It uses supervised models for known error patterns and unsupervised models that learn normal behavior per component and compare it to peer groups. Multivariate models combine temperature, power, clocks, and utilization to detect degradation before failure, even when no static threshold is crossed.
How can operators interact with Minerva?
Operators use the native AI agent Sherlock to ask questions in natural language about specific jobs, intervals, or component changes. Sherlock is grounded in live telemetry and the knowledge graph, presenting supporting evidence and recommended remediation without requiring knowledge of query languages.
Autonomous Recovery is a closed loop that senses degradation, classifies a fault as transient or hardware, recovers workloads and quarantines hardware in parallel, then verifies against an RTO.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.