---
title: "How NVIDIA Uses AI Agents to Streamline AI Factory Operations"
description: "NVIDIA developed the AI Factory Operations Agent to streamline internal DGX SuperPOD operations, using specialized AI agents that communicate through Model Context Protocol (MCP) to reduce manual effort for SREs managing GPU clusters. The architecture centers on a NemoClaw agent runtime that routes queries to domain-specific agents for job recovery, hardware triage, observability, research, Kubernetes awareness, and cluster management, all using locally served Nemotron open models for inference. The system runs as a Kubernetes control-plane application with Helm deployment, using OpenShell for sandboxed shell execution and read-only kubectl tools with RBAC for security. In production, agents detect training failures and goodput degradation, correlate root causes across scheduler, application, hardware, and network layers, and surface recommendations through a chat interface with full auditability. NVIDIA shares five lessons from production use: clear agent boundaries, read-only first approach, cross-layer correlation, precise MCP tool definitions, and transparent reasoning for SRE trust."
---

# How NVIDIA Uses AI Agents to Streamline AI Factory Operations

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Shara Tibken · Nov 4, 2026

**Type:** Tutorial

## Summary

NVIDIA developed the AI Factory Operations Agent to streamline internal DGX SuperPOD operations, using specialized AI agents that communicate through Model Context Protocol (MCP) to reduce manual effort for SREs managing GPU clusters. The architecture centers on a NemoClaw agent runtime that routes queries to domain-specific agents for job recovery, hardware triage, observability, research, Kubernetes awareness, and cluster management, all using locally served Nemotron open models for inference. The system runs as a Kubernetes control-plane application with Helm deployment, using OpenShell for sandboxed shell execution and read-only kubectl tools with RBAC for security. In production, agents detect training failures and goodput degradation, correlate root causes across scheduler, application, hardware, and network layers, and surface recommendations through a chat interface with full auditability. NVIDIA shares five lessons from production use: clear agent boundaries, read-only first approach, cross-layer correlation, precise MCP tool definitions, and transparent reasoning for SRE trust.

## Context

Site reliability engineers managing large GPU clusters often need to check multiple dashboards across compute, network, hardware, and scheduling layers to diagnose training performance drops. NVIDIA developed the AI Factory Operations Agent to automate this correlation through a chat interface using specialized AI agents.

## Approach / What changed

NVIDIA built the AI Factory Operations Agent using a NemoClaw blueprint with domain-specific agents (Job Recovery, Hardware, Observability, Research, Kubernetes, Cluster Management) that communicate via MCP. The system runs as a Kubernetes control-plane application with Helm deployment, using OpenShell for sandboxed shell execution and read-only kubectl tools with RBAC. Inference uses locally served Nemotron open models.

## Takeaways

- The AI Factory Operations Agent uses specialized sub-agents for different operational domains (job recovery, hardware, observability, etc.) that communicate through Model Context Protocol (MCP) to correlate root causes across layers.
- Security is enforced through read-only kubectl tools with RBAC, sandboxed shell execution via OpenShell, and per-session Kubernetes sandboxes, ensuring agents cannot modify production workloads.
- Five key lessons from production use include: clear agent boundaries to avoid overlapping ownership, read-only first to build SRE trust, cross-layer correlation to find missed root causes, precise MCP tool definitions for accurate routing, and transparent reasoning for auditability.

## FAQ

### How does the AI Factory Operations Agent ensure security when executing commands?

The agent uses read-only kubectl tools with a pinned binary and registered credentials, sandboxed shell execution via OpenShell per-session Kubernetes sandboxes, and RBAC to restrict access to read-only operations. Kubernetes service accounts and RBAC define the cluster resources the tool can inspect.

### What domain-specific agents are included in the AI Factory Operations Agent?

The agent includes Job Recovery, Hardware, Observability, Research, Kubernetes, and Cluster Management agents. Each handles specific operational domains like failure detection, hardware faults, telemetry queries, documentation search, and cluster awareness.

### Can the AI Factory Operations Agent be deployed in multi-tenant environments?

Yes, for multi-tenant AI factories, scoped agents, tools, sandboxes, and Kubernetes permissions control cluster data per tenant. OpenShell enforces boundaries through isolated sandboxes with kernel-level restrictions, and each tenant can use its own domain-specific agents and dedicated inference hardware.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [GPU](https://yomu.fyi/topic/gpu), [Kubernetes](https://yomu.fyi/topic/kubernetes), [MCP](https://yomu.fyi/topic/mcp), [NVIDIA](https://yomu.fyi/topic/nvidia)

- Source: [NVIDIA](https://developer.nvidia.com/blog/how-nvidia-uses-ai-agents-to-streamline-ai-factory-operations)
- Source URL: https://developer.nvidia.com/blog/how-nvidia-uses-ai-agents-to-streamline-ai-factory-operations
- Ingested by Yomu: 2026-11-05T00:00:19.457Z

[Read original post](https://developer.nvidia.com/blog/how-nvidia-uses-ai-agents-to-streamline-ai-factory-operations)
