Audit Is a Provenance Chain, Not a Log File
An audit is a provenance chain: what was stored, where each fact came from, and what was used for each decision. This explainer maps the three links that keep agent behavior traceable and fixable.
NVIDIAShara TibkenTutorial
NVIDIA developed the AI Factory Operations Agent to streamline internal DGX SuperPOD operations, using specialized AI agents that communicate through Model Context Protocol (MCP) to reduce manual effort for SREs managing GPU clusters. The architecture centers on a NemoClaw agent runtime that routes queries to domain-specific agents for job recovery, hardware triage, observability, research, Kubernetes awareness, and cluster management, all using locally served Nemotron open models for inference. The system runs as a Kubernetes control-plane application with Helm deployment, using OpenShell for sandboxed shell execution and read-only kubectl tools with RBAC for security. In production, agents detect training failures and goodput degradation, correlate root causes across scheduler, application, hardware, and network layers, and surface recommendations through a chat interface with full auditability. NVIDIA shares five lessons from production use: clear agent boundaries, read-only first approach, cross-layer correlation, precise MCP tool definitions, and transparent reasoning for SRE trust.
Site reliability engineers managing large GPU clusters often need to check multiple dashboards across compute, network, hardware, and scheduling layers to diagnose training performance drops. NVIDIA developed the AI Factory Operations Agent to automate this correlation through a chat interface using specialized AI agents.
NVIDIA built the AI Factory Operations Agent using a NemoClaw blueprint with domain-specific agents (Job Recovery, Hardware, Observability, Research, Kubernetes, Cluster Management) that communicate via MCP. The system runs as a Kubernetes control-plane application with Helm deployment, using OpenShell for sandboxed shell execution and read-only kubectl tools with RBAC. Inference uses locally served Nemotron open models.
The AI Factory Operations Agent uses specialized sub-agents for different operational domains (job recovery, hardware, observability, etc.) that communicate through Model Context Protocol (MCP) to correlate root causes across layers.
Security is enforced through read-only kubectl tools with RBAC, sandboxed shell execution via OpenShell, and per-session Kubernetes sandboxes, ensuring agents cannot modify production workloads.
Five key lessons from production use include: clear agent boundaries to avoid overlapping ownership, read-only first to build SRE trust, cross-layer correlation to find missed root causes, precise MCP tool definitions for accurate routing, and transparent reasoning for auditability.
The agent uses read-only kubectl tools with a pinned binary and registered credentials, sandboxed shell execution via OpenShell per-session Kubernetes sandboxes, and RBAC to restrict access to read-only operations. Kubernetes service accounts and RBAC define the cluster resources the tool can inspect.
The agent includes Job Recovery, Hardware, Observability, Research, Kubernetes, and Cluster Management agents. Each handles specific operational domains like failure detection, hardware faults, telemetry queries, documentation search, and cluster awareness.
Yes, for multi-tenant AI factories, scoped agents, tools, sandboxes, and Kubernetes permissions control cluster data per tenant. OpenShell enforces boundaries through isolated sandboxes with kernel-level restrictions, and each tenant can use its own domain-specific agents and dedicated inference hardware.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
An audit is a provenance chain: what was stored, where each fact came from, and what was used for each decision. This explainer maps the three links that keep agent behavior traceable and fixable.
Actionable banking wires real-time event triggers, unified customer context, and agent memory into one pipeline that ends at a human decision, not an automatic transaction.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.