---
title: "How to Choose Full-Stack Observability for NVIDIA AI Factories"
description: "AI infrastructure spans compute, networking, storage, orchestration, and applications, so a symptom at one layer may originate elsewhere. The post proposes selecting observability tools by failure domain, mapping components to telemetry sources, and reducing signals to a top-k alert set in a unified Prometheus/Grafana triage dashboard. For a DGX cluster using InfiniBand, BCM, and Slurm, it recommends Redfish/IPMI, DCGM, NVSM, UFM, and BCM, while deferring NetQ, NMX, Run:ai, and NIM until their corresponding capabilities are needed. The framework emphasizes DCGM for GPU metrics, NVSM for DGX system health, UFM for InfiniBand, and BCM for cluster and job aggregation, with specialized tools retaining low-level telemetry. An elevated InfiniBand bit error rate illustrates how retransmissions can slow one rank during NCCL all-reduce, causing synchronized training throughput to follow the slowest rank; alerts should connect concise signals to owners and remediation actions."
---

# How to Choose Full-Stack Observability for NVIDIA AI Factories

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Jorge Cardoso · Aug 12, 2026

**Type:** Explainer

## Summary

AI infrastructure spans compute, networking, storage, orchestration, and applications, so a symptom at one layer may originate elsewhere. The post proposes selecting observability tools by failure domain, mapping components to telemetry sources, and reducing signals to a top-k alert set in a unified Prometheus/Grafana triage dashboard. For a DGX cluster using InfiniBand, BCM, and Slurm, it recommends Redfish/IPMI, DCGM, NVSM, UFM, and BCM, while deferring NetQ, NMX, Run:ai, and NIM until their corresponding capabilities are needed. The framework emphasizes DCGM for GPU metrics, NVSM for DGX system health, UFM for InfiniBand, and BCM for cluster and job aggregation, with specialized tools retaining low-level telemetry. An elevated InfiniBand bit error rate illustrates how retransmissions can slow one rank during NCCL all-reduce, causing synchronized training throughput to follow the slowest rank; alerts should connect concise signals to owners and remediation actions.

## Context

AI infrastructure has multiple interdependent layers, and performance symptoms can originate away from the layer where they appear. The source describes a distributed training job whose throughput declined after an InfiniBand link developed an elevated bit error rate. Because bulk synchronous parallel training waits for the slowest rank, this gray failure caused cascading impact and wasted GPU hours.

## Approach / What changed

The framework enumerates observable failure domains, maps each component to telemetry tools, and selects the fewest tools needed to close coverage gaps. For an InfiniBand DGX cluster, it combines Redfish/IPMI, DCGM, NVSM, UFM, and BCM, integrates their signals into Prometheus and Grafana, and uses a short top-k alert set tied to owners and remediation actions. A two-layer dashboard supports high-level triage followed by vendor-interface investigation.

## Takeaways

- Bulk synchronous parallel training can amplify a single InfiniBand link’s bit errors: retransmissions delay one rank during NCCL all-reduce, while other ranks block and throughput follows the slowest rank.
- The framework assigns DCGM to GPU telemetry, NVSM to DGX system health, UFM to InfiniBand visibility, and BCM to cluster and job aggregation; no single tool covers every required domain.
- The initial alert set prioritizes platform, GPU, InfiniBand, and job signals, expanding only after incidents expose coverage gaps rather than collecting every available metric.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [NVIDIA](https://developer.nvidia.com/blog/how-to-choose-full-stack-observability-for-nvidia-ai-factories)
- Source URL: https://developer.nvidia.com/blog/how-to-choose-full-stack-observability-for-nvidia-ai-factories
- Ingested by Yomu: 2026-08-28T12:00:56.196Z

[Read original post](https://developer.nvidia.com/blog/how-to-choose-full-stack-observability-for-nvidia-ai-factories)