How to Choose Full-Stack Observability for NVIDIA AI Factories

Summary

AI infrastructure spans compute, networking, storage, orchestration, and applications, so a symptom at one layer may originate elsewhere. The post proposes selecting observability tools by failure domain, mapping components to telemetry sources, and reducing signals to a top-k alert set in a unified Prometheus/Grafana triage dashboard. For a DGX cluster using InfiniBand, BCM, and Slurm, it recommends Redfish/IPMI, DCGM, NVSM, UFM, and BCM, while deferring NetQ, NMX, Run:ai, and NIM until their corresponding capabilities are needed. The framework emphasizes DCGM for GPU metrics, NVSM for DGX system health, UFM for InfiniBand, and BCM for cluster and job aggregation, with specialized tools retaining low-level telemetry. An elevated InfiniBand bit error rate illustrates how retransmissions can slow one rank during NCCL all-reduce, causing synchronized training throughput to follow the slowest rank; alerts should connect concise signals to owners and remediation actions.

Takeaways

1. 1

   Bulk synchronous parallel training can amplify a single InfiniBand link’s bit errors: retransmissions delay one rank during NCCL all-reduce, while other ranks block and throughput follows the slowest rank.

2. 2

   The framework assigns DCGM to GPU telemetry, NVSM to DGX system health, UFM to InfiniBand visibility, and BCM to cluster and job aggregation; no single tool covers every required domain.

3. 3

   The initial alert set prioritizes platform, GPU, InfiniBand, and job signals, expanding only after incidents expose coverage gaps rather than collecting every available metric.