Validate GPU Cluster Readiness Before AI Workloads Land

Summary

GPU clusters can pass standard health checks yet fail under real AI training workloads due to a single degraded GPU, network link, or misrouted traffic. NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that runs real distributed workloads across topology-aware node groups before production jobs land, measuring results and reporting which nodes failed each test. Its layered API uses Custom Resource Definitions for Certification, Workflow, and Job resources, attributing each failure to a specific node and category such as NCCL bandwidth or training goodput. The WorkloadRun API handles platform detection, NCCL settings, shared-memory volumes, and gang scheduling, so operators do not need to write those manifests by hand. NVCRE integrates with NVSentinel for passive telemetry and is available under Apache 2.0, with a roadmap for inference and automated lifecycle validation.

Takeaways

1. 1

   NVCRE runs real distributed workloads across topology-aware node groups, measuring results and reporting which nodes failed each test, eliminating manual bisection of GPU clusters.

2. 2

   The API hierarchy consists of Certification, Workflow, and Job custom resources, attributing failures to specific nodes and categories such as NCCL bandwidth or training goodput.

3. 3

   NVCRE integrates with NVSentinel to translate certification results into health events, enabling automated quarantine and drain workflows while allowing corrective actions to clear the failure signal.

FAQ