---
title: "Validate GPU Cluster Readiness Before AI Workloads Land"
description: "GPU clusters can pass standard health checks yet fail under real AI training workloads due to a single degraded GPU, network link, or misrouted traffic. NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that runs real distributed workloads across topology-aware node groups before production jobs land, measuring results and reporting which nodes failed each test. Its layered API uses Custom Resource Definitions for Certification, Workflow, and Job resources, attributing each failure to a specific node and category such as NCCL bandwidth or training goodput. The WorkloadRun API handles platform detection, NCCL settings, shared-memory volumes, and gang scheduling, so operators do not need to write those manifests by hand. NVCRE integrates with NVSentinel for passive telemetry and is available under Apache 2.0, with a roadmap for inference and automated lifecycle validation."
---

# Validate GPU Cluster Readiness Before AI Workloads Land

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Michelle Horton · Sep 23, 2026

**Type:** Explainer

## Summary

GPU clusters can pass standard health checks yet fail under real AI training workloads due to a single degraded GPU, network link, or misrouted traffic. NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that runs real distributed workloads across topology-aware node groups before production jobs land, measuring results and reporting which nodes failed each test. Its layered API uses Custom Resource Definitions for Certification, Workflow, and Job resources, attributing each failure to a specific node and category such as NCCL bandwidth or training goodput. The WorkloadRun API handles platform detection, NCCL settings, shared-memory volumes, and gang scheduling, so operators do not need to write those manifests by hand. NVCRE integrates with NVSentinel for passive telemetry and is available under Apache 2.0, with a roadmap for inference and automated lifecycle validation.

## Approach / What changed

NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that validates GPU cluster readiness by running real distributed workloads across topology-aware node groups before production jobs land. It provides a layered API with Certification, Workflow, and Job custom resources, and a WorkloadRun API that handles platform detection, NCCL settings, shared-memory volumes, and gang scheduling. NVCRE measures results and reports which nodes failed each test by category, with pass/fail criteria defined in Common Expression Language. It integrates with NVSentinel for passive monitoring and automated remediation workflows.

## Takeaways

- NVCRE runs real distributed workloads across topology-aware node groups, measuring results and reporting which nodes failed each test, eliminating manual bisection of GPU clusters.
- The API hierarchy consists of Certification, Workflow, and Job custom resources, attributing failures to specific nodes and categories such as NCCL bandwidth or training goodput.
- NVCRE integrates with NVSentinel to translate certification results into health events, enabling automated quarantine and drain workflows while allowing corrective actions to clear the failure signal.

## FAQ

### How does NVCRE isolate the failing node in a multi-node GPU cluster?

NVCRE uses adaptive fault isolation with topology-aware hierarchical group testing. It splits failing groups, reruns halves, and continues until reaching minGroupSize, then flags suspect nodes with reasons for failure.

### What are the built-in test categories in NVCRE?

The built-in catalog covers five NCCL communication variants, the DCGM level-4 diagnostic suite, and NVIDIA NeMo pretraining with Nemotron 5 models at 8B and 56B parameters.

### How does NVCRE handle pass/fail criteria for tests?

Pass and fail criteria use Common Expression Language expressions evaluated against measured metrics. No thresholds ship by default; users define values such as busBandwidthGBps >= 900 for communication tests.

**Tags:** [AI](https://yomu.fyi/topic/ai), [GPU](https://yomu.fyi/topic/gpu), [Kubernetes](https://yomu.fyi/topic/kubernetes), [NVIDIA](https://yomu.fyi/topic/nvidia), [Open Source](https://yomu.fyi/topic/open-source)

- Source: [NVIDIA](https://developer.nvidia.com/blog/validate-gpu-cluster-readiness-before-ai-workloads-land)
- Source URL: https://developer.nvidia.com/blog/validate-gpu-cluster-readiness-before-ai-workloads-land
- Ingested by Yomu: 2026-09-23T21:00:11.127Z

[Read original post](https://developer.nvidia.com/blog/validate-gpu-cluster-readiness-before-ai-workloads-land)