---
title: "NVIDIA Exemplar Cloud: Lessons for Unlocking Full Performance on AI Infrastructure"
description: "Identical AI computing clusters can exhibit 8% to 12% training throughput deficits compared to NVIDIA reference architectures due to compounded configuration choices across kernels, hypervisors, BIOS, and NCCL settings. Profiling with Linux perf, turbostat, and NCCL tests uncovers performance degradation across multiple layers of the compute stack. In virtualized Grace deployments, enabling Arm SMMU Virtual Command Queue support eliminates invalidation serialization bottlenecks on memory-heavy workloads. On x86 systems, adjusting BIOS C-states to allow idle cores into C6 frees thermal headroom for active cores to achieve rated turbo frequencies, while cpuset isolation resolves NUMA contention. Further gains stem from tuning NCCL queue pairs per connection and properly bind-mounting topology files inside workload containers."
---

# NVIDIA Exemplar Cloud: Lessons for Unlocking Full Performance on AI Infrastructure

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Elizabeth Goodman · Jul 30, 2026

**Type:** Problem & solution

## Summary

Identical AI computing clusters can exhibit 8% to 12% training throughput deficits compared to NVIDIA reference architectures due to compounded configuration choices across kernels, hypervisors, BIOS, and NCCL settings. Profiling with Linux perf, turbostat, and NCCL tests uncovers performance degradation across multiple layers of the compute stack. In virtualized Grace deployments, enabling Arm SMMU Virtual Command Queue support eliminates invalidation serialization bottlenecks on memory-heavy workloads. On x86 systems, adjusting BIOS C-states to allow idle cores into C6 frees thermal headroom for active cores to achieve rated turbo frequencies, while cpuset isolation resolves NUMA contention. Further gains stem from tuning NCCL queue pairs per connection and properly bind-mounting topology files inside workload containers.

## Context

Partner deployments of NVIDIA H100, B200, GB200 NVL72, and GB300 NVL72 clusters routinely show 8% to 12% performance gaps compared to NVIDIA reference architectures on identical training workloads, causing clusters to miss the 95% threshold required for NVIDIA Exemplar Cloud validation.

## Approach / What changed

Isolate and remediate cumulative multi-layer bottlenecks using Linux perf, Nsight Systems, turbostat, and nccl-tests to diagnose Arm SMMU contention, x86 CPU power and NUMA affinity issues, NCCL queue-pair concurrency settings, and container topology file mounts.

## Takeaways

- Under virtualization on Grace systems, enabling CMDQV/VCMDQ prevents SMMU invalidation serialization and excessive host trapping, reducing DeepSeek-V3 MoE iteration times by approximately 12%.
- Restricting idle x86 cores to C1 limits package power budget for busy cores; allowing idle cores to enter C6 enables active cores to reach maximum turbo frequencies and recovers around 4% performance.
- Unpropagated NCCL topology files and unmounted /etc/nccl/topo.xml inside job containers can degrade collective operations like AllGather and ReduceScatter by 2x to 4x without throwing explicit runtime errors.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Observability](https://yomu.fyi/topic/observability), [Performance](https://yomu.fyi/topic/performance)

- Source: [NVIDIA](https://developer.nvidia.com/blog/nvidia-exemplar-cloud-lessons-for-unlocking-full-performance-on-ai-infrastructure)
- Source URL: https://developer.nvidia.com/blog/nvidia-exemplar-cloud-lessons-for-unlocking-full-performance-on-ai-infrastructure
- Ingested by Yomu: 2026-08-27T20:00:58.527Z

[Read original post](https://developer.nvidia.com/blog/nvidia-exemplar-cloud-lessons-for-unlocking-full-performance-on-ai-infrastructure)