---
title: "Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing"
description: "NVIDIA Confidential Computing (CC) enables secure AI inference by using memory-encrypted confidential VMs, confidential GPUs, and encrypted NVLink. Running inference frameworks like TensorRT LLM in a CC environment introduces performance overhead because secure execution changes assumptions about memory movement, timing, and multi-GPU communication. To quantify this overhead, the post presents a controlled methodology comparing CC-on and CC-off performance on a workload with long input context, extended output generation, and low concurrency. With CC-aware adaptations—including pageable memory selection, asynchronous readback, GPU %globaltimer for autotuning, and CC-aware multi-GPU communication—TensorRT LLM retained 96.1–98.2% of baseline throughput on eight NVIDIA B200 GPUs while keeping per-token latency overhead below 5%. The post concludes that security configuration and inference optimization should be treated as a single deployment problem for organizations moving private inference into production."
---

# Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Tanya Lenz · Sep 22, 2026

**Type:** Explainer

## Summary

NVIDIA Confidential Computing (CC) enables secure AI inference by using memory-encrypted confidential VMs, confidential GPUs, and encrypted NVLink. Running inference frameworks like TensorRT LLM in a CC environment introduces performance overhead because secure execution changes assumptions about memory movement, timing, and multi-GPU communication. To quantify this overhead, the post presents a controlled methodology comparing CC-on and CC-off performance on a workload with long input context, extended output generation, and low concurrency. With CC-aware adaptations—including pageable memory selection, asynchronous readback, GPU %globaltimer for autotuning, and CC-aware multi-GPU communication—TensorRT LLM retained 96.1–98.2% of baseline throughput on eight NVIDIA B200 GPUs while keeping per-token latency overhead below 5%. The post concludes that security configuration and inference optimization should be treated as a single deployment problem for organizations moving private inference into production.

## Context

The post examines how AI inference frameworks like NVIDIA TensorRT LLM can maintain high performance when running in a confidential computing environment on NVIDIA Blackwell GPUs. It describes the performance overhead introduced by secure execution and presents a methodology for measuring it, along with specific adaptations to mitigate overhead.

## Approach / What changed

The post presents a controlled CC-on vs CC-off comparison methodology to quantify overhead, using a workload with long input context, extended output generation, and low concurrency. It details TensorRT LLM adaptations: using pageable memory for host-to-device transfers, asynchronous worker for device-to-host readback, GPU %globaltimer for kernel autotuning, and CC-aware multi-GPU communication that detects NVLS availability.

## Takeaways

- To expose CC overhead, select a workload with long input context, extended output generation, and low concurrency, as high request volume can amortize fixed encryption costs.
- With CC-aware adaptations, TensorRT LLM retained 96.1–98.2% of baseline throughput and kept per-token latency overhead below 5% on eight NVIDIA B200 GPUs.
- In B200 CC, NVLS multicast is unavailable, so frameworks should detect NVLS availability and choose communication algorithms that minimize latency for the given message size and topology.

## FAQ

### How can I measure the performance overhead of confidential computing on my AI inference workload?

Run the same workload under CC-on and CC-off conditions, holding model, hardware, framework version, sequence lengths, parallelism, and concurrency constant. Measure output throughput and time per output token (TPOT) to compute retained throughput and latency overhead.

### What specific adaptations does TensorRT LLM use to reduce CC overhead?

TensorRT LLM uses CC-aware memory selection (pageable memory for host-to-device transfers), asynchronous worker for device-to-host readback, GPU %globaltimer for kernel autotuning, and CC-aware multi-GPU communication that detects NVLS availability.

### What workload characteristics best expose CC overhead?

A workload with long input context, extended output generation, and low concurrency. Long context stresses data movement during prefill, extended generation amplifies small per-token overhead, and low concurrency limits hiding costs across concurrent requests.

**Tags:** [Confidential Computing](https://yomu.fyi/topic/confidential-computing), [LLMs](https://yomu.fyi/topic/llm), [NVIDIA](https://yomu.fyi/topic/nvidia), [Performance](https://yomu.fyi/topic/performance), [TensorRT LLM](https://yomu.fyi/topic/tensorrt-llm)

- Source: [NVIDIA](https://developer.nvidia.com/blog/enabling-private-high-performance-production-ai-inference-with-nvidia-confidential-computing)
- Source URL: https://developer.nvidia.com/blog/enabling-private-high-performance-production-ai-inference-with-nvidia-confidential-computing
- Ingested by Yomu: 2026-09-22T18:00:12.289Z

[Read original post](https://developer.nvidia.com/blog/enabling-private-high-performance-production-ai-inference-with-nvidia-confidential-computing)