Streaming ASR vs. Batch ASR: The Buffer-Latency Trade-off
Streaming and batch ASR pipelines compared: how input buffer length, speaker diarization, and where audio is processed trade latency against transcript quality and privacy.
Loading post…
NVIDIATanya LenzExplainer
Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing
Summary
NVIDIA Confidential Computing (CC) enables secure AI inference by using memory-encrypted confidential VMs, confidential GPUs, and encrypted NVLink. Running inference frameworks like TensorRT LLM in a CC environment introduces performance overhead because secure execution changes assumptions about memory movement, timing, and multi-GPU communication. To quantify this overhead, the post presents a controlled methodology comparing CC-on and CC-off performance on a workload with long input context, extended output generation, and low concurrency. With CC-aware adaptations—including pageable memory selection, asynchronous readback, GPU %globaltimer for autotuning, and CC-aware multi-GPU communication—TensorRT LLM retained 96.1–98.2% of baseline throughput on eight NVIDIA B200 GPUs while keeping per-token latency overhead below 5%. The post concludes that security configuration and inference optimization should be treated as a single deployment problem for organizations moving private inference into production.
Context
The post examines how AI inference frameworks like NVIDIA TensorRT LLM can maintain high performance when running in a confidential computing environment on NVIDIA Blackwell GPUs. It describes the performance overhead introduced by secure execution and presents a methodology for measuring it, along with specific adaptations to mitigate overhead.
Approach / What changed
The post presents a controlled CC-on vs CC-off comparison methodology to quantify overhead, using a workload with long input context, extended output generation, and low concurrency. It details TensorRT LLM adaptations: using pageable memory for host-to-device transfers, asynchronous worker for device-to-host readback, GPU %globaltimer for kernel autotuning, and CC-aware multi-GPU communication that detects NVLS availability.
Takeaways
To expose CC overhead, select a workload with long input context, extended output generation, and low concurrency, as high request volume can amortize fixed encryption costs.
With CC-aware adaptations, TensorRT LLM retained 96.1–98.2% of baseline throughput and kept per-token latency overhead below 5% on eight NVIDIA B200 GPUs.
In B200 CC, NVLS multicast is unavailable, so frameworks should detect NVLS availability and choose communication algorithms that minimize latency for the given message size and topology.
FAQ
How can I measure the performance overhead of confidential computing on my AI inference workload?
Run the same workload under CC-on and CC-off conditions, holding model, hardware, framework version, sequence lengths, parallelism, and concurrency constant. Measure output throughput and time per output token (TPOT) to compute retained throughput and latency overhead.
What specific adaptations does TensorRT LLM use to reduce CC overhead?
TensorRT LLM uses CC-aware memory selection (pageable memory for host-to-device transfers), asynchronous worker for device-to-host readback, GPU %globaltimer for kernel autotuning, and CC-aware multi-GPU communication that detects NVLS availability.
What workload characteristics best expose CC overhead?
A workload with long input context, extended output generation, and low concurrency. Long context stresses data movement during prefill, extended generation amplifies small per-token overhead, and low concurrency limits hiding costs across concurrent requests.
Source and freshness
- Source
- developer.nvidia.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
How a 3D CT Model Reads a Volume: From Voxel Grid to Structured Report
A walkthrough of how a 3D CT model turns a voxel volume into a structured diagnostic report: isotropic resampling, 3D patch tokens, spatial encoding, and language-model reasoning.
How BioNeMo Turns a Target List Into Folded Structures
BioNeMo Inference Runtime runs one pipeline per target: parse, tokenize, generate features, infer on a GPU replica, and write a structure. This diagram traces those stages and the throughput trade-off.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.