NVIDIATanya LenzAnnouncement
NVIDIA TensorRT multi-device inference enables a single TensorRT network to execute across multiple GPUs using NCCL-backed distributed collectives, supported from TensorRT 11.0. Integrated into Dynamo-Triton release 26.07, one model instance can own multiple GPUs and expose a single gRPC endpoint so the client does not coordinate GPU ranks. The post demonstrates this with Cosmos 3 Nano video generation, where the transformer uses Ulysses context parallelism to distribute 44,160 video tokens across up to eight GPUs. End-to-end latency drops from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs, achieving a 6.09x transformer RPC speedup. Teams can trade additional GPU resources for shorter request latency while keeping the application interface and surrounding workflow stable.
Generative AI workloads increasingly exceed single-GPU compute and memory limits. Organizations need a way to distribute inference across multiple GPUs without complicating the application interface. NVIDIA TensorRT multi-device inference and its integration into Dynamo-Triton aim to close this gap by allowing a single model endpoint to span multiple GPUs.
NVIDIA TensorRT multi-device inference uses NCCL-backed distributed collectives to execute a single TensorRT network across multiple GPUs. Dynamo-Triton integrates this by allowing one KIND_MODEL instance to own multiple GPUs, create per-rank TensorRT execution contexts, CUDA streams, and NCCL communicators, and launch ranks together for each request. The application calls one named model through a gRPC endpoint instead of coordinating GPU ranks itself.
TensorRT multi-device inference enables a single network to run across multiple GPUs using NCCL collectives, supported from TensorRT 11.0.
In the Cosmos 3 Nano demo, end-to-end latency dropped from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs, with transformer RPC speedup of 6.09x.
The client does not coordinate GPU ranks; Dynamo-Triton exposes a single gRPC endpoint and handles multi-device execution internally.
The distributed Ulysses graph is compiled into each TensorRT plan before deployment. Dynamo-Triton loads the plan, creates multi-rank execution state, and exposes one gRPC model endpoint. The client sends a transformer request to that endpoint without coordinating GPU ranks.
Each context-parallel output was compared to the single-device result using the same seed. CP2, CP4, and CP8 passed thresholds of MAE ≤ 25 and PSNR ≥ 18 dB. Outputs are not pixel-identical; CP8 measured MAE 16.316 and PSNR 19.400 dB.
Teams can trade additional GPU resources for shorter request latency. The benchmark does not measure concurrent throughput, cost per video, or TCO. Teams should evaluate against their own SLOs and deployment economics.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
BioNeMo Inference Runtime runs one pipeline per target: parse, tokenize, generate features, infer on a GPU replica, and write a structure. This diagram traces those stages and the throughput trade-off.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
