Topic

Triton Inference Server

2 posts about Triton Inference Server. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

NVIDIA
Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton
NVIDIA TensorRT multi-device inference enables a single TensorRT network to execute across multiple GPUs using NCCL-backed distributed collectives, supported from TensorRT 11.0. Integrated into Dynamo-Triton release 26.07, one model instance can own multiple GPUs and expose a single gRPC endpoint so the client does not coordinate GPU ranks. The post demonstrates this with Cosmos 3 Nano video generation, where the transformer uses Ulysses context parallelism to distribute 44,160 video tokens across up to eight GPUs. End-to-end latency drops from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs, achieving a 6.09x transformer RPC speedup. Teams can trade additional GPU resources for shorter request latency while keeping the application interface and surrounding workflow stable.

Grab
Modernising Grab’s model serving platform with NVIDIA Triton Inference Server
Catwalk, Grab’s machine learning model serving platform, accumulated technical debt by maintaining separate inference engines for ONNX, PyTorch, TensorFlow, and vLLM, contributing to higher latency, lower throughput, and rising costs. The team evaluated NVIDIA Triton Inference Server and began consolidating those engines around it, initially adopting ONNX. The Triton manager combines a server manager, which retrieves and verifies models, prepares configurations, launches Triton, and monitors health, with a proxy that translates legacy API requests without requiring client code changes. Benchmarks showed at least five times the traffic capacity for a transformer model; rollout reached over 50% of online deployments in 10 days, while some larger models saw p90 latency fall from 120 ms to 20 ms.

The best engineering reads, once a week.

Topic

2 posts about Triton Inference Server. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

NVIDIA
Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton
NVIDIA TensorRT multi-device inference enables a single TensorRT network to execute across multiple GPUs using NCCL-backed distributed collectives, supported from TensorRT 11.0. Integrated into Dynamo-Triton release 26.07, one model instance can own multiple GPUs and expose a single gRPC endpoint so the client does not coordinate GPU ranks. The post demonstrates this with Cosmos 3 Nano video generation, where the transformer uses Ulysses context parallelism to distribute 44,160 video tokens across up to eight GPUs. End-to-end latency drops from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs, achieving a 6.09x transformer RPC speedup. Teams can trade additional GPU resources for shorter request latency while keeping the application interface and surrounding workflow stable.

Grab
Modernising Grab’s model serving platform with NVIDIA Triton Inference Server
Catwalk, Grab’s machine learning model serving platform, accumulated technical debt by maintaining separate inference engines for ONNX, PyTorch, TensorFlow, and vLLM, contributing to higher latency, lower throughput, and rising costs. The team evaluated NVIDIA Triton Inference Server and began consolidating those engines around it, initially adopting ONNX. The Triton manager combines a server manager, which retrieves and verifies models, prepares configurations, launches Triton, and monitors health, with a proxy that translates legacy API requests without requiring client code changes. Benchmarks showed at least five times the traffic capacity for a transformer model; rollout reached over 50% of online deployments in 10 days, while some larger models saw p90 latency fall from 120 ms to 20 ms.

The best engineering reads, once a week.