Topic

ONNX

2 posts about ONNX. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

NVIDIA
Build Local AI Apps with C++ and NVIDIA TensorRT RTX Samples
DIN Deploy is an open-source collection of C++ samples combining ONNX Runtime with NVIDIA TensorRT RTX for local AI inference on Windows and Linux. It separates model conversion from deployment: Python exports models to ONNX, then native C++ runs inference via ONNX Runtime session and tensor APIs. Supported workloads include automatic speech recognition (Whisper offline, Parakeet TDT streaming), interactive segmentation (SAM 2.1), and prompt-driven image generation (FLUX.2-klein-4B). The FLUX.2 sample includes Vulkan and DirectX graphics interop and demonstrates post-training quantization with NVIDIA Model Optimizer, producing quantized ONNX models as drop-in replacements without code changes. On DGX Spark, GPU acceleration provides speedups: Whisper large-v3-turbo achieves 58.5x real-time factor, Parakeet TDT 206.41x, and SAM 2.1 runs at 38.3 FPS on GPU versus 0.5 FPS on CPU.

Grab
Modernising Grab’s model serving platform with NVIDIA Triton Inference Server
Catwalk, Grab’s machine learning model serving platform, accumulated technical debt by maintaining separate inference engines for ONNX, PyTorch, TensorFlow, and vLLM, contributing to higher latency, lower throughput, and rising costs. The team evaluated NVIDIA Triton Inference Server and began consolidating those engines around it, initially adopting ONNX. The Triton manager combines a server manager, which retrieves and verifies models, prepares configurations, launches Triton, and monitors health, with a proxy that translates legacy API requests without requiring client code changes. Benchmarks showed at least five times the traffic capacity for a transformer model; rollout reached over 50% of online deployments in 10 days, while some larger models saw p90 latency fall from 120 ms to 20 ms.

The best engineering reads, once a week.

Topic

2 posts about ONNX. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

NVIDIA
Build Local AI Apps with C++ and NVIDIA TensorRT RTX Samples
DIN Deploy is an open-source collection of C++ samples combining ONNX Runtime with NVIDIA TensorRT RTX for local AI inference on Windows and Linux. It separates model conversion from deployment: Python exports models to ONNX, then native C++ runs inference via ONNX Runtime session and tensor APIs. Supported workloads include automatic speech recognition (Whisper offline, Parakeet TDT streaming), interactive segmentation (SAM 2.1), and prompt-driven image generation (FLUX.2-klein-4B). The FLUX.2 sample includes Vulkan and DirectX graphics interop and demonstrates post-training quantization with NVIDIA Model Optimizer, producing quantized ONNX models as drop-in replacements without code changes. On DGX Spark, GPU acceleration provides speedups: Whisper large-v3-turbo achieves 58.5x real-time factor, Parakeet TDT 206.41x, and SAM 2.1 runs at 38.3 FPS on GPU versus 0.5 FPS on CPU.

Grab
Modernising Grab’s model serving platform with NVIDIA Triton Inference Server
Catwalk, Grab’s machine learning model serving platform, accumulated technical debt by maintaining separate inference engines for ONNX, PyTorch, TensorFlow, and vLLM, contributing to higher latency, lower throughput, and rising costs. The team evaluated NVIDIA Triton Inference Server and began consolidating those engines around it, initially adopting ONNX. The Triton manager combines a server manager, which retrieves and verifies models, prepares configurations, launches Triton, and monitors health, with a proxy that translates legacy API requests without requiring client code changes. Benchmarks showed at least five times the traffic capacity for a transformer model; rollout reached over 50% of online deployments in 10 days, while some larger models saw p90 latency fall from 120 ms to 20 ms.

The best engineering reads, once a week.