Build Local AI Apps with C++ and NVIDIA TensorRT RTX Samples

Summary

DIN Deploy is an open-source collection of C++ samples combining ONNX Runtime with NVIDIA TensorRT RTX for local AI inference on Windows and Linux. It separates model conversion from deployment: Python exports models to ONNX, then native C++ runs inference via ONNX Runtime session and tensor APIs. Supported workloads include automatic speech recognition (Whisper offline, Parakeet TDT streaming), interactive segmentation (SAM 2.1), and prompt-driven image generation (FLUX.2-klein-4B). The FLUX.2 sample includes Vulkan and DirectX graphics interop and demonstrates post-training quantization with NVIDIA Model Optimizer, producing quantized ONNX models as drop-in replacements without code changes. On DGX Spark, GPU acceleration provides speedups: Whisper large-v3-turbo achieves 58.5x real-time factor, Parakeet TDT 206.41x, and SAM 2.1 runs at 38.3 FPS on GPU versus 0.5 FPS on CPU.

Takeaways

1. 1

   Model conversion and deployment are fully separated: Python exports a model to ONNX, and native C++ runs it with ONNX Runtime without requiring a model-specific runtime.

2. 2

   Shared inference code uses ONNX Runtime tensor APIs for portability; vendor-specific CUDA APIs appear only in optional accelerated paths.

3. 3

   Post-training quantization with NVIDIA Model Optimizer produces quantized ONNX models that are drop-in replacements requiring no application code changes.

FAQ