The Global Performance Tuner in TensorRT 11.1 provides automatic, accuracy-aware, end-to-end performance tuning, replacing reliance on a single default scheduling strategy with explicit build routes. Each build route is a concrete combination of compiler knobs controlling scheduling decisions; TensorRT 11.1 exposes over 200 such knobs. The tuning loop iterates over candidate routes, builds each engine, measures its accuracy and end-to-end performance against user-supplied references, and excludes any that fail to meet a tolerance threshold. This process decouples performance optimization from accuracy constraints, addressing the performance-accuracy entanglement problem described in Part 1. In evaluated workloads, tuning delivered 1.14x to 1.63x speedups over the default route, and best practices include re-tuning after any change to the model, input shapes, hardware, or software stack.
Context
The Global Performance Tuner in TensorRT 11.1 provides automatic, accuracy-aware performance tuning for deep learning models by exploring multiple build routes defined by compiler knobs.
Approach / What changed
The tuning loop iterates over candidate build routes, builds each engine, measures accuracy against user-supplied references, and selects the fastest engine that meets the accuracy tolerance. TensorRT exposes APIs to apply routes and discover available knobs; trtexec provides a ready-to-use tuner implementation.
Takeaways
1
The Global Performance Tuner replaces a single default scheduling strategy with model-specific build routes, enabling automatic, accuracy-aware performance tuning.
2
Accuracy and performance are decoupled by a user-defined tolerance threshold; only routes meeting the tolerance are considered for performance comparison.
3
The optimal build route is specific to the model, input shapes, target GPU, and software stack; re-tuning is necessary after any change.
FAQ
How does the Global Performance Tuner handle the accuracy-performance trade-off?
It uses a user-defined accuracy tolerance threshold. Only candidate engines whose outputs meet the tolerance relative to reference outputs are eligible, and the fastest eligible engine is selected.
Can I reuse a build route found for one deployment in another?
No. The best route depends on the specific model, input shapes, target GPU, and software stack. A route that improves one deployment may cause regressions in another, so re-tuning is required after any change.
How can I implement a custom tuner using TensorRT APIs?
TensorRT provides IBuilderConfig APIs: setBuildRoute, getBuildRoute, and getAllBuildRoutes to apply routes and discover available knobs. Developers can build custom search strategies using these APIs, as demonstrated by the open-source trtexec implementation.
A case study sells a peer's outcome; a benchmark proves a repeatable delta. This side-by-side shows what each kind of evidence establishes, what it hides, and how to pick the right proof before you commit.
A tradeoff map for synthetic audiobooks: natural-language style control buys character differentiation, but spends captioned data, tokens, and compute. See which lever to pull first and when plain narration is good enough.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.