# Make Long-Running NVIDIA TensorRT Engine Builds Observable and Cancelable in Python or C++

[NVIDIA Developer Blog](https://yomu.fyi/company/nvidia-developer-blog) · Michelle Horton · Jul 22, 2026

**Type:** Tutorial

## Summary

Long-running NVIDIA TensorRT engine builds can freeze workflows and waste GPU hours when they lack visibility or abort mechanisms. Developers can resolve this by subclassing the IProgressMonitor API in Python or C++ and attaching it to the builder configuration. The monitor tracks hierarchical build phases through start, step completion, and finish callbacks, requiring thread synchronization because TensorRT calls monitor methods across internal builder threads. Returning false from the step completion callback requests early build cancellation, which unwinds active phases in reverse order at the next step boundary. Beyond terminal rendering with ANSI escape codes, teams can stream these progress events to IDE protocols, HTTP service endpoints, or agent tool-call traces.

## Context

NVIDIA TensorRT engine builds can take minutes due to large strongly typed models, deep tactic searches, or cold timing caches. Most integrations report no progress and provide no mechanism to abort early, leaving developers, users, or agent workflows stuck in frozen sessions with wasted GPU-hours.

## Approach / What changed

Subclass TensorRT's IProgressMonitor abstract base class in Python or C++ and assign it to IBuilderConfig. Override phase\_start, step\_complete, and phase\_finish to track nested phase hierarchies under a thread lock, and return false from step\_complete to trigger cancellation on SIGINT, webhooks, or agent timeouts.

## Takeaways

- The step\_complete callback is the sole IProgressMonitor method capable of halting a build by returning false, which prompts TensorRT to unwind active phases in reverse order.
- Thread safety is mandatory when implementing IProgressMonitor because TensorRT executes monitor callbacks from multiple internal builder threads.
- IProgressMonitor output can be forwarded to various surfaces, including ANSI terminal progress bars, Language Server Protocol notifications, FastAPI Server-Sent Events, or agent tool traces.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Observability](https://yomu.fyi/topic/observability), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://developer.nvidia.com/blog/make-long-running-nvidia-tensorrt-engine-builds-observable-and-cancelable-in-python-or-c)