CCCL Runtime: A Modern C++ Runtime for CUDA

Summary

NVIDIA CCCL runtime introduces modern C++ abstractions for fundamental CUDA programming concepts starting in CUDA Toolkit 13.2 and CCCL 3.2. Designed as an alternative to the traditional CUDA runtime API, the library eliminates implicit global state by requiring explicit dependencies, such as binding streams directly to specific device references. The API enforces strong typing through owning types and non-owning reference types, omits default stream usage to ensure all streams remain non-blocking, and handles errors via standard C++ exceptions. In addition, CCCL runtime simplifies kernel launches through kernel functors that enable automatic template argument deduction and automatically transforms owning buffer arguments into device spans.

Context

As CUDA programs grow more complex with multiple libraries sharing devices, streams, and memory, the traditional CUDA runtime API relies on implicit global state, such as associating new streams with whichever device happens to be active and depending on manual error-checking macros or status return codes.

Approach / What changed

CCCL runtime provides a collection of modern C++ headers including cuda/stream, cuda/buffer, and cuda/launch with strong typing and explicit dependencies. It distinguishes owning types from non-owning ref types for interoperability, excludes the implicit default stream, uses C++ exceptions for error handling, supports kernel functors with automatic template deduction, and automatically converts buffers to spans during kernel launch.

Takeaways

• CCCL runtime replaces implicit global state with explicit dependencies, requiring streams to be constructed with a specific device reference and making all created streams non-blocking.
• The runtime uses paired owning types and non-owning ref types, allowing seamless conversions and lifetime management alongside legacy raw handles like cudaStream_t.
• Kernel functors with device call operators enable automatic template argument deduction during launch, while buffers passed to cuda::launch automatically transform into cuda::std::span.

Related reading

• NVIDIA Developer Blog · Jul 10, 2026

  Kernel Fusion in NVIDIA CUDA: Optimizing Memory Traffic and Launch Overhead

  GPU compute often outpaces memory bandwidth, causing separate kernels to suffer from intermediate global memory round-trips and launch overhead. Kernel fusion resolves this by executing multiple operations within a single kernel to keep intermediate data inside registers. Using the sum(abs(x)) operation as a benchmark on an NVIDIA GeForce RTX 4090, a naive two-kernel setup moved 3 GB of global memory over 3.51 ms. In contrast, manual CUDA C++ fusion, PyTorch compiler fusion via torch.compile, and explicit Python fusion using cuda.compute all achieved a 3x speedup. Across all three approaches, global memory transfers dropped to 1 GB by computing absolute values inline during reduction.

  Michelle Horton

• NVIDIA Developer Blog · Jun 25, 2026

  Streamlining Resource Binding with End-to-End Support for Vulkan Descriptor Heaps