Kernel Fusion in NVIDIA CUDA: Optimizing Memory Traffic and Launch Overhead

Summary

GPU compute often outpaces memory bandwidth, causing separate kernels to suffer from intermediate global memory round-trips and launch overhead. Kernel fusion resolves this by executing multiple operations within a single kernel to keep intermediate data inside registers. Using the sum(abs(x)) operation as a benchmark on an NVIDIA GeForce RTX 4090, a naive two-kernel setup moved 3 GB of global memory over 3.51 ms. In contrast, manual CUDA C++ fusion, PyTorch compiler fusion via torch.compile, and explicit Python fusion using cuda.compute all achieved a 3x speedup. Across all three approaches, global memory transfers dropped to 1 GB by computing absolute values inline during reduction.

Context

GPU compute speed frequently outpaces device memory bandwidth, causing un-fused kernels to waste bandwidth on intermediate global memory round-trips and incur separate kernel launch overheads.

Approach / What changed

Fusing operations such as element-wise transforms and reductions into single kernels keeps intermediate data in registers. This can be implemented via manual CUDA C++ with CCCL and CUB, implicit compiler fusion via torch.compile, or explicit Python composition using cuda.compute and TransformIterator.

Takeaways

• Manual CUDA C++ kernel fusion reduced global memory traffic from 3 GB to 1 GB and execution time from 3.51 ms to 1.18 ms on an RTX 4090 for sum(abs(x)).
• CUDA Graphs capture dispatches and reduce host launch overhead, but they do not fuse kernel bodies or eliminate global memory round-trips for intermediate buffers.
• cuda.compute brings CUB-backed iterator-based explicit fusion to Python, avoiding intermediate allocations without requiring custom CUDA C++ kernels.

Related reading

• NVIDIA Developer Blog · Jun 22, 2026

  CCCL Runtime: A Modern C++ Runtime for CUDA

  NVIDIA CCCL runtime introduces modern C++ abstractions for fundamental CUDA programming concepts starting in CUDA Toolkit 13.2 and CCCL 3.2. Designed as an alternative to the traditional CUDA runtime API, the library eliminates implicit global state by requiring explicit dependencies, such as binding streams directly to specific device references. The API enforces strong typing through owning types and non-owning reference types, omits default stream usage to ensure all streams remain non-blocking, and handles errors via standard C++ exceptions. In addition, CCCL runtime simplifies kernel launches through kernel functors that enable automatic template argument deduction and automatically transforms owning buffer arguments into device spans.

  Piotr Ciolkosz

• NVIDIA Developer Blog · Jun 16, 2026

  NVIDIA Blackwell Tops MLPerf Training 6.0 with Industry-Leading Scale and Performance