Kernel Fusion in NVIDIA CUDA: Optimizing Memory Traffic and Launch Overhead

Summary

GPU compute often outpaces memory bandwidth, causing separate kernels to suffer from intermediate global memory round-trips and launch overhead. Kernel fusion resolves this by executing multiple operations within a single kernel to keep intermediate data inside registers. Using the sum(abs(x)) operation as a benchmark on an NVIDIA GeForce RTX 4090, a naive two-kernel setup moved 3 GB of global memory over 3.51 ms. In contrast, manual CUDA C++ fusion, PyTorch compiler fusion via torch.compile, and explicit Python fusion using cuda.compute all achieved a 3x speedup. Across all three approaches, global memory transfers dropped to 1 GB by computing absolute values inline during reduction.

Takeaways

1. 1

   Manual CUDA C++ kernel fusion reduced global memory traffic from 3 GB to 1 GB and execution time from 3.51 ms to 1.18 ms on an RTX 4090 for sum(abs(x)).

2. 2

   CUDA Graphs capture dispatches and reduce host launch overhead, but they do not fuse kernel bodies or eliminate global memory round-trips for intermediate buffers.

3. 3

   cuda.compute brings CUB-backed iterator-based explicit fusion to Python, avoiding intermediate allocations without requiring custom CUDA C++ kernels.