The Modern CUDA Toolbox in Practice: A Step-by-Step Optimization Walkthrough

Summary

An image-processing pipeline serves as the test case for a modern CUDA toolbox, starting with RGB-to-grayscale conversion, 32-by-32 tile median computation, and host/device transfers. It advances through six incremental changes: adopting CCCL and Compute Sanitizer for indexing bugs, adding NVTX for Nsight Systems benchmarks, replacing hand-rolled work with CUB algorithms, using pooled device containers and pinned host containers, and assigning each image its own stream with asynchronous copies. The implementation uses cuda::stream, cuda::copy_bytes, cuda::device_buffer, and CUB calls on per-thread streams, allowing kernels and transfers to overlap instead of executing sequentially on the default stream. The reported total duration falls from 6.8 seconds to about 23 milliseconds for three images, a roughly 300x improvement, while the post characterizes the code as safer and easier to maintain without low-level optimizations.

Takeaways

1. 1

   The pipeline converts RGB images to grayscale, divides each grayscale image into 32-by-32-pixel tiles, computes each tile's median, and copies the medians back to the CPU.

2. 2

   CCCL supplies tools and APIs including Compute Sanitizer, cuda::stream, cuda::copy_bytes, cuda::device_buffer, and pinned host buffers for safer management and asynchronous execution.

3. 3

   Using a separate CUDA stream for each image enables overlap among kernels and memory transfers; the reported runtime drops from 6.8 seconds to about 23 milliseconds for three images.