Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler

Summary

The post introduces torch.profiler through a beginner-oriented walkthrough of profiling a matrix multiplication and addition, explaining how to configure the profiler and interpret its outputs. It uses a small function with torch.matmul and torch.add, annotates it with record_function, captures CPU and CUDA activities after warm-up iterations, and exports a profiler table plus a Chrome trace. The table summarizes event counts and CPU or GPU time, while the trace shows execution order, call hierarchy, CPU-GPU gaps, kernel launches, and overlap. Comparing 64x64 and 4096x4096 matrices demonstrates a shift from overhead-bound behavior to compute-bound behavior; the guide also covers dispatch chains, hidden device-to-device copies, profiler buffer activity, and torch.compile's runtime and caching behavior.

Takeaways

1. 1

   For 64x64 matrices, GPU work takes less than 1% of CPU time because kernel preparation, launches, and data movement dominate; the example is overhead-bound.

2. 2

   Increasing the matrix size to 4096 makes self CPU and CUDA totals comparable, with the GPU GEMM kernel taking most CUDA time and the workload becoming compute-bound.

3. 3

   In torch.compile traces, Dynamo cache lookup and the AOTAutograd-to-Inductor runtime stack remain visible on each call, so compilation overhead may exceed eager execution time for tiny operations.