Speeding up GPU kernels by 38% with a multi-agent system

Summary

The post reports a three-week experiment in which a multi-agent system optimized 235 CUDA kernel problems generated from workloads spanning more than 124 production open-source models. Using SOL-ExecBench on 27 NVIDIA Blackwell 200 GPUs, the system used a planner to distribute and rebalance work, while autonomous workers benchmarked, debugged, and refined kernels without developer intervention. It ran separate CUDA C with inline PTX and CuTe DSL runs, reaching assembly-level optimization and learning a documented higher-level API. Against a single-agent optimized PyTorch baseline, it outperformed 149 of 235 problems, achieving a 1.38x geometric mean ratio, while 45 solutions exceeded 2x. The highlighted cases included an 84% geomean gain over a human FlashInfer baseline for paged-prefill attention, a 39% gain for NVFP4 MoE linear, and a GEMM reaching 86% of a tuned cuBLAS baseline; median SOL was 0.56, leaving room for improvement.

Takeaways

1. 1

   SOL-ExecBench evaluated solutions against software baselines and theoretical hardware limits on 27 NVIDIA Blackwell 200 GPUs, and invalidated results using tactics such as caching that exceeded what a B200 could support.

2. 2

   For BF16 grouped-query attention with paged prefill, the system used hardware-specific memory and math instructions, persistent kernels, and input-size hyper-optimization to reach a 0.9722 SOL score and an 84% geomean speedup over FlashInfer.

3. 3

   The system’s median SOL score was 0.56 despite the overall 38% geomean speedup, and the post says additional GPU compute could enable deeper exploration and further optimization.