Unlocking asynchronicity in continuous batching

Summary

Continuous batching improves GPU utilization with tightly packed batches, but its synchronous loop makes the CPU and GPU take turns, leaving the GPU idle while the CPU prepares each next batch. Profiling 8K-token generation with a batch size of 32 and an 8B model recorded 300.6 seconds, with the GPU idle for 24.0% of runtime. The post builds asynchronous batching by overlapping CPU preparation with GPU computation, using non-default CUDA streams, non-blocking transfers, and synchronization events to coordinate dependencies. In the same experiment, GPU activity rose to 99.4% and total generation time fell to 234.5 seconds, a 22% speedup, without new kernels or model changes while maintaining model accuracy.

Takeaways

1. 1

   In the 8K-token, batch-size-32, 8B-model experiment, synchronous generation took 300.6 seconds and left the GPU idle for 24.0% of runtime.

2. 2

   Asynchronous batching raised GPU activity from 76.0% to 99.4% and reduced generation time to 234.5 seconds, producing a 22% speedup.

3. 3

   The implementation overlaps CPU scheduling with GPU work through non-default CUDA streams and event-based synchronization, without adding kernels or changing the model.