---
title: "Unlocking asynchronicity in continuous batching"
description: "Continuous batching improves GPU utilization with tightly packed batches, but its synchronous loop makes the CPU and GPU take turns, leaving the GPU idle while the CPU prepares each next batch. Profiling 8K-token generation with a batch size of 32 and an 8B model recorded 300.6 seconds, with the GPU idle for 24.0% of runtime. The post builds asynchronous batching by overlapping CPU preparation with GPU computation, using non-default CUDA streams, non-blocking transfers, and synchronization events to coordinate dependencies. In the same experiment, GPU activity rose to 99.4% and total generation time fell to 234.5 seconds, a 22% speedup, without new kernels or model changes while maintaining model accuracy."
---

# Unlocking asynchronicity in continuous batching

[Hugging Face](https://yomu.fyi/company/hugging-face) · Rémi Ouazan Reboul, Pedro Cuenca, Aritra Roy Gosthipaty · May 14, 2026

**Type:** Tutorial

## Summary

Continuous batching improves GPU utilization with tightly packed batches, but its synchronous loop makes the CPU and GPU take turns, leaving the GPU idle while the CPU prepares each next batch. Profiling 8K-token generation with a batch size of 32 and an 8B model recorded 300.6 seconds, with the GPU idle for 24.0% of runtime. The post builds asynchronous batching by overlapping CPU preparation with GPU computation, using non-default CUDA streams, non-blocking transfers, and synchronization events to coordinate dependencies. In the same experiment, GPU activity rose to 99.4% and total generation time fell to 234.5 seconds, a 22% speedup, without new kernels or model changes while maintaining model accuracy.

## Context

Synchronous continuous batching leaves the CPU and GPU alternating between useful work, so the GPU waits while the CPU samples outputs, updates request state, reschedules batches, and prepares inputs. In the profiled workload, this CPU overhead left the GPU idle for 24.0% of total generation time.

## Approach / What changed

Asynchronous batching separates CPU batch preparation from GPU computation. It uses non-default CUDA streams for GPU work, assumes non-blocking transfers, and coordinates dependencies with events so batch N+1 can be prepared while batch N computes. GPU-to-CPU output processing still synchronizes when outputs are ready.

## Takeaways

- In the 8K-token, batch-size-32, 8B-model experiment, synchronous generation took 300.6 seconds and left the GPU idle for 24.0% of runtime.
- Asynchronous batching raised GPU activity from 76.0% to 99.4% and reduced generation time to 234.5 seconds, producing a 22% speedup.
- The implementation overlaps CPU scheduling with GPU work through non-default CUDA streams and event-based synchronization, without adding kernels or changing the model.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/continuous_async)
- Source URL: https://huggingface.co/blog/continuous_async
- Ingested by Yomu: 2026-08-29T01:48:54.293Z

[Read original post](https://huggingface.co/blog/continuous_async)