ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale

Summary

ThunderAgent is a program-level scheduler for high-throughput agentic inference, developed by researchers at Georgia Tech, UIUC, CMU, and Together AI. It solves KV cache thrashing, which occurs when request-level engines like vLLM and SGLang evict an agent’s KV cache during tool calls and recompute it on resume, creating a cascade of evictions. ThunderAgent tracks each workflow as a schedulable program, pausing low-priority workflows under memory pressure and routing resumed programs through a global waiting queue to the node with the most available capacity. In synthetic data generation pipelines, it achieves up to 2.5x single-node throughput and 2.4x speedup on 8 nodes with near-linear scaling, and was accepted as an ICML 2026 Spotlight paper.

Takeaways

1. 1

   KV cache thrashing in agentic inference occurs because request-level schedulers evict a paused agent’s KV cache, forcing recomputation that cascades across agents under high concurrency.

2. 2

   ThunderAgent treats each multi-turn agent workflow as a schedulable program and pauses low-priority workflows under memory pressure to improve cache hit rates for remaining active agents.

3. 3

   ThunderAgent replaces static node-pinning with a global waiting queue that routes resumed workflows to the node with the most free capacity, balancing cache locality and load across the cluster.

FAQ