---
title: "ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale"
description: "ThunderAgent is a program-level scheduler for high-throughput agentic inference, developed by researchers at Georgia Tech, UIUC, CMU, and Together AI. It solves KV cache thrashing, which occurs when request-level engines like vLLM and SGLang evict an agent’s KV cache during tool calls and recompute it on resume, creating a cascade of evictions. ThunderAgent tracks each workflow as a schedulable program, pausing low-priority workflows under memory pressure and routing resumed programs through a global waiting queue to the node with the most available capacity. In synthetic data generation pipelines, it achieves up to 2.5x single-node throughput and 2.4x speedup on 8 nodes with near-linear scaling, and was accepted as an ICML 2026 Spotlight paper."
---

# ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale

[Together AI](https://yomu.fyi/company/together-ai) · Jul 29, 2026

**Type:** Announcement

## Summary

ThunderAgent is a program-level scheduler for high-throughput agentic inference, developed by researchers at Georgia Tech, UIUC, CMU, and Together AI. It solves KV cache thrashing, which occurs when request-level engines like vLLM and SGLang evict an agent’s KV cache during tool calls and recompute it on resume, creating a cascade of evictions. ThunderAgent tracks each workflow as a schedulable program, pausing low-priority workflows under memory pressure and routing resumed programs through a global waiting queue to the node with the most available capacity. In synthetic data generation pipelines, it achieves up to 2.5x single-node throughput and 2.4x speedup on 8 nodes with near-linear scaling, and was accepted as an ICML 2026 Spotlight paper.

## Context

LLMs are increasingly used as agents that multi-turn reason and use tools, requiring large-scale synthetic data generation since natural web data lacks agentic trajectories.

## Approach / What changed

A lightweight scheduling layer that abstracts each agentic workflow as a schedulable program, monitoring execution phase, KV cache footprint, and node placement. It pauses low-priority workflows under memory pressure to reduce cache contention, then resumes them through a global waiting queue onto the node with the most free capacity.

## Takeaways

- KV cache thrashing in agentic inference occurs because request-level schedulers evict a paused agent’s KV cache, forcing recomputation that cascades across agents under high concurrency.
- ThunderAgent treats each multi-turn agent workflow as a schedulable program and pauses low-priority workflows under memory pressure to improve cache hit rates for remaining active agents.
- ThunderAgent replaces static node-pinning with a global waiting queue that routes resumed workflows to the node with the most free capacity, balancing cache locality and load across the cluster.

## FAQ

### How does ThunderAgent avoid KV cache thrashing?

It monitors memory pressure per node and selectively pauses low-priority workflows when memory is tight, reducing the number of competing agents. When a paused workflow resumes, the global waiting queue routes it to the node with the most available capacity.

### Does ThunderAgent work with existing inference engines?

Yes, it is a drop-in layer requiring only a program\_id field and is OpenAI-compatible, working with engine-level optimizations like speculative decoding.

### What throughput gains does ThunderAgent provide?

On a single 8xH100 node with HiCache offloading at batch size 192, ThunderAgent achieves 803 tokens/s versus 390 tokens/s for the baseline, with mean latency of 10.6s compared to 65s.

**Tags:** [Inference](https://yomu.fyi/topic/inference), [Performance](https://yomu.fyi/topic/performance), [Scheduling](https://yomu.fyi/topic/scheduling)

- Source: [Together AI](https://www.together.ai/blog/thunderagent)
- Source URL: https://www.together.ai/blog/thunderagent
- Ingested by Yomu: 2026-09-19T10:13:43.147Z

[Read original post](https://www.together.ai/blog/thunderagent)