---
title: "Scaling real-time AI agents with session-aware load balancing"
description: "Real-time AI agents require infrastructure that manages continuous, bidirectional conversations rather than isolated requests, including interruptions that halt speech generation and update context without dropping the connection. The post argues that QPS and instantaneous CPU utilization are insufficient because they miss long-lived committed work and silent sessions that can become active simultaneously. It proposes application-level active-session tracking around the stream lifecycle, with cleanup in finally blocks and consistent snapshots, then combines session pressure with CPU and memory utilization in a hybrid routing model. The model normalizes session counts into a rate and uses target utilization, smoothed utilization, cost per session, and a safety scaler to estimate effective capacity and regulate new assignments. Validation should use long-lived, varied workloads and metrics such as session distribution, overloaded assignments, startup latency, dropped sessions, and counter behavior; high-concurrency trackers may require sharded or LongAdder-style aggregation."
---

# Scaling real-time AI agents with session-aware load balancing

[Google](https://yomu.fyi/company/google) · Simerus Mahesh · Aug 3, 2026

**Type:** Explainer

## Summary

Real-time AI agents require infrastructure that manages continuous, bidirectional conversations rather than isolated requests, including interruptions that halt speech generation and update context without dropping the connection. The post argues that QPS and instantaneous CPU utilization are insufficient because they miss long-lived committed work and silent sessions that can become active simultaneously. It proposes application-level active-session tracking around the stream lifecycle, with cleanup in finally blocks and consistent snapshots, then combines session pressure with CPU and memory utilization in a hybrid routing model. The model normalizes session counts into a rate and uses target utilization, smoothed utilization, cost per session, and a safety scaler to estimate effective capacity and regulate new assignments. Validation should use long-lived, varied workloads and metrics such as session distribution, overloaded assignments, startup latency, dropped sessions, and counter behavior; high-concurrency trackers may require sharded or LongAdder-style aggregation.

## Context

Standard request-based load balancing relies on request throughput and current resource utilization, but long-lived, stateful AI streams create committed workloads that QPS and instantaneous CPU measurements do not fully represent. Open bidirectional connections also contain application state that generic infrastructure cannot distinguish accurately.

## Approach / What changed

Track active sessions inside the backend when streaming lifecycles begin and end, ensuring cleanup covers timeouts, cancellations, and disconnects. Report consistent session snapshots and combine active-session pressure with CPU or memory utilization. Normalize session pressure into a rate, apply a safety scaler, and use the result to estimate effective capacity and control new assignments.

## Takeaways

- Active-session counts expose committed concurrency that QPS misses, while CPU utilization alone can make silent sessions appear cheap until users speak simultaneously.
- The tracker increments at stream start and decrements in a finally block; timeout, cancellation, and disconnect paths must avoid duplicate or missing cleanup.
- For JVM services, high-concurrency tracker tests should measure contention and may favor sharded counters or LongAdder-style aggregation over a heavily contended AtomicInteger.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Load Balancing](https://yomu.fyi/topic/load-balancing), [Streaming](https://yomu.fyi/topic/streaming)

- Source: [Google](https://developers.googleblog.com/scaling-real-time-ai-agents-with-session-aware-load-balancing)
- Source URL: https://developers.googleblog.com/scaling-real-time-ai-agents-with-session-aware-load-balancing
- Ingested by Yomu: 2026-09-09T14:28:02.719Z

[Read original post](https://developers.googleblog.com/scaling-real-time-ai-agents-with-session-aware-load-balancing)