---
title: "Choosing a low-latency infrastructure layer for conversational AI"
description: "Conversational AI call-center latency can feel like dead air, and the delay may come less from LLM inference than from the infrastructure moving audio among telephony, media servers, ASR, the agent, and TTS. The described approach uses a unified voice path: ConversationRelay provides a WebSocket API to stream live call audio, handle speech-to-text and text-to-speech in one connected pipeline, connect with a backend agent, and support barge-in. It is model-agnostic, allowing teams to retain their LLM, knowledge bases, AI tools, and third-party voice providers, while Conversation Orchestrator, Conversation Memory, and Agent Connect address context and integration. Twilio’s internal benchmarks report median latency below 0.5 second and below 0.725 second at the 95th percentile, though relevance and context also matter alongside speed."
---

# Choosing a low-latency infrastructure layer for conversational AI

[Twilio](https://yomu.fyi/company/twilio) · Luke Morgan · Sep 8, 2026

**Type:** Problem & solution

## Summary

Conversational AI call-center latency can feel like dead air, and the delay may come less from LLM inference than from the infrastructure moving audio among telephony, media servers, ASR, the agent, and TTS. The described approach uses a unified voice path: ConversationRelay provides a WebSocket API to stream live call audio, handle speech-to-text and text-to-speech in one connected pipeline, connect with a backend agent, and support barge-in. It is model-agnostic, allowing teams to retain their LLM, knowledge bases, AI tools, and third-party voice providers, while Conversation Orchestrator, Conversation Memory, and Agent Connect address context and integration. Twilio’s internal benchmarks report median latency below 0.5 second and below 0.725 second at the 95th percentile, though relevance and context also matter alongside speed.

## Context

Call-center voice AI latency can result in pauses that feel like dead air and may cause callers to hang up. The source attributes much of this delay to audio moving through separate telephony, media server, ASR, LLM, and TTS systems, rather than to LLM inference alone. It also states that maintaining context and relevant responses matters for keeping callers engaged.

## Approach / What changed

ConversationRelay provides a WebSocket API and a single connected voice pipeline for live call audio, speech-to-text transcription, backend agent processing, and text-to-speech synthesis. It supports barge-in, works with an existing LLM, knowledge base, AI tools, and third-party voice providers, and integrates with Twilio Voice. Conversation Orchestrator, Conversation Memory, and Agent Connect support context retention and agent connectivity.

## Takeaways

- The source attributes most call latency to handoffs among telephony, media servers, ASR, the LLM agent, and TTS, rather than to LLM inference alone.
- ConversationRelay combines live audio streaming, speech-to-text, backend agent processing, and text-to-speech in one WebSocket-based path, with support for natural caller interruptions through barge-in.
- Twilio’s internal benchmarks report ConversationRelay latency below 0.5 second at the median and below 0.725 second at the 95th percentile; the source also emphasizes context and response relevance.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Latency](https://yomu.fyi/topic/latency), [LLMs](https://yomu.fyi/topic/llm), [WebSockets](https://yomu.fyi/topic/websockets)

- Source: [Twilio](https://www.twilio.com/en-us/blog/insights/low-latency-layer-conversational-ai)
- Source URL: https://www.twilio.com/en-us/blog/insights/low-latency-layer-conversational-ai
- Ingested by Yomu: 2026-09-09T13:36:35.067Z

[Read original post](https://www.twilio.com/en-us/blog/insights/low-latency-layer-conversational-ai)