How to handle real-time interruptions in your AI voice agent

Summary

Real-world AI voice agent calls are messy, with background noise, overlapping speech, and callers interrupting mid-sentence. Twilio Conversation Relay handles interruptions through attributes like interruptible, which controls whether caller input stops the agent’s speech, and ignoreBackchannel, which filters out short backchannel phrases. Background noise that sounds like speech can cause false interruptions and garbled transcripts; Twilio’s testing found Deepgram Flux produced zero transcription failures under combined noise, while a tuned Nova-3 setup also succeeded. Turn detection uses either a traditional silence timer or Deepgram Flux’s fused model, which reduces false interruptions by roughly 30% and trims response latency. When a caller interrupts, the app receives an interrupt message with the exact text played before cutoff, allowing it to trim the LLM’s conversation history so the model only remembers what the caller heard.

Takeaways

1. 1

   Twilio Conversation Relay’s interruptible attribute controls which caller input can stop agent speech: none, dtmf, speech, or any (default).

2. 2

   Deepgram Flux fuses transcription and turn detection, reducing false interruptions by roughly 30% and trimming response latency by 200–600 milliseconds compared to Nova-3.

3. 3

   When a caller interrupts, the app must trim the LLM’s conversation history using the utteranceUntilInterrupt value from the interrupt message to prevent the model from referencing unheard content.

FAQ