Add an AI Voice Assistant to a Twilio Video Room with Conversation Relay

Summary

The tutorial addresses wasted patient time in telehealth waiting rooms by adding an AI voice assistant to a Twilio Video room for intake collection. It combines Twilio Programmable Video, Conversation Relay, and an LLM, with the assistant joining as an audio-only participant and handling questions about visit reasons, symptoms, and medications. A two-leg Voice call bridges the room to Conversation Relay: one leg participates in the room, while the other sends transcribed audio to a WebSocket server and returns spoken LLM responses. The tutorial uses Node.js, Express, the Twilio helper library, OpenAI, ws, dotenv, and ngrok, and explains setup, browser interaction, and troubleshooting. Its stated result is a working application in which a patient converses naturally with the AI before a provider joins, with GPT-4o-mini processing speech.

Takeaways

1. 1

   Twilio Programmable Voice calls can join a Twilio Video room as audio-only participants, allowing an AI assistant to hear and speak with room participants.

2. 2

   Conversation Relay manages transcription, speech synthesis, interruption detection, and turn-taking, so the application WebSocket handler exchanges text with the voice system.

3. 3

   The demonstrated application lets a patient interact with an AI intake assistant before a provider joins; the test flow uses GPT-4o-mini for response processing.