Loading post…
TwilioWen Zhu, Xinghao HuangTutorial
How to Build a Voice AI Agent with OpenAI's GPT-Live, Twilio Agent Connect, and TypeScript
Summary
The post addresses the common problem of voice AI demos failing in production by introducing a solution using OpenAI’s GPT-Live and Twilio Agent Connect. GPT-Live is a full-duplex speech-to-speech model that separates conversation from background work, and Twilio Agent Connect provides a GPTLiveProvider to bridge Twilio Media Streams to GPT-Live without custom WebSocket plumbing. The tutorial walks through building a TypeScript application that connects inbound callers to a GPT-Live-powered voice agent, including setting up a Fastify server, configuring the session with a welcome instruction, and wiring tool calls via Responses delegation using a get_weather example. It also covers placing outbound calls using the same provider and demonstrates features like per-call customization and transcript logging. The result is a production-ready voice agent that can greet callers, answer questions, and execute backend functions, with the same pattern applicable to scenarios like appointment booking or order status lookups.
Context
The post provides a tutorial for building a production-ready voice AI agent using OpenAI’s GPT-Live and Twilio Agent Connect. It addresses the common failure of voice AI demos in production by offering a declarative config interface that bridges Twilio Programmable Voice Media Streams to GPT-Live without custom WebSocket code. The tutorial covers inbound and outbound calling, tool calls via Responses delegation, and features like welcome instructions and per-call customization.
Approach / What changed
Use Twilio Agent Connect’s GPTLiveProvider to bridge Twilio Media Streams to GPT-Live. Set up a VoiceChannel with a GPTLiveProviderConfig containing defaultSessionConfig (model, instructions, audio format) and welcomeInstruction. Define tools using defineTool() and register them in the provider config and delegation.responses.tools. Use TACServer to expose a TwiML endpoint. For outbound calls, call voiceChannel.initiateOutboundConversation().
Takeaways
GPT-Live is a full-duplex speech-to-speech model that separates conversation from background work, allowing tool execution outside the real-time voice loop.
Twilio Agent Connect’s GPTLiveProvider bridges Twilio Media Streams to GPT-Live using a declarative config interface, eliminating the need for custom WebSocket plumbing.
Tool calls are delegated to a Responses API model, and the actual TypeScript function is executed by Twilio Agent Connect, which then speaks back the result.
FAQ
How do I set up a welcome message that speaks first?
Set the welcomeInstruction property in GPTLiveProviderConfig. The agent will speak the provided text immediately upon connection without waiting for the caller.
How do I handle tool calls with GPT-Live?
Define tools using defineTool() and register them in the provider config's tools array and in delegation.responses.tools. The model delegates tool calls to a Responses API model, and Twilio Agent Connect executes the TypeScript function and speaks the result.
How do I place an outbound call?
Use voiceChannel.initiateOutboundConversation({ to: '+15551234567' }) with the number in E.164 format. TAC handles placing the call, connecting the Media Stream, and bridging to GPT-Live.
Source and freshness
- Source
- twilio.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.