TwilioAmanda Lange, Dylan FrankcomTutorial
Voice agents can converse, but need external APIs to retrieve live data or take actions such as checking inventory and booking appointments. This tutorial builds a Python and FastAPI voice agent with Twilio Conversation Relay, OpenAI tool calling, and an unauthenticated Cat Facts REST API. A FastAPI webhook returns TwiML that connects callers to a websocket, while Conversation Relay messages are converted into conversation history and streamed responses. When the model requests a cat fact, the application executes the tool, sends its result back to OpenAI, and streams the follow-up answer to the caller. The setup uses environment variables for the OpenAI key, httpx for asynchronous external requests, and ngrok to expose local webhooks; the completed test lets a caller ask for and receive a cat fact.
The tutorial demonstrates how a Twilio voice agent can use LLM tool calling to obtain real-time information from an external HTTP API. Its example uses the Cat Facts API, which requires no additional authentication.
Create a Python virtual environment, install FastAPI, Uvicorn, OpenAI, Twilio, python-dotenv, and httpx, then configure the OpenAI key in a .env file. The application exposes a POST /voice endpoint that generates TwiML and a /ws websocket for Conversation Relay. It sends caller messages to OpenAI, executes the selected external API tool, returns the tool result to the model, and streams the final response back through the websocket. ngrok exposes the local server for Twilio testing.
The /voice endpoint reads the incoming Host header to construct the Conversation Relay websocket URL, avoiding manual hostname configuration when the ngrok forwarding hostname changes.
When OpenAI returns a tool call, the application executes the named tool, appends its result as a tool message, and makes another streamed completion request for the answer delivered to the caller.
The Cat Facts demo needs no additional API authentication, but expanded applications must store and provide credentials for external APIs that require authentication.
The application provides OpenAI with a tool function schema and a system prompt describing when external calls should be made. If the model needs a cat fact, it returns a tool call instead of ordinary response text, and the application executes it.
The tool result is appended to the OpenAI conversation as a tool message. The application then requests another streamed completion and sends its text tokens through the Conversation Relay websocket.
The ngrok URL in the Twilio console must match the URL currently shown by ngrok and include /voice, with the incoming HTTP method set to POST. API keys must also be correct, and any expanded external API must be authenticated according to its requirements.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Axum is a Rust web framework for composing routes, middleware, and extractors. This explainer shows how different login flows become verified identity before a handler responds.
A practical asyncio workflow for moving independent work through typed events, queues, concurrent tasks, and awaited API calls. Learn where validation and fallback belong.
Apple integrations are boundary designs: native capability, server-side mapping, and trust checks must meet a platform-agnostic product layer. See where each responsibility belongs.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
