No Dependencies, No Problem: Streaming AI Over the Phone

Hi, everyone. I'm Marius. I work on the developer relations team at Twilio, and that means I speak to a lot of developers who use our APIs, such as the text messaging API or the voice API. And one thing that a lot of developers recently wanted to build is an AI agent that can make a phone call or receive a phone call. Let me show you how this story often goes from a developer perspective. You have all the great models that you want to combine, one for automatic speech recognition, one for interrupt detection, and the text-to-speech model. You combine them all together, and, in theory, it works nice, but then they quickly realize latency.

The latency is just you say something, you wait, nothing happens, and then you say something again, and then the model starts to talk, and that kills the entire experience. So they need to find a way to work around this. Something that we provide at Twilio is that you can basically left shift a lot of responsibility on our infrastructure, such as automatic speech recognition, text-to-speech, and we work with best-in-class providers, such as 11 Labs or Google Cloud, to provide these services, and you just need to focus on the configuration.

And what you actually need to do in the end is to host a WebSockets server that receives text, and then you can process it with your own LLMs. You can post it to an LLM that runs close to your machine with low latency, and you just stream text back. And, actually, you can build an agent in three minutes. Let's do that together. So I use BUN to host a simple web server. You can just, for the fun of using a new stack every now and then, I expose it on port 5050, and then I have this fetch function that the only thing it does, it upgrades HTTP to WebSocket, and then I attach a data object, so I can recognize the same stream again.

It would make sense to use the phone number here, but I would just use a timestamp, because I would be the only one calling it anyway. And then in the WebSocket configuration, I just have a callback when the socket is open. When it's closed, let's format it a bit. And then the interesting part happens here, when a message comes in. I pass the JSON payload, and then when the message is of type prompt, which it will mostly be. Anyway, I log this to the console, and for now, let's have a hard-coded answer to say that's a great question. And then I log it to the console, and I just stream it back. And what I also do is I log the other types so you see them, but we don't have to worry about them for now.

Okay, let's give it a run. I run my server. I expose this port to the internet, and if I go over here to my configuration, you see whenever a call comes into a phone number, it connects to my WebSocket server. This is the I use 11 labs for text-to-speech. This particular idea of the voice, and this is the starting sentence. Let's call it, and let's hope it works. Is the audio? Let me check. That's a great question. Hey, what's the capital of France? That's a great question. You see, I always get the same response back. Why? Because I don't do anything here. I just say return a static response, but you saw how fast the latency was. So, if I look at the logs, you see the text-to-speech, and speech-to-text happened instantly. Now, let's actually involve some AI here, and if I drill down into that, that's auto-imported.

You see, what do I do here? Well, I have an if statement to see do I already know this conversation ID. If I don't know it, I just use the GPT4 or mini model. I add a system prompt, and the prompt of the user. I add a web search tool, and I make sure to store the history on the server side at OpenAI, and then if the response comes back, I save it. So, when I ask another question the next time, the if statement is triggered here, and then I can refer to the previous conversation, so I don't have to carry on that messages array all the time. I just have to add the most recent prompt. Let's try that again. I restart the server.

Call it again. Hello, you are connected to the AI voice assistant powered by Twilio Conversation Relay. What's the capital of France? The capital of France is Paris. And the one of Germany? The capital of Germany is Berlin. Berlin. Who won the UEFA Nations League last weekend? Portugal won the UEFA Nations League final against Spain on June 8, 2025 in Munich. The match ended 2-2 after extra time, and Portugal secured a 5-3 victory in the penalty shootout. So you got the idea, right?

And here I was logging the response time from OpenAI to the console. You could tell when it had to do a web search. It took a little bit longer. Things you could do, you could say, oh, you could add a message like, oh, let me think about it. And then you triggered a request, so there are ways to work around it to make it appear quicker. But you can already tell basically the latency from speech to text and text to speech was basically zero, I would say.

And then you only had the one from the LLM, which is probably also not the best on this network on my local MacBook. But that's my demo and whoop, just like that, the latency problems were gone. Thank you. I'm Marius. If you have questions about this or other AI use cases with voice, our booth is up there. You can find me in the next break. Thank you.