Beyond Chat Bubbles: Building React Apps Where AI Generates Your UI

I'd like to start by showing you two different ways to interact with AI. Both are trying to solve the same problem, helping someone learn a new skill, but the experience couldn't be more different.

Here's what most of us are using today, a chat interface. You type in a question like, help me learn React books. And what do you get? It responds with text. Lots and lots of text. It's helpful information, but let's be honest, who wants to read all of this?

And if you have a follow-up question, you type that again. You get more text. And yes, it's a conversation, but it's slow. It's tedious. You have to scroll up and down to look at previous messages. And frankly, it's not how we actually want to learn and interact with these large language models.

Now check this out. If I type a question, I'll type, help me learn React books. And watch what we get. It actually generates a flash card for us. The AI is responding, but it's not just generating text, it's generating interfaces. Each one is a component. You can flip them and see the answer on the back. I could ask it for a multiple choice card. I could say quiz me with a multiple choice question. And let's see, it will generate another one. So let's go ahead and answer this. Which React hook should you use when you need to perform side effects on your components? I know it's use effect. Let's just say use state and see what happens. It shows us that was the wrong answer. Here was the actual correct answer, and here's why. So it gives us immediate feedback. So here's my question for you. Which experience would you rather use? Which would your users prefer? We've been building AI applications for over two years now.

ChatGPT launched in late 2022. And since then, almost every AI app we've built has been some sort of chat interface. Text in, text out. But here's the thing. We're not stuck because users want text. We're stuck because LLMs were designed to output strings. That's it. The technology literally returns text. And so we build chat bubbles around it and call it a day. But think about the questions that users are actually asking. Show me sales data for the last quarter. They don't want a paragraph, they want a chart. Or find properties near downtown. They don't want a description, they want cards with photos.

Or help me book a flight. They don't want just instructions, they want a form with options. And we've been forcing people to consume information through this one modality our technology could output. And that's not user-centered design. That's technology-constrained design. We have an LLM that generates text and then puts that text into chat bubbles and the result is frustrated users. And you know what? We can do better than this. The technology has evolved. LLMs can now trigger actions, call functions, execute tools, which means they can do more than generate words. They can generate interfaces.

And so that's what we're going to talk about today. It's not theory, it's not a future vision. I'm going to show you how to build this and to go beyond chat bubbles and create AI applications where the UI itself is generated in response to what your users need. So quick intro, I'm Jesse Hall. You might have heard of my YouTube channel CodeStacker. I'm a staff developer advocate at LiveKit, where we build open-source frameworks for voice and video applications.

And personally, I've been building web applications for over two decades. And in the last few years, I've been focusing heavily on AI-powered experiences. So before we dive into the how, how do we build generative UI applications, we need to clear up some confusion. Because when people say generative UI, they often are talking about completely different things. At its core, generative UI is simple. It's the process of allowing an AI to go beyond text and generate visual interfaces. Instead of an LLM returning a string that you render in a chat bubble, the LLM triggers the creation of UI components, charts, cards, forms, dashboards, whatever makes sense for the user's request.

But here's where it gets interesting. There's this debate happening right now in the AI community about what generate actually means. And honestly, the confusion is justified because there are fundamentally different approaches. I think of this as a spectrum with four levels. Level one is enhanced markdown. Level two is component selection. Level three is declarative UI generation. And level four is true code generation. Each one gives you different tradeoffs between safety and flexibility. So let's dive into each one of these levels.

Level one is what most chatbots today use, enhanced markdown. The AI outputs rich text like tables, code blocks, formatted lists. And so this is better than plain text, but it's still static content. Chat GPT rendering a markdown table, that's level one. It's an improvement, but it's not really generative UI. It's just formatted text. Level two is where things start to get practical. This is what I call component selection, and it's what we're starting to see with some production applications today, including the demo that you just saw. So here's how it works. You build a library of React components ahead of time, a weather card, a line chart, flash cards, quiz components, whatever your application needs. The AI doesn't create these components, you do.

But the AI decides which components to use and what data to pass to them. So the AI is not a designer writing UI code. It's more like an intelligent router deciding which one to display. The AI says the user is asking about weather, so I should call the weather tool and then I should render the weather card component with the data. The components are pre-built, tested, and predictable, and they are safe. But they're composable in infinite combinations based on what the user is asking for.

So notice what's happening here. The AI isn't writing React code. It's calling this tool, just like it would call a function to fetch weather data or search a database. The tool returns data, and then we map that data to a component. And this is production-ready. It's safe. It's testable. And think about this, the users, they don't care if the UI is generated or selected. They just care that it works. And this works today.

Level three is the middle ground, declarative generation. So instead of selecting pre-built components, AI generates a specification that describes what the UI should be. Your application then interprets that specification at runtime. So think of it like a domain-specific language for UI. This gives you more flexibility than pure component selection. And the AI can create novel combinations and layouts. But you're still maintaining your quality control because your interpreter defines what's possible.

And then level four is really the wild west, true code generation. The AI writes actual React or HTML code as a string. Your application compiles or interprets it at runtime and then renders it. This is what tools like vZero, Cloud Artifacts, Bolt, and others do.

It's maximum flexibility. The AI can literally create interfaces from scratch. But today, this is dangerous. For production applications, this is risky. You're executing code that an AI wrote at runtime. So this opens you up to security vulnerabilities, unpredictable output quality, accessibility issues that you can't control, performance problems, and maintenance nightmares.

It's amazing for prototyping and internal tools, but for customer-facing applications, most teams aren't ready for this today. So let's go back to this debate. The component selection camp says this is real generative UI. The AI dynamically chooses what to show based on context. Prebuilt components give us safety, consistency, and production readiness.

We're not limiting the AI. We're giving it reliable building blocks. And then we have the true generation camp that says component selection is just glorified function calling, not real generative UI. You're forcing the AI into boxes instead of letting it truly innovate. Why constrain AI to prebuilt options when it could generate anything?

Both have valid points. But here's what I've learned by building real applications. The right approach depends on your context. Are you prototyping? Go wild with code generation. Are you building internal tools for your team? Maybe declarative generation is right for you. Are you shipping to users who expect reliability and consistency? Component selection is your friend.

We're using component selection for the AI tutor demo today because it's the right tool for the job. Predictable, safe, and production ready. For the AI tutor, we chose the Brazil AI SDK and here is why. Number one, we wanted full control over the educational experience. Teaching requires consistency. Every flash card needs to work the same way. Every quiz needs to be reliable. Prebuilt components let me test and refine that experience.

Number two, the AI SDK supports any LLM provider and so there's no vendor lock-in there. Number three, the tool to component mental model is incredibly clear. When I explain that to other developers, how this works, they just get it immediately. Tools are just functions. Components are just React and the AI connects them. For number four, it is production ready. This isn't experimental. This is the same technology powering AI features in real products right now. Again, this mental model is key to understanding generative UI with component selection.

Here's the complete flow. Step one, define the tools. Step two, the LLM decides to call the tool. Step three, the tool, a result gets streamed to the client and step four, the client renders the component based on the tool type. And that's it. It's just React components triggered by AI tool calls. The AI isn't writing code. It's not compiling anything. It's just calling functions that you define and those functions result in rendering components that you built. The generative part is that the AI dynamically decides which of these tools to call and when to call them based on what the user is asking. But under the hood, it's React doing what React does, rendering components based on data.

Okay, now you understand what generative UI is. You've seen the options. You know the mental model. What if we could make this even better? The AI tutor that you saw at the beginning works great with text. You can type in questions. You get interactive flashcards. It's already 10 times better than a traditional chatbot. But I'd like to show you something that transforms the experience even further. Voice, and I don't mean voice as in faster input. I mean voice as in a completely different interaction paradigm.

Think about how you learn with the text-based tutor. You think of a question. You type it out, which is slow. It requires a keyboard, breaks your flow. You wait for the response. You read the explanation. You look at the flashcards. You think of another question, and you repeat. Every step requires your full attention. Your hands are on the keyboard. Your eyes are on the screen. And you're locked into this device.

Now imagine you're trying to study while you're making dinner, going for a walk, doing the laundry. You can't do that. The interaction model does not allow that. But watch what happens when we add voice. Let's click Start Voice. Hey there. Great to meet you. What would you like to dive into today? Help me learn React Hooks. Great. Let me create your first flashcard about React Hooks. What does the useState hook do in React? Show me the answer.

It lets you add state to functional components. Want to dive into more details or try another flashcard? Give me a multiple choice question. Which React Hook is used for performing side effects in function components? Let's see. I think that might be D. No. I got it wrong. The AI hears me, responds with voice, and while it's speaking, flashcards appear on the screen.

I can ask it to flip the card. It flips the card. I can ask it for another question, multiple choice questions. And when I answer the question verbally, it tells me if I'm right or I'm wrong, and it can give me extra explanation. My hands never touch the keyboard or mouse, and I could do all of this while cooking, exercising, commuting, whatever.

Voice doesn't replace the visual UI. It enhances it. I'm getting audio feedback and visual components. It's multimodal, and that's the magic. Now I built an AI tutor because education is a great demonstration. But think about how this pattern applies everywhere. Like for instance, data exploration.

You could say, show me sales by region. The AI could produce a chart and say here's your regional breakdown, and then you can ask which region is growing fastest. The chart could highlight a specific place, saying Northeast is up by 34%. And then you could say drill down into that region, and a new chart could display the city level details. Wouldn't that be amazing? Or maybe for e-commerce, you could say, find winter jackets under $200.

The AI could show 12 different options, product cards with those options. And then you could say, show me the warmest one, and it could highlight the card and say it's rated for negative 20 degrees. And then you could say, add that to my cart, and it appears in your cart.

Or how about workflow automation? You could say, schedule a meeting with the design team. A calendar UI could appear, and it could say, I found three open slots, and you could say the first one works. And then the meeting's scheduled, and a confirmation appears. And so the patterns are the same.

Voice for natural conversation, UI for rich information display, and both work together. From a technical perspective, adding voice is more straightforward than you might think. We're using LiveKit for the voice layer. LiveKit is an open source framework for building voice and video agents. And it's actually what powers ChatGPT's advanced voice mode. So here's what that looks like.

You probably use LiveKit every day. So here's what that looks like. First, we define our LiveKit agent, and that handles voice input and output. Next, we define our entry and connect. And then we create a voice session. Now, the beautiful part is that our tools stay the same. Generate Flashcard still does what it did before, but now instead of just returning data through a REST API, the agent can respond with voice and send UI commands to the client simultaneously. And so the client receives both the voice audio stream and the component data, rendering them together.

So we've gone from text-only chat bubbles to interactive components, to voice-enabled multimodal experiences. But where is all of this going? Well, the future is closer than you think. Soon, the AI won't just hear your voice. It'll see what you're looking at. Hold up your homework to the camera and ask it, can you help me with this problem? An AI will be able to see the page, understand the questions, and generate a step-by-step visual walkthrough. That's not five years away. That's this year.

In some ways, that is already here. The AI will be able to remember your preferences. If you're a visual learner, it's going to generate more diagrams for you. If you learn by doing, maybe it will create more interactive exercises. The interface adapts to you, not the other way around. And imagine voice-enabled AI that lives with you throughout your day. Studying during your commute, getting help while cooking, learning while exercising, the device fades into the background and the learning becomes seamless.

But here's what I want you to take away. You don't have to wait for that future. The foundation is here now. Everything that I've showed you today is production ready. The Vercel AI SDK has over three and a half million downloads weekly. It's used in real applications. LifeKit powers Chat GPT's advanced voice mode. It handles billions of calls. It has production-grade LLMs with excellent tool use. And React, well, React is React. It's not going to go anywhere. So this isn't a concept demo. This isn't a research project. This is technology that you can start using today.

So the question isn't, can we build this? The question is, what's the right approach for your application? And that depends on your constraints. Do you need full control? Well, look at component selection, the Vercel AI SDK Copilot Kit. Do you want to prototype fast? Use an API approach like Thesis C1. Are you building MCP tools? Well, look for the MCP UI framework. Or do you need code generation? Well, run it at your own risk and be sure to carefully sandbox. There's no single right answer. There's only the right answer for your context.

And so let me close with this. For two years, we've been building AI applications with one hand tied behind our back. We constrained ourselves to text because that's what the technology could do. But we're developers, and we don't accept arbitrary constraints. When the web could only show documents, we built applications. When phones could only make calls, we built ecosystems. When chat was the only option for AI, we built lots of chat interfaces. Yeah. But now, now we can build interfaces that adapt to what users need, not chat bubbles when they need charts, not paragraphs when they need interactive components, not typing when they need to have their hands free. So my challenge to you is simple. Pick one feature in your app, one place where users are suffering through text responses. Build a component, connect a tool, ship it, learn from it. Go beyond the chat bubble, and let's figure out together where this technology can go next. Thank you.