Chat With Your Components

Hi. I hope you're enjoying the conference so far. My name is Teodor. I'm a web developer building Proxima, an open source analytics platform. In my talk today, I'll try to explain how you can build smart user interfaces using React and artificial intelligence.

In my honest opinion, React has been a game changer for web development. JSX index allowed us to repurpose the way we build user interfaces. Imagine you have a bunch of components in the design system. These components are like Lego bricks, but way more powerful and flexible. A component can fit data, generate content, and pass down the result to other components. Then you can start building using these primitives and compose them to build complex user interfaces.

But there's a fundamental problem. Every single time you build something, well, you have to actually write the code, right? Every time you want to make a change, you have to combine the data, the UI, and the logic. So, the end users are somehow tightly coupled with their release cycles of the developers. When chat2BT came out, though, it reintroduced an idea coming from the early 2000s. The chat interface. A symbol and so common user experience. Asking questions and getting answers back in plain text, it's the most straight forward way to interact with artificial intelligence. With the public release of new AI APIs, access to advanced technology became more open, allowing us to create applications that push beyond typical limits.

For this talk, we'll try to experiment a bit with the AI packets. This is a thin wrapper around open AI and other model providers along with some helpers, hooks, and abstractions built on top of React and the Node.js ecosystem. The AI packets allow us to directly interact with AI models from our React components or API routes. So, we can pass a prompt, the desired model, and get a response back in plain text format. And this is just the simplest use case. But there's so much more we can actually do. A model operates like a black box. It has no understanding of your application's context or the data at hand. However, we can guide it by providing a runtime schema to ensure it produces the correct output. For this purpose, we're going to use Zod Schemas. And as an example, we can start exploring how we can solve common user interface problems.

Here, we've got a table component alongside a few dropdowns for filtering the data. While the dropdown seems to guide the user, interacting with the data isn't as straightforward as it should be. In larger applications, this could lead to unnecessary clicks and visual clutter.

Now, let's explore how we can tackle this issue using AI. Inside our application, we have our filter schema and use it to generate a filter subject that is passed along the table component. We have just three fields. Category, language, and country. We can pass the schema as context in the model and receive a response.

With the AI SDK, we can stream the response as a partial object, while the schema defines the structure of the data returned by the model. Streaming is a powerful feature, allowing us to update the UI almost instantly. Finally, we set up a basic input to trigger the filtering process. It's important to note that in frameworks like Next.js, we can fully take advantage of the stream capabilities of React Server components.

And this is what the implementation looks like in the user interface. We can gradually add more fields. And as the schema updates, the interface will adapt accordingly.

Now, if you are anyhow familiar with Zod, you may have used it to validate form inputs as well. We can build on top of the previous example by pre-filling a form directly from the model's response. The source of the data isn't crucial, as it can be passed through the context of the generation. The best part is that we can reuse the same Zod schema for the AI generation, front-end form validation, and back-end data validation as well.

Finally, here's another example that I really like, and that's a booking engine. Once the prompt is injected with the correct context, we can pre-fill the slot in the calendar and generate the event accordingly. Everything is starting in the front-end. All the previous demos highlighted micro-indirections and how we can address common UI challenges, such as filtering, completing complex forms, or managing date pickers. If you want to dive deeper into this topic, I have written a detailed article on my website for you to explore.

And actually, this was just a glimpse of what we can achieve with a large language model. Undeniably, the most powerful aspect of the AI package is its ability to integrate full-prompt engineering with generative components. So, we'll try to create a simple weather application that lets us check the weather in different cities around the world. Let's call it Breeze. The most crucial part is to start creating the engine that will guide the model to produce the correct output. In this case, we are simply instructing the model to act as a weather assistant.

We can adjust the language prompt as needed and provide as much context as required. In this scenario, we can not only guide the model, but also specify the different UI elements it can generate using brackets and shortcut notations. For example, we can ask for the weather in Paris.

Finally, we can begin building the chat integration as a server action in NextJS. Since we aim to create a fully functional chat application, we also need to track the conversation history between the agent and the user. And this is why we're using the mutable AI state. The return result is a stream containing plain React components. We can set up the initial response, which is a spinner, then pass along the agent prompt to set up the behavior of the assistant. Finally, we can attach the conversation history within the object.

Our next step is to add the weather generation functions. We need to guide the model to understand what the user is asking for and bring all the weather-related information and trigger the appropriate task in the agent. Its task, also referred to as a tool, is defined within the tool subject linked to the stream UI function.

Now, let's explore how we can implement the so-weather UI tool. Every tool comes with a description that outlines its purpose and functionality. The description is used as a guidance for the model. We also have the parameter subject that describes the arguments that the tool accepts. And finally, the generate function. A generator that yields plain text or React nodes. Inside the generator, we can progressively update the UI, starting with the loading state, and then retrieve all the necessary data from the database to finally display the weather UI.

Now, let's bring everything together and see the full picture. The input field on the bottom allows us to ask the assistant for weather in different cities. Once triggered, the agent will pick up the prompt and the response will be streamed back to the user almost in real time, as you can see in the chart. If you would like to learn more about Fully Fledged Generated User Interfaces, I have put together a detailed article on my website, complete with examples and use cases. As always, feel free to check it out.

But there's still a problem that I should have to mention. The system is once again static. Everything is hard-coded, and you have to manually guide the model to produce the correct output with the appropriate tools on every single scenario. But what if we could somehow dynamically generate complex React components based on the user's intentions and all the data available? If you're familiar with site builders like Wix, or applications like Notion, you might have noticed that they let you drag and drop UI components to create pages or dashboards according to your needs. If you examine how Notion defines the UI of a page, you will see that it's essentially a JSON object. The structure outlines the visual outcome of the page in a highly detailed manner.

So we can generate a similar structure based on the available data and the UI components at hand. Let's explore how we can do that using a framework called LangChain.

LangChain is an open-source framework for building language model applications. With LangChain, we can interact with various data sources, models and tools, orchestrating the flow of data and generation. It also works nicely along with the AI package.

By data source, we can basically mean different things. It can be an API, a graphical endpoint, a database, a file, or even a third-party tool. LangChain can read the data structure of different data sources, interact with them through a uniform interface, and pass them along to the language model.

Now, let's query the database and provide data to our components. First, we need to create a data source object that will be used to connect to our database. Here, we're using just a local database. Next, we need to create a prompt template that will guide the model to generate the correct output. We will describe the input and output using placeholders for all the parameters.

Our goal here is to prompt the model to produce an SQL query based on the user's question. The context for the prompt is automatically provided by the database schema. So, if the user asks for the top two bestselling albums, the model will know which tables to query. Once that's set up, we can create a chain that takes the incoming prompt, turns it into an SQL query, runs it, and gives back the result.

Finally, we can invoke this chain using a question and get the answer back as a string. It's quite obvious, but I have to mention that since we're interacting directly with the database, we need to make sure that the model has limited permissions. Database operations aren't always predictable, and the model can sometimes get prompted. We can improve the results, though, by using a more detailed, strict, and specific prompt.

Our last step is to add another link in the chain that converts the output into a JSON object. Now, we have a complete chain that takes the user's question, creates an SQL query, runs and returns the results in JSON format. Its link in the chain is modular and can be adjusted as needed. Once we're satisfied with the data parser, we can combine everything and use the right adapter to connect our chain with the AI module.

The final piece of the puzzle is to force the model to map the data response with the user's intention. By intention, we mean the expected result of the database query shown in the visual interface, widely accepted by the end user. For instance, if we're querying multiple rows, we would expect to see a table view. For time series data, on the other hand, a chart would be more appropriate, and so on. This time, we're directing the model to work as a designer.

This will generate a JSON data structure that we will use to build the UI. As you can tell, the problem engineering here is key because it outlines what we want to create. We can further improve the model's understanding by giving it samples and clear instructions on how to convert the JSON data structure. This is crucial to ensure the model's visual output is consistent across screens and aligned with our visual requirements.

Next, we need to give the model all the available components we want to use. Since we're using React, we can document our components, including their props, the props that they do accept, and the data they need. To do this, we can extract their types, any zone validation schemas they might use, and create a mini documentation guide for the model.

This is where all the pieces come together. The generator will receive the data structure, compose the UI components, and stream the results back to the client. In our final demo, we can see how the whole process of data flow and UI generation works. Users are asking the Assistant for information about their favorite artists and albums, and the Assistant responds with custom mini widgets created on the spot. We have limited the building blocks to titles, cards, lists, and charts, but the visual results on every single run are still quite appealing and consistent.

So, I hope you enjoyed the talk, and I truly want to encourage you to start building your own tools and applications with a piece of technology that looks so promising. All the code from the demos and slides are also available on my website, and if you have any questions or feedback, feel free to reach out on Twitter or through email. I'm looking forward to hearing back from you. So, I want to thank you a lot for your time and patience, and I hope you enjoy the rest of the conference. Up until the next one. Bye!