The UI That Builds Itself: Exploring the Generative Front-End

Hello, I am Kiko and today I want to share with you a concept that I have been working on which I call Dynamic User Interfaces. It's the concept where an LLM, an AI, generates user interface on the fly based on the user desired prompt, just personalized for them. But before that, let me tell you a little bit more about myself. So I am Kiko and I come all the way from Bulgaria. If you haven't heard about Bulgaria, we just won Eurovision. So in the past 10 years I have been working for a company called Progress and I have been focused on UI components development and design systems specifically. And lately, I have been exploring how AI can affect this exact user space.

My journey begins with exploring the history of a web as a platform. So if we take a look back, so really like back, back, back, I wasn't even born there, but we had the first static websites, I call them. You can think of them like the first space jam website if you have seen it. And what is specific about them is that every single person experiences the website the same way. So it's the same for everyone. This is what I call like a static website. Moving forward to today, we have the modern web applications. So we now have dynamic data coming from servers, it's personalized for each individual user, but there is something distinctive about web applications that remain static.

And this is the user phase around them. You can do A-B test, you can do all kinds of things. But at the end of the day, it was the developer that has predefined this user phase. So a little bit of a spoiler alert. We are going to explore if that can change what the future can be enabled by an AOM. So what exactly is the problem with static user interfaces? This is your typical SAS landing page. You are loaded with all possible paths that the user can see. And you have one user that probably wants to start a trial. And you give him not one, not two, but actually three call to action buttons. Then you have the user that's ready to purchase a license. You also have a user interface for it. And then you have this poor guy that's trying to apply for a job at your company. Not only do they have to scroll all the way down to your careers page, they should probably go to three pages just so they can send their CV. I want to define the problem differently. So it's not the number of pages.

It's not the duplication of elements that we have on that page. It's that we have predefined as the developers all of the available paths that the user can take. So we have limited them in this UX maze to save for them to navigate. And what we are going to challenge today is that if we can reverse this from first principles, what if we start the journey from the user and give them the ability that our application exposes? So how are we trying to approach this is through this formula. The user interface is a function of data and state in the most basic way.

The state is dynamic by default. So it's different every time, depending on the user, depending on the conditions. The data has become dynamic. So you know this user's exact data. So if both the data and the state are dynamic, the only thing that is not dynamic is the function. It's us, the developers. We are providing, we're generating this user interface, but we are doing it once built time. So I want to make like a clear separation between LLMs creating user interfaces during build time, and what we will try today is the opposite. We will deploy the LLM into the actual application.

So the barrier that we are crossing here is that this is after the deployment boundary. So we have already deployed our application. There is no developer between the user and the application. So how do we do that? We start by what everyone grows accustomed to, which is the most basic user prompt. And what we try to do on the code side is like do a function, which takes the user's prompt, the user's data, the current state, all of our application capabilities, and our design system to try to create user interface on the fly. So I'm jumping into the code for now. And I would expect that not a lot of you are manually coding. So today we are going to be looking at prompts and reviewing the code that they generated instead of live coding.

So the first thing I have prepared is this imaginary company called Boxcell that sells software. And we are going to try to purchase a license for it. We have our application shell around it and just a single prompt in which we will say what do we want to achieve in this exact web application.

The first addition that we want is that we want the absolute minimum. We will wire this input to do a post to the server and the back end will call an LLM with the system prompt and the user input. Then we will request a structured output that you try to render on the client. And I will show you only the important parts of the code so we can identify them.

The first thing is the LLM call and I will try to keep the system prompt as small as possible and instead of dumping instructions over there, we will build around it a whole system to support our dynamic user interfaces. So we will have a system prompt, the user intent, and the actual LLM call. And the important part here is what do we request from the LLM as a structure to represent the user interface? We do not ask for HTML. We do not ask for React code. We ask for a structure that can represent the DOM tree. So this is the backbone.

This definition of a single node is the backbone of our application. And then from this structure, we will do a client-side rendering, it's just a recursive rendering of the element and its children.

So moving from here, the first checkpoint, what would happen is if we ask, I want to purchase a license, the LLM would take everything into consideration and generate some arbitrary user interface. It still does not know anything beyond the user prompt. So this data is not ours. It's just hallucinated by the LLM.

The next step that we will try to do is feed or ground the model response based on the data that we have created locally. That will be a single JSON file with some simple data for our licenses. For the data, we have a team license and a business license. And the next time that we request this data, we would expect the results to match this. So we see the team license, we have the business license and the prices are exactly the one that we fitted.

So the LLM will now know what our data is. For the data, we have a team license and a business license. The next time we request this data, we expect the results to match. The interface generates buttons that currently do not function. Our next step is to make the user interface interactive by adding the capability to purchase a license separately defined with its arguments. This sets the groundwork for interactive features.

The LLM is taught to generate forms and pass fields for each argument of the capability. Testing the purchase of a team license shows backend response success. Moving forward, we aim to make the LLM state aware by sending the client state back to teach embedding in the application. Logging capability with credentials in the client state is added, making the LLM aware of the application's state, allowing actions like logging in and purchasing licenses.

So it should look like this. This is our first capability that our application exposed. And what I did is I have a utility function that takes these comments and feed it to the LLM so it knows how it works with our application. So what we would expect next, if we generate this user interface, is for the LLM to generate working user interface and how does it interact actually. It turns out the WebAssembly platform has this thing called the form.

So in the LLM side we will be teaching, this is the only thing that we're adding to the system prompt, we'll be teaching the LLM to generate forms and pass fields for each of the arguments of our capability. So we will see that now we have like three distinctive fields, each of those maps to a specific argument on the backend. So if we do test and try to buy a single team license, nothing would happen. But going over to the Network tab, I will zoom in a bit. We will see that our backend responded that we have successfully purchased one license.

Moving on, what we want to do is make the LLM state aware. So we have some type of client state. Then the first thing is that we will send this client state back to the LLM so it can work with it and teach it how to embed it in the actual application. So I would also be adding somewhere over here a logging capability and we will be keeping the logging credentials in the actual client state. So I did a logging panel over here. Right now we are not logged in and if I request a license, now the LLM should... Yup, now the LLM knows that we can log in and we can purchase a license. So if we try to log in over here, we will do another request for logging in. The backend will return that we have successfully logged in and we will once again request the user interface from the LLM. And right now we can see that our account ID is already populated because the LLM is state aware of what is actually our application state. So moving on, I do not want to wait for the whole user interface. If you have seen it's appearing as once, but to showcase this, I'll add a little bit more capabilities. So that will be our next checkpoint where we will add ads to cart, review update quantities and some other capabilities, but we will still let the LLM dictate the actual user flow. And through this, we will request from the LLM to not give us out the whole tree, but rather stream all of the individual elements one by one so we can show feedback to the user as soon as possible. So a single ads to cart action does not take like five seconds or so. So how does that look in practice? So if I click go over here, you can see that the LLM is generating all of this stuff on the fly. And now one of the benefits of working for a big company is that I have access to all the new tooling so we can see what is actually happening on the background and how does the LLM stream those stuff. So first we have system prompt, then we have fitted all of the data. Then this is our capabilities that we gave it. Oops, this is the previous request.

It's streaming commands, so it's saying us, it's telling us, create this node, create this node, create this node, create this node. And the way that we are reconstructing the tree is that we reference them by ID and parent ID. So we say this node goes under this node and this node goes under this node. And this way we can have streaming from the LLM but at the same time we can reconstruct a tree-like structure which is the DOM.

The next thing I want to add is to improve the current user interface design by integrating our custom design system. We aim to enhance the visual appeal by applying our design system elements to the schema. This includes not only our node types but also custom design system components like buttons, cards, and more. By enabling the LLM to utilize our components and design primitives, we ensure consistency in the user interface appearance, incorporating our colors, spacings, and typography.

Moreover, transitioning to utilizing real production data, we plan to empower the LLM to query its own data rather than rely on hardcoded information. By leveraging a RAC database and live coding to the LLM, we enable dynamic data retrieval based on user queries. This approach ensures relevant data is fetched efficiently, optimizing the user experience and delivering tailored responses based on user interactions, leading to a seamless and personalized workflow.

Not only that, the LLM remains consistent by using our design primitives through utility classes to also use the typography, the spacing and everything else. And it also started generating this grid layout, it's not really good at responsiveness but I will give him that. So finally, we started by giving him just the sample data and I want to change that in production. Data is coming from a lot of sources, most of it is static. So instead of hardcoded data, I want to give the LLM the capability to query for its own data. For this I will use a RAC database which I have prepared in advance. So we dump all the information that we currently have for this demo, which is the licenses, the additional add-ons, everything that is happening in our app and we will just let the LLM decide what's relevant to the user's query and not fetch gigabytes of data. So switching to our last checkpoint, giving it a little bit of time, it's live code to an LLM, I use the fastest model possible. So we will let it finish and then we will see what's happening in the background and how it is requesting the data. So for this exact user interface, I believe it should do a query data, tools, somewhere. So it's getting the data from the actual backend by constructing like I need all the relevant licenses. So if I do our next thing, which is I want to apply for a job, the LLM will not request the license data because this is not relevant for our current request, but rather would ask for the actual current open positions. So I want to complete the whole flow with the license to do one final run. I will try to add a single team license, actually two licenses and I will add some extra. I will add them to the cart. And we will see that the LLM is generating additional user interface instead of replacing the whole user interface. And finally, it will give us the capability to check out. So let's try to check out. That's my not real credit card number, but clicking the checking out the LLM have constructed a series of form elements to move the workflow forward. So with this wrapped up, I want to get back to our presentation and see what we have done. We took the users from the data, the state, the capabilities and our design system. We give all those arguments to a large language model and it generated user interface on the fly, moving the user forward without any predefined user interface by other developers beyond our design system primitives. And what this allows us to do is remove the maze. So just we as the developers are free to add any functionality, focus on the business part of our application and just let the user or the flow start from the user and not backwards. And this demo took a little bit of time to generate user interfaces. And today I believe the bottleneck is actually the LLM, the actual AI model that's sitting behind it. But I believe that in the future that won't be the case. And if you want to build the experiences of the future, we should start building the tooling of tomorrow from today. So this was my talk. Thank you very much.

We have a booth. You can check it out. First of all, that was a really good talk and a nice live demo as well. I really, really enjoyed the fact that you showed us the prompts because I think engineering is changing. So live demos of engineering are going to change. That's going to be very interesting.

The first question that came in is talking about the subsidisation. So, for example, tokens, the price of a token. Given the fact that we're getting the discontinuation of subsidised LLM invoices, tokens are getting more expensive, eventually we're going to go in this big path of starting caching similar responses. And we realise there's a really nice pattern that we like.

The users might jump straight to cutting the distance between the user landing on your website and purchasing something, doing their job. In the end, it could be cheaper to do it dynamically instead of having predefined user interfaces and patterns that users struggle to go through. I think there's a lot of people who are really excited about this technically cool concept but are asking the bigger question as to why. There are a few questions on this. One of these questions was about use cases and what kind of use case would be really good for this idea of a dynamic UI being generated.

So why don't we just lock this in? And eventually we just find ourselves back at static websites. That's absolutely possible. The users might jump straight to, you can cut the distance between the user landing on your website and purchasing something on this, doing their job. And in the end, it could be cheaper to do it dynamically instead of having those predefined user interfaces and patterns that the users are just struggling to go through. That's interesting.

I think there's a lot of people who are really excited. This seems technically really, really cool. But they're asking the kind of bigger question as to the why. There are a few questions on this. So I'm kind of going to group them together because one of these questions was about use cases. So, like, what kind of use case would you say it's really, really good for, like this idea of a dynamic UI that is being generated?

Oh, that's really nice. I can think of two, actually. One is you can have like a Swash Explore version of your website. So you can have your predefined paths, but you can have an explore page which you only greet your user with a prompt and just see what they are coming from. They could be exploring the documentation, they could be looking to invest in your company and just exploring it through your website. And the second one is coming up lately, which is the MCP apps, where you already have an LLM and it wants to generate user interface inside the prompts.

So the concept is there. It's just the consumer is different. It's not your website. It's your user through a chat interface. It's really interesting because I think this is a case where the technology exists and we are figuring out the use cases for it as it goes on. Thank you so much for that. Those were like a bunch of different questions that I'm just going to group together under that topic and then go through the others.

The next one, and there is kind of this next set of themes, which is all about testing. Because, I mean, think about engineers. We love when things are deterministic. We've all been there when we're trying to recreate an error or even trying to recreate things working, and it's not on our machine. So I'm going to ask this question, and there are a few other questions that were related. How would you perform end-to-end regression tests in such applications? Are there any tools that would you use? We avoid flaky tests, but this seems like a breeding ground for flaky tests. And then how, if you're not going to use an end-to-end testing tool, how would you ensure app quality? I would say the same way you test any agent basically. Basically, it's just you request user interface from it and not something else. So things like benchmarking, evaluation of problems, and things like that. And not the typical end-to-end user testing that you're testing the LLM and its capabilities, not the user interface itself.

That's a really good answer as well. And the fact that we'd be evaluating it just the way we evaluate other agents and the way that they work. And this spins off another question, which is, we've evaluated it, we're keeping track. We notice sometimes it's doing its own thing, it's going off the rails. What guardrails are necessary to ensure a proper user experience and prevent users from doing anything we don't want? And this is where it branches into two, accidentally and deliberately. So why don't we start with the accidental, a user who is very innocently trying to work through the app. Make sure they go down approved journeys, even if they're creating their own journey. I would say you can have a bigger collection with predefined components that do more stuff. This is why I included the design system part. So you can give them buttons, you can give them inputs, but you can also give them entire card component that would maintain the actual payments, so they do not steer around that. And that's really interesting as well because the next question is very, very linked to the answer, which is, is the generative UI only able to create components we gave context from? How do we avoid other than creating components with data that doesn't exist? So I think you spoke a little bit about giving it some context, here is a card or here is data and how it plugs in to our system. So it can use that. How do you make sure it's going to use those instead of coming up with something on its own? I believe hallucination is a common problem in MLMs.

So the idea is not to fit it with do not hallucinate, do not hallucinate in the system prompt, but rather build a system around it that's reliable and have all of the necessary validation step to make sure it's using relevant data. So give it the tool to query a data and it would prefer it every time instead of coming up with arbitrary data. And what are some of the, and this is, I'm really interested in this, what are some of the ways that we can make sure we create those systems? Are there any checklists or advice that you can tell us to think about as we're going through making sure we give it all of the information it needs so it doesn't hallucinate? So you basically validate the things before you pass it to the LOM, you then validate the things that you pass to the LOM, which is basically tools and the system prompt itself. And after that, you give it a tool to validate itself. This is the thing that I have explored lately, is that if you give the LOM the tool to validate its own output, it can go around and around until everything is okay. Absolutely. Absolutely.

Okay. I'm going to skip the testing question because we kind of covered it before. But the next one is about switching models. Now the good thing about right now is a lot of people feel that you can switch through models quite easily and things. And oftentimes the user, maybe they can see a difference in text generation, but it's not necessarily going to have a massive impact. But now we're doing components, UIs, things that require a lot more complexity. Have you seen, maybe as you're working with different models, any big difference in the output and the quality of the components that are generated and streamed based on the model that you use? Not specifically. As long as the model can utilize tool calls, which I believe all of the latest models do, it should be okay. And I have deliberately used the cheapest and fastest model, which is the nano version of GPTs. And you can use Heiko for a quote or even a local model you can use just to describe those things. But the other thing is taste. And you cannot teach the LLM taste. So maybe work with skills or just give it your design system. And if your design system is structured okay, it will be intuitive for the LLM itself. Yeah. We can't teach LLMs taste, so they're still going to need us engineers.

Thank you. Folks, let's give him one more round of applause. Thank you so much.