Building a Hybrid AI Chatbot with Next.js 16

All right, hello everyone, my name is Sam and I'm excited to be talking with you today about Next.js. So about a month ago we released Next.16 and it has a feature called cache components, which is really like an evolution on the programming model in the app router. It finishes a lot of the things that have been in swing in Motion for a couple of years since Next released the app router, started adopting server components and server actions. And so it's really becoming a joy to use.

But a lot of folks use Next.js when they have content-heavy sites because Next.js has always had a lot of powerful static pre-rendering capabilities. And when you build a static site with Next.js, it's really fast, it's really snappy, you can serve it on edge CDNs. But in the past, if you wanted anything dynamic in your app, anything that depended on the request or the user's personalized data, runtime data from a database, you kind of are faced with this decision.

If you want a static initial page really quickly, now you have to fetch data kind of on your own in the client and you have to make an API endpoint and you have to set up something to fetch. You're kind of on your own. Or you could render on the server at request time using something like GetServerSideProps in the old Next or just using React server components in the app router. But you would sacrifice that snappy initial load. So today I want to show you building some aspects of this little chatbot I have here.

And this is a fully dynamic app with personal data. And there's a database and it uses cookies to find the header. And you can just chat with it just like you would expect. It's like a little GPT clone. I'm giving a talk. It's about you, actually. So this is an app that's dynamic.

And you would normally expect to build maybe like an SPA for this sort of thing. And maybe in the past you wouldn't have used Next to do this, because what help would it be? It's all data I need at the request. But kind of the big thing I want you to take away from today is that in Next 16 with cache components, pages are both static and dynamic. And they let us build apps like this that stay snappy, that feel really fast, in spite of doing a lot of the work on the server.

So that's kind of what I mean by hybrid app. And I really just want to go through a few demos with you that kind of showcase some of my favorite aspects of the model in the context of this app. And then you can look at the full code and run it yourself on GitHub. So let's come over and go to demo one here. And for this demo, we are going to talk about PPR. And this is just a static page right here. And we want to fetch the chats in the sidebar.

So in Next, we have an async component here. And this runs on the server. It's a server component. So we can actually just access our database. I'm using Drizzle here. I've got it open over here in a tab. You can see all the data. There's the data.

So we want to get all the chats. We can just say query chats. Let's do many. And if we look at this, it's going to be nice and typed. And then we can just iterate over each chat. And we'll render the chat's title. Oops. And we'll give this a key of chat.id. And there we have our chats.

We can make them small. Give them a little bit of padding. And these are looking good. And that's how easy it is to fetch data in RSCs with Next. But if we reload, we're actually going to see an error here. And this error comes from cache components. By the way, in Next 16, if you want to play with this, you need to enable it as an opt-in feature because it's a change to the programming model. There's a lot of things that still work, but you couldn't just upgrade with just a flag.

There's some changes you need to make, for example, like this one. And so once you've opted in the cache components, we see this error. It says, uncached data was accessed outside of suspense. This delays the entire page from rendering, resulting in slow user experience. Next uses this error to ensure your app loads instantly on every navigation. And this goes back to what I was talking about at the beginning.

If we didn't fix this, this whole page would be blocked on these chats. And Next is wanting us to make sure we have some static content that we can render immediately so that we can put it on a CDN, for example, and get those snappy initial loads that we're used to when we have SSG or pre-rendered sites. So one thing you can do here is create a loading template. And if you export a default loading function, you can say loading. And now the error goes away.

If you haven't seen this yet, this is really cool. This is something new from the React team. It's called the Suspense DevTools. And if you open this up, you'll actually see... We can click here to see our initial paint. And so you can come here and keep this actually in this suspended state. I can just do something like replace this with a spinner. And maybe text center this and add a little margin. And now if I refresh, and let's actually just add a little bit of artificial latency here. We can see our spinner. And then this goes over. So that's pretty cool. Suspense DevTools, by the way, are really neat.

If we look just in the root app real quick, it's a lot more neat to look at it in like a production app. You can see here the sidebar, the client. I can see the initial page right here and see everything that's rendered. So let's actually come back and make ours a little bit better. Because this is fine, but it's not really useful to give the user a spinner, right?

So instead of using loading, let's just delete this. And we're going to see that error again. Uncache data. And what we can do is move this down into a new component. Let's just say this is going to be called chat list. I'm going to grab this. And I'm going to grab this. And if I do this and then we render this here, and I fix this like this. Then we're still rendering it. So we're going to see the error.

Now we can wrap this in suspense. And now we have... Let's put this delay back just to simulate some latency. Now we have an error. Which we shouldn't. Oh, I need to put this down here. The promise is considered suspending as well. Because it's asynchronous work. So now we have our app. And now you can see, none of the content is blocked by the chat list. And so this part of the UI can be served from CDNs. But in the same request, our origin server can stream in the dynamic UI. So it's all one request. And we didn't have to bail out and start setting up client-side data fetching to get this dynamic data into our app. We're in a server component. No API routes.

No fetch calls. No waiting for the browser to load to start doing waterfalls between the client and server. So this is a really awesome feature of cache components. And it's partial pre-rendering basically. It's kind of the latest version of that. It's the actually shipped version. We've had a couple versions of it in the past. But this is like the final version.

Again, if you look over here in the full app, this is the initial paint. And you can see it's much more useful. Because you can have things like you can start letting the user type in. Even while you're streaming in slower data on the side. So that's partial pre-rendering. And I just wanted to show that off. And those Suspense DevTools, because they're really cool to use. And it's nice not to have to start fetching the client if you don't need to.

Okay, let's go to the second demo. And this one is about actually fetching the AI stream. So if you've worked with any AI providers before, you usually call them. And they give you back a stream of some sort that you can iterate over and display the message. And you might, again, set up an API endpoint for this. With Next 16, we have server actions. You might have used these before if you've used Next. But something that a lot of people don't know about server actions is that they participate in the RSC serialization protocol in the same way that server components do. So what do I mean by that? Well, here we have a form and we have the message. And right now, we're not doing anything with it. But let's actually get the stream. So how might we do that? Well, I'm going to say getStreamMessage. And this is going to be a server action. So let's just create an actions.ts folder.

We'll use server and we'll export default async. No, export async function getStream. And this is going to take in a message, which is a string. And we'll go ahead and import this right here. And if you haven't used this before, server actions, now when we type something, we'll see, oops, I meant to console.log it over here. We will see the console log over here in the console in the server. So this is kind of like an API route, but it's a lot more convenient because we just wrote a function. We called useServer up here and we get the function right here, and we can just call it.

And if we were to change this, for example, to be a number, we would know right away, you don't get that with fetch. So all these things are just built in now to next, really built into React. Any framework that uses server components get this. But this is how we can set this up so that we can call a function on the server from the client. And now we can actually get the stream.

And the way we would do that is I have OpenAI installed. This is just like the hello world from their guides. And the way you get a stream is by saying client.responses.create. And you can pass it a model. Let's go with 35 turbo. And you can pass it some input, which is an array of messages. We'll just pass the user message now. So role user, and then our content will be the message we pass in. And then we'll stream it by saying stream is true. And we'll await this, actually, so we get the stream. So this thing that OpenAI gives us is a stream. The stream is their own class, but it's basically an async iterable. And when you're working with it here in the server, it's great because you can just for of this thing and get a chunk from the stream if you await it. And the chunk has all this information. But we want to get in the front end, right? Because we want to update the log as the stream comes in. So we can actually just return the stream. And here, if we await it, we can get the stream.

We can await each trunk of the stream. Let's get some state. I'm going to show you that working first. We console.log each chunk, open the console in the dev tools, and see all the chunks coming in. Now, we can use state to create some state for the response. We'll show the response in a new div and set the response to the current value plus the delta from OpenAI.

This stream is a custom class from chat GPTs in OpenAI's SDK, implementing async iterable. It provides chunks with delta events for text updates. By iterating over the stream, we can update the response with the text delta events. The stream handling allows for real-time updates of the response displayed in the front end, reflecting the incoming textual information from OpenAI.

The integration of state management enables the dynamic updating of the UI as new information arrives. Utilizing the delta events from the stream, we can selectively append relevant text updates to the response shown on the front end. This interactive process showcases the seamless flow of information from the server-side stream to the client-side display, enhancing the user experience.

And then when we're iterating over our stream, we can just set the response to the current value plus. And what do we need here? Well, I don't know. Let's see. We have a chunk. This is all typed. So I actually want only the text delta event from OpenAI. So if the chunk is a delta, then we know we're going to have a delta property. And now I can just set this to the current plus the delta. And with any luck. Whoops. Oh, chunk.delta. Look at that. Giant red underline. Missed it.

So why is the sky blue? And there you go. So how cool is that? This stream is, again, this custom class from chat GPTs from OpenAI's SDK. It returns this stream, which is something that implements async iterable. But basically if you read the guides, you're going to be told to do exactly this in Node. But we can just return it from our server action and do it in the client if we want to update the screen immediately. So pretty awesome.

Now, sometimes the DevTools in Chrome don't show RSC payloads because they're kind of in the streaming format. So I like to show it with Firefox. But if we do this with Firefox, why is the sky blue? You can take a look at this function here, which is returning our stream. And you can see here, this is the RSC protocol. This is like the serialization format. It's kind of the secret sauce of RSC. When you first render your app with RSC, it uses this to turn it into HTML and also JSON. When you navigate to pages, again, in Next or any framework that uses RSC, it can just fetch this JSON and update the UI without losing any React state. It's basically like a React render across the network. But it handles all sorts of things that can be serialized across the network, including these async iterables.

So you'll see this $2 is a promise. It's kind of how the initial render works with suspense so that you can be updating the UI in a single response, again, without having to go back and forth. So it's pretty cool to look at this. And you'll see these are all the events we're getting from OpenAI. Now, there's quite a bit of data in here, since we're just returning the stream directly from OpenAI, log probs and obfuscation, all these properties. It'd be nice to just send over the text. Well, we can do that in our server action with a new function that's going to be a generator.

So if you've never used a generator before, they're pretty cool. But let's pass this in. And this is going to be a stream from OpenAI that gives us a stream of response stream events. So now I'm just going to grab this logic. And here, if we're in an output text delta chunk, we just want the delta. Instead of setting state or anything like that, I just call yield. So yield chunk dot delta. And if you've never played with generators before, they're very cool. They're basically like stateful functions. You get to kind of go part of the way through them, get a return value, call them again.

The caller gets to decide if they want to keep going, all these cool things. But this basic API is how you do it. You do it with a star, and then you yield out data progressively. And so if we return this generator, stream text, around our stream, we save it. Now we see a type error over here because our server action is now not returning a stream of chunks, but a stream of strings. We get that feedback immediately. So now we don't need this. We don't need this. And now the chunk is just a string. So let's save that and come over here to Firefox and refresh.

Why is the sky blue? And if we take a look, now we're just getting the text. So super powerful way to restrict what you send to the client, keep your apps performant, not over send too much data, make sure you don't send sensitive things. Generators are a great way to do that when you're working with streams.

So I thought that was just a really cool feature here of server actions. Okay, last demo. This page is rendering a chat message input here, and two messages from our database. I have this little demo messages table that just has fewer fields. We're fetching these and we're passing them into our client component.

And this client component has our kind of same form right here. And what it's doing is kind of what my main app, my main demo does, which is, you know, it sets up the data that needs to be saved. So there's a user message, an assistant message. And right now it doesn't do anything. So if I said test three, we hit enter, nothing happens.

If we refresh, nothing is there. So this is using a server action already. So let's go ahead and just await DB. And this is going to be insert into demo messages. And we'll give it the values here. And again, this is cool because this is all typed. We know we're sending the right thing from the client. We know that this is going to work with Drizzle because Drizzle is typed. So if we do that, then we say test three. We don't see anything. But if we refresh, we'll see it.

So that form did work. And usually here, if you're familiar with something like SWR, React query, you know, when you make a mutation from your front end, usually the mutation comes back and then you do some sort of revalidation, right? You have a key, you say something needs to be revalidated, so you go and fetch the new data. Well, in server actions here, we can actually just call this new refresh function right from the server action. If you've used Next, you might have used revalidate path for this, but this is kind of the blessed way to do this particular pattern now.

And if we say test four, now we see the page updates. And one cool thing about this, again, thanks to RSCs, the way it can serialize parts of our UI in addition to just data is if we refresh and clear out our network log, and I say test five, we actually see one request. So in the same server request, the server action request, we update the data. And thanks to refresh, we re-render the dynamic parts of the current page and send that JSX backup, and it re-renders in the client. So again, no round trips is kind of a theme in this release of Next. And it's pretty awesome.

It's just less state to manage on the client side, you know, kind of like David was talking about. A lot of times when you have a lot of excess state that's really just derived from data whose source of truth is in the server, it can cause problems. It can lead you to stale state bugs, all sorts of things. And so that's part of the motivation behind these APIs as well.

This works and it's cool. But if you notice when I hit test six and hit enter, we don't see any feedback. And if we were to like slow this down maybe by a second, it's basically taking that long, you know, to run this and then however long to re-render. And in the meantime, we don't see anything. Now we could use transitions and put our form in kind of a pending state, but I want to show you how to use optimistic inside of React.

And if we come over here to our client, we're going to see that our client is getting the messages and it's rendering from them down here. Now optimistic, use optimistic is a hook from React. And the way it works is you pass in some canonical state. And this is going to give you a set of optimistic messages and set optimistic messages. And if we render from optimistic messages, what we can do is right before we submit our call our server function, our server action, we can set optimistic messages. And we'll take the current and we'll create a new array with current and then our new user message and our new assistant message. So if we do this and refresh, now we say test eight and we hit enter, we see it pushed to the array. Let me go ahead and clear some of these out so we can start fresh.

And now I can just say test one, you know, test two. And again, you can look at the main code, but this pattern is used a lot. Again, part of this theme of having temporary client state to make your app responsive and instant, but ultimately, it falls back to the server side value. And you can actually see this if we were to, for example, throw a new error.

And in our client, we were to do something like just wrap this in a try catch for now. If I were to create, you know, a message, then we'll see it reverts back. And of course, you could do error handling here. But again, this is just to point out that the way you use optimistic works, and I think you should check out the docs for it, because it's a very cool hook.

And again, it's not specific to Next, but Next is built, you know, around the idea of using this for this sort of thing. It only keeps the changes you've made around as long as the action is running. And once it stops, the revalidation comes in. So you're basically always guaranteed, once your app is kind of in a settled state, you're guaranteed that this UI will match as if I hit command R, right? And so again, on this theme of like, how can we make these snappy SPA like experiences without having to copy a lot of state and sync a lot of state to the browser.

It's kind of the approach that we're taking. So those are three of the features I wanted to show you. And really quickly, at the end here, I want to talk about use cache. Because if you've followed Next at all, the last couple of years, and especially this year, like I said, we just released Next 16 with cache components.

You know, I've been talking about Next 16, but I haven't mentioned use cache at all. But that's because cache components is, again, more of resetting the foundation of Next to be a fully dynamic framework by default. Components render dynamically by default. There's no hidden caching. If you've used Next in the past, you might have run into times where Next tried to cache something or make it static, and it was confusing. So all that's gone with cache components.

And the way you cache things is with the use cache directive. If you want to learn more about that, check out my section from the NextConf keynote, because I talk about using that for things that we've used getStaticProps for in the past. So you want to pre-render a blog, pre-render a content site, so it's instant. That's a great use case for use cache. But there's another use case for use cache that I want to talk about in the context of this app, which is runtime prefetching.

If we go to our chat page, we can see we're getting the current user, the param from the URL, and then we're fetching the chat to render. This runs on a local database, making it fast, with fast initial shell thanks to partial pre-rendering. As we click around, a loading spinner indicates the data is being streamed in, and Next prefetches to ensure instant loading. In a cache components world, dynamic data feedback is instant, unlike before.

There's a feature in development called runtime prefetching, crucial for model enhancement. Enabling it allows the app to prefetch and pre-render more of the page, leveraging cookies and params at runtime, not build time. Prefetching is off by default in dev but can be enabled for production builds. Enabling runtime prefetch improves the pre-rendering of pages based on available data, providing a smoother user experience.

For specific functionalities like runtime prefetching, using the useCache directive is essential. This directive is crucial for handling dynamic data that is not available during build time but needs to be fetched at runtime. By calling useCache strategically, the app can pre-render specific content, enhancing performance and user experience significantly.

But it would be nice if we could actually render more of the content. Well, we're working on a feature which is a really important part of the model, but it's not quite ready yet, called runtime prefetching. And once you enable this, once you're in the app and clicking around, you can get cookies and params and search params, because you're clicking a link that already has them. So now what Next will do is we'll prefetch and even pre-render more of the page than otherwise would. You don't have cookies at build time, so you can't pre-render that. But at runtime you do. And so that's what this helps you do.

In dev, we'll still see this, because prefetching is not turned on by default in dev. But if I were to run a production build of this... And we check this out locally... Now, when I refresh... Oops! Oh, I forgot that I had useCache. In order to actually use this, you actually call useCache right here. So that's the first step. And again, if that doesn't depend on request data, it can be pre-rendered at build time. This does, because we need the user ID and the actual prop from the params.

Once I have that, and I enable runtime prefetch, if we do another build here... Whoops! And we take a look. Now when I click to these links, look at that. It's instant. And I can make new chats. And I can say, tell me a story. We can see it's there. It's all instant. It feels like an SPA. But we don't have any API routes. We have no client-side data fetching. We have no data on the cache that's long-lived that we have to maintain, or update, or surgically insert. We have a guarantee that every time the app settles, it's going to look exactly the same as if we refresh it. So it's kind of early days with this stuff.

But it's very powerful. And I think finally we have some of the primitives and tools that we need to make server components actually work for apps like this. So I had a great time building this demo. That's all I wanted to show you for today. But if you want to check it out, you can look right here and poke around at both the demos we looked at and the main code. So that's it for my talk. Thank you so much.

So you created some really great material on build UI. What should we look for to Vercel's education platform with you now that you're on the team? Thank you, by the way. I definitely want to be making a lot of the same kind of content. I've been at Vercel for about four months, and so I've just kind of been learning a lot of this stuff. But we're actually going to be building a lot of recipes, which is something that was pretty popular from build UI, where we have little isolated examples showing off some of the kinds of techniques that we looked at today. So that's on a roadmap. And then definitely some YouTube videos coming soon. That's so exciting.

Is the next JS error stack at the bottom a Google extension or is included in the next by default? What was the question? The next JS error. The error? Yeah. Ah. Is the accessing uncached data. Is that included by default? Yeah. When you turn on cached components, that error, I think that's what the question is asking. Yeah. I think it's like the bottom left. Oh, yeah. Oh, the dev tools. Yes. And that shows you errors and lets you see if your page is fully static if you wanted like a fully SSG site. Yeah. That's part of next dev. Nice.

All right. Last question. Is there a way to integrate next JS into smaller features for an existing large application? For example, having a large SPA rendered by a different stack, but use next JS explicitly for the AI chatbot and newer features. Yeah. Actually, a lot of people do that. I think Grok actually does this on Twitter, on X, that they kind of use next and then they mount an SPA. So if you wanted to go the other way, I think you could do it. I can't think right now exactly how you would. But next is pretty flexible. People use next for all sorts of things. I mean, tons of people use next for all sorts of things. And a lot of the features that I just showed you, things like server components and server actions, we're trying to provide more first class APIs to help you build your app so you don't have to reach for libraries right away and that they integrate with our caching story and everything. But they're really opt in if you think about a lot of these kind of server first features are opt in. And so a lot of people do that. They will mount another library or an SPA at a route and have separate things. So yeah, I think you could do that. Sounds good. That sounds like it works really well. That's all the time we have. Thank you so much for the talk. You got it.