Out-of-Order Streaming – The Future of Web Development

Let's talk about performance and more specifically about web performance. And whenever you talk about web performance, especially these days, the conversation usually very quickly drifts towards things like server-side rendering and maybe even streaming as a way to make your rendering as fast as possible and as efficient as possible. Now, in this talk, I want to explore what that landscape looks like of web rendering at the moment, how we got there, some of the limitations of traditional server-side rendering and streaming, and then specifically dive deeper into how out-of-order streaming solves some of those limitations so that hopefully by the end of it, not only do you get an idea of what out-of-order streaming is and how it works, underneath the hood, but maybe more importantly, you will hopefully get an idea of what problems it actually solves and what the core motivation behind this technology is.

Quick disclaimer before I get started. This is not going to be a talk where I'm going to tell you you have to use server-side rendering and out-of-order streaming and otherwise you're stupid and you're wrong. I just want to give some details on how it works and why it might be useful so that you can go away and still make a hopefully more informed decision for yourself based on your projects and your team and all the other requirements. Whatever the best solution is for you. With that out of the way, let's get started with why. What problems does out-of-order streaming actually solve? To understand that, we need to take a quick look at the history of web rendering and how we got to where we are today. We have to start at the very beginning, which I would call the good old days, with plain HTML and maybe some server-side language on top of that. I'm using PHP here, mainly because that's what I used when I started. It doesn't really matter. It could be any other server-side language. Fundamentally, they all do the same thing, which is generating that HTML on the fly. If we visualize that, we have a browser that makes a request to the server for a specific page. That server then generates HTML, returns that HTML, and the browser renders it.

And that's it. Much simpler times. Because that HTML is purely static, we don't have any dynamic elements just yet. The response time of the server really depends on what we're doing on the server. If we're just looking up static HTML files, that's really quick. The more we do on the server, say we connect to a database or we're fetching data from third-party APIs, the more we do, the slower the response time gets. And that's annoying on the initial load because we're just waiting for that HTML to come in for a long time. But it gets even more annoying as you navigate around the page because every time you click on a link, we have to do that whole full loop again. We go back to the server, the server generates the whole HTML for the new page, returns that, and then renders that. This is really frustrating for pages where we do more on the server. So we tried to solve that problem by introducing JavaScript or more. Specifically, this is kind of the jQuery era and the introduction of technologies like Ajax. So we're talking 2006-ish year. And the idea was with JavaScript, A, we can make our applications more interactive and dynamic.

So now when we're requesting a page, we're still going back to the server, the server still returns static HTML. But now the server can also return a JavaScript file. And then the browser pauses and executes the JavaScript file. And that JavaScript might inject some event listeners for click handlers on buttons and things, or some local state and make your app more dynamic. Now, this is great for an overall user experience. But to solve that long wait time when you navigate around, the introduction of Ajax really helped. So Ajax allowed us to make network requests from the client even after the initial page load has already ended. Which means when we navigate to a new page, say we have a lot of static elements like a navigation and layout elements that are all the same on all the pages, we can be much more specific about what we're actually requesting from the server and only request like a subset of the HTML. What that means is that the server response time usually is much quicker, but it also means that all of this happens on the client. So we're in full control over any kind of loading states that we might want to show while the user is waiting for that content to come in, which massively improves the user experience on that navigation reads.

Now, once we introduce JavaScript in good old JavaScript fashion, we kind of get addicted and quickly went all in on that. And into the next phase, which is kind of like the single page application era. So we just plain jQuery. Then quickly we built application frameworks that helped us build the shape of normal applications with like Knockout.js. And then you start seeing the frameworks that we still know, love and use like React, like Vue, like Angular.

The idea is that we cut out the server completely. So now when we request a page, all the server really does is return an empty shell with your documents from sharing, like an empty div and a JavaScript file. And the idea is that that JavaScript will build up all the HTML dynamically on the client and inject it into that empty div. So once the JavaScript is all downloaded, passed and executed, we get our application and it's immediately interactive. And all of it is interactive because all of it is built in JavaScript, so there's no barrier anymore to add event listeners and local state and all of that at that time.

This is where kind of building much more enhanced applications and much bigger user experiences kind of exploded because it was much easier to build them at large scale. And navigating around because all of it happens on the client is usually pretty fast. It's only limited by do we need to go to do some network requests to fetch new data for the new page or that kind of stuff. But just rendering new HTML for a new page is usually reasonably fast. The real downside here is that initial load where the user is waiting, not only for the server response time, but then for the JavaScript to pass.

The real downside here is that the initial load where the user is waiting, not only for the server response time, but then for the JavaScript to pass. The larger and more complex those applications get and the slower your network might be either because you don't have a fast network connection or you're on a not-that-fast old mobile device, the longer this wait time becomes. It got to the point where we realized we might have gone a little bit too far with full-on client-side and single-page applications.

So we started moving back to the server. But we wanted to keep all the benefits from all those nice frameworks like React, like Vue that we started enjoying. So we took advantage of Node which gave us a JavaScript runtime on the server. And the idea was, when the browser requests a page, the server can run the same framework on the server to generate the snapshot of that initial HTML to give a better initial server response.

So instead of just returning that empty shell, we can return that snapshot. So that doesn't have any interactivity in it because by definition, the server doesn't have a concept of interactivity. But it can have all of the visual elements. And then we still return the normal JavaScript alongside it and the framework will build a copy of the page on the client and then inject all of the dynamic parts into that static snapshot that we initially returned.

So that injection of dynamic parts is called hydration. And this is basically where all the frameworks do the heavy lifting to inject the dynamic parts and then take over from there where if you navigate around after this, it's mostly like a single-page application feel. So this is trying to get the best of both worlds where we get a better and faster initial paint, but then we still have to do the hydration bit.

And the hydration bit can be really slow because, like I said, you're essentially building the application twice, right? Once on the server and then on the client again to be able to then inject all the dynamic parts into it. The second problem with this approach is that we're now back on the server. So we have the original problem again from the HTML PHP days where if we do a lot of work on the server to get that initial render, the initial server response time can get really slow, really fast. So how do we solve that?

We introduced server components. So React became famous for this, but they weren't the only framework or not even the first framework to play with the idea. But fundamentally, the concept is to give the bundler the knowledge about which components in our application actually are dynamic and actually need client-side JavaScript, and which components are purely static and can be rendered once on the server and then be done with it. And if you take a look at your application on a larger scale, like most of your application is static and only very few components actually need client-side stuff like event listeners, like local state, etc.

So the idea is we do the server request, we still do that initial render on the server, but then we don't need to send the whole application to the client and re-render the whole thing again. We can do what we call partial hydration. So if the bundle knows which components are dynamic, it only needs to send those dynamic components to the client and only those need to get hydrated and get their dynamic stuff injected.

Now this visualization might look very similar to what we had in the jQuery and Ajax days, right? Because fundamentally, it's still trying to do the same principle where we're just much more selective about what we want to do with the JavaScript and where we need JavaScript. And this is where the internet and Google fashion started mocking the idea a little bit by saying, ah, okay, so a component is just PHP. Ooh, the nice tree occurred on top of it. And that we're evolving, but backwards.

All of this is to say, this kind of solves the hydration thing with partial hydration, that JavaScript that we need to download is much smaller, the hydration itself is much smaller and much quicker, but it doesn't really solve the second problem. So if the server does a lot of work and the server response, the initial server response is really slow, how do we fix that?

Now, to understand that, we'll build a solution from scratch, like a very simplified solution from scratch ourselves. And for that, we're going to look at traditional server-side rendering and where that brings us. Then we'll look at some stuff like classical streaming and some of the limitations that come with it. And finally, we'll take a look at how auto-streaming kind of addresses those final limitations and what all of that gives us from a DX perspective, but also from a user experience perspective. All right. Let's write some code.

For that, I prepared a small demo application. It's fairly simple. It's just a kind of movie or TV show app where you can look at a TV show, see some details about it, what the plot is about, see details about the cast, and maybe some similar movies and TV shows that you might like if you like this one. It's called Notflix. I don't want to get sued. But you can see if you look at the code for this, I also tried to keep it fairly simple. So this is just an Express app. Express doesn't really matter here. It's just the simplest service setup.

And we're rendering individual components here for the different sections of the application. So we have one component for the header, one component for the title, image, and details, then one component for the cast, and one component for the simpler movies. If we look at those components, you can see that they look mostly like regular React components.

It's just the simplest service setup. And we're rendering individual components here for the different sections of the application. So we have one component for the header, one component for the title, image, and details, then one component for the cast, and one component for the simpler movies. If we look at those components, you can see that they look mostly like regular React components.

You might notice that they're asynchronous functions, not normal functions, because we are on the server. So I'm treating all of these components as server components that only need to run on the server once. And in that case, we can do asynchronous stuff within them, which is nice because that means data fetching and stuff can be done right in line. So in this case, all of these components will go to the server to fetch their respective parts that they need from an external API.

And we're not in control of that API, so we can't really make that faster, but we're in control of our components here. And then on the server, what I'm doing right now is basically what a classical server-side renderer is. I'm starting to collect the HTML, so just starting with the document structure, and then this JSX gives me an array of children. So then I just loop through those children. Like I said, all these children can be asynchronous. So we await, run the function, basically the component function, with our props, and then we render them to strings. I'm just stealing that from React on.

And in the end, I'm concatenating all of those strings from all of the components. I'm closing the document text, and I'm sending all of that back to the server. If we click around here, we'll see that fundamentally works. However, due to the asynchronous processes and server-side fetching, the response time is notably slow, impacting user experience. Users face delays as the HTML generation completes, leading to a spinner display until the new page loads.

The initial step to enhance performance involves classical streaming. In Node, this process is straightforward. By using Express and the response write method, we initiate streaming by writing the response incrementally, maintaining an open stream to continue writing until completion. This method optimizes response time and user experience by allowing elements to render progressively. By shifting from collecting HTML to writing it directly into the stream, we achieve a more efficient rendering process.

Despite the improvements, there are noticeable challenges in maintaining the order of asynchronous components during streaming. While a for loop is a simplistic solution, it lacks efficiency as it runs asynchronous components sequentially. Although there are better methods to parallelize and resolve components in the correct order, for this demonstration's simplicity, the for loop approach suffices. This method ensures a faster rendering experience compared to the initial version, even though further optimizations are possible for enhanced performance.

And then because we've already written everything, all we need to do is end. So this tells the browser, hey, the stream is finished. Everything is closed now. You can stop listening to the server. And if we now refresh the page, what we'll see is that the logo renders immediately because it doesn't do anything asynchronous. And then we see what we kind of expected, like the content streams and when it's ready.

But you also see that the sections are a little bit out of order. It's kind of expected because before we were running all of those asynchronous components in parallel because we could, because we were just awaiting all of them and then stitching them together. Now that we're streaming them whenever they're ready, we can't really do that anymore. So for now, to keep things simple, I'm replacing this with a simple for loop. This is not the most efficient way to do it because now we're doing one asynchronous component at a time.

So there's better ways how you could still run them in parallel, but then resolve them in the right order. But to keep things simple for this demo, let's use a for loop. And what that does is again, we see the logo and then we wait until the individual sections come in. Now this is definitely a lot better than the initial experience. It's still not great though. So if I click now on the new thing, I get the immediate visual feedback that something happened, but I'm still mostly staring at a blank black screen.

And what I ideally want is some form of loading state, right? So some fancy skeleton experience that I can use to kind of indicate to the user that something's loading. And how I would do it on the client is to use suspense. And then we give it some kind of fallback that allows us to render that loading state. In this case, I have a skeleton component prepared. And basically I want to render this fallback while any of the children still do their asynchronous stuff.

So how would we do that on the server? Let's implement a very simple version of this ourselves to kind of see how it would work underneath it. And for that, we need to first keep track of some form of map or object where we can store all of the suspended items, all of those asynchronous components that we can't immediately. And then we do the suspense component itself, which really just does two things. One is it takes children like we've seen, and a fallback. And what we want to do is put those children, like we create a unique ID for each instance of the suspense component, and then we store the potentially asynchronous children in that map with the ID and only render the fallback.

And we store the suspense ID here as a data attribute. You're going to see why in a second. CSS tip on the side, we give that diff that we wrap everything in a display of contents. What that means is the browser will ignore it for any layout purposes, so it won't break any git grid or flex layouts. So that's pretty neat. But if we do that and we import suspense here, what we can then do is actually just refresh the page and see it does kind of what we want to do. It renders that fallback. And right now we don't have any logic yet to tell it to replace that fallback with the content once it's ready.

So let's add that. So we take that suspended object and we check if there's any items in it. And if so, we want to render those suspended items. And then once they're rendered, we do what we've done in the very beginning. We just concatenate all of the children of each of those suspense components and write them to the string. And if we do that, fundamentally, that works. That does what we want it to do. So now, once the content is ready, it pops into existence and it's added to the stream. But that's the fundamental limitation of classical streaming, right? We can only add to the stream because we're only depending on that streamed document that we've already started sending to the server. So how can we actually replace this loading state with the real content? Short answer is in classical streaming, we can't. So this is where out of order streaming comes in.

So this is where out of order streaming comes in. And for out of order streaming, we need a little bit of JavaScript, at least right now. So there's two things that we want to do here. We want to wrap the content from the suspense boundary, like the suspended content in a template tag. This just tells the browser to ignore it for rendering purposes, but it is still in the DOM. And then we want to add JavaScript logic to replace the placeholder with the actual content once it's ready.

There's a million different ways how you could do this. For the purpose of this demo, I'm going to use a custom HTML element. And we're just rendering that after the template tag with the target ID. That's the suspense ID, which allows us then to link it back to the loading state with that same ID. So that's why we edit that as a data attribute here. So now we just need the JavaScript to do the replacing bit.

To do that, we essentially can do cursors helping here, something like this. So we are registering that custom element. Like I said, this is our tag name that we want. Extending HTML element. And all we really need is a connected callback. This connected callback is called every time this custom element is being rendered for every instance of it. So when this runs, we want to grab the previous sibling of our current element, because we're in control of the rendering. We know this is always going to be the template tag that it's relevant to.

Then we get the ID from the target ID attribute. Then we look in the DOM for that data suspense ID from the suspense boundary. So we're basically looking for the loading state that belongs to that ID. And then we're clearing the HTML of that loading state and replacing it with the HTML of that template tag. So fairly straightforward, just a few lines of JavaScript. But what that does is basically give us exactly what we want. When we refresh the page, we have immediately the loading state available. And then it gets replaced by the real content once that is there.

Again, when I click on a link, I immediately get that nice loading state. And everything else comes in. And we're not limited to a single suspense boundary either. We can easily have multiple of these for each of the sections. So I would only recommend that if you know exactly the heights of each of the sections to minimize layout shift. But if you do know your heights, you can do this and then have a much richer loading experience. And you can then allow things to pop in the moment they're ready.

You don't have to wait for all of these sections to be ready to replace your individual loading boundaries, which is pretty cool. The other cool thing that you get with this is because it's streaming and it's all coming from the server, you get a lot of cool browser functionality, essentially for free. What I mean with that is things like caching and history back and forward. If I now that this is loaded, go back in history, you see this is immediate. Because we've streamed the whole response, including the resolved suspended components. All of that in one stream, in one final document that gets put into the cache. Meaning the browser has all of that immediately available if you go back and forward.