Out-of-Order Streaming – The Future of Web Development

Let's talk about performance and more specifically about web performance. And whenever we talk about web performance, especially these days, the conversation usually very quickly drifts towards things like server-side rendering and maybe even streaming as a way to make your rendering as fast as possible and as efficient as possible. Now in this talk, I want to explore what that landscape looks like of web rendering at the moment, how we got there, some of the limitations of traditional server-side rendering and streaming, and then specifically dive deeper into how out-of-order streaming solves some of those limitations so that hopefully by the end of it, not only do you get an idea of what out-of-order streaming is and how it works, but maybe more importantly, you will hopefully get an idea of what problems it actually solves and what the core motivation behind this technology is.

Quick disclaimer before I get started. This is not going to be a talk where I'm going to tell you, you have to use server-side rendering and out-of-order streaming, and otherwise you're stupid and you're wrong. I just want to give some details on how it works and why it might be useful so that you can go away and still make a hopefully more informed decision for yourself based on your projects and your team and all the other requirements, whatever the best solution is for you.

With that out of the way, let's get started with why. What problems does out-of-order streaming actually solve? And to understand that, we kind of need to take a quick look at the history of web rendering and how we got to where we are today. And we have to start at the very beginning, which I would call the good old days, with plain HTML and maybe some server-side language on top of that. I'm using PHP here, mainly because that's what I used when I started. It doesn't really matter. It could be any other server-side language. Fundamentally, they all do the same thing, which is generating that HTML on the fly.

So if we visualize that, we have a browser that makes a request to the server for a specific page. That server then generates HTML, returns that HTML and the browser renders it. And that's it. Much simpler times, because that HTML is purely static. We don't have any dynamic elements just yet. The response time of the server really depends on what we're doing on the server. So if we're just looking up static HTML files, that's really quick. The more we do on the server, say we connect to a database, or we're fetching data from third-party APIs, the more we do, the slower the response time gets. And that's annoying on the initial load, because you're just waiting for that HTML to come in for a long time. But it gets even more annoying as you navigate around the page, because every time you click on a link, we have to do that whole full loop again. We go back to the server, the server generates the whole HTML for the new page, returns that, and then renders that. So this is really frustrating for pages where we do more on the server. So we tried to solve that problem by introducing JavaScript. Specifically, this is kind of the jQuery era and the introduction of technologies like AJAX. So we're talking 2006-ish here. And the idea was with JavaScript, A, we can make our applications more interactive and dynamic. So now when we're requesting a page, we're still going back to the server, the server still returns static HTML, but now the server can also return a JavaScript file.

And then the browser parses and executes the JavaScript file, and that JavaScript might inject some event listeners or click handlers on buttons and things. Or some local state, and make your app more dynamic. Now this is great for an overall user experience. But to solve that long wait time when you navigate around, the introduction of AJAX really helps. AJAX allowed us to make network requests from the client even after the initial page load has already ended. Which means when we navigate to a new page, say we have a lot of static elements like a navigation and layout elements that are all the same on all the pages. We can be much more specific about what we're actually requesting from the server. And only request like a subset of the HTML. What that means is that the server response time usually is much quicker, but it also means that all of this happens on the client. So we're in full control over any kind of loading states that we might want to show while the user is waiting for that content to come in. Which massively improves the user experience on that navigation needs.

Once we introduce JavaScript in good old JavaScript fashion, we kind of get addicted and quickly went all in on that. And enter the next phase which is kind of like the single page application era. So we just plain jQuery. Then quickly we build application frameworks that helped us build the shape of normal applications with like Knockout.js. And then you start seeing the frameworks that we still know, love, and use like React, like Vue, like Angular. And the idea is that we cut out the server completely. So now when we request a page, all the server really does is return an empty shell with the documents from sharing, like an empty div. And a JavaScript file. And the idea is that that JavaScript will build up all the HTML dynamically on the client and inject it into that empty div. So once the JavaScript is all downloaded, passed, and executed, we get our application and it's immediately interactive. And all of it is interactive because all of it is built in JavaScript. So there's no barrier anymore to add event listeners, and local state, and all of that at that time. Which is great for user experience.

So this is where building much more enhanced applications and much bigger user experiences exploded because it was much easier to build them at large scale. And navigating around because all of it happens on the client is usually pretty fast. It's only limited by, do we need to go to do some network requests to fetch new data for the new page, or that kind of stuff. But just rendering new HTML for a new page is usually reasonably fast. The real downside here is that initial load, where the user is waiting not only for the server response time, then for the JavaScript to pass. And the larger and more complex those applications get, and the slower your network might be.

Either because you don't have like a fast network connection or you're on a not that fast old mobile device, say. The longer this wait time becomes. And it got to the point where we realized we might have gone a little bit too far with full-on client-side and single-page applications.

So we started moving back to the server. Well we wanted to keep all the benefits from all those nice frameworks like React, like Vue that we started enjoying. So we took advantage of Node, which gave us a JavaScript runtime on the server. And the idea was, when the browser requests a page, the server can run the same framework on the server to generate the snapshot of that initial HTML.

To give a better initial server response. So instead of just returning that empty shell, we can return that snapshot. So that doesn't have any interactivity in it. Because the server doesn't have a concept of interactivity. But it can have all of the visual elements. And then we still return the normal JavaScript alongside it.

And the framework will build a copy of the page on the client. And then inject all of the dynamic parts into that static snapshot that we initially returned. So that injection of dynamic parts is called hydration. And this is basically where all the frameworks do the heavy lifting to inject the dynamic parts.

And then take over from there. Where if you navigate around after this, it's mostly like a single-page application feel. So this is trying to get the best of both worlds. Where we get a better and faster initial paint. But then we still have to do the hydration bit.

And the hydration bit can be really slow. Because, like I said, you're essentially building the application twice. Once on the server and then on the client again. To be able to then inject all the dynamic parts into it. The second problem with this approach is that we're now back on the server. So we have the original problem again from the HTML PHP days. Where if we do a lot of work on the server to get that initial render. The initial server response time can get really slow, really fast.

So how do we solve that? We introduced server components. So React kind of became famous for this. But they weren't the only framework or not even the first framework to play with the idea. But fundamentally, the concept is to give the bundler the knowledge about which components in our application. Actually are dynamic and actually need client-side Javascript. And which components are purely static and can be rendered once on the server and then be done with it.

And if you take a look at your application on a larger scale. Most of your application is static. And only very few components actually need client-side stuff like event listeners like local state, etc. So the idea is we do the server request. We still do that initial render on the server. But then we don't need to send the whole application to the client and re-render the whole thing again. We can do what we call partial hydration.

So if the bundle knows which components are dynamic, it only needs to send those dynamic components to the client. And only those need to get hydrated and get their dynamic stuff injected. Now this visualization might look very similar to what we had in the jQuery and Ajax days. Because fundamentally, it's still trying to do the same principle where we're just much more selective about what we want to do with the Javascript. And where we need Javascript. And this is where the internet and Google Fashion started mocking the idea a little bit by saying, OK, so our component is just PHP with a nice tree occurred on top of it.

And that we're evolving but backwards. And I just want to mention that because I think that's just a very interesting way to look at progression. And I actually think it's wrong. I think using old best principles and ideas from old technologies and applying them to new technologies and new capabilities that come with those technologies is actually a really good way to evolve your tech stack. So I actually think this is a really good progression through. We got into Javascript. We got everything dynamic. And then we took those learnings and took the learnings from the past of how to build stuff on the server and kind of combine them.

All of this is to say this kind of solves the hydration thing. With partial hydration, the Javascript that we need to download is much smaller. The hydration itself is much smaller and much quicker. But it doesn't really solve the second problem. So if the server does a lot of work and the server response, the initial server response is very slow, how do we fix that? Now, to understand that, we'll build a solution from scratch, like a very simplified solution from scratch ourselves. And for that, we're going to look at traditional server side rendering and where that brings us. Then we'll look at some stuff like classical streaming and some of the limitations that come with it.

And finally, we'll take a look at how autostreaming kind of addresses those final limitations and what all of that gives us from a DX perspective, but also from a user experience perspective. All right, let's write some code. For that, I prepared a small demo application. It's fairly simple. It's just a movie or TV show app where you can look at a TV show, see some details about it, what the plot is about, see details about the cast, and maybe some similar movies and TV shows that you might like if you like this one. It's called NotFlix. I don't want to get sued. But you can see, if you look at the code for this, I also try to keep it fairly simple. So this is just an Express app. Express doesn't really matter here. It's just the simplest server setup. And we're rendering individual components here for the different sections of the application. So we have one component for the header, one component for the title, image, and details, then one component for the cast, and one component for the simpler movies. And if we look at those components, you can see that they look mostly like regular React components. You might notice that they are asynchronous functions, not normal functions, because we are on the server.

So I'm treating all of these components as server components that only need to run on the server once. And in that case, we can do asynchronous stuff within them, which is nice because that means data fetching and stuff can be done right in line. So in this case, all of these components will go to the server to fetch their respective parts that they need from an external API. We're not in control of that API, so we can't make that faster. But we are in control of our components here.

On the server, what I'm doing right now is basically what a classical server-side renderer is. I'm starting to collect the HTML, so just starting with the document structure. Then this JSX gives me an array of children. So then I just loop through those children. All of these children can be asynchronous. So we await, run the component function with the props, and then we render them to strings. I'm just stealing that from React. In the end, I'm concatenating all of those strings from all of the components. I'm closing the document text, and I'm sending all of that back to the server.

Clicking around here, we'll see that fundamentally works. But with a lot of asynchronous stuff and fetching on the server, the response time is slow. It's frustrating from a user experience perspective as nothing happens until all the HTML is generated. The first step we can take is classical streaming. With Express in Node, we use the response write method instead of send to open a stream and keep writing until finished, then close the stream. We start with headers, write collected HTML to the stream, and end to indicate the stream is finished and everything is closed.

With Express, all we need to do is use the rest or response write method. So instead of doing the send at the end, we can actually start writing the response. And that opens up a stream but keeps the stream open so we can keep writing until we're finished. And then we just need to tell it to close the stream.

If we do that, we want to actually start with header. And then everywhere where we just collected HTML instead, we now write it to that stream that we just opened. Same with the closing text. And then because we've already written everything, all we need to do is end.

So this tells the browser, hey, the stream is finished. Everything is closed now. You can stop listening to the server. And if we now refresh the page, what we'll see is that the logo renders immediately because it doesn't do anything asynchronous. And then we see what we kind of expected, like the content streams and when it's ready.

But you also see that the sections are a little bit out of order. It's kind of expected because before we were running all of those asynchronous components in parallel, because we could, because we were just awaiting all of them and then stitching them together. Now that we're streaming them whenever they're ready, we can't really do that anymore.

So for now, to keep things simple, I'm replacing this with a simple for loop. This is not the most efficient way to do it because now we're doing one asynchronous component at a time. There's better ways that you could still run them in parallel but then resolve them in the right order. But to keep things simple for this demo, let's use a for loop. This does is again, we see the logo and then we wait until the individual sections come in.

Now, this is definitely a lot better than the initial experience. It's still not great though. So if I click now on the new thing, I get immediate visual feedback that something happened. But I'm still mostly staring at a blank black screen. Ideally, what I want is some form of loading state. So some fancy skeleton experience that I can use to indicate to the user that something is loading. Let's implement a very simple version of this ourselves to see how it would work on an etherhoop.

And then we give it some kind of fallback that allows us to render that loading state. So in this case, I have a skeleton component prepared. And basically, I want to render this fallback while any of the children still do their asynchronous stuff. So how would we do that on the server? Let's implement a very simple version of this ourselves to kind of see how it would work on an etherhoop.

And for that, we need to first keep track of some form of map or object where we can store all of the suspended items. All of those asynchronous components that we can't immediately. And then we do the suspense component itself, which really just does two things. One is it takes children like we've seen and a fallback. And what we want to do is put those children, like we create a unique ID for each instance of this component. And then we store the potentially asynchronous children in that map with the ID and only render the fallback. And we store the suspense ID here as a data attribute.

You're going to see why in a second. CSS tip on the side, we give that diff that we wrap everything in a display of contents. What that means is the browser will ignore it for any layout purposes, so it won't break any Git, grid, or flex layouts. So that's pretty neat. But if we do that and we import suspense here, what we can then do is actually just refresh the page and see. It does what we want it to do, it renders that fallback.

And right now we don't have any logic yet to tell it to replace that fallback with the content once it's ready. So let's add that. So we take that suspended object. And we check if there's any items in it. And if so, we want to render those suspended items. And then once they're rendered, we do what we've done in the very beginning. We just concatenate all of the children of each of those suspense components and write them to the string. And if we do that, fundamentally that works. It does what we want it to do.

So now once the content is ready, it pops into existence and it's added to the stream. But that's the fundamental limitation of classical streaming. We can only add to the stream because we're only appending to that streamed document that we've already started sending to the server. So how can we actually replace this loading state with the real content? Short answer is, in classical streaming, we can't. So this is where out-of-order streaming comes in. And for out-of-order streaming, we need a little bit of JavaScript, at least right now. So there's two things that we want to do here. We want to wrap the content from the suspense boundary, like the suspended content, in a template tag. This just tells the browser to ignore it for rendering purposes, but it is still in the DOM. And then we want to add JavaScript logic to replace the placeholder with the actual content once it's ready.

There's a million different ways how you could do this. For the purpose of this demo, I'm going to use a custom HTML element. And we're just rendering that after the template tag with the target ID. That's the suspense ID, which allows us then to link it back to the loading state with that same ID. So that's why we added that as a data attribute here. So now we just need the JavaScript to do the replacing bit. To do that, we essentially can do—the cursor is helping here—something like this. So we are registering that custom element. Like I said, this is our tag name that we want, extending HTML element. And all we really need is a connected callback. This connected callback is called every time this custom element is being rendered, or every instance of it.

So when this runs, we want to grab the previous sibling of our current element. Because we are in control of the rendering, we know this is always going to be the template tag that it's relevant to. And we get the ID from the target ID attribute. Then we look in the DOM for that data suspense ID from the suspense boundary. So we're basically looking for the loading state that belongs to that ID. And then we're clearing the HTML of that loading state and replacing it with the HTML of that template tag. So fairly straightforward, just a few lines of JavaScript.

But what that does is basically give us exactly what we want. When we refresh the page, we have immediately the loading state available. And then it gets replaced by the real content once that is there. And again, when I click on a link, I immediately get that nice loading state. And everything else comes in. We're not limited to a single suspense boundary either. We can easily have multiple of these for each of the sections. So I would only recommend that if you know exactly the heights of each of those sections to minimize layout shift.

But if you do know your heights, you can do this and then have a much richer loading experience. And you can then allow things to pop in the moment they're ready. You don't have to wait for all of these sections to be ready to replace your individual loading boundaries, which is pretty cool. The other cool thing that you get with this is because it's streaming and it's all coming from the server, you get a lot of cool browser functionality essentially for free. What I mean with that is things like caching and history back and forwards. So if I now that this is loaded, go back in history, you see this is immediate.