How to Show 10 Million of Something: Frontend Performance Beyond Memoization

Hello, today, I'm going to talk about performance in React apps. Now, when I started working on the project that I'll discuss in this talk, I found that there are many resources on React performance and frontend performance in general. But most of them were not actually all that relevant to the specific type of performance that I was looking for.

Probably the most common conversation around performance is the one of web vitals. Now, web vitals are super important. They determine how fast your app loads, how fast it is interactive, and so on. And nowadays, there are many great solutions for this. If you're in the React ecosystem, you can try Next.js or Remix. And there are also many other great tools, such as Astro or Quick. However, I mostly work on what I would call applications or dashboards, so I find that users don't actually care that much how fast the site loads as long as it's reasonably fast. And they care much more about how it performs once they're using it.

That brings us to the second topic that is discussed quite often, which is rendering performance. This is a topic I deal with fairly often, but I find that there are many excellent resources on this nowadays. For example, I'd recommend the writing of Nadia Makarovic. I found that once you know how to optimize React a little bit, it gets quite fast, and the entire problem of rendering performance is mostly solved. Additionally, React Compiler is coming soon, which will help us even more with this. And then, of course, there are other frameworks that focus even more on rendering performance, such as Solid.js.

Another topic on which there are already plenty of resources is the one of having very heavy runtime compute. And there are many things you can do here. One of them, for example, is to write RustWasm, and if you're interested in that sort of thing, then Ken Wheeler has some really good talks about this topic. But the kind of performance that I found myself struggling with is how to deal with enormous amounts of data, tens of thousands, hundreds of thousands, or even millions of items. Now, as this tweet here shows, I think a lot of people don't understand why you would ever want to do this, and it's a very reasonable question, which is why I want to talk about it today. So, what we're looking at today is how to show 10 million of something.

Let's start with a demo. The application we're going to look at today is Axiom's Trace Viewer. It has a fairly normal feature set for this kind of application. So, for example, you can page through error spans, you can look at the details of various spans, and if a span is particularly slow, you can take a look at how it compares to usual spans of that type. Of course, it also has many other features, but the main point I want to show is that it's quite performant, even though we currently have a Trace Open with just over a million spans, and this is in local host, which has quite a bit of overhead compared to running the app in production.

So, now that you've seen the app, let's talk about how you can achieve something like this. Some quick details about me.

My name is Chris. I work for Axiom, which is the application you just saw, and in the open source world, I'm known for some contributions to CreateT3App and TRPC and some other projects.

So, how do you handle 10 million items? Well, my recommended thing to do is just to not do it. Now, of course, this is not very helpful advice, but I want to be serious here. Try to avoid it. This type of work takes a lot of time that you could spend on fixing bugs, writing new features, and so on, and it also makes your code base much more difficult for the next person to work on.

I think many of us love this idea of solving very interesting technological problems, but what you need to think about is, is this the best way you can spend your time to make your users' lives better? The way to know whether you need this or not is to listen to your customers and see what their frustrations are. The other thing that can help you figure out if you need this or not is to consistently develop against a real server with real data of a similar scale as your biggest users have.

So, if you should avoid it, how can you do that? It's going to depend on your situation, but there's many options here. You can use pagination, so getting in 10 or 20 or 100 results at a time. You can use streaming, so only getting in the specific results that are currently needed. Or maybe you don't even need the individual items, you just need to run some sort of aggregation on them. In that case, you can do it server-side, or in many cases, even in the database. And the final thing I want to point out is that if your product owner asks for this, I would really suggest to negotiate the requirements and figure out why it is that they want this. Maybe they're actually presenting you with an XY problem, and there's really a much better solution, such as one of the three things above.

But let's say you have negotiated, you have thought about this, you have considered the alternatives, and you've come to the conclusion that the only way to make your app good is to show millions of items at a time. What do you do now? The first thing that's very important is to measure before you start optimizing. There's a good chance that you're completely wrong about where your bottlenecks are. There are three main things to measure, compute, memory, and network, and we'll look at each of them later on.

So now that we're through the introduction, let's talk about the specific things that helped us out in our situation. I'm going to omit some implementation details here, so I'll show a few things that don't one-to-one represent how Axiom behaves in production. But these details are very specific to our situation, and I think we'll end up with a more useful talk this way.

Here's the starting point. It's late 2023, and tracing in Axiom works great, up until about 5000 spans. But we didn't know that last part at the time, because why would anybody ever want to create traces that big? And then we launched a new pricing plan, and people realized that they could use our tracing as effectively a profiling tool without incurring huge costs, and so they did, and we couldn't handle it. This is what happened if you tried to open a trace with even just 10,000 spans. I think you can see from the error message that we hadn't even considered this possibility or this way of failing.

After some quick investigation, we realized that we really had to rethink the entire architecture of the trace viewer. So let's look at what we did step by step to improve our capability by several orders of magnitude. Now this initial set of errors didn't actually originate in the front end.

We had a small node server that we were relaying the data through that was communicating to our actual back-end, which is written in Golang. We could have probably kept optimizing that node server, but we very quickly realized that the requirements weren't just twice or ten times as high as what we currently supported, but probably closer to a thousand or even ten thousand times. So the first thing we had to do was to stop relaying the data through the back-end for front-end.

Having a BFF can be great. It gets deployed together with the front-end, it helps with security, version skew, and many other things, but it's not the right fit for every situation, and it clearly wasn't here anymore. The solution here is simple, right? Instead of relaying through the BFF, you just talk to the back-end directly. But this does bring some difficulties with it. For example, you might need to rethink how auth and security works. But for huge amounts of data, it becomes very important to pipe it through as few places as possible. So we changed the architecture to have the front-end talk directly to the main back-end for this use case, and this worked great.

But very quickly, we started hitting other limits, and the first one was rendering performance. This is probably not surprising if you've tried to render thousands or tens of thousands of items in React before, but it doesn't go very well. The good news is this is the easiest one to solve, and the solution is to virtualize the data. All that means is that instead of rendering every single row, you only render the ones that are in the viewport and a couple above and below to make scrolling smooth.

And there are two libraries I'd recommend for doing this. If you want something that's very small and light, less than 2 kilobytes of bundle size, and basically plug and play, then I would recommend the library Virtua. And if you're looking for something more powerful and configurable, then Tanzac Virtual is an excellent choice. This solved our actual rendering, but we still had a lot of compute that was blocking the application. Let's look at how to investigate that.

The main tool for this is the performance tab in the Chrome DevTools. Before we get into how to analyze this, there's two magic numbers I want you to know about. The first one is 16 milliseconds, which is one frame at 60 hertz. So if you block for more than 16 milliseconds, you're going to drop a frame. The second number is 200 milliseconds, which is approximately the longest amount of time you can block on a click before it starts feeling quite bad for the user.

Typing should of course always stay at 60 frames per second, but especially if you need to make network requests, users tend to forgive latency of up to 100 or in most cases, even 200 milliseconds. In terms of how to read this diagram, what you mostly care about is the self time, which is how much time is spent not in children of the current span, but in the span itself. However, if you scroll down, you'll mostly end up in library functions or system calls and stuff like that. So at that point, you want to go back up and find the last thing that you recognize or that is part of your own code base.

A quick rule of thumb here is to look for the last blue thing. But if you have source maps, you can do much better than that. And you can see on the bottom of the screen here, it's referencing the parse.ts function, which is part of our own code base. And if it is instead referencing a function inside of a library, the file name would instead start with node modules. But also having said that, there's no one quick trick that will let you understand every frame graph immediately. So if this is something you do often, you should really get used to skimming these top to bottom and finding the patterns that are relevant to your app and becoming familiar with the names of things that happen quite often in your app.

Now the performance problem we had in our app was that we were now virtualizing the list, but we were actually still processing all the data upfront. I'm sure you've seen an example like the component on the left before, where you get a list of items, and then you process them, and then you render them one by one. The problem here is that if you have a million items, then you need to process a million items even if you're only rendering five of them.

The most common solution here is to move the processing of the data into the component that does the rendering for one piece of the data. This does mean that the app will run a little bit slower after the initial load, because every time a new row is rendered, it needs to process the data for it. But usually that overhead is negligible.

And that was actually, in our case, all of the optimization we had to do in terms of compute. But there was another much, much bigger problem. After a certain amount of spans, the app would just run out of memory. This makes sense as each additional item requires additional memory, and there just wasn't enough of it to go around.

Memory in Chrome is a bit complicated, but a rule of thumb that I use is that you can usually get four gigabytes if you're the active application, but you should really try to stay under one gigabyte if at all possible. And the way to know how much you're using is to open the memory tab in the dev tools and take a memory snapshot. In this case, we have opened a trace with 8,210 spans, and you can see that the memory snapshot takes 750 megabytes. So we could only handle maybe 30,000 or 40,000 spans before running out of memory.

The memory tab in the Chrome DevTools is quite powerful, but to give a crash course, what I often do is sort by retain size and then drill into that and look for the first thing I recognize. And then in the lower half of the tool, you can see what is actually using that thing. So in our case, you can see that even though we only have about 8,000 spans, there are 1.9 million of these observable values and they're overwhelmingly used in the Traces store. This made the cause of the issue quite clear.

We were using MobX as our front-end state management solution to have fine-grained reactivity in the UI. But the way MobX works is that it wraps a proxy around each object, which is usually a very good solution. But in this particular case, it gets expensive very fast if you have millions of deeply nested objects. We experimented with other state management solutions or even just using built-in React state, but we weren't really happy with any of it. So what we decided to do is to just use JavaScript primitives. We kept the spans, which are large objects, outside of MobX and even outside of React, instead of just using regular JavaScript objects, and kept something much smaller in MobX that was just enough to let us know when to update the UI.

Now, of course, React state and other state management solutions are built for a reason, right? We don't want to have to tell our application when to re-render the UI. However, in this case, we realized we don't actually need the most fine-grained reactivity, as we get all of the spans at once, and then we just need to notify the application that we have them. And once they're loaded in, they're not going to change. Keeping the expensive stuff in raw JavaScript was enormously successful. We suddenly had support for around 100,000 spans, or in some cases even 200,000 or 300,000. But the problem is that spans can have an arbitrary number of attributes, and as you can imagine, a couple million of arbitrarily large objects get expensive even if you don't have the overhead of a state management solution. Also, loading them in over the network was taking forever. So we needed to go one step further. Until now the server had been sending all the data for all the spans, and then we used that to render the UI. And that just wasn't feasible anymore. What we needed to do was to denormalize.

What we needed to do was to denormalize. Denormalization means introducing redundancy into a data model, typically in the database, but in our case, in the front end, to improve read performance or to simplify query execution. More specifically, if you think about the trace viewer, there's two parts to it. The span tree on the left and the span details on the right. Now the span details on the right do actually need all of the attributes because you need to be able to scroll through all the details of the span, but it turns out that the waterfall view on the left only actually needs a few attributes. If you think about what the waterfall is showing, you need to know the span's name and the service that emitted it, you need to know its start time and duration, and you need to know the id and the parent id in order to be able to create a hierarchy of spans. But that's it. You don't need any of the other attributes.

So what we started doing is only sending those necessary things for all of the spans, and then fetching the detail for each span in the sidebar as necessary. Of course, having a loading spinner in the sidebar is not ideal, so we needed a solution for that as well. What we ended up doing was a fairly simple prefetch on hover using react-query. Once the cursor enters the span, we wait 300 milliseconds, and if the cursor is still over the span, we fetch its details. A neat trick we used here is to put the span id into an attribute in the DOM itself in order to not need to pass the span id to the event handler. This means that all rows share one handle mouse enter, one handle mouse leave, and one timeout ref. To take this further, you can also do cute things like preload the next and previous search results or the next and previous error spans so that users can tap through those fast.

But of course, there are still going to be situations where this behaves worse, for example, if the user pages through errors or search results quite fast, or if they click very fast. So what I'm showing here is not actually the whole truth. What we actually did is, depending on the size of the trace, we would either load the full span or only the necessary information for the waterfall. This means that most of our users who deal with small traces have the real-time performance, but for the users who do need us to be able to handle huge traces, we can do that as well. Another example of where there's different behavior for smaller and larger traces is in search. So when we have full spans, we do full text search in JavaScript, but when we only have the necessary information for the span tree, we of course can't do that because we don't have any of the other attributes. So what we do in that case is we send a network request with the search parameters and then return a response with all the span IDs that match the search. And once all of that is implemented, here are the five types of requests we end up with. Now this slide is a bit of a fabrication because we don't actually use a traditional REST API for any of this. Instead, we use Axiom, the database, as our API. But if you were building a REST API to behave like what we're doing, then it would look something like this. The first thing you would need is an endpoint that gets you the information of a trace based on its trace ID. And all that means is it has a start time of x, an end time of y, and it has a certain number of spans. Based on the number of spans, you know if it's a small or a big trace, and if you should fetch the full span details or only what's needed for the waterfall. And the start and end times can help enormously with query performance if your table has an index on the timestamp.

Then, of course, we need the two endpoints that return the spans. So either you get a full trace and that just takes a trace ID, or in the case of fetching the trace waterfall, you also pass a trace ID and then you get back what is necessary for the waterfall. In our case, we actually batch this, so we also provide a count and an offset. If you're not fetching full spans, then you need a separate endpoint that will get you all the details for a span. This is used to populate the span details on the right of the screen.

Finally, you need the endpoint that handles search. This takes a trace ID and some search text and returns a list of span IDs that match the search. This is the architecture we ended up with in JavaScript. The first thing we have is the spans, and this in our case is an object where the key is the span ID and the value is the span itself. The reason we do it this way is so that accessing any span is O(1). The second thing we have is called the span ID tree, and the reason we need this is because spans have parents and children, and we need to be able to traverse that hierarchy quickly. So we build this up where each object has only the span ID and the information about its children, and then if we need to know the details about a specific span, we get it from the spans object.

The first thing we have in MobX is called filtered spans, and this is just an array of span IDs that represents the order that the span should be rendered in the trace viewer. Users can expand and collapse individual spans and their children, so this needs to be able to change and have the UI react to that. The second thing we have is search results, which is a set of span IDs. And the purpose of this is that if the user types in some search text, we want to highlight all of the spans that match their search. Finally, we have a get span function that just returns a particular span based on its span ID.

This is the point where we broke through being able to display a million spans at a time, but there were still some things that needed to be improved. One problem that remained is that sometimes if we mapped over a large set of spans, we would still get out of memory. The reason for this is that the way map works is if you map over 1 million elements, you will briefly end up with 2 million elements, as it creates a second array for the results of the map. What that means for you is that if you do need to process data, I would recommend using a for loop and manually processing each element. After you've processed it, you can remove it from the original array and hope that the garbage collector does its job. To take that a step further, shift and unshift are each O(n), but push and pop are O(1). So if you're dealing with very large arrays, I would recommend sticking to push and pop.

At this point, we had solved most of the big issues around memory and compute, but there was still one very big problem and this is actually the biggest remaining problem in the application currently. This problem is getting the actual data in. Denormalization helps a bit because fetching spans that only have the necessary information for the waterfall view is a lot less data than fetching the full spans, but at some point, it's still a lot of data. Now, if you want to understand what your network requests are and why they're slow, you can use the network tab in Chrome DevTools and there are two sections there that people don't use that often. The first is the initiator tab, which I find great. If you have source maps, you can see which function and which line a network request actually comes from in sort of a stack trace way, so that you can find out where in your code this network request is actually being caused.

And the second one I find very helpful is the timing tab. In the timing tab, you'll get the durations for the request broken up into a few different categories. The first one is queuing, and if you spend a long time queuing that generally means that the concurrency limit is reached. You're generally allowed either six or eight concurrent requests, so if more than that are in the queue, some of them will have to wait. But queuing can also be related to some other things such as DNS, the browser deciding it's more urgent to deal with other things like HTML and CSS, waiting on the main thread, or things related to service workers. But generally if you're spending a long time queuing, it's due to concurrency.

The second important one is waiting for server response, and all this means is it's the amount of time your back end actually needs to process the request and send out a response. So if this takes a long time, you should see if there's anything you can optimize server side.

The last one is content download, and other than getting faster servers, the only way you can optimize this is to send a smaller payload. Now sending a smaller payload isn't always easy. If you're sending unnecessary data, then of course you should stop sending that. But if you're already only sending the necessary data, then it can be tough to find optimizations. But here's one that we found that helps quite a lot. That optimization is to change the wire format from thinking of data in terms of rows to thinking of data in terms of columns.

In Axiom, we didn't specifically make this change for tracing, but it also had great performance impacts on tracing. If you implement this, what it means is that you don't need to include the column names in every row. So in the example on the left, you can see that every row has ID and every row has service. This means that in your Stringify JSON, you're sending that ID and that service tens of thousands or hundreds of thousands or millions of times. Now it gzips to some extent, but even so there's a bit of overhead. And switching to a columnar response can help a lot with that. Also if your backend is in a language with a more efficient type system and memory model than JavaScript, then arrays of equally sized items tend to be much faster to work with than structs or objects, so it can also be beneficial in that sense.

Of course if you want to display a bunch of rows of data in React, then React usually likes to have that data in a row format. So you end up paying a serialization cost in the frontend, but we found that to be absolutely worth the performance overhead and everything is much faster in total. Now as I said before, at the scale we're dealing with here, which is tens of thousands or hundreds of thousands or even millions of items, even with these optimizations it's still a ton of data. So we also looked into one other optimization. In a typical trace, even if there are a lot of spans, they are usually only produced by a small number of services.

So for example, you might have one service that represents your API, maybe you have another one that represents auth and another one that represents a database and so on. And even in the case of ten thousand spans, it's probably less than a thousand services. So instead of sending an array that has these service names in it over and over, would it maybe be cheaper to instead create a lookup table of the service names and then just send the lookup table index to the front end along with the table itself? What we found is that there are minor savings in the payload that get sent this way, but it just wasn't worth the effort. gzip and v8 both already optimized very very well for repeated string values, so this ended up just not making a very big difference.

But we had another problem. If this slow request for a huge amount of data fails, then we need to start all over. And that's not great. So what we started doing instead is chunking our data. What chunking means is that instead of sending all the results at once, we send them little by little and use cursor-based pagination to know what the next block of results should be. I mentioned pagination at the start as a way to avoid dealing with the whole problem this talk is about, but even if you decide that this is an issue you need to deal with, a paginated API can still help you a lot.

Several small requests will often finish faster than one large request, and if one of the smaller requests fails, it's much cheaper to retry just that small one than to retry the large one if it were to fail. There is also a downside to it, which is that several small requests tend to be more expensive server-side unless you're quite good at caching. But we found the difference made by chunking to be so big that it was non-negotiable to end up chunking the data. Finding the ideal chunk size can be a bit of trial and error. In our case we found that about 10,000 elements was the right size. But this is going to depend enormously on what type of data you're dealing with.

My final bit of advice around chunking, and we're currently not actually doing this but I wish we were, is to avoid ID-based cursors. There's roughly two ways of paginating. The first one is to say, give me the first 10 results, okay now give me results 11 to 20, and so on. But in some case, the backend doesn't actually know where result number 11 starts, so what you can do instead is use the ID of the last row that was returned as the cursor for the next request. This fixes the problem of the backend knowing what data to send out next, but it means that you need to wait for the first batch in order to be able to fetch the second batch, so you lose any advantages that would be gained by parallelism. If you can use offset-based pagination, I'd strongly recommend it, but you don't lose as much performance by not doing it as you think you might.

The last point I want to make is that you need a culture of caring about performance. What this means is that you need to constantly look at where the bottlenecks are, and if there's anything you could do to minimize them. There are a ton of smaller improvements that I haven't talked about today because they won't be relevant anywhere except axiom-specific code basing goals, but the point I want to make is that making many of these small improvements in the hot path does actually pile up. If you make a bottleneck 10% faster 24 times, you've actually made it 10 times faster, which is a huge improvement. But of course this applies only to your bottlenecks, and as I said at the beginning, excessive optimization in places where it's not important can end up being a huge waste of time that doesn't give your users any benefit. So before spending time on optimizing, you should always make sure that what you're optimizing actually does impact your users. That's all I wanted to talk about today, so thanks for listening and I hope it was helpful for you.