Comparing JavaScript Frameworks Performance Using Real-World Data

Hello everybody, I hope you're enjoying the JS Nation conference. Today I'm going to talk about web performance, and specifically about how your choice of framework impacts the performance of the websites and web apps that you build. In order to do this, I'll be using real-world data to compare the performance impact of the leading JavaScript frameworks and meta-frameworks.

First though, a few words about myself. My name is Dan Shapir, and I'm the Performance Tech Lead at Next Insurance. We're an InsurTech unicorn aiming to revolutionize the way in which small businesses get insurance. And as everything is done online, it's critical that performance is as fast as possible. I'm also a host and panelist on the popular JavaScript Jabber Weekly podcast. I'm an invited expert on the W3C Web Performance Working Group where the performance metrics I will describe today are discussed and standardized. If you want to contact me about web performance or web development in general, you're welcome to do so on Twitter or Masterdon.

Speaking about web performance, we all know that building fast websites can be hard. And it's not just hard for you or me, it's hard for everybody. And that's the reason that according to Google, the majority of the websites it indexes do not have good performance. For this reason, when we select which tools to use for building websites, we want to choose tools that will set us up for performance success. Certainly, we don't want to choose tools that will set us up to fail. And probably the most important tool we choose as web developers, the most impactful item in our arsenal, is the framework we use. The reason for this is that when we build a website, it's the framework that's in the driver's seat. Our code is sitting in the back, telling the framework where to go and hoping that it takes us there. It's the framework that decides where and when to run our code, which parameters and state to pass to our code, what to do with the values our code provides to it, when to update the display, how to handle user interactions, etc. So, it's really not surprising that our choice of framework can have a huge impact on the performance of the websites we build with it.

Fortunately, these days we are spoilt for choice when it comes to deciding which framework to use. In fact, it seems like a new framework is being introduced almost on a daily basis. And what's more, a prime motivation for the creation of new frameworks is often web performance. They aim to enable us to build faster websites, more easily and consistently. In addition, there are a whole bunch of meta-frameworks implemented on top of these frameworks. For example, if you want to build websites using React, you can do so with Next.js or Emix or Gatsby or Astro and others. So the question is, which framework or meta-framework should we choose? The answer to that is that with all else being equal, we should prefer to use the framework that is the most likely to help us produce a fast website. But how can we know which framework that is? The answer to that is that we need to measure them and compare. There are two main methods for measuring performance on the web. The first method is known as lab tests.

This means measuring website performance in a controlled environment. Ideally, it should replicate the environments in which real users access our website. The main advantage of using this method is that it provides total control over the tests and complete visibility into the results. We can inspect every facet and behavior of our code and recreate it as required.

But lab tests are also difficult to do because setting up the lab environment can be challenging, and determining exactly which scenarios we should replicate can also be difficult. Lab tests are especially problematic when we want to compare performance of the various frameworks because we need to build the exact same applications using every framework we want to test. There are some tools that can help us automate this process, but significant time and effort may still be required.

The other approach that we can take is to use field data. That means collecting performance data from actual sessions of real users, preferably as many as possible. Random user measurements, or RUM for short, is about getting performance data from live sessions and using that data for the comparisons. This way, we can be certain that what we measure actually reflects what our users are experiencing. But how can we gain access to the performance data for all those sessions? Is it possible to instrument every website built using any framework so that it reports the performance data for every session, and then to collect this data into some sort of database so that we can generate comparisons?

Obviously, we cannot do something like that. But it turns out that someone can, and that someone is Google. That's because instead of instrumenting the websites, they instrumented the platform on which these websites run, the Chrome browser itself. Unless you opt out, the Chrome browser collects performance information about every session of every website that you visit and sends it up to Google Database in the cloud. This database is the Chrome User Experience Database, or CRUX for short. Google uses all this performance information that it gathers as a ranking signal for its search engine. That means that when Google decides how to order results in the search engine results page, it gives a ranking boost to pages that have better performance according to this data.

The primary performance information that is stored in CRUX are the Core Web Vital measurements for each visit to every page. I'm guessing that many of you are already familiar with the Core Web Vitals, so I'll make this description brief. There are three Core Web Vital metrics. The first is Largest Contentful Paint, also known as LCP. It measures the time that elapses from the start of navigation to a page until its largest piece of content is displayed. First Input Delay, or FID for short, measures the time from the first user interaction with a page until the page can respond to that interaction. And Community Layout Shift, or CLS, measures how much the content on a page shifts around on its own, not in response to some user interaction.

For each one of these metrics, Google has defined ranges, which specify measurements as being good or green, needs improvement or yellow, and poor or red. For example, for LCP, good means any measurement under 2.5 seconds, needs improvement is anything from 2.5 seconds to 4 seconds, and anything above 4 seconds counts as being poor. A session is considered as having good performance if all its metrics are green, which means that all these three metrics measured for it are in the good range.

In addition to using the data in Crux as a ranking signal, Google also provides us, and by that I mean everybody, with free access to this data. For example, in the Google search console, there is a panel which shows the performance data from Crux to all the pages in a website. This way we can know which pages we need to focus on in terms of performance. Likewise, we can use the Google PageSpeed Insights service to get the Crux performance data for any URL in addition to some synthetic results. But in this case, what we want to do is something different. We want to aggregate the performance data for all the websites built using a specific framework.

How can we do something like that? Luckily, Crux has some friends that can help it provide us with this information. Friend number one is the HTTP archive. Whenever a URL gets added to Crux, it's also handed over to the HTTP archive, which runs a collection of tests on it, extracting a whole bunch of information about how this page is constructed and how it operates. In particular, one of the tools that HTTP archive uses is a service called WAPLizr. This service examines the page and determines which web technologies are being used by it, in particular which frameworks and meta-frameworks it uses or is built on. All this information is saved in the database alongside the Crux data. This enables us to perform queries and extract, for example, the aggregated performance data for all the websites built using React.

But to make life even easier for us, this awesome guy that works at Google, Rick Viscomi, created something called the Core Web Vitals Technology Report, which is an interactive dashboard that does the queries for us and then graphs the data. This dashboard is open to all and it's free to use, and you can use it to graph the aggregated performance for all the websites that use any technology identified by WebAnalyzer. And this is exactly the tool that I used for analyzing the data for this presentation.

But before finally looking at this data, there's an important point I need to make. We must always remember that correlation is not the same as causation. In our case, it means that when we see a particular performance pattern for a framework, it doesn't mean that by definition it's the framework itself that's the primary cause of this pattern. For example, a leading framework that has many websites implemented using it is likely to have a long tail of websites that are still using older versions and aren't being updated or optimized.

Conversely, a new framework that promotes speed is likely to draw performance-minded developers that will optimize the heck out of the relatively few websites that they built using it. That said, despite the difference between correlation and causation, I wouldn't go back to using Internet Explorer. And likewise, all else being equal, I would tend to prefer a framework that has a better performance profile, on the assumption that I'm not really that much better than all the other web developers that are using those frameworks.

Okay, then, having said all that, let's finally get into the data. In this first graph we see two lines. The darker line represents the ratio of websites having good performance out of all the websites in the CrUX database using any technology. Remember how I said that the majority of websites don't have good performance? Well, we can now clearly see it. Only approximately 40% of all websites have good performance according to the Core Web Vitals. The rest do not. At least we can see that the situation is improving and the graph is trending upwards. I also put the data for WordPress on the graph, the lighter blue line, because WordPress sites represents approximately a third of all the websites. Far more than any framework. In fact, more than all the frameworks combined. We can see a strong correlation between WordPress and all sites in general, but WordPress is noticeably lower, which means that something else is pulling the general line up. What do you think it is? Is it the frameworks? We will see.

One more thing, all the graphs I'm showing are for mobile. I did this because we're living in a mobile-first world and the majority of web sessions are on mobile and also mobile is usually more challenging in terms of performance. So here, finally, are the results for the top frameworks. And when I say top frameworks, I'm not making a value judgment. I'm simply looking at the frameworks that have the largest number of websites built using them in the crux database. As we can clearly see, it's not the frameworks who are pulling the performance of the web up. Of all the leading frameworks, only React and Preact have better performance than WordPress. And all of them have poor performance than the web at large. But we know that a whole bunch of new frameworks have been introduced in recent years, and that these frameworks often focus on performance. Let's look at those as well. Qwik is a new framework that is especially focused on performance, yet it seems to be all over the place. And it's not that it got really good, and then really bad, and then really good again. It's just that there are still very few websites out there that are built using Qwik, because it's so new. The number is so low, in fact, that it's hardly statistically significant, but it does seem that it's heading in a good direction. You might also be wondering about the hump in the Preact data.

It's not that Preact started fast and then got slower. Rather, there was a bug in how Apple ISR identified Preact usage in websites. As a result, the data for Preact prior to November 2021, when that bug was fixed, is meaningless and should be ignored.

Another thing that's kind of surprising is how poorly Svelte seems to be doing compared to the other frameworks. We know that Svelte is supposed to be really good in terms of performance, yet appears to be worse than the others. And that brings us again to the issue of correlation vs. causation, and how carefully we need to be when drawing conclusions from this data.

The data that we're looking at is for the entire world, and as we know, some countries have much slower internet than others. Likewise, in some countries, slow mobile devices are more common than in others. It's very possible that when developing websites for use in such countries, people gravitate towards frameworks that are lighter-weight. And by doing so, they actually adversely impact the overall global score for these frameworks, causing them to appear to be slower than they actually are.

To check if this is indeed the case for Svelt, we can limit the graph to a particular geographical location. I chose the US simply because it has the most sessions in Crux. Indeed, now we see that Svelt jumps to the top, above all the leading frameworks, and only slightly worse than the web in general. React remains in a good position, and surprisingly, above Preact, which is supposed to be a lighter weight alternative. We will see why that might be later on.

Another interesting filter that we can apply is that instead of looking at all the websites in crux, we can look at the top 100k websites in terms of traffic. That means to filter out the less popular websites. Why is this interesting? Well, because we can assume that the most successful websites can afford the best developers, and also they can afford to invest the most in their infrastructure, and their CDN cache is more likely to be hot and primed. Given all that, we can assume that the top websites will likely have better performance. Is that indeed the case? So here we are looking at the top 100k websites, but I don't expect you to remember the numbers, so let's just compare them side by side. WordPress is indeed up by 18%, which seems to corroborate my assumptions. Angular is also up and by a significant 22%, which goes to show that you apparently need to be an expert in order to build fast websites using Angular. But its ratio is still pretty bad. Vue is essentially the same, which seems to indicate that you don't really need to be an expert to get as much out of Vue as you're likely to get. And it's the same with Preact. Surprisingly, Svelte is down by 8%, and I don't really know why that is. But even more significantly, React is down by a whopping 16%, which seems to indicate that the better React developer you are, the worse you are in terms of performance. Kind of surprising. To try to figure out these surprising results, I decided to look at the performance of the MetaFrameworks, since that's how frameworks are often used these days.

Moreover, MetaFrameworks are especially intended to improve performance. For example, by enabling server-side rendering, known as SSR, and static-side generation, known as SSG.

First, let's look at MetaFrameworks in general in the US. Because of how it works, I'm counting QUIC as a MetaFramework as well. And because I'm filtering for the USA, the number of websites built using QUIC is even lower. In fact, it's just 8, which means that it's really premature to talk about it, but again, it does look promising.

The same is true for SolidStart, which only has 9 websites built using it. It's less focused on performance than QUIC, but Ryan Corneato, Solid's creator, is doing some very cool things. Again, time will tell. Astro is another newcomer showing promise. It's got a bit more usage at 186 websites, but it's still early days. At the bottom we've got Nuxt, slightly below Next.js. The good thing I can say about it is that over the past two years, it's improved by more than three-fold. That's a lot, but it still has got a long way to go.

Now, let's focus on the React frameworks. I've removed the old timeline and WordPress, and instead added React itself for reference. Both Gatsby and Remix are doing well, and are above the aggregated React line. Based on what we've seen before, the way that Remix fluctuates indicates that there are only a few websites using it, and that's indeed the case at 254. Gatsby matches Remix performance, but it has many more websites at around 5,000.

One thing that is surprising is that Next.js is worth worse than React in general. It's surprising because, as I mentioned before, like other meta-frameworks, Next.js enables SSR, which is supposed to improve performance, and it also has some other performance optimizations, for example, for images. Given that Next.js is now 8% of all React websites, this is really unfortunate. On the positive side, it looks like it's improving and I know, and we know, that it's doing additional things to improve performance going forward, so there is hope.

You may also be surprised as to why I threw Wix into this graph. After all, Wix is not a React meta-framework. Or is it? Turns out that it kind of is. By the way, for those of you who don't know what Wix is, it's a popular drag-and-drop And I used to work there before I went to Next Insurance. And the reason that it's kind of a React meta-framework is that Wix uses React for displaying the websites built using it. So every Wix website is also a React website, and that's how it's identified by crux. And it turns out that Wix has a much better performance than all these React meta-frameworks. Interesting. So, it turns out that people dragging and dropping their way to a website are getting better performance than custom websites built manually with React or one of these meta-frameworks. Just think about that.

Moreover, it turns out that almost 20% of all React websites are actually Wix websites. Way more than any React meta-framework. As a result, Wix has a significant impact on the overall React performance. In fact, it appears that Wix is pulling React up. So the reason that React has better aggregate performance than all the other top frameworks is because Wix is built using it. When I change the filter to view only the top 100K websites, we can see that the amount of Wix websites drops significantly. This is because Wix focuses on small businesses. Consequently, Wix has very little impact on React in this scenario. This explains why we saw such a large performance drop in React by 16% when we looked at the top sites, because by doing so we took Wix out of the equation. This means that the numbers we see for React in that scenario, which are on par with View and Preact, are much more representative of that framework when it's being used by web developers using MetaFramework or manually rather than by Wix. So, to sum things up.

First of all, Crux is a great resource of information. I think we can all agree on that. Sure, we may have various feelings about the fact that Google is collecting so much information about us and our sessions, but the fact that it also provides us with access to this information is very useful.

The second thing is that while we saw that the good performance ratio of some websites was pretty low, none of them had a ratio of zero. This means that you can build a fast website using any framework, even Angular. Conversely, we saw that no framework has a good performance ratio of 100%, which means that you can build a slow website using any framework, even Quick.

But having said that, we saw that there are differences in good performance ratios between the various frameworks and meta-frameworks. This means that the probability that you will build a fast website does vary between them, depending on which framework and meta-framework you choose to use. Your choice of a framework or meta-framework can have, in fact, a significant impact on the probability that you will build a fast website and the amount of know-how and effort that will be required in order to do so.

We also saw that, unfortunately, the top frameworks have worse performance than the web at large. So all those great websites that we're building using React or Vue or whatever? Well, guess what? Many of them are actually making the web slower and worse. And that, if you've got a neighbor who knows nothing about web development and is using Wix to drag and drop their way to a working website, that website your neighbor builds will likely be faster than if you build using the same website using your favorite framework. Again, think about that.

This means that we definitely need to do better, because currently for the majority of web developers, frameworks and metaframeworks are not delivering performance success, at least not for the majority of them. I'm really hoping that the new frameworks coming out and the improvements being introduced by leading frameworks will change this unfortunate situation. Time will tell.

Thank you for attending my talk on comparing framework performance using real world data. If you want to contact me about this presentation, about this data, or about web performance in general, please do so on Twitter or Mastodon where my handle is Dan Shapir.