Scaling React-Three-Fiber Applications beyond the Hello World

Hi! My name is Giulio. I'm a software engineer, and I've worked with React and JavaScript for quite a while now. Recently in my career I've been developing some stuff with WebGL, and today I wanted to talk about my experience doing it. To talk about it, I think it makes sense first to discuss about what is it and its history.

WebGL is a browser technology that allows using the GPU to build 3D graphics-based applications with JavaScript. It has been around for quite a while now, almost 12 years, and its usage changed a lot during that time. It all started in the beginning with some power users building cool 3D graphics demos, which were showcasing the technology but not yet delivering value to users. After a while, though, we started building real applications that used WebGL to enhance user experience for some specific use cases, which answered a specific business need and enhanced the user experience. Those experiences, though, were still delivered as special applications, an isolated section of a website with a clear scene from the rest of it, not something that you would use every day.

This changed, though, with time, and today WebGL is used in all sorts of applications that we use every day and professionally, even. Think about Google Maps or Figma. You don't even know that WebGL is there, but it's using the GPU to accelerate the rendering, both for 3D and 2D applications, and it's what makes the application possible in the first place. So since WebGL is today so widespread, can we say that 3D graphics on the web are nowadays a commodity, something that is easy to build? Well, to answer this question, I think we have to look at what is the current tooling ecosystem that is available for it.

One example of this is 3.js, a library that accelerates the development of WebGL applications by providing utilities and by wrapping its API, providing an API that is more similar to the DOM. Nowadays, we can even use React together with 3.js to build WebGL applications. And this really boosted its adoption recently.

In fact, thanks to a library called React ReFiber, we are now able to use React declaratively to build applications and scenes using 3.js. The power of React is that it allows to easily reuse components and bind your data without imperative mutations. What's really powerful about React ReFiber is the ecosystem that is built around it. There are many open source projects that facilitate some common 3D rendering tasks such as post-processing, layouting, physics, and accessibility. There are even some component libraries, such as Tri, which provide advanced rendered techniques such as lighting, skyboxes, shadows, and keyboard and mouse controls as reusable React components.

This is really powerful since it allows everybody to build really cool applications with very few lines of code. For example, this demo really showcases what you can do with it. Physics, mouse controls, advanced materials, and reflections. The code that is needed to build something like this seems a lot, but actually it's surprisingly low. This demo is in fact around 60 lines of code. This is possible because it's able to reuse a lot of code from the component library.

We have seen a lot of cool demos like this one recently around on Twitter, and a lot of talks have been made in conferences on how to get started with this new color technology. But does this mean that now I can easily build my Figma or my Google Maps just by using those libraries and writing a few lines of code? Well, unfortunately, not really. For example, if we look at something that implements a similar thing as what Google Maps is doing, such as procedural.js, we can see how it quickly turns into a giant library with over 10,000 lines of code. This is because if you want to develop something that is more data intensive and uses 3D graphics, you have a lot of challenges to overcome. For example, you have to efficiently load and download lots of data, triangles and textures which represent the terrain. You have to stream your data in tiles from a server, and you need to handle level of details between zoom levels so that if you zoom out, you get a lower resolution of your scene that is renderable. And you also want fast collision checking for your mouse-based interactions. Unfortunately, the answer to those questions is not something that you're really going to find on a web development book. It's something that is way more common between game developers, not web developers. Game developers use different languages and different tools, such as C++, so there is not much around on the web that you can find. But all this insider knowledge is what made open-world video games possible in the first place. For example, algorithms that dynamically simplify a complex scene when you look at it from a distance. Using these kinds of tricks is crucial as you're dealing with massive amounts of data. If you look at complex applications on the web, such as Figma, they're actually doing the exact same thing after the hood. If you artificially slow down your computer and zoom into a complex Figma document, you'll see how it is actually rendering stuff asynchronously and showing you the lower resolution first while it's loading. Let's look at it again. You zoom in and you see how you can get a pixelated version first and then it loads when ready without blocking the main thread or without blocking user interactions. Using a lot of techniques like these is what makes Figma possible and not painfully slow. Unfortunately developing these kinds of tricks is not easy. It involves writing a lot of code that is tailor-made for your data, for your specific use case, so it's difficult to reuse and to encode into a library. We can say that for this reason building applications like those is actually a data problem. You need efficient data structures to represent your data. You need efficient algorithms to get information you need from data structures. You also need indexed representation of data, such as binary trees, so that you can accelerate your algorithms. You also need streaming to be able to dynamically load data from the web. Most importantly, you need to pre-process as much data as possible so that when it's loaded it's ready to be rendered on the client machine without the need for further processing.

We have seen the challenges that building complex 3D graphics applications on browsers takes. I want to show you what's my experience with it by discussing a real-world scenario of a complex 3D graphics application.

I'm working on the rendering engine of a new product called Flux. It's an online, browser-based, collaborative CAD editor. Similar to Figma, but instead focused on electronic engineering and printed circuit boards.

In Flux, we want to be able to handle very complex engineering documents. It needs to render very complicated scenes with lots of components. Each component consists of 2D shapes and 3D objects. We want users to be able to design complex electronic circuits and even display them in 3D. While keeping always a smooth user experience.

Our application is built with React, 3JS and React 3 Fiber. Flux when compared to Figma and Google Maps, we can say that it comes with its own set of challenges. In fact, we want to be able to support very big documents with up to 10,000 electronic components. We want each electronic part to have a lot of different shapes inside of it, like circles, rectangles, text and 3D models. The entire document also can't be viewed at once. We want the user to be able to zoom out and see all of it without performance drops. We also want to provide, as seen before, a 3D view of your document, not only 2D, so tricks like caching tiles cannot work. And we also want real-time speed picking so that mouse interactions are fast and snappy. All this needs to be able to run at 60 frames per second, even on mobile and low-end hardware.

So when you want to overcome challenges like those, it's important to keep a performance methodology. I wanted to share with you what I found are the most important points to follow when scaling up your React 3 Fiber application. And those are the steps that we are following internally to ensure that the application stays fast. We want to try to limit React 3 renders as much as possible. We continuously profile the rendering and interaction times. We try to profile and reduce the memory usage as much as possible. We use efficient algorithms and data structures to represent our data and for some tasks. And last, we use a lot of GPU optimization to exploit the hardware that we have at our disposal as much as possible. The first things are the Reactor Renders. As described in the React 3 Fiber documentation, you want to use transit values, refs, and imperative mutations in your frequently running code, which can be sometimes hacky, but it's the only way to obtain smooth animation without going through Reactor Renders. We went even further than this guide by optimizing our state management solution to use partition store to reduce the cost of subscriptions.

Even if you apply all the React level optimizations though, you still need to optimize your render loop. We use the concept of frame budget. This means that if you want to reach 60 frames per second you have 15 milliseconds to render your entire scene. So it's crucial to go there with the Chrome profiler and see what is really taking up a lot of time in your frame loop and optimize those things first. Sometimes the result of those profiles can be even unexpected. Finding out that you introduced a function call with too much complexity in your critical path.

Another thing that we do is memory profiling as keeping a low memory footprint is crucial when you're handling lots of data. By using the Chrome profiler you can take a snapshot of the memory and see what is taking up too much space. By using this method for example we're able to notice how we're using hundreds of megabytes just to store string-based UIDs. This suggested us to use numbers instead which were able to save up a lot of space and make the load of certain documents possible.

Another thing you need are efficient algorithms and data structures. This really helps for example for optimizing raycasting which is the process of finding out which objects are present in your scene at a given position. This is crucial for mouse interactions as you need to know what's under the cursor pointer. The naive approach to this problem as it's also implemented in ReactryFiber is by doing a linear search which unfortunately can be very expensive when you have thousands of objects. Since this is something that needs to run in real time having faster data structures such as R3 and BinaryTree can really speed up your application a lot.

Another thing you have to be mindful about are GPU optimizations. GPUs are really powerful parallel processors but you need to be mindful of the CPU GPU bottleneck. In fact the communication between the CPU and the GPU is not free. Every time you ask the GPU to do something also called a draw call you get some hoverhead. For this reason you want to keep your draw calls as low as possible especially on low-end mobile devices. This means that if you want to draw for example 10 000 instances of the same object over and over again doing that the naive way, by using a for loop can be very inefficient. You can see that if we want to render 10 000 instances of the same object we will end up with pretty poor performance around 33 FPS. Luckily there is a way to optimize that and it's called instanced rendering. With this technique we can draw several instances of the same object with a single draw call to the GPU reducing the bottleneck by a lot. This can yield great performance improvements in some cases like here for example where we were previously limited at 33 FPS on average. The cool thing about instanced rendering is that it's very customizable. Even if you're limited to have the same geometry material for each instance you can program how the GPU is processing each vertex and pixels and even pass custom parameters for each one. For example you can give each instance a different position or size and program on the GPU how to interpret those parameters. Instanced rendering was crucial in Flux to optimize our text rendering even though it required some special setup. You see text rendering is unfortunately not provided by WebGL and it's something that you have to handle with your own custom solution.

It's also quite complicated since it involves shapes and geometry that can have a lot of triangle and vertices which can become expensive to draw. For Flux we needed a solution that was able to scale up to thousands of characters a screen while keeping each character as a 3D extruded shape. For this reason a simple sprite based 2D solution wouldn't have worked. We needed real 3D geometry.

To optimize our text rendering we needed to use instance rendering. But as we said previously instance rendering only works when you want to repeat the same geometry multiple times and this is obviously not the case if you have different characters. We also said though that we can customize how each instance is displayed by providing custom parameters which can encode stuff like position and size. So we asked ourselves what if we represented the desired cliff as an instancing parameter? If that was possible we could encode our string to display into an instancing parameter and have the GPU figure out how to draw each character by itself.

At this point we also needed to pass to the GPU the data about the geometry of each cliff as sequences of 3D points. We did that by using textures. Normally textures are used for attaching images to the surfaces but since they are after all a big array of numbers we can use them also to encode 3D positions so that the GPU can use them to represent characters like those one. We can use the RGB values to encode our XYZ positions so that we can encode geometry into a texture.

The end result is something like this in the example here. We are able to display 20 000 3D characters at the same time with a single draw call and animate each one independently while keeping a steady frame rate. You may ask, well but how would have this behaved without those optimizations? We tried it and even with a fraction of the characters on the screen it was painfully slow, leaving almost no processing power left for other more interesting stuff. At this point I can already hear someone saying maybe have a lot of text to render too And for all you guys, we just published on NPM our 3D and 2D text rendering engine which provides high performance and efficiency with a React-friendly API. Just wrap your scene with a provider component and use the InstancedText component wherever you like. Thank you for following my talk.