Using React to Build Performant Game UIs in Minecraft

Welcome to Using React to Build Performant Game UIs in Minecraft. My name is Paolo. I'm a tech lead at Mojang Studios here in Stockholm. And I work in a game that you're probably familiar with, which is Minecraft.

I'm part of this fantastic team composed of artists, designers, C++ developers, JavaScript developers, and a producer. And our main goal is to change how UIs are built in Minecraft. And we're doing that not only by changing the tech stack, but also by introducing a design system into the product.

This is some of the components that we have, and you might see these rolling out in the upcoming years. And yes, we are actually already even in production. We rolled out a screen last year, which is the achievement screen. So if you play Minecraft Bedrock Edition in your Xbox or PlayStation, you probably have seen the screen. And this screen is completely based on web standards, and it's built using React. And this is all the platforms that we have support, which is also one of the main challenges for our project. So we need to run on the Xbox One, on the PlayStation, the Switch. We have Android, phones and tablets, iOS, phones and tablets, Windows 10, Mac OS, almost all platforms that exist, Minecraft runs on it. And then we need to support it. And that's not only about device capabilities, but also different input types. So we need to support touch, gamepad, all sorts of things. VR, even, is supported by Minecraft.

But why are we using web technologies to, or like web standards, should I say, to build UIs for Minecraft? There are a lot of reasons. And some of those I did cover on this preview talk that I gave in 2018. So if you're curious, you can also check that out. But if you don't want to take a look at that, here's the too-long-didn't-read. So basically, Minecraft is a completely custom engine game. So it doesn't use something like Unreal or Unity. So the UI is completely built in-house, it's completely custom. And currently that's been a problem for us where when we onboard new developers, it takes a long time for them to get used to the tech and figure out how to use it. And we wanted to move to a solution that will lead us to better maintainability and would make it easy for us to get people onboarded and also improve the iteration speed. So in short, we wanted to make something that was based on open standards, W3C, and want to get the benefit of all the great tooling that the JavaScript ecosystem has.

But how are we doing this? Are we just taking Chromium and embedding it within Minecraft? So that could technically be a possibility, and there are some games that actually do this. What we're doing is instead something else. So there's a company called Korean Labs, and they've been doing solutions based on Chromium for a while, for game UIs. But they, for a while already, they built another solution which they call Gameface, where they took the standards which are open of the web, and then they build just a subset of them into something they call Gameface. So Gameface, it could be considered kind of like a tiny browser I'd say. If you build a UI for Gameface, it will most likely run in Chrome or Firefox, but it might not be the case if you just pick a random UI that you build on Firefox and put it on Gameface, it might not work. For example, it has support for Flexbox, but it doesn't have support for Floats. And it's built with a main goal for targeting game UI, so performance is the main thing, it's one of the main goals. But otherwise taking Gameface out of equation is a fairly standard stack. We use React, we use Webpack, we do unit tests with Jest. So if you're a web developer working in a shop today and you join Mojang, you will probably be productive from day one and you'll be able to contribute with a UI for the game, which is pretty awesome and it's a very different experience from what we have today in the studio.

But before I go and talk about performance, which is the main thing we want to discuss on this presentation, first we need to talk about how state management works and how that's different when we are in a game environment.

So when we look at a traditional web application, we can think about it like this. We have your browser, for example, here I have Firefox running the UI and that has its own JavaScript runtime and then we have, let's say, a server on the internet running Node.js and then we have HTTP requests and response coming back. So it's a very kind of like request-response separation and then we have a copy of the state in the browser.

When you go into a game world, what we usually have is just a single process, right? We have the JavaScript engine still running, which will have our React code and our UI code in there, but we have the C++ game engine right there and that's what we call our backend and that's usually the holder of the state. If you want to understand this, it's better to look at an example and a good example is Minecraft.

So if you look at this screen, which is basically what you see if you're playing the game, this is the HUD. We see basically the information about how much health your character has, the fullness because Minecraft is a survival game, so if you don't eat, you start to starve and you can die. We also see the items that you have access to in your hotbar, so this is items that you can quick switch using the gamepad, and which item you have selected. So here you can see we have the first item selected, which is the sword, but a player can also by interacting with the game, can switch to change the selected item to be the Rotten Flesh and that not only affects the game itself, but also needs to be reflected in the UI. And the player can also take damage, right? Let's say a mob comes and a creeper blows up next to the player, and then we need to take some health damage and here the health has dropped to eight. So we can see here that we have kind of like two categories of data, two groups of data. And we call those facets.

So facets a term that we coined internally in the studio because we needed something that didn't exist within the code base. So we couldn't use model, for example. But this is kind of like what it is. It's just a slice of data. But all like that, like facets conceptually, they're very similar to an observable where available over time. So we subscribe to a facet, like let's say the player stats, and then we keep getting updates about new values as they change, as the player plays the game, right? So the bottom there we can see as the time progresses and the player loses health, we have a hypothetical player health component updating with how much health the player has. Conceptually then what we want is that the state shouldn't live in the JavaScript side, but it rather should live on the C++ side, which is our back end, but it's living right there within the same process. We want kind of like a global store and that global store is the C++ side of the game. And we think about global stores, and we look at the JavaScript ecosystem, there are a couple solutions available. Some of the popular ones these days are Recoil or Jotai and they're very kind of like on this fashion where you have a global store. And we took a very similar approach to these solutions, and we got inspired by Recoil's API to build something that feels very familiar so that anyone that would come to the studio would understand what the concepts were trying to imply. But there are some differences of course that we're going to go through in Next.

RackFacet is actually a pack, a collection of packages. We have a lot of packages. The part that deals with the communication between the JavaScript bits and the C++ bits we call RackFacet Remote. And that's the first part I'm going to talk about now. And I guess the first thing that we need to look at is how do we define the facets, right, the state bits that we have that is shared between the C++ side and the JavaScript side.

And if you looked at... If you used Recall.js, for example, at one point, this might feel a bit familiar. The only thing is that we're using TypeScript here. So the first thing that we start with is defining the API contract, which is just an interface for this data type. This is something that we usually talk with the C++ developers in a line. So we say, for example, that we want to have a health that has a number and we want to have fullness that has a number. Then we have a unique identifier. And this is how we, from the JavaScript side, can request a particular piece of data to the C++ side. Then we have the definition itself, which is very similar to an atom, on a Recall.js lingo. And then we also have selectors, which allow us to pick the full facet and slice just a piece of it, kind of like of a map function. And here we're picking, for example, just the health attribute from the entire place stats facet.

And when we want to consume it, it's also very familiar, you would have just some extra hooks and some different naming here. But we basically need to first use the useRemoteFacet hook to kind of like subscribe to this remote facet, to actually kick off and start listening to the events. Then we turn it into a React state, which is the health there, it's going to be the actual string where the number containing the health, and then we can just render the result as a normal React component. The key difference between our solution and the existing solutions in the open sourcing space is subscription. Our facets, they don't really have any value until we start subscribing to it. And after that, they act pretty much like observables. So in our solution, we do have to have a provider that will wrap the application and this provider, we have the responsibility of implementing the glue from the JavaScript side to the C++ side. So in this case here, we have an engine, which is like a global object that we have access to. And it's kind of like an event emitter, where I need from the C++, from the JavaScript side, I need to set up a listener, which in this case is listening for the event facet-updated player stats. Let's say I want to access the data for the player stats. And then I need to emit an event to notify the C++ side or the game engine that I'm interested in that data. And finally, I can return a function that can do a cleanup. So what happens in practice is that when a component subscribes to, let's say, the player health selector that we showed in the previous example, if it's the first time this function will be called, we'll set up the subscription and we'll start listening to that data. And once that component gets unmounted, the cleanup function gets called and we stop listening to it. And we tell the C++ side to stop notifying us from updates. That's pretty cool. And it works pretty well for us.

So the next step, which is the interesting part for all of you is, of course, how we handle performance. Where do you optimize stuff for a solution so that it can work well in a gaming development world? And the first thing I guess to really make it clear is that React by itself is already plenty fast.

If you're building web applications, it's usually sufficient just to use React as it is. And solutions like Recoil or Jotite, they help kind of like narrow down and follow updates so that only specific parts of your React tree are updated. And when we run into performance issues in React, usually the recommendation is that we take a look at our components and we see if we have any kind of re-rendering that we're doing unnecessarily and see if we can avoid that at all costs.

So when we look at this concept, it's kind of like, I mean, what if instead of just avoid unnecessary renders, we could avoid all re-renders at all? Or could we have a situation where we basically don't trigger the React reconciliation? So here's kind of how it's happening as our previous example, right? The facet gets updated through an event from the C++ side that triggers the, that turns, that of course updates some state internally in React, which triggers the reconciliation and then Reacts through the virtual DOM, figures out, oh, I need to actually update this text node and that's how an update happens. So what we're proposing is that we could just skip the middle man and just go straight from the facet update to the DOM update. So if we go back to the example that we had before, what we're proposing is instead of like taking the facet, which is kind of like this nice data structure that holds a value over time, instead of unwrapping it and turn it into a state, a React state, can we just pick the facet and pass it directly to the React render and let the React render figure out how to unpack the value, unwrap the value from the, from the facet? And that's exactly what we did. So we want to pass the facet, and that directly updates a text node. So let's see how we've done that. So taking the example from before, I just expanded it a bit and turned it a bit more interesting. So here I have now a bit more complex progress bar. So I'm actually rendering some divs to make it like a nice style. You can see at the top there how visually it will look. And then we see the hard-coded result. We have like a hardcode variable there with the progress set to two. So that's a very simple React component, just rendering a progress bar. So if you want to take this example and make it so that it skips reconciliation altogether, how could we do that? So I think the first step of course is to subscribe to the facets in the first step. So we changed that to do the use remote facet, to take the value from the selector, and grab it to be used in a component. But progress here is not actually a number per se, it's actually just the facet still, like it's a data structure that holds the value inside of it. So I'm getting actually a typo error here because I can't make an arithmetic operation with a facet. It's not a number. So we need to then figure out a way now to be like, okay, I have this facet and I want to do some transformation to it and turn it into something else. If this was a recoil in a global state, I would probably just use a selector. Because this is inside a React component, what we need is some way of making a selector that is defined within a React component. And we do support that through a custom hook that we have that's called use facet map, and it's very similar to a selector conceptually. It takes an input facet, you can actually take multiple ones, but in this case, we're only passing one, which is the progress that we subscribed to in the line before. Then we have a map function that will take the progress, which is the number, coming from the facet, and transforming it into the string containing the pixel unit. And then the result of it is also another facet, which is holding now in the width variable. So the next step now, of course, is to take this width and pass it to the div. But if we do that, React is not going to be happy. It's going to be like, hey, I don't know what to do with this data type, right? You're trying to give me something that's not a string, it's not a number.

I don't know how to render this. So the next step in this is, we need to change React Renderer itself to support this data type. And that's what we did, by introducing a new div which we call FastDiv. And we actually created a whole series of components that match, kind of like one-to-one. So we have a FastB, a FastSpan. Instead of accepting just regular numbers, strings for the props, they also accept facets as a prop. And in the end, what we have is basically this, right? A component that is subscribing to a remote facet, turning to like a variable progress that holds a facet of progress. Then we're mapping that into a Width facet that has the width. And then I'm taking that and passing to the FastDiff that actually then updates that prop in the DOM. We have basically created a renderer that natively supports facets, or you could say a lightweight observable, as values for the props. We could also go one step further and make custom components that also natively supports facets as value.

Let's see how that will look like. If we go back again and take a look at this example of the player health, what if we want to also do the fullness? We want to render also the fullness progress bar. Could we do that? I guess the first thing is that we want to extract the components so we can reuse in both places. We want to take this part here that handles the progress bar and extract that to a progress bar component. But we want to pass the progress there as a facet. We don't want to pass as a number. Because if we unwrapped that in the player health to turn it into like a regular React state, we would go back to basically triggering conciliation, and we would defeat all the benefits we were trying to achieve.

So here we have them, basically the player health and the player fullness, two components using the progress bar, and both of them passing a progress, each with their own facet containing their own data. And the progress bar implementation looks like this. Instead of taking a progress that's just a number, it's taking instead a facet of a number. And we have the type specifically available for there. But otherwise, the implementation here in the body is exactly the same as we had before. So this is kind of like what's happening then in the end. We have at the top our player stats remote facet. Once that gets notified of an update from the C++ side, that data then gets passed down to the player health remote selector that then picks just the health from the data. That then is passed to their use remote facet, which kind of like triggers the subscription. Then it goes through the use facet map that transforms that. And then the way that it finally gets down to the facet div, and then the DOM is actually updated. And the cool thing about all of this is that all of this is happening behind the scenes. We're triggering no rewrapped renderings at all. And this whole implementation is built as a custom render, as I said, and it's available in this package, react-facet-dom-fiber.

But this is all fantastic and it's pretty cool, but like how well, how much does it really improve the performance? So comparing, and I put here two different platforms just to give it some perspective. So I'm running these scenarios on both an Xbox One and also on Chrome. So this Xbox One is running the Minecraft game, and we have also Chrome on my MacBook. This example that I'm showing here is basically taking the progress bar implementation that we did before and just changing its value every frame and see how much time I'm taking between each implementation. On the left, we have implementation that's state-based from React and it's taken like 100% of the time. And on the Xbox One, we can see that a Facet based solution takes only 32% of the time to do the same work. When we look at Chrome, the difference is a little smaller, but it's also a lot faster when we look at the Facet's implementation. Another scenario that's very interesting to look at is that imagine if you have like a big list and you're running that through a bunch of components and you're memoizing it. So each component is memoized.

When we look at the solution based on Facets on the Xbox, the difference becomes massive, taking just 12% of the time. And on Chrome, it's also about 30% of the time. This example I brought up because it's quite interesting to look at if we have like a big list, but all the items are being updated at once, here we can see that the Xbox is actually a lot faster when we compare it to the standard React implementation. When I looked at the Chrome on my MacBook, the difference is not that big. And the main thing for this that is worth bringing up is that when we're running V8 in our Chrome, in our Macs, we have just-in-time compilation. But unfortunately in some platforms like the Xbox, PlayStation, and Switch, we cannot have JIT enabled. So optimizations and libraries that we're used to will behave and will have different performance characteristics when we run them. So, and also another important thing is like the more complex the React trees become, then the wider the gap will be and the faster, actually the faster the implementation will be in relation to the standard React implementation. We have more data, of course, available at our GitHub repository. All the examples that I mentioned here, you can go there and see their actual implementations to understand better exactly what they're doing. I don't have much time to go into the details here, but we do have all of that available on the GitHub repository.

But not only that, I mean, if you look at facets, we have global stores, we actually have a bunch of more hooks that turn into like a much more React friendly API. So for all of the React hooks that are on the right here, like use state, useReduce, or useEffect, we have counterpart implementations on the facet world, but instead of like, you know, the difference being that it actually can accept facets as input, and they might return a facet as an output. And this is all available in the react-facet-core package.

Just to go really quickly, here's how like that looks like, where I have this component here, where I'm using the useFacetState, that is very similar to useState, but instead of like value there being just a string, it would be, for example, a facet of a string. Then I have also, useFacetCallback here, that is very similar to useCallback, but instead it allows you to pass value there, which is a facet instead. There's more hooks and documentation available also at our GitHub page.

So in summary, a facet is not only just a library, but it's a pack of libraries. We have the new data type that we introduced, which we call facets, of course, which is a lightweight observable that lives inside the React facet core, and you can use just React facet core to build something without necessarily using the custom renderer or anything like that or the remote package. We have the remote package, which is the mechanism that we developed to have like this global store state to share the state between the JavaScript side and the C++ side, and we have our custom renderer that we built within the DOM fiber. So this is all the packages that we have, and you can take a look at the docs to figure out what they do. I don't have much time on the talk to go through them.

But it seems a bit daunting, but it's actually quite easy to get started if you want to just get your feet wet with facets. We have a package called DOM components. What it allows you to do is it implements this fast components that we did, but instead of being like host components in a custom React renderer, they're actually just regular React components. So you can see here them being used on a regular React DOM to render this component. And all of this is being open sourced. We're open sourcing under the Oryui brand. Oryui is kind of like an initiative that we're doing at Mojang studios to try and standardize UI development in the gaming industry. So the stuff we're doing is trying to push that. So Oryui will have React facets to start, but we'll push other packages that we're developing as well. So this is the fantastic logo. And this is our GitHub repository. So if you want to look faster at the codebase, if you want to learn more, go to GitHub and take a look at Mojang slash Oryui. And of course I have to mention that we are hiring at Mojang. It's quite an exciting project. I mean, it's not everywhere you can go and use your web skills to contribute to one of the biggest games in the world. It's quite exciting to be able to work here. And if you want to do it, we have positions open for tech leads, web developers. Feel free to reach out to me or go straight to jobsmojang.com. So yeah, thank you so much. I had to run a bit through the slides, but yeah, if you have any questions, I think we're going to have a Q and A soon. So thank you so much and feel free to reach out to me as well if you have any questions.