React Native as a Gaming UI System in Unreal Engine

So hi there. This is a talk called React Native as a Gaming UI System in Unreal Engine. We'll be talking a little bit about why this is a good thing, how it's achieved, and when and how you can engage with it. So first of all, a little bit about me. My name is Danny. I've been coding for 24 years. Since about the age of 13, you do the math, 18 of those years was professionally. My journey is very mixed. I've seen the mobile world, the web world, which is where I've acquainted myself very well with React Native. And I've also been participating in some game dev productions, which I think provides me with a great unique point of view of what each industry is doing well.

Right now, I'm an engineering director at Star Atlas. Star Atlas is a game dev studio which is making a triple-A game set in space. Yeah, you should really check it out. Before I introduce my React Native integration for Unreal Engine, I want to explain how we arrive at this idea. So we will go into a little bit of history of why React Native is good for mobile and what parallels we can draw from that to Unreal Engine.

So if we look at the history of mobile apps before around the year 2015, most apps that did not want to go the native route were forced to do sort of one of two things. Either they would make an HTML component-based app. It would just be a regular JavaScript and HTML application. They would then package it into a WebViewRenderer that looks like a native app, but it doesn't behave like one. The other option that people had was to take an imperative UI construction script, package it in JavaScript, create a bridge that constructs UIs imperatively in Kotlin or before Kotlin and Java and Objective-C. And that would be a really poor way of doing things. So come React.js Conf in 2015 and React Native was introduced with kind of a new way of looking at things.

And this new way, we would do declarative UI in the React way. And we would deal, most importantly, we would deal with native components that are fully platform rendered. So the question then becomes, what would an evolution like this look like for Unreal Engine? And to answer that, I want to kind of give you a little bit of an introduction to how game UIs are made today.

So right now, if you're building a game, you're going to build it one of three ways. The first way that you would build a UI would be just to utilize the native UI of the engine that you're building your game in. So if you're lucky and your engine actually has a native UI system, in Unreal 5, this would be Slate and UMG. That's just the name of the native C++ UI toolkits. You'd have Unity with a UI toolkit. You'd have proprietary UI systems like Anvil. You probably played a lot of games that do this.

The second way, which is also very popular, is utilizing some sort of a middleware. So this is usually and commonly done when the proprietary engine of choice for the game doesn't really have a framework. And they want to utilize a framework built by someone else. So these are also native UI frameworks. They are kind of like modules that you put onto the engine. And I've listed some games that you probably played that have this technique.

And I've listed some games that you probably played that have this technique. The last choice, and this is the choice that I feel that the React Native integration evolves, is taking a browser and rendering that browser headlessly onto a texture and then slapping that texture on top of existing UI. This is done in a surprisingly large amount of games. So the way it works is that currently you would have something like a game engine that would render all the regular game engine stuff. And then on the side, you would have a headless browser that does not live in a window. It still executes HTML. It still executes CSS and JavaScript. But it also takes all of that and packages it into a buffer. And that buffer has the texture with all of your UI. And that is always drawn on top of the frame that the engine provided for the game. So just for you to understand kind of like where you've seen it before, probably one of your favorite games does that. And this is before the C programmers in the back row start throwing tomatoes at me for even suggesting putting web browsers in a game. So if you've played FIFA or Football Club, you've seen an HTML UI. What you're looking at is an HTML UI. If you've played PlayStation games like Control, that's an HTML browser rendered UI. If you've played simulation games with complex data like City Skylines, that's also a web HTML UI. And there are many, many, many more. Just off the top of my head, I can name games like Sea of Thieves, PUBG on mobile, SimCity, The Sims by Maxis, and many, many other games.

So the question is why would anybody ever do that? Why would someone take a browser and ship a game with a browser that's rendering UI on top of a game? That sounds like it is very redundant and non-performant. Well, it turns out that first of all, data-heavy user interfaces are very hard to build. Layouting for data-heavy interfaces is very hard to build performantly. And the web way of doing things, the React way of doing things, and by extension the React Native way of doing things is battle-tested by 5.5 billion Internet users and counting. And there's this not insignificant detail that hiring React front-end talent is hiring from a global gigantic pool with millions of people. It's all good and well. Today you could do things in a similar fashion to how things used to be done for mobile apps before React Native came about, but it has some problems. The first one of those problems is redundant systems. What happens is a browser has a text rendering engine, a browser has a vector graphics engine, a browser takes up memory and CPU, and most likely your game engine is going to have the same stuff. And same goes for integration with native engine constructs. If you want to do transparency, opacity, colors, effects like blur, filters, color correction, you're going to be dealing with the browser engine that has zero talk with the game engine. And I was wondering if we can do better.

What would a better way look like? So, the perfect way would be to not have a browser, to still have React Native code or React-like code that is then being fed to some sort of a native runtime just like on mobile that has a native integration with all the native game engine constructs and would render a frame that has the UI on top of it. Before I show my integration, I want to say that you've probably already interacted with something like this. This is not a completely new idea. This is an idea that has been explored, and I think some of you will be surprised to hear that, but if you own a PlayStation, you are playing around with a React Native UI on the PlayStation. So, the whole entire PlayStation UI, the whole entire main PlayStation experience with all the apps is actually a collection of dozens and dozens of React Native apps that are being rendered in 3D. So, to recap, until now, when we wanted to use web technologies in games, we would do it in a way that resembles apps before 2015. We would grab HTML components, render them inside of a web view, and my hope is that after this conference, we can think of it similar to how the conference in 2015 opened up new ways of building things. We could think in terms of native slate components in Unreal Engine that are natively rendered by the platform, but while we are doing that, we can also still write JavaScript and TypeScript code.

How does that work? How would a system like this work? React Native has something called platforms. In-tree platforms and out-of-tree platforms. So, if you've been building any React Native code for iOS and Android, you're familiar with the fact that there is a C++ core that is being compiled to a shared object and then being distributed with bindings to the native platforms. What you might not be as familiar with is that there are out-of-tree platforms like Mac OS and Windows being developed by Microsoft and Vision OS, which I think is being made by Callstack currently. These are also in a surprising amount of places. I've heard that if you currently click on the start menu in Windows, you're interacting with the React Native mini application that powers the start menu. So, my proposition is an Unreal Engine out-of-tree platform. That platform is then packaged as a standard Unreal Engine plugin. This plugin has bindings for all the different React systems. So, React has native core components like a scheduler, surface manager. It has a React host class that helps bootstrap everything. It has interfaces for HTTP, WebSocket, timer, all sorts of native API functionality. All of that is satisfied by this Unreal Engine native React plugin. All of that talks to the game. With this kind of architecture, we can then take the familiar React views, the familiar React text components, image components, and then translate them directly to slate components, such as React views, React paragraphs, React images, which are just child components, child classes of regular native slate components. That sounds great. I have been working on this for quite a while, and until now, everything is easy peasy, but reality is not so fast, not so easy. The first challenge to make all of this work was that React Native currently assumes that you're going to be working with iOS and Android, despite the existence of many out-of-tree platforms. If you're trying to build a React Native codebase for C++, it's going to assume one of two things. Either you're building for Android or you're building for something called Phantom, which is a private meta headless tester that they use to confirm that the core code is working as it should. The solution to this was a lot of ugly CMake lists overrides, and I'm going to use this platform to say that it would be nice if meta made the build system more modular, because I think that out-of-tree platforms are a big part of the future. There are many more platforms that are mature and would benefit from an integration with React Native.

The second challenge is that JavaScript overrides were, or maybe are, still a nightmare. If you are writing code in React Native, you would know that to write code in React Native, you need to import the React Native JavaScript code base on the JavaScript site. That code base would have things like the image component, for example. There would be image components for iOS, overrides for Android, and then all sorts of supporting types and functions. Now, when you're building a custom out-of-tree platform, besides writing your own, for example in our case, Unreal Components, you still need to import and use those binding, general purpose, common data structures and functions. Unfortunately, if the metro bundler is redirecting your game to consume the out-of-tree platform, you need to have those files present there. You cannot just extend the platform with additional files. You have to do something that merges the two together. Of course, the solution is that the build process for the platform has a bunch of really ugly folder merges. This is modeled after the way React Native Windows currently handles this situation, where we take the React Native code that has all the iOS and Android and general purpose stuff, and then we copy it into our own React Native Unreal platform before we bundle and ship the code.

The second challenge could be also solved by meta-making the build system more modular in the future. So, big ask. Enough talk, I want to show a quick demo of how a plugin like this looks in real life. What we have here is we have an Unreal project, and this Unreal project has a very minimal system set up for showing a currently running developer server with Metro shipping a standard JavaScript React Native application for the Unreal platform. So, on the Unreal Native side, we have a widget blueprint, which has currently just a React surface, and this React surface is all that we need to drop in to be able to connect to the React Native dev server. I will show this first without any kind of effects or 3D game world, just to see what we're dealing with. So, this is a simple UI that I've made as a demo. This is a fictional hacking session that we're in. We're trying to get into some sort of a secret laboratory in a game. We need to enter a passkey, and we need to get into the system. This is powered by code that would look very familiar to you if you've ever done anything with React. So, we're looking at the code right now. So, this is a quick and simple dirty one pager. You see all the different pieces that you would expect from a regular React Native application. So, we have an in-memory router with login, multiple authorized locations, and what more is we can go and we can do stuff like this. We can, for example, decide that we want our items, which are the digits, to not be navy. We can make them red, and if we go back to the game and we refresh, we instantly get the change hot reloaded into the engine. Let's make them blue again, because that's much nicer. And let's refresh again. Now, this is a very rudimentary UI.

What is cool when you actually have a UI that is in-engine is that you can now do things to it that a native UI would leverage, which the biggest of all is shaders. So, I prepared this little demo. So, going with the theme of our secret clandestine operation laboratory, we have this kind of little world here where some mysterious stuff is going on. Abandoned healthcare facility with mysterious looking rooms, and what we're going to try to do is we're going to try to hack into it in game, of course, and sort of interact with our little UI. But, as you can see, something is different.

Now, what we've done is we've taken the basic React Native UI and we've applied some cool shaders to it. So, this shader is kind of going with the analog horror theme. So, let's try a few passwords. I wonder what the password to a secret facility would be. Maybe someone's birthday. No. Let's try 1234, because I think people in secret clandestine facilities are, yes, they're using really stupid passwords. So, now we have our UI being rendered in this kind of shaded texture on top of a 3D world that we've created for this. So, we're now watching the security camera footage for this shady facility, trying to understand what's going on. It looks really, really abandoned.

It looks like not much is going on there. I wonder if the logbook will have something useful. All right. So, specimen transfers, holding temperatures, biohazard filtration broken, air purity exceeds baseline by 300%. This looks really, really bad. Let's look at our experiments. We don't have access to that. Right. So, this is just like a tiny little demo, but it shows you how a simple game UI really can be elevated by applying shaders and just putting it in-game. And all of this is powered by React Native. So, we can go on and go back to the login key. And just like before, we could edit things in real-time. I can go back to the editor and, again, change the background color to red and reload it and boom. We have the hologram and this holographic analog UI showing a different color. All right.

So, this is a workflow that you would be very, very familiar, but fully in-engine, fully native, and everything that you've just seen is rendered natively through the actual slate C++ native UI system.

What's next? I think if you've seen this and you think it's cool, you probably want to get your hands on this as soon as possible. I really would like people to get involved. The first thing that I would like you to do is, while this is being polished for sharing with the rest of the world, go to rnue.dev. There will be a form there to tell me how you're planning to use React Native on Unreal Engine.

I'm looking for partners to inform how this should grow and to inform architectural decisions for the first version 1.0 of the system. After some initial testing with some interested projects, the code will be made available on GitHub. So, please go to rnue.dev. Let me know what you're planning to use this for. Become part of the community, and we will release this to the public on GitHub shortly, hopefully within the next few months. That's it. Thank you very much. I'm very much looking forward to the future where we can use React Native to make games in a much better, much more performance-integrated way.