Create Collaborative VR Environment in the Browser with React and GraphQL

This is the end of the video. Thanks for watching. We'll see you in the next video.

Until then, have a great day. And as always, thanks for watching.

Here's my Twitter. Thanks for watching. Have a great day. Bye-bye. Thank you. Have a great day. Bye-bye. Bye-bye. Thank you. Have a great day. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye. Bye-bye.

Bye-bye. Bye-bye. Bye-bye. So, what are our solutions today if we want to still create some kind of VR experience? And I'm using Chrome here, and we'll still be using VR. So, with webVR and webXR, if you don't have webXR enabled in the browser, it defaults to, like, 3D environment, and, like, game-like experiences, and so on. Which is also good, and it's nice. We'll see today.

So, with webVR and webXR, you can actually get your sensor positioning, you can get your headset positioning, and based on that, you can do interactions with the UI. For that, you can use different tools. One is for developing that. One is Aframe, which is an amazing library and huge community that basically wraps around, like, 3GS and gives you lots of capabilities, like physics, particle effects, movement, controllers, breakcasting, you name it. Lots of things. And you can actually create really complicated experiences with that, just by using its so-called NT component system and using JavaScript. Now, if it uses JavaScript, obviously, we can embed this in React, as well, and we'll see today how it's done. Another option is just to fall back to best practices in terms of game engines. So we can use Unity game engine and develop something in C-Sharp, or we can use Unreal Engine and develop in C++, or their so-called feature, feature so-called Blueprints. But we are not here to talk about Unreal Unity, so we are here to talk about both A-Frame and React 360. Mostly about React 360. So, let's continue.

Let's take a look at the A-Frame and let's look at the examples of experience that you can get. I don't have VR enabled here because I'm on Chrome, but I still can navigate the environment by pressing arrow keys and by just looking around this wall of basic shapes, right? So this is a nice experience, but it's just like a really simple one, right? I want to see something more complex. So for more complex, let's get into a directory of different examples. One of such examples is Minecraft-like world, VR blocks, you can go around it, navigate And it's pretty awesome what you can do just with the power of A-Frame and JavaScript, right? You can get all this cool stuff here. So apart from that, let's take a look at something more complex, something more like Beat Saber type of thing, if you played on the Quest or other devices, so we can get something like this also developed in A-Frame. I think it's pretty nice, also music is kind of nice here. You absolutely cannot create something like that in React 360. So with A-Frame you can do really amazing hypnotizing experiences, right? So pretty cool what you can do with that. Now, what A-Frame uses. A-Frame uses coordinate systems. Not just A-Frame.

Basically every 3D environment uses coordinate systems and one of these... This system is basically either left-handed or right-handed. And the idea is if you take your hand and you point the middle finger at you, so the Z axis will be the point at you. So it means that positive X will be to the right side and the negative X to the left side. If you need to position something in front of the viewer, it will be minus Z and so on. So we use right-handed coordinate systems.

So when you see coordinates, keep in mind that we'll have these coordinate systems, both in A-frame, in React 360, and so on. So X direction also changes when applied rotation, so if we rotate something to the right, then all the Xs change, and we need to take this into account.

A-frame uses entity-component system, which means it consists of entities, components, and systems. Entities are containers into which components can be attached. So when you'll see in the code, it looks more like entities are like React components, components are like props, and systems are the global systems like services and stuff. The reason why it's so is because in game engines, entity-component system is a pretty common pattern, and A-frame mimics that system. Unity also uses entity-component system, so it's pretty close.

If we look at the A-frame code, we'll see something like that. Let's take it. This just loaded, so I can navigate around and have these basic shapes. As you can see, I need to just import A-frame, bring a bunch of styles, and then this is the scene component, these are assets component, entity. I call it component but in A-frame, this is called entities, and like, background, fog, and scale, roughness, width segments, all of that are components. It's kind of misleading because in React we're used to that the whole thing is component, right? And these are props, but in A-frame, in documentation, you will see these are entities and these are components. The main point of showing that is that you can easily see that you can just take this and abstract that into React component and start using it. Right? So there is React A-frame library but I'm not sure it's maintained, I think it's not maintained for a while now. So you can just create wrappers yourself and as you can see it's just fairly straightforward here to pass different data to them and so on.

So what about React 360? So React 360 is built for 360 user interfaces, hence the name. The main idea is that you don't create these like fancy experience. You don't walk around. You just look at the curved screens and interact with curved screens in VR or in general in a 3D environment on the web. So the architecture of React 360 is actually based on React native architecture. We have React running in a web worker. When you bootstrap a new project, you will see an index.js. This code doesn't have any access to browser APIs.

It's just running in a web worker. We have a React native bridge that uses MessageQueue to pass all the asynchronous messaging to the browser, to the client.js. Now, in client.js, you need to define native modules, which basically have access to browser APIs. To get started, you run npx React 360 CLI init. You get your demo app and then you get these kind of components. This is client.js, the one that has access to browser APIs. You have the init function, you have a bunch of React instance creation and you pass a bunch of stuff, such as getting default surfaces, setting background to the whole environment, and so on. We'll see more complicated examples here, but this is the thing that you will get once you create a new project.

What you can do now, you can render something on different surfaces. And there are several types of surfaces. The most important ones are the cylinder one and the flat one. Actually, it's supposed to be... This one's supposed to be cylinder, but anyways. Cylinder is a cylinder that surrounds you and rendering something on top of this cylinder will have something curved in front of you, so it will look better in UI. It's the best practice in VR user experience, in VR design, to put flat UIs on curved surfaces so it will look more natural, so to speak. And if you put a flat surface, then it's just a flat surface that you can position however you want, like set an angle, and that's it.

So to interact with the browser, you need to define different native modules. This is how we can define them. We import module from Reactor 16. We just define a class here, and we have the game state and game ID and user ID. And you understand in a bit what is this game state because it relates to the demo I will show in the end of the talk. Then you register these modules and you register them by adding native modules array here and adding a function that will get the context. And you can return a new game module, passing the context back. So you can use the context to pass callbacks around and so on. So this is pretty cool thing that you can create different surfaces, can create the game module to basically share your logic between different surfaces, because they are rendered in different places, basically. So on the React side, you use regular React components and you use different components provided by React 360. For example, you have VR button, and you have different callbacks for that, like on enter, on exit, on click. Now in VR, you have a concept of in general, in 3D environments. You have a concept of raycasting. When you project an invisible ray going into a specific direction, when it hits the target, it will return back the target that it hit.

When dealing with VR controllers or VR headsets in general, there is a raycast projected from the camera or projected from the controller, and it hits the button or hits the target or hits something like that. So in React 360, VR button does that under the hood. If you just look at the button in VR, it depends on the headset or you point with a controller and enter the button space with the reticle of the controller, then it will trigger on enter or trigger on exit if you leave that. It's like on mouse enter or mouse leave, but in VR.

We have our component here, GameTile, and that's about it. It's pretty similar to React Native. I have views. We have different styles we can provide like so for the buttons. There are specific styles for positioning and so on. We can animate using Animated, similar to React Native, and basically use practices we're aware of and using for several years.

So what about the data? How we can bring in the data into our VR environment? And in addition to that, how we can bring it in real-time? We're talking about collaborative VR environments. We need to bring some data. We need to have data updated constantly, and so on.

So for that we'll use GraphQL. GraphQL, as you all probably know, is a query language for the API, and it's like magic that just works. I'm really passionate about GraphQL. I talk a lot about GraphQL, so this talk is no different. We'll talk about that as well.

So the idea with GraphQL is that we can send one request to the server and get all the data that we need, and the shape of the data will be the same as we requested. So it gives us a lot of benefits of having the data when we need and in the shape that we need, and having all the introspection of what is happening on the server.

So this is how GraphQL looks like in sandbox environment, pretty simple one. And under the hood, we're sending the query and getting the data back. The query is sent in stringified format with variables, and we're getting back the data in JSON. Now this is more like a realistic format, how it will look like in production. This is like a Star Wars API. We have different directives, we have variables, we have different types, so GraphQL is all about types. So yeah, GraphQL is amazing, and we will use it for our talk as well.

So what about architecture and especially how about architecting something collaborative, right? So lots of things happening asynchronously. We have different third parties APIs, we have lots of moving parts and obviously, architecture wise, we need to think about more like microservices or architecture.

So having different microservices here and third party APIs and everything combined into one GraphQL facade that will return everything back. And the best practice to my experience is to use something called three-factor-up architecture. And architecture is something similar to that.

You have an app which talks with the real-time GraphQL API. It updates some kind of state in the database or wherever you like and will trigger events. Now, events will trigger a serverless function that will update the state and basically do the processing that it needs. And when state is updated, everything is propagated back to the client using subscriptions.

So, for our example, we will use Hasura, which is an open-source and free engine, gives you auto-generated real-time API, and on top of new and existing PostgreSQL. And it has a bunch of features that I won't go through right now but you can read that after the talk in the slides. There are lots and lots of features. This is just the tip of the iceberg.

To see in action our environment we can go to Hasura console and see something like this. We can have, we have a game with a bunch of moves happening. We can see these moves, like a subscription is running, and we get a bunch of data back. This is a subscription that I am calling. And I always can insert a new move and this will be updated. Now, of what this serves, this backend serves this demo.

So basically, I have a tic-tac-toe game and I have here a bunch of... I can change my background, I can change it from being a grid to being like a Skyrim background. I can have a chat, I can scroll on this chat. The coolest thing here is that we have subscriptions in place, and it's all real-time. For example, if I will click on something here, you see this one was updated. And let me click here again, I got another move updated here. Now in turn, if I will go and I'll stop that, it will execute a move. Let's say we have a row number 0 and column 2. Let's write move here, 0, 2. We insert a move. What will happen? Oh, yeah, it's like a race.

It starts from 0. So we have this one added, or let's add another one, insert move. And you see everything is updated in VR and in real-time. This is pretty cool, how we can do this with React 360 and Hasura together and create this kind of environment.

And also, we can chat here. The code is in GitHub. So you're welcome to look through the code. We will briefly look through that. But you're welcome to look more closely at the code and comment on that, and so on.

So how we add things to VR, how we add things, for example, here in A-Frame, we basically add Apollo, Apollo Provider. We have our HTTP link. We have a WebSocket link. And we wrap everything with Apollo Provider.

Now here, in A-Frame example, it looks pretty ugly, because A-Frame is not built for something like this. It's built more for games and interaction. So I did the other things around. The tic-tac-toe game was a game in React 360. In A-Frame, it's like a blog post list in A-Frame. So maybe it's a bad example. But anyway, you can go into posts here, and it will get the subscription, getting all the data back. And you can see that you can update lots of stuff. There's different endpoints, so you will see different things here. So you're welcome to play around with that after the talk.

How to add stuff to React 360? You add a poll provider. You add our game board. And basically, you execute subscriptions. You execute queries.

Now, before finalizing this talk, let's look briefly through the code here. And here we see our client.js. And client.js will have multiple things.

We'll have cylinder surfaces rendering the whole game board. And we have our VR chat panel rendering on the right side of things and game gathering rendering on the left side where you can change the background. And the whole logic is done through the native modules.

Here we have one native module called GameModule, and it will be shared later on. Not now, but it can be shared between these instance and other instances here. We have, yeah, it can be shared across the whole thing.

We can also look into Index.js. Index.js is registering multiple components. Now you have to register a component, VRChatPanelComponent, to use that or GameGallery. You have to register that, because these are different roots. And you basically, you basically wrap each one of them with a polar wrapper. And you can use subscriptions inside.

So here I can render the bold. And the whole idea with the bold rendering is that I have the matrix that I go through and I basically execute mutations whenever I click. One caveat, the React 360, unfortunately, is not updated to the recent version of React. So unfortunately we cannot use hooks. So we still use like the old method of render props with mutation and you have a componented mount and all of this stuff. I wish the hooks will be there soon, but it doesn't depend on me. So whenever it will be ready, it will be ready. Meanwhile, you can use A-frame easily with hooks and context and all the new React by just dropping A-frame components and so on.

So let's get back here, and before wrapping up, I just want to, first of all, thank you, thanks for React Summit organizer for inviting me, and I want to basically encourage you to get into VR and AR try things, and try to create these amazing demos that were created for A-frame. Create amazing demos for React 360. Experiment. Play around with the code. VR is just starting, it's just getting into the browser. It's still not supported everywhere, but it will be. So get into this stuff, try to learn new things, thanks for having me, and enjoy the conference. Bye-bye. See ya.