WebXR? Virtual Reality and Augmented Reality Natively on Browsers

So, I'm going to show you a bunch of examples of how you can work with WebXR, create your own experiments. So please, take as many pictures as you can, mention me, mention the event. This helps us a lot with the work we've been doing right here.

Okay, who's the first time at C3? C3 conference. Wow, that's amazing. Yeah. Oh, a lot of people. So yeah, I hope you enjoy this. This is my third time in Amsterdam. I was in JS Nation and this is one of my favorite people ever because people are always interacting. So, feel free to publish your questions later or your suggestions too on Slido and it's going to help a lot.

Okay, who has played with virtual reality before? Nice. Who actually coded something to VR? Ooh, that's nice. I saw that they have a prize now, right, for a meta quest. So, probably we're going to get a lot of ideas if you want this. Let me know later what you're going to build. What about augmented reality? Has anyone still played Pokemon Go? Ooh, I see a lot of faces like, oh no, I'm not sure you know this. No, Pokemon Go is fine. It's pretty nice. Okay, I hope you guys are not like this in the future, like people walking around and this is going pretty viral but I don't know about you but I feel very weird like people walking the streets with those glasses, but it's pretty amazing.

Okay, as always, my talks, we're going to have just 20 minutes here. So, all the content I prepared here you can follow along later, create your own experiments, but I want to show you some things so you can keep in mind throughout the whole presentation. So, one of my goals here is for you to motivate you to create your own experiments. This is not just to watch, okay, you can get there, clone the projects, and try out at home. And here I would say let's think outside the box, right, let's try working outside our 9 to 6 PM work, let's try using software as our hobby because this is what the way we are improving our CV, or actually just having fun with JavaScript and having fun with technology in general. And I would say this is something like for me, people say, wow, I don't code after work, but no, we can use it as something fun as well and I'm going to show you a bunch of possibilities.

Okay, so I bought the MetaQuest a few times, not long time ago, and I was like, wow, I want to code with this, I want to try something out, but I haven't seen many web developers talking about this. It was pretty rare, I don't know if you noticed but during conferences it's very rare to see this. However, those experiences, they are there for like 40 years and this is something that I'm going to show you today. Who here has played with the Google Cardboard, who had this before? Well, I had this one and it was pretty terrible, I don't know about you, like I felt a bit dizzy, and in my opinion, well, virtual reality, the graphics are not so good, the experience is not so good, so I wasn't paying so much attention to this.

And when I saw Uncle Zuckenberg playing with this, I was like, come on, we are in 2024, this graphics is like Nintendo Wii, way long before, he's betting a lot on Metaverse, but I felt like, no, this is just hype or something. Although I saw this game, come on, look how beautiful is this, there are a bunch of graphics there, the animations, and it's pretty beautiful. So when I noticed this I was like, wow, there's a world here that I'm not seeing. And I got even crazier when I saw this AR game. So you can have your characters, the monsters going through your furniture, analyze your environment and so on. So I was like, wow, there's a lot of attention here and I can start working with it. So back then I bought the MetaQuest 2. So this MetaQuest 2, I think is obsolete right now, but it's the one I could buy back then, but it works pretty fine. So when I started playing with it, I started asking my friends, well, what should I do? And then I found out this store, which is, I would say, for independent developers, most of the games and applications here, they are open source, so you can start using right away. And here you can also pay the developers as well. And most of the games here, they use C sharp and they use unit to create all those things. I started asking my friends, well, what game should I use or what experiences should I try on, on my virtual reality glasses? They're like, go on Beat Saber. When I saw the price I was like, oh, no, I won't buy this, it's too expensive. And then I was trying to figure out other games and I found this one, Moon Rider. When I found this game, I saw a very interesting button, WebXR. So my mind exploded, wow, WebXR, this means this game or this experience is going to be executed in the browser. And this means that I can run in my browser, I can run on MetaQuest, on mobile phone, in whatever places I want. So how was it? So Moon Rider here, it's a reinterpretation of the game. And for me it was amazing because the graphics and the experiences, I felt as a native application. I couldn't find any lags. So here I'm just showing how the graphics is. It's pretty simple, it works really fine. I'm going to just jump a bit in the future to see it playing. But yeah, you have the same experience, same immersion you were seeing back then. I'm just a bad player here, but it works just fine. So I was like, wow, there's a lot of possibilities here. JavaScript running there, I don't know, let's search more. What about this kind of thing? Before we go on, I would say this kind of technology is dangerous. It's pretty dangerous. So take care when you're developing something, you're creating something, because accidents might happen.

Look again, my god, I punched so hard this. Once I was playing, I punched like a box near me, I punched my wife, so you should be very careful while executing this kind of operation.

Okay, Moon Rider, when I was looking through the project, I saw there A-frame and webXR. So I started asking myself, webXR, A-frame. So webXR is basically AR and VR, we use the same API in the browser to create our projects. So Google released a lot of things on Android so you can use right away. Apple just released some news about this, but it's just the technology that you can integrate with your environment. You can place 3D objects, and it's pretty straightforward. To make everything work pretty standard, you can use A-frame.

Has anyone played with A-frame before? Wow, nice. So this is, I don't know, it's the piece of art, right? Because with the same code you can use on MetaQuest, Apple Vision, in the browser, and you can have different layers of rendering, and they have fallbacks and work just fine. Behind the scenes, they are using 3JS to get those modules, and it's pretty standard too, but it's amazing how they look like. So you can get all these objects placed on your web application and see them working on VR or AR. And this game is basically an experience, here I'm using my phone, so you can see, well, if you are not in the glasses, you can have fallbacks to control it, you can have joysticks, or if you don't want to use joysticks, you can use... Here you can use the cardboard, pretty straightforward. I didn't have to touch any of the things. Everything was coming from the actual library.

So okay, now we understood the potential of the thing, and now it's time for starting coding, right? We should see how the tools are, and how hard it is to code something like this. So first we're going to need an emulator. So I was using this extension, the browser, so it makes it way easier. I would say, in the first applications, I was deploying to my phone or trying to update the thing, and it was taking a long time. So with this, you can save a lot of time. Or, you can use 8th Wall, which is the owner of Pokemon, the Niantic, they built this so you have an emulator there that can send videos, you can see the depth of the environment, and try coding before deploying or updating your code.

Well, for Apple devices, Apple doesn't allow us to use WebXR in the web right now, right? I think they just released the beta one that you can activate with flags, but for making all those experiments, I had to install third party browsers to start using them. And this is how our Hello World works. So I can place components, I can place entities, animations, and with just, I would say, less than 30 lines of code, I have a whole experience. I can see how moving objects work. I can even touch with the environment or the emulation and it's nice. Pretty straightforward, pretty easy. However, I don't know if you guys know, but I'm a back-end developer.

I don't know how I'm going to place objects. If you ask me, try doing something in CSS, I'm in trouble. So I wanted to see how I can do some magic using actual characters or using complex animations on my own applications. So the highlight of this talk for you guys to go further later is, you can download stuff, you can edit, customize as you wish, and just start using on your applications.

So there are other stores here that you can find a lot of assets for free. You can go there, or for the most complex ones, more richer environments, and so on, you can buy or you can pay developers. There's a huge community behind this and you can get everything, not just the modules, but the textures and everything there you can get for free. So once you find something that you liked, in this case a pretty complex object, I'm going to just download it. So I download it, I place it on my file system, and when I go to the web application, it's like an image. I have the source, I just point to the module, and that's it. Everything works. It will work on MetaQuest, and any device you wish, just with a few lines of code. So this is pretty straightforward, and for me it's amazing what we can do with it.

So this is pretty straightforward, and for me it's amazing what we can do with it. Okay, now I'm starting to like it, right? Because I'm a backend developer, when I start coding I like to use like WrapableStrap, I like to use jQuery, so I like this just to start working with something to have fun, and then we think of some product or creating something later. Okay, what about animations? Well, video games are basically animations, right? You are interacting with the user, but you change how your character is moving or dancing. So how hard is it for me, like a backend developer, to start working with it? Same idea. You go to the store, you find an object you like, and then you can upload it to Mixamo. So you can place the animations there, like dancing, you can dance Brazilian capoeira there too, you can put some objects to fall, there are a bunch of free animations that you can go there, or you can even go and edit if you want on Blender, which is the most popular feature here for editing it. But I would say this is the next step. My goal here is to show you can get everything for free, you can start working, you build your MVP, and then you start customizing and improving your software, right? Okay, A-Frame is amazing because it comes with a lot of built-in functions. Like you can see the background, and look how amazing it is to activate my animation. I have just a prop, and it starts working. I just have all my environment working for VR, I can use in whatever device I can use. So here I have, I would say, over a hundred lines of code because I have lights and more complex objects, but it's amazing how easy this is going to be. Like in just a weekend you can build something actually functional, right? Okay, I've been talking a lot about virtual reality, but augmented reality for me is the best thing, right? We can interact with the audience. So I could build an animation, someone running and flying or sitting around. So augmented reality is something that for me is where the powerful thing is for software development, right? So I would say the best use case is Pokemon Go. So if you are on a lake, you're going to see different Pokemons, you're going to see different animations, so they can understand objects and understand how your environment is working. To do so, modern devices, they use the LiDAR scanner. It's basically, they throw a light to the environment and they calculate how long the light took to come back. This is how they can map the obstacles in the environment and check how it's going. However, I was building this whole presentation on an older version of iPhone and I was asking my friends, wow, I was telling them, how amazing is this? iPhone can map all the environment. And then I show my phone. And they were like, you know what? Your phone doesn't have a LiDAR. You have a microphone there. I was like, wow, but how was I able to create this thing then? So what we have there, using WebXR, you can have computer vision behind the scenes. So if you don't have the sensor, you can use older versions to map your environment based on computer vision. So in this case, I'm using the older iPhone to see that I can interact with the environment and so on and so forth.

OK. There are a lot of types of virtual reality. This is something really new for me. So one of the best demos I found on the internet was this one. So you can have your cards. Have you seen Yu-Gi-Oh! Far Beyond Memories? Has anyone played this before? It's nice. So you can have your monsters. You can have animation. It's a world of possibilities for this. We have GPS-based, too. So you can see museums. You can have your friends to reach your house. But this works better outdoors. I would say for indoors, it doesn't work so well because GPS is not so precise. And the last one is the most common, I would say. Like I've seen Coca-Cola or other companies, you just aim your cell phone to a QR code and you can see something moving. So here, the guy just placed a video on top of the object and you can see it moving. Pretty easy, pretty straightforward, right? This is something an application can do in 20 minutes. It's incredible.

All right. So one of my favorites are the heat tests. So this is how you can create like Iron Man experiences. You can move things, you can place it in the back, you can place it in your own environment. And you can start moving on your environment and the objects, they're going to be fixed in those places you want it. So if you have a huge room, you can work with it pretty nicely.

All right. Has anyone seen the announcements from Apple recently? The WWDC, has anyone watched it? Nice. So four days ago, I think, they just said one amazing thing. You can use the Apple Vision to create your own things. You can use WebXR to create applications on Apple Vision.

I know you're thinking, it's too expensive for now. But as a developer, imagine the world of possibilities. This is a new world, right? There's not so many people work on this. And Apple Vision is just new. So if you have ideas, you can use all those technologies I've been showing to you to create your experiences and be like a step forward of everyone else. So using their technology, it's pretty the same idea of other devices. However, the Apple Vision, they have the eye tracking, right? So you're looking at that component and you use your finger to pinch. Behind the scenes, they are going to dispatch the same events. So for us developers, we shouldn't do anything differently. So with the same code, we have different environments, and it's going to be working in all of them with no problem. So this is pretty amazing.

All right. I know it's a lot of information, right? I want to see now how many of you are going to try and experiment with it. Oh, nice. So I gather all the links, all the research I've done for this presentation in just a Google Docs, and I'm going to send to you later. Okay? So basically, the summary of this talk is we are living in the best area for development. We have web APIs, we have audio, we have AI, we have WebXR, and we can start thinking outside of the box because we can build for fun or even create new products with it. Okay? So keep in mind.

And as promised, here is the QR code, so take your time to take a picture or go through the link. And before I finish, as tradition, let's do a selfie. What do you think? Woohoo! Let's do a selfie! Are you still with me? I feel some people are sleepy! All right. I'm going to count on three, and we just say, like the other one, right? Yeah, yeah, yeah. So one, two, three! Ah! Thank you so much, C3! Is there optimism about iOS support of WebXR in production? I believe it is based on your statement about it's hidden behind the flags. Oh yeah. What do you think? I mean, Apple just said it. I would say when they released, I was trying to find the resources they were using, but it's better. So I would say we have some company pushing forward now. I would say we're going to get more improvements in the browser because someone is paying hard now, right? But before, I would say everything was experimental. But now, with Apple, with this huge announcement, and they are targeting the company through the glasses, I was like, wow, this might be good. So I'm very optimistic about this.

Cool, cool, cool. But we don't know, right? And let me take the next one. And by the way, one of the ideas I had was to scare my mom. Imagine there are a lot of horror movies. I remember it was like a girl, she goes through the, like you can see her on the screen and she just jumps on your face, right? So this kind of experience might work really good, right? You just point the cell phone to the TV and then, wow, jump on her face. Might be fun. Lots of questions came just recently.

OK, let's take this one. Is there a separate marketplace for these kind of applications? Yes, you can use either the Facebook MetaQuest store, but you have all the licensing and other things. But I was showing the SideQuest, which for me is the best library ever. So you can have free games, free applications. They have even applications for testing. Like I want to test application examples. How this environment is going to work. So it's beautiful and most of the code is on GitHub. So it's incredible how this community is there. But I've never seen them during the conferences, right? So something that if you guys know, guys and girls know, anyone working on this kind of stuff, invite them to these events because we can exchange a lot of content, right? It's amazing. That's a cool idea. Maybe a very good idea for GitNation to have a conference on that topic. OK, done.

So let's take this one. How much physics knowledge is needed to understand settings and the X, Y, Z coordinates? Mostly of the applications I saw, we used libraries. So we won't have to deal with math. I mean, depending on how customized or how complex is your experience, you might touch there. But A-Frame, there are a lot of plugins that you just drag and drop, start using right away. If you ask me, I'm terrible with math also, so I wouldn't be able to do anything there. Awesome. Awesome. Let's take this one. How well does WebXR abstract over platform conventions? This is amazing because some cell phones that I was showing you, some of them, they don't have LiDAR and some of them doesn't have so much resources as the modern devices now.

So they have a layer for fallback. So if your cell phone doesn't support WebGPU, they're going to use CPU. If you cannot use WebGL, they are going to use other stuff. So it's like, I would say, when you're playing on PCs and you have a PC with lower resources, the graphics are a bit uglier, I would say. It's not so rich, but they work. So with a single code, you can make it work everywhere.

3JS is the key thing behind the scenes. Yeah, that's super great to hear that code once works everywhere. I love it. Kind of. And fallbacks and fallbacks for fallbacks. That's super. It's more actually a request than question. Can you build a course on WebAR and WebXR on Eric Randall training? So homework for you. My God, I had so much ideas. Or conferences. We could have someone just going around with a 360 camera and people who's on the screen now could interact with each other. Like there was a, I don't remember, it's like an RPG game that people gather around. Gather.io. Yeah. So imagine a Gather.io for AR or VR, and it's a lot of ideas. Way better than the metaverse, right? And that's your answer.

It's a great segue to our last question. Let me put it on screen. Yeah, actually, I have a different one. And it's a very short one. What do you think about the best ever use case for WebXR? I would say the best ever experience is you are a web developer, you know web APIs, you know browsers, so you can get all your experiences and start using right there. So for me, you don't feel like you're in a different environment. You have exactly the same workflow you are doing for the web but you can use there. So for developer experience, I think it's one of the best things.

Other, I think, is the community. So the community is pretty extensive. They are helping a lot. And for us, who's trying to make money on this, I think the best scenario is there are no, there's not so many people doing this so we can start building something now. So in a few years we can start being the reference on the comment. Yeah.

Start earlier. Exactly. Let's finalize with this statement. Start early. Build something today. Thank you very much, Erik. It was great, great, great to have you. Thank you so much. Thank you.