Building 3D Web Apps in 2025 – React, XR & AI?

Hey everyone, I'm Bela, open source maintainer at Pomondros and founder of Drawcall.ai. Today we're going to take a look at how to build 3D web apps in 2025, especially with how we can use our React knowledge to build XR apps and how AI can help us in the process. As I said, I'm Bela Bolender, I maintain the library React to XR and I'm also the maintainer of React UI Kit, the most popular UI framework for virtual reality and augmented reality applications on the web. I've been doing 3D and XR web development now for over five years, building various web pages, 3D web games, 3D web apps, and now majorly maintaining open source libraries for the 3D web. 3D on the web has arrived.

What started as a gimmicky usage of 3D on the web has now become a key differentiator. Today, you'll find stunning web pages, full-blown 3D productivity tools, and even fully immersive VR games, all running in a browser and often powered by React. In this talk, we look at the abstraction layers commonly used when building 3D web apps on the web, what role React can play in building 3D web apps and why it's useful, and finally, what role AI, specifically large language models, can play in developing for the 3D web and where LLMs are still struggling.

Let's start by taking a look at the foundation of 3D web development, the lowest abstraction layer, and how different abstraction layers simplify the development. The first layer, the WebGL browser API, exposes access to instruct the GPU to draw individual triangles, which is very powerful but also incredibly verbose because you need to communicate with the GPU in buffers and shader languages. On top of the WebGL API, 3GS, which is the most popular 3D web development library, builds an abstraction layer that is made up of a scene containing elements such as camera, lights, meshes, and materials. We'll take a closer look at those elements later.

And 3GS has made 3D web development far more accessible, and it's the foundation that most higher-level 3D web engines are built upon today. Now instead of imperatively building the scene by hand, we can use the declarative style of React to orchestrate our scene. This is possible using the React 3 fiber library, and because the scene is a tree structure, just like HTML use interfaces, React turned out to be a great abstraction for declaring 3D elements in a scene.

Let's take a closer look at React 3 fiber. It is a React renderer for 3GS, just like React DOM is a React renderer for HTML. And therefore, React 3 fiber manages 3GS elements instead of HTML elements. Now people might ask, why is React a good idea for building 3D web apps? Don't 3D web apps require high performance? The reason React is great for 3D web apps is because it simplifies creating complex 3D scenes. Using the React model, we can co-locate our code, which reduces the likelihood of not disposing resources, and therefore memory leaks. Using the declarative syntax is far less verbose, so we can express our ideas in less code.

And we can leverage the whole React ecosystem and 3GS ecosystem combined, which lets us tap into a lot of libraries, allowing us to access common things like state management, but also key 3D functionality. The code at the bottom gives us an idea on how we can use the React syntax to create 3GS elements, such as groups and meshes. If you are building with React 3 fiber, you get access to the whole React 3 ecosystem to kickstart your development process. This is very similar to how a game engine provides you with elements such as physics and the user interface system out of the box.

The key difference here is that this is not a monolith, but a text stack, like we know it from the web. Every part is interchangeable, fostering innovation. But in some cases, the functionality you need is missing. And the question comes, how can we use existing 3GS libraries that already solve our problem with React 3 fiber?

Since React 3 fiber is a React runner, it effortlessly integrates with any 3GS library, allowing you to access the whole 3GS ecosystem, such as the React 3 VRM library for rendering VRM avatars, the Nufcat library for pathending for your AI agents, or the 3Mesh BVH library, which is super useful for speeding up custom mesh computations. Now the question often arises when building 3G web apps, shouldn't we build a native app for this, since browsers are always behind native apps in performance and hardware access?

However, this is changing, since the gap is closing in key areas thanks to two major APIs. WebGPU, the next generation graphics API that brings near-native performance and modern GPU features, and WebXR, allowing developers to build web apps for AR and VR devices, including hand tracking, spatial anchors, and more. These APIs allow us to build apps that feel native, with nothing but JavaScript, and therefore React. This is where React 3 XR comes into play.

React 3 XR is part of the React 3 ecosystem, allowing us to build web apps using the WebXR standard. Therefore, these apps work across almost all AR and VR headsets. This means we can build apps that are truly cross-platform, from smartwatches, mobile phones, desktops, to our immersive XR headsets, all using our declarative React syntax and without worrying about individual device compatibility.

In practice, this even means we can turn our existing 3D app into a VR web app with three lines of code, as seen in this ragdoll example. We can give our application some advanced features, including spatial understanding of the user's environment. When building an augmented reality application, some devices expose the position and size of certain objects, such as chairs, couches, windows, walls.

Also, using the concept of layers, we can render high-quality 2D media, such as videos, efficiently to a virtual screen in the user's environment, enabling us to build high-quality cinema experiences or high-quality user interfaces. In some cases, a web application might want to anchor something to the user's environment that should persist even when closing the application. With all these features out of the way, let's actually start simple by building our first React 3 XR app using the npm create react3 command.

This starts up a CLI where we can select the 3, handle, XR, and UIKit package from the React 3 ecosystem, which we'll need for this sample application. Let's now take a look at the project's source code. Here we can see the canvas component from React 3 Fiber and the XR component from React 3 XR wrapping our application. Also, we see an environment component from React 3 Drei and a standard ambient light from 3JS and two boxes, which are our custom components displaying clickable rotating boxes.

This CLI automatically starts our project, which we can access either through our headset or the XR emulator right in our desktop browser. Let's now take a look at the project's source code. Here we can see the canvas component from React 3 Fiber and the XR component from React 3 XR wrapping our application. Also, we see an environment component from React 3 Drei and a standard ambient light from 3JS and two boxes, which are our custom components displaying clickable rotating boxes.

The key component in this setup is the canvas component. It creates a scene, automatically resizes the canvas to its parent, sets up a camera and forwards events from the canvas into the 3D scene, making our application immediately interactive using events such as onClick. Next, to express what elements are rendered to the screen, we need to introduce the concept of a mesh. A mesh combines both the geometry, also called its shape, and the material of an object. Combined with the position, rotation and scale, they represent exactly where and how the mesh is displayed to the user from the user's camera.

Now when it comes to adding state interaction and building custom components, we can leverage all the knowledge from React with local state, including global state solutions such as Zastand and interactions such as onClick and all other pointer events working across all devices. The big difference for developing 3D apps is the frame loop. It exists because apps often change every frame, which includes animations, physics, user input and more. The frame loop needs to make sure all the user input state changes and custom update functions are applied before rendering a new frame to the screen. This is crucial for fluent user experience and the frame loop needs to run typically at around 30 or up to 100 times per second. To maintain this speed, we cannot re-render our whole React app every frame. Instead, we need to use the useFrame hook to run a function on every frame.

As seen in this example, we imperatively modify the rotation of the cube every frame. This imperative way of updating a value is very atypical for React development, but necessary to maintain high performance. The rule of thumb is that once something needs to be regularly updated multiple times per frame, we should move it into a useFrame hook and update it imperatively.

Now let's talk about AI, because it's also changing how we build for the 3D web. Specifically, AI changes how we can code 3D web apps, but also how we can generate assets, such as 3D models, textures, sounds, and animations. AI approaches to, for instance, 3D model generation have come a long way and are great for prototyping and especially useful for new developers who are not yet familiar with the sometimes very complex tooling landscape in the 3D world.

Developing AI, specifically large language models, for the development of 3D web apps is challenging because of three reasons. First, most large language models haven't seen enough 3D specific examples. They can create basic scenes, but once you ask for advanced features or ecosystem-specific APIs, they start hallucinating. Secondly, the React 3 and WebXR landscape move quickly. New libraries, new APIs, breaking changes, so models get outdated fast. And lastly, LLMs don't think well in 3D yet. They can describe but not visualize geometry or spatial relationships accurately.

Let's use AI to convert our boring startup project into a sneaker configurator. First, we use Trellis, a 3D model generation AI, to convert an image of a white sneaker into a 3D model. Next, we remove the rotating boxes from our app and add our sneaker model using the glTF component from the React 3.3 library. And we use the handle component from the React 3.0 handle library to make the model grabbable. Then, we use the DrawCode.ai app to generate a 3D user interface for our sneaker configurator, which generates code using the React 3.0 UIKit library, which we can directly copy into our app.

With all these capabilities available to 3D web developers, I believe that 3D web can become or might already be the new creative frontier. As traditional front-end development is more and more automated by AI, building 3D web apps is still an area where engineers and designers have the edge to express their creativity. If you want to get started or continue with your 3D web development journey, I can recommend to take a look at the documentation of React 3 Fiber and the whole React 3 ecosystem, the 3GS journey from Enno Simon, the React 3 Fiber course from Baba Sensei, the Fancy Learning Pages course from Paul Hentschel, and I can also recommend to join the WebXR Discord, the Pomondos Discord, of course, and also the Web Game Dev Discord.