HTML in Canvas: Bridging UI and GPU on the Web

Hi, everyone. I'm Santiago Colombato. I work as a front-end developer at Basement Studio. We build sites and experiences with companies like Brazil, Daylight, MrBeast, and a lot of what we ship uses WebGL.

Today, I want to talk about a gap that we keep running into, and that gap is making the DOM and WebGL move as one. When you build a visually heavy website, you usually have two phases to manage. One side, it's the DOM. It gives you layout, text, scroll, accessibility, and the other is WebGL, shaders, GPU rendering, cool effects, basically. The challenge starts when both need to stay in sync, and the hard part is making them move as one.

So, I'm going to focus on one specific problem today, and that is scroll sync. But first, most of you write JavaScript, but not everyone writes WebGL, so let me take a minute to cover the basics so we can understand the full context. When you work on a normal web page, you write HTML and CSS, the browser calculates layout, paint text, and elements, and everything works. You don't think too much about the GPU.

WebGL changes that. WebGL is a JavaScript API that lets you render graphics inside a canvas element. You work directly with the GPU. But the trade-off here is that you leave the normal rendering pipeline of the browser. Inside the canvas, you don't have HTML. You don't have CSS. So, you're just drawing pixels with little programs that run on the GPU. Those programs are called shaders. We have two types of shaders. The first one is the vertex shader. Every shape on canvas is made of triangles, and triangles are made of points or vertices. So, as you can see on the slide, this code snippet is a basic vertex shader.

That program runs once per vertex, and the job is simple. Take a 3D position, output a screen coordinate. If you want to move, scale, or transform something on WebGL, you will be working with the vertex shader. The second one is the fragment shader. Once the GPU knows where the triangles are, it fills them in. So, for every pixel inside a triangle, the fragment shader decides what color to write. Think of visual effects, lights, textures. All of that is the fragment shader, answering the same question millions of times per frame. What color should this pixel be?

So, basically, the vertex shader decides where, and the fragment shader decides what it looks like. If you look at visually heavy websites like this one, they will use this hybrid model for sure. The DOM is still there, but the canvas is doing a lot of the heavy visual work. The results can look great, but there's a cost. The browser understands the DOM, but your WebGL scene is like a black box. So, we need to make them agree. And scroll is where things start to break. When you scroll a page, the browser updates the visual scroll position immediately. But your WebGL layer updates from JavaScript, inside a frame loop, from a sample value after the scroll. So, you have two systems that are close in time, but they are not the same.

The browser moves the page, and the canvas reacts a frame later. Usually, you barely notice this, but we don't want that. So, let me show you a real case. We have a demo here with a WebGL scene that draws this leader effect on top of images. And if I scroll, you will see that the WebGL layer drifts. So, we need to fix this. And the first thing we can do is fix the canvas, and pass the top offset to match the scroll on every frame. So, basically, you're making the canvas position absolute, and you are updating the top offset to match the scroll. This removes the drift. But please check this, the edges of the canvas. They are clipping. So, we are getting close.

And to fix the previous clipping, we can add overscan. Overscan is basically padding on every edge, extra pixels that act as a buffer. The delay is still there, but now it has somewhere to hide. And the last approach goes one level deeper. I'm going to demo first, and then I explain it. But basically, you will see that now the DOM and the WebGL scene are logged in. If you check the canvas, there is no clipping, and we have the WebGL layer logged with the DOM. So, let me explain that. Basically, what you do is read the current scroll position, and you pass it into the Vertex Shader, and you apply that offset to the mesh position. Because remember, as we said, the Vertex Shader decides where things go. So, we need to update the Vertex Shader. The GPU reads the scroll Y at draw time, and the element is positioning at rendering time. And as we saw, the DOM and the canvas fill up together. No visual drift, no clipping. But look at what we built to get here. Overscan, scroll pipelines, we have to create a common Vertex Shader for each component of the page. So, we also have another feature. We have resize, zoom, focus, text. Every feature that the browsers give you for free in the DOM, you either rebuild it or you let it go. So, if only we could have HTML elements in a WebGL canvas, that would be perfect. So, we will get into that.

But first, Tomas, my partner, is going to show you what happens when you try to render text on the GPU. Thank you. Hi everyone, I'm Tomas Herreras, I'm a front-end developer at Basement Studio. My job lives somewhere between the DOM and a canvas, because we build sites that most of the time lean on WebGL and shaders. And once you spend enough time in that work, you start to notice that everything that lives inside a canvas is essentially a black box. For crawlers, for screen readers, and for robots. An input field, for example, in WebGL, it's a mesh in 3D space with a shader material pretending to be one. But that isn't trackable and isn't accessible, and we are losing every single benefit that native HTML gave for us for free. So, for me, one of the biggest pain points are typographies.

Because today, rendering fonts in WebGL usually involves a pipeline similar to this. Basically, you pick a font, a TTF or an OTF font format, and you have to define the charset that you want to support. Then you have to bake the MSDF Atlas. An MSDF Atlas is basically a png plus a JSON file with glimpses and metrics. Then you have to load that Atlas and build them. And finally, you have to write a custom MSDF fragment shader. Then if you want DPR, text wrapping, or text overflow, you have to handle it by yourself. So the result is roughly 150 lines of rendering logic plus 30 lines of GLSL. And the font distillers know nothing about CSS syntax or accessibility. If you want bold text, you have to rebake the font. If you want them to add an italic, you have to rebake it again, and so on. You have to rebake every time you have any change in that previously defined charset.

And even then, we are still losing text selections. We are still losing finding page, screen reader, CSS syntax for decoration, proper line breaking, among others. But how do we solve this today? Because this pipeline is getting really complex. And to be fair, there are some ways to work around these issues. We have workarounds, we have abstractions, we have libraries, which are really good, and they produce a great result. We actually use them at Basement Studio website in the arcade page. But the closer you try to get to a native browser behavior, the complex it gets. You start recreating selection systems, cursor behaviors, text wrapping, accessibility, all of that stuff by yourself. And then you're basically rebuilding the browser inside WebGL.

So, these tricky stuff and complex things are starting to change. And it's started to change with something called HTML in Canvas. HTML in Canvas is a proposal that allows developers to render HTML elements, real buttons, real forms, real complex text directly inside the canvas. Currently, it's on experimental phase, so you have to enable a flag inside Chrome. But it works around four pillars. The first one is the layout subtree. It's an attribute. You put it onto the canvas and it allows children inside it. Basically, you're telling the browser, hey, don't display these elements like you should, but keep them alive and ready so I can draw them whenever I need.

The second one is the draw element image. This is a rendering function. So, you take real HTML elements that exist inside the canvas, and you draw it wherever you want inside the rendering context. So, these are real elements, keep your shaded. So, it's really cool.

Then, the third one is the paint event, a new event that fires whenever the underlying HTML changes. For example, if you have an input and the user starts to type inside that input, that event triggers, and you know exactly how to redraw it and keep the things in sync.

And the last one is the capture element image. Basically, it takes us a snapshot of an element, so you can transfer it to a worker. This is useful if you're working with off-screen canvases. These are four pieces, one attribute, two methods, and one event. That's the whole proposal. So, you're probably thinking right now, okay, so, what this proposal unlocks for developers? And basically, once you stack all the four pieces together, the things suddenly become possible, and you have unlimited possibilities, and that's pretty wild. Basically, you can have complex text, wrapping, interactivity, accessibility, you have visual effects, you can apply WebGL, shaders, post-processing, or whatever, to actual DOM elements like a div or a form. And then you can shade it with any shader, any post-processing that you like. We have an example with a CTR effect that is pretty good. So, then we can show you that.

And for scroll sync, talking about the scroll sync, the problem is that Santi spends half of the talk just disappearing. Basically, the browser draws all the DOM elements as a child of the canvas. You wrap the whole website with a canvas, so there's no more lag and hacky stuff. Right now, it's the first time that we have the GPU and the document model as just one. There's no bridges, no copy, no weird stuff on workarounds. It just syncs the HTML and the WebGL stuff.

The thing you draw inside the browser already understands that it's inside a canvas or anything like that. So basically, right now, we have two examples that are related to WebXR. Since I'm a developer of WebXR, I actually spent a large amount of time doing it. But what you're seeing here are two demos. These are real HTML in a 3D space. The first one is I played a lot of Fallout games, so I decided to make a BitBoy inside WebXR. This is really complex with the current solutions because of the complex layout, but here it's just using HTML and CSS, so it's really simple to make it.

And then the second one is another experiment that we made in WebXR. We are actually using our hands to destroy a physics-based world. The world is containing the JS Nation speaker section. This is all WebGL and HTML, but there are no workarounds. It's the plain DOM inside a canvas. This is the part that gets me the most excited, because there was always a trade-off, or used to have a trade-off, of do you want it to look beautiful or do you want to make it accessible and crawlable. Right now, with HTML in canvas, it's the first time that we don't really have to choose.