HTML in Canvas: Bridging UI and GPU on the Web

Hi, everyone. I'm Santiago Colombato. I work as a front-end developer at Basement Studio. We build sites and experiences with companies like Bracel, Daylight, MrBeast, and a lot of what we ship uses WebGL.

Today, I want to talk about a gap that we keep running into, and that gap is making the DOM and WebGL move as one. When you build a visually heavy website, you usually have two phases to manage. One side, it's the DOM. It gives you layout, text, scroll, accessibility. And the other is WebGL. Shaders, GPU rendering, cool effects, basically. The challenge starts when both need to stay in sync, and the hard part is making them move as one.

So I'm going to focus on one specific problem today, and that is scroll sync. But first, most of you write JavaScript, but not everyone writes WebGL. So let me take a minute to cover the basics so we can understand the full context. When you work on a normal web page, you write HTML and CSS. The browser calculates layout, paint text, and elements, and everything works. You don't think too much about the GPU. WebGL changes that. WebGL is a JavaScript API that lets you render graphics inside a canvas element. You are directly with the GPU. But the tradeoff here is that you leave the normal rendering pipeline of the browser.

Inside the canvas, you don't have HTML, you don't have CSS. So you are just drawing pixels with little programs that run on the GPU. Those programs are called shaders.

We have two types of shaders. The first one is the vertex shader. Every shape on the canvas is made of triangles, and triangles are made of points or vertices. So as you can see on the slide, this code snippet represents a basic vertex shader. That program runs once per vertex, with the task of taking a 3D position and outputting a screen coordinate.

The second type is the fragment shader. Once the GPU identifies where the triangles are, it fills them in. For every pixel inside a triangle, the fragment shader determines the color to write, addressing visual effects, lights, and textures. The vertex shader defines the position, while the fragment shader determines the appearance. Visually heavy websites often employ this hybrid approach where the canvas handles significant visual work alongside the DOM.

When scrolling, discrepancies arise. The browser updates the visual scroll position instantly, while the WebGL layer updates in a frame loop after scroll from a sampled value. This delay causes misalignment, where the browser moves the page, and the canvas reacts slightly later. To address this issue, adjustments are needed to synchronize the scroll between the DOM and WebGL layers for seamless movement.

The results can look great, but there's a cost. The browser understands the DOM, but your WebGL scene is like a black box, so we need to make them agree. And scroll is where things start to break. When you scroll a page, the browser updates the visual scroll position immediately. But your WebGL layer updates from JavaScript inside a frame loop from a sample value after scroll. So you have two systems that are close in time, but they are not the same.

The browser moves the page, and the canvas reacts a frame later. Usually you barely notice this, but we don't want that. So let me show you a real case. We have a demo here with a WebGL scene that draws this effect on top of images. And if I scroll, you will see that the WebGL layer drifts. So we need to fix this. And the first thing we can do is fix the canvas and pass the top offset to match the scroll on every frame. So basically you're making the canvas position absolute, and you are updating the top offset to match the scroll.

This removes the drift, but please check this, the edges of the canvas. They are clipping. So we are getting close. And to fix the previous clipping, we can add overscan. Overscan is basically padding on every edge, extra pixels that act as a buffer. The delay is still there, but now it has somewhere to hide. And the last approach goes one level deeper. I'm going to demo first, and then I explain it. But basically, you will see that now the DOM and the WebGL scene are logged in. If you check the canvas, there is no clipping, and we have the WebGL layer logged with the DOM. So let me explain that. Basically, what you do is read the current scroll position and you pass it into the vertex shader, and you apply that offset to the mesh position. Because remember, as we said, the vertex shader decides where things go. So we need to update the vertex shader. The GPU reads the scroll Y at draw time, and the element is positioning at rendering time. And as we saw, the DOM and the canvas fill up together.

No visual drift, no clipping. But look at what we built to get here. Overscan, scroll pipelines, we have to create a common vertex shader for each component of the page. So we also have another feature. We have resize, zoom, focus, text. Every feature that the browsers give you for free in the DOM, you either rebuild it or you let go. So if only we could have HTML elements in a WebGL canvas, that would be perfect.

So we will get into that. But first, Tomas, my partner, is going to show you what happens when you try to render text on the GPU. Thank you. Hi, everyone. I'm Tomas Ferreras. I'm a front-end developer at Basement Studio. My job lives somewhere between the DOM and a canvas because we build sites that most of the time lean on WebGL and shaders. And once you spend enough time in that world, you start to notice that everything that lives inside a canvas is essentially a black box for crawlers, for screen readers, and for robots.

An input field, for example, in WebGL, it's a mesh in 3D space with a shader material pretending to be one. But that isn't trackable and isn't accessible. And we are losing every single benefit that native HTML gave for us for free. So for me, one of the biggest pain points are typographies, because today rendering fonts in WebGL usually involves a pipeline similar to this. Basically, you pick a font, a TTF or an OTF font format, and you have to define the char set that you want to support. Then you have to bake the MSDF atlas. An MSDF atlas is basically a PNG plus a JSON file with glimpses and metrics.

Then you have to load that atlas and build them. And finally, you have to write a custom MSDF fragment shader. Then if you want DPR, text wrapping, or text overflow, you have to handle it by yourself. So the result is roughly 150 lines of rendering logic plus 30 lines of GLSL. And the font knows nothing about CSS syntax or accessibility. If you want bold text, you have to rebake the font. If you want them to add an italic, you have to rebake it again. And so on. You have to rebake every time you have any change in that previously defined char set. And even then, we are still losing text selections. We are still losing finding page, screen reader, CSS syntax for decoration, proper line breaking, among others.

But how we solve this today because this pipeline is getting really complex. And to be fair, there are some ways to work around these issues. We have workarounds, we have abstractions, we have libraries, which are really good, and they produce a great result. We actually use them at Basement Studio website in the arcade page. But the closer you try to get to a native browser behavior, the complex it gets. You start recreating selection systems, cursor behaviors, text wrapping, accessibility, all of that stuff by yourself. And then you're basically rebuilding the browser inside WebGL. So this stuff, this tricky stuff and complex things, are starting to change. And it's started to change with something called HTML in Canvas.

HTML in Canvas is a proposal that allows developers to render HTML elements, real buttons, real forms, real complex text directly inside the canvas. Currently it's on experimental phase, so you have to enable a flag inside Chrome, but it works around four pillars. The first one is the layout sub-tree. It's an attribute. You put it onto the canvas and it allows children inside it. Basically, you're telling the browser, hey, don't display these elements like you should, but keep them alive and ready so I can draw them whenever I need. The second one is the draw element image. This is a rendering function, so you take real HTML elements that exist inside the canvas and you draw it wherever you want inside the rendering context. So these are real elements, keep your shaded. So it's really cool.

Then the third one is the paint event, a new event that fires whenever the underlying HTML changes. For example, if you have an input and the user starts to type inside that input, that event triggers and you know exactly how to redraw it and keep the things in sync. And the last one is the capture element image. Basically it takes a snapshot of an element so you can transfer it to a worker. This is useful if you're working with off-green canvases. These are four pieces, one attribute, two methods, and one event. That's the whole proposal.

So you're probably thinking right now, okay, so what this proposal unlocks for developers? Basically once you stack all the four pieces together, the things suddenly become possible and you have unlimited possibilities and that's pretty wide. Basically you can have complex text, wrapping, interactivity, accessibility, you have visual effects, you can apply WebGL shaders, post-processing, or whatever to actual DOM elements like a div or a form. And then you can shade it with any shader, any post-processing that you like. We have an example with a CTR effect that is pretty good. So then we can show you that. And for a scrollsync, talking about the scrollsync, the problem that Santi spent half of the talk, just disappears.

Basically the browser draws all the DOM elements as a child of the canvas. You wrap the whole website with a canvas so there's no more lag and hacky stuff. Right now, it's the first time that we have the GPU and the document model as just one. There's no bridges, no copy, no things, no weird stuff and workarounds. It just syncs the HTML and the WebGL stuff. The things you draw inside the browser already understand that it's inside a canvas or anything like that. So basically, right now, we have two examples that are related to WebXR. Since I'm a developer of WebXR, I actually spent a large amount of time doing it. But what you are seeing here are two demos. These are real HTML in a 3D space. The first one is I played a lot of Fallout games. So I decided to make a BitBoy inside WebXR. This is really complex with the current solutions because of the complex layout. But here, it's just using HTML and CSS. So it's really simple to make it. And then the second one is another experiment that we made in WebXR. We are actually using our hands to destroy a physics-based world.

The world is containing the JS Nation speaker section. This is all WebGL and HTML, but there are no workarounds. It's the plain DOM inside a canvas. So this is the part that gets me excited the most. Because there was always a trade-off or used to have a trade-off of, do you want it to look beautiful or do you want to make it accessible and crawlable? Right now, with HTML in canvas, it's the first time that we don't really have to choose. So there's a lot of options, a lot of things to play.

So I hope this eventually becomes a standard and people start building cool demos in the web. To close, we wanted to show a few experiments that came out of basement. Projects where the line between the DOM and the canvas start to disappear. HTML inside 3D scenes, text that distorts, effects running on top of real divs, but still keeping the things that make HTML valuable in the first place. Accessibility, SEO, real browser behavior mixed with GPU effects.

Today, we show a lot of workarounds we use to make both worlds work together and they work, but they are often fragile, hard to maintain, and performance becomes part of the problem real quickly. That's why HTML in canvas is exciting, because it gives us a bridge between both systems. We can build this kind of experience in a simpler way with less glue and less overhead. And we think that unlocks a completely different generation of a website. So we are really excited to see where HTML in canvas goes. Thank you for watching and thank you, Shades Nature.