What's Under the Pointer?

Let's talk about Tealdraw and let's talk about, of all things, hit testing and clicking, stuff like that. Just curious, how many people know about Tealdraw or have ever heard of Tealdraw? Hey, all right, that's not so bad. Cool. Well, Tealdraw is a couple of different things. It is this. It's an application, tealdraw.com. Free whiteboard. You can move things around. Works kind of like Miro, works kind of like Excalitraw, if you've ever used that.

Tealdraw is also a startup that makes tealdraw.com. And in addition to tealdraw.com, Tealdraw, the startup, also makes tealdraw.dev, which is the SDK that you can use to make things like this. I have a long list of favorites, but I always like this one a lot where you have a, I didn't make this, by the way, but this is a great liquid simulation. Fantastic. Runs in the browser, super fast. But the author also used Tealdraw as a kind of like editing environment using the whiteboard. Not necessarily for whiteboard things, but using some of the geometry inside of Tealdraw to interact with the liquid simulation. So pretty far out what you can do with Tealdraw.

And the reason why this works, and the reason why I guess it works as an SDK, is because we solve a lot of, like, super gnarly problems involved in the canvas for you. And you can build on top of that. If you've ever tried to build one of these things, it's just like a map or a text editor or something. Quite internally complex, but also those complex bits are just the way it should work. So good place for an SDK. Not that far. Today, I'm going to be talking about what's under the pointer. Originally, this was what's under the cursor. But then the cursor happened, and we thought people would be confused. So it's what's under the pointer. It's all about hit testing. General idea of if you've done game development or things like that, you might have spent a lot of time on hit testing, the whole point of, like, what am I... Where's my pointer? What am I hovering? What am I about to click on? Things like that.

So let's get into it. When Tealdraw started, it was me wanting to play around with arrows and shapes and text and things like that. And making sure that things would bind perfectly and do all this type of sizzle reel type stuff. But in order to do that, I had to solve some of these problems. So one of them is when you have a ton of different types of shapes, some of them are closed and open and filled and not filled. They're all overlapping and stuff. How do we determine what the user is hovering? Typically in the browser we don't have to think about that, because buttons just work when you click on them. They all have event listeners that kind of hide that type of hit testing, et cetera, from you.

In the old Tealdraw, the original Tealdraw, I used DOM events for this. So here's how this would work. I'll kind of show you also over here. Let me come back to that. So you had hollow shapes, you had filled shapes, you had these kind of open shapes, which are neither. And we want to select a hollow shape if we're kind of near the edge. We don't want to have to go exactly on the edge, because that might be very, very thin. But we want to just be near it. And for a filled shape, we want to be near the edge or in the middle of it. This works for open shapes as well.

And so the way that I did that in the original Tealdraw was by using kind of a hidden SVG path. So everything in Tealdraw is both old Tealdraw and new Tealdraw is DOM, right? It's a bunch of React components on the canvas. The canvas itself doesn't use a canvas element, it just has React component, React component, React component, React component. And believe it or not, that just works. You can put a lot of stuff on the canvas, even if it's React. You have to be very careful, but you can do that. So in that world, when I was using DOM elements, you'd have the normal shape. And then behind it, I was kind of doing a hidden path just to absorb that hit test from the browser, right? So the black shape has pointer events none. But this kind of hidden path behind it is the one that has the pointer events. And that's how you get this kind of like, you know, close hover hit, right? Okay. Now I'd have to adjust the size of that based on the zoom. Right? Because I needed it to be in page space or screen space rather than page.

Anyway, that worked pretty well. But it had some problems. Those problems kind of looked like this. If things are overlapping, then where do what happens here, right? Because like visually, like this is easy. But once they're hollow, like the Z order of the stuff, you can't tell because they're both hollow.

However, with DOM elements, like, whichever thing gets hit first, that's the thing that gets the hit. So that's a problem. Is that the actual hit areas would kind of look like this. But that wouldn't be available to the user. So that was one problem. The other one was that when shapes were really close to each other, well, hey, that's just what I talked about. I need to zoom out a little bit.

When shapes were really close to each other, that's another place where those overlaps would kind of hide things. And so, like, if your pointer is like right here, right in between it, one would be selected, even if you were really close to the other one. That's not what you want. So can't do that. Z stacking kind of screws up the whole, like, using these pseudo, like, touch target SVG paths. Can't do that. Also, like, hit testing is not free in the browser. In teal draw, you can have thousands and thousands of shapes on the canvas.

And we were noticing that it would just be spending time on every frame, just kind of working through that DOM, trying to find what's actually beneath the pointer. Surprisingly inefficient, given that it's kind of a browser thing that you're supposed to get for free. And to make matters even worse, with the kind of DOM style approach to hit testing, is arrows. Because when I'm dragging an arrow, which is something that I'm hopefully doing a lot in teal draw. Let me get rid of this so the arrows work a little bit better. I'm not actually... Like, my pointer is kind of, like, captured by this. Like I'm dragging... And it's not doing hit tests against the DOM anymore, because the pointer is captured by that element that I'm dragging.

And so we needed a different way to recognize, like, when the user is hovering on the canvas. Which meant we needed to kind of make a geometric representation of the canvas in JavaScript anyway in memory, and then be hit testing against that. Which was awful, because now we have two different ways of hit testing. I don't want that. So let's go all the way back. We had these two different strategies. We had these DOM-based click, touch target area paths that were hidden from the user, but they were actually there. And then we had this other system, this very rudimentary system of polygons. And you do, like, normal is point and polygon type of hit testing against that. And I decided... All right. Let's get rid of one of them. Which one can we get rid of? We can get rid of the DOM. So let's go all the way here. And let's do a whole different geometric system.

So this is kind of fun. Because this was... Sorry. I have fairies here, hiding in front. More on those later, maybe. I'll tell you what.

This is the debugging geometry view of the canvas. This is what the editor sees. Because the editor is kind of like a class that manages the state of the canvas, that manages all the data in the canvas. And that, like, exists separately from the canvas itself. Which is rendering using React. But we want to keep that data going one way. As much as possible.

So before it was fine. You know? You're moving the pointer around. It's sending events back to the editor. But... And now you're still sending around your mouse position. But the hit is being determined by the editor itself. So as I'm moving around my cursor, you can kind of see that it's measuring the distance to all these different polylines and polygons and so forth. And it uses that distance to determine what's hovered. But there's more. And that more is going to be what I talk about for the rest of the talk.

Of how exactly do we determine what's hovered and what's not? To... And like... Yeah. This... Although this is a big canvas. I don't know if you can see it. I don't know if I have this mode on. But you can kind of see the way that it works. We have essentially a polyline being created for open shapes. And for closed shapes, we have the shape itself. And this square in the middle is actually its label. So we want to be able to...

So we want to be able to... I can't show you. Because we're not showing the canvas. But yeah. Believe it or not, I am typing into that. And I think if I show the shapes again... Oh, no. It did require that. Whatever. There's the... What I mean by label. So the good news is that this solves a lot of problems for us. The lots and lots of shapes problem and the browser itself. It's hit test being kind of slow. We don't have to deal with any of that. We just slap on pointer events, none across the whole thing. And we're done. Right?

Because we're gonna handle that hit testing ourselves inside of... Our own JavaScript land. Even more than that, we actually don't want... We want to guarantee that there's no hit tests being done against the canvas. Even with indicators or other things that might be clickable. So when you are doing an interaction that would make it impossible to click on something, we actually bring up another element in front of the canvas with pointer events all on it to absorb any hits. Because the browser will work its way down from kind of front to back with hit tests. So if you ever want to just not do any hit tests, you just put something in front, capture that event, and it won't make it any further.

There are some challenges, though. What happens when you have a lot of these kind of complicated hand-drawn shapes or something? And I'm moving around. I don't want this system to be iterating through every one of those polylines, essentially, trying to find... Okay, what's the distance? Is that the closest? Or is that close enough to register as a hit? Et cetera. So we do something pretty common.

Again, especially if you've done any sort of game development, especially. Which is we do... Or any algorithms. Right? You do a broad phase, and then you do a narrow phase. You get rid of the work. And then you do the work that you have to get rid of. So in this case, I can just look at the bounding boxes of all these things. And if this is my cursor, pointer, whatever, then it's not inside of any of those bounding boxes. So I actually don't have to do that kind of like find... Okay, what's the distance to this line work? Because I know that it's not inside of any of the bounding boxes, or I can kind of expand these bounding boxes a little bit. Which is something I think we do. And either way, I can just not deal with any of that work.

It's only when the pointer is inside of one of these bounding boxes that then I actually check like... Yeah, okay, we're inside of the bounding box, but are we really clicking on it or not? The only... Things do get complicated when things are overlapping. But in general, they just don't. And either way, you don't have the whole canvas overlapping. That's one way that we solve that. And of course, the other problems. Things like... All right, this is... There's two shapes that are really close together. Which one should I hover? Should I hover this one or this one? Which one is being clicked on?

Or... All right. There's actually a couple of shapes. There's one inside of another one. The user is clicking. Which one is he clicking on? Which one are they clicking on? That's actually a solvable problem, too. We just write a bunch of code. We go through and we decide...

Yeah, you know? You just write the code. You just... You know, it's like a decision tree that just kind of keeps going. This is majority comments. Because this is a complicated thing. I'll go through it together. This is the decision tree. As best as I can extract it from that. Which I'll send you a link to this whole board if you want.

And then we can... But in the meantime, I'll kind of walk through some of the big decisions that we make. First off, the basics. Shapes can be hollow, shapes can be filled. We start at the top. Very top of the canvas. Everything is kind of in Z order. Front to back. And we go until we hit the canvas.

Or until something stops us. So an example of something stopping you would be this filled shape here. This is hollow. It's on top. Let's make this red so that you can see it. Even though this thing is on top, it's hollow. It has no fill. So we kind of like pass through it. But we pass through it. And we hit something that is filled. And so that's when we stop. Simple.

If we click near to the edge of a hollow shape, then we select the hollow shape. It's kind of the easiest ways to start. If we... Here we go. So, yeah. Filled shapes will stop the search. We kind of pass through the hollow shapes. But as soon as we hit a filled shape, that's the shape that we're gonna return. If we did reach the canvas, then we're gonna hit the smallest shape that we are inside of. The smallest hollow shape that we're inside of. And this works for arrows as well as for hit testing. Now, that is regardless of the Z order, right? Like if I have... By the way, is this making any sense whatsoever? I normally just show like AI stuff. All right. Cool.

Which I can and will if this other guy's Uber doesn't show up. So even though this shape is... Let's put it all the way in the back. This will still be the kind of shape that we hit because it's the smallest shape that we're inside of. Because it's kind of like the way that it should work. There's actually a lot of kind of intuitive user experience inside of a canvas. Which is both awesome because as a user, I don't have to learn everything. If I grab something that is in front of something else, I expect to grab the thing that's in front of it, right? You don't get that often in software. You definitely don't get it always. However, it just turns out that real life is very complicated. And so trying to capture all that stuff within algorithms, within code is really fun. But also hard.

If the pointer is between... Is kind of close enough to an edge to select it, but there's multiple edges that you're close enough to select, then you have to pick the closest one. So you reach all the way to the canvas.

You've reached the canvas. You check... Was I close to any edges? And if I was, what's the closest one? That's the one that you use. Which is great to be able to do this kind of like... Obviously in the DOM method that we used at the beginning, you couldn't do this stuff. Because as soon as it hit any element, it would stop. But now I get to do all this kind of comparison stuff. Build some more logic into it. Such as labels.

Even if the label... Even if the label is kind of like further back and you're pretty close to an edge, if you're in a label, then we're gonna select the label. Because generally that just seems like the way to do it. And finally, there are frames in TealDRAW which kind of contain the clip... They clip the shapes that are inside of them. But they also... So I can select the shapes inside of a frame, but I can't select things behind the frame. So if I've reached the kind of the back of a frame... Even if there are shapes still beneath it, I just stop. I don't select the frame. I just stop. It's like another canvas.

And that prevents you from kind of clicking things inside. And of course, you know, we only check the masked part of the shape. So I can't click or select the parts of this shape that are kind of hidden. Sidebar, just because I think it's cool. If I drag this shape out, it falls out. If I resize this, it'll fall out. If I have a shape that is like this, and I resize it like that, I haven't let go. The bounding box is still in the frame. Right? It's still overlapping the frame, but the geometry isn't.

And since we have that geometry system before, we have that, you know, whatever, like little wire view, this data representation. Like we can check against that and have it fall out, you know, using that. It's actually a really, really cool thing to have these two separate systems.

Anyway, frames stop selection. That's actually my talk. That is actually how you do hit testing in TealDRAW. Which is the first time I've finished a talk on time in a long time.

TealDRAW.com if you want to play with the white board. TealDRAW.dev if you want to build something with the white board. We do have little AI fairies coming on Monday, which are kind of like little agents on the canvas, and this should actually be pretty fun. They'll build stuff for you. I'll talk about this next time. What you wish I talked about that? All right. I promised them that I wouldn't. But yes. Follow TealDRAW on Twitter or X or whatever. And... I'm doing this. Catch me later. I'll be around. I'll show you the rest of this stuff. Thank you so much.

We have a lot of questions coming in. I am the emcee, so I get to seed my question first. I love how intuitive you've created the controls and stuff. How did you think about that hierarchy of different steps, different checks, all of those kind of... That logic flow, in order to get that really intuitive feel? Yeah. I think it's... I just use the app a lot. I mean, this is the same for most products. You'll find the kind of latent design of a product just by using it a lot. And saying... If I'm clicking on this, and it's not selecting, why not? If there's this heuristic, this rule that I'm using over here about distance to edge, how does that intersect with this other system about labels or something?

There's a game called Dwarf Fortress, which maybe some people in the room... This room might know about. Where you have all these different systems, like dwarves can get drunk, but they can also bleed. And when they added the drunken system, that affected the bleeding system, because if you have less blood, then you get more drunk. And that's what TealDRAW feels like. As we added things like labels, we'd have to say, oh, this system interacts with this other system. And then you just layer those systems on again and again and again. And again, this is what makes Canvas so hard, because there are so many of these systems. But really just brute force. Just use it, use it, use it, until you figure out what the logic should be. It may be hard, but you're doing a great job with it.

Let's jump into some of the questions. Let's go with a guy who is afraid of QA engineers. They have given up a lot of questions in today. What's the weirdest shape that broke hit testing code? I believe this is self-intersecting closed drawn polygons. So if you can imagine, like, the infinity symbol, right? Or if you can imagine maybe, like, is it an Airbnb symbol, where it has a loop inside of it? Like this? Like a paper clip type of a thing. That middle of the loop is really hard to identify, whether you're inside of the shape or outside of the shape. Because a lot of the normal kind of like polygon, am I inside of the polygon or outside the polygon break in that scenario. Unless you do a lot of extra complicated things to build essentially multiple polygons out of the self-intersecting polygon, and then check against every one of those and see if you have an odd number.

Unless you do a lot of extra complicated things to build essentially multiple polygons out of the self-intersecting polygon, and then check against every one of those and see if you have an odd number. Or even, you know, like that type of crazy stuff. So yeah, if you go on to draw, and you kind of draw like a paper clip type of a shape, some of the hit testing might not work as good as you're used to. But most people don't. Got to take your losses somewhere. There's always going to be one. Watch the QA engineer bring that into the next process. We are hiring, by the way, if you like this type of thing. Nice.

Gotta say it. And you spoke about how this is TLDraw 2.0 or .whatever. Have you ever wished, and maybe if there's a 3, that you built on the canvas rather than the DOM? No, actually this was probably one of the better decisions that I made with TLDraw. The good news is if you don't want to use the DOM for rendering, you can actually just take that layer out completely and add your own. So people have brought in WebGL renders for TLDraw. However, most of the teams that work with TLDraw as an SDK, and most of the things that we do that are special, I suppose, about TLDraw, really leverage the fact that it's just made out of normal web stuff. So putting videos on the canvas is trivial. Putting iframes on the canvas is trivial. Putting iframes on the canvas and interleaving them with other shapes is trivial. And yeah, I think from an SDK perspective, like the customers that we work with, the businesses that we work with, most of their products are not graphics editing products where you really benefit from that tall ceiling or high ceiling of like, we control the whole renderer, which you get with Canvas and WebGL, et cetera. Most of them are like, I want to take my dashboard, like my AWS dashboard and put it on a canvas so I can zoom out and see them all. Or I want to, you know, I mean insert here, like I want to have this the same to-do lists, you know, if you're ClickUp, like from the rest of ClickUp. I want to have them on the canvas as well. So no, actually the web is like a pretty good technology for this type of application. You've got, this is the web on steroids to a certain degree. It is just a React app. It is a really complicated React app, but it is honestly React all the way down. Okay.

Speaking of that, do you use WebAssembly ever for web complex processes? I mean, the things that would use WebAssembly would be like that liquid simulation that I showed before. That's another case where, well, they haven't removed our renderer. They just kind of removed the background and put another renderer behind it, right? But that is running in, I believe WebAssembly and certainly the, if I had another five minutes, I'd show you the weird camera thing that you can do with the, with that one where you can, whatever, it uses in-browser ML to kind of like use your webcam and create geometries based on your, where your hands and face are and stuff like that.

That's using WebAssembly. But those are, those are like things that other people built with TealDRAW, not me, but you know, it's the web. It just plays well with other things. Just it's what you want. That's awesome.

And remember, if you do want to go check it out with him, he's going to be over downstairs in an in-person Q&A so you can also look at his laptop and see some other cool examples. I really like the liquid one.

And this one from a guy who's afraid of QA engineers again, if the pointer lands exactly between two shapes and maybe not just between two shapes, cause you talked about it, but you had this kind of cascading set of logic, is it possible for two objects to kind of still sit at the same level of that logic? And then how do you pick between? That's a really, really good question. I'm guessing we break the tie based on whatever has the highest Z index, Z order, because that's going to be unique for every layer. So we break a lot of ties using what's in front versus what's behind. If you need another solution, you can run a mini Hunger Games. I could run a mini Hunger Games. That could be a good solution. Iterate until there's only one.

And then another one, which is calculating inside a frame. Is it like a recursive calc only within the context of inside the frame? Functionally, yes, but not actually. When we, so the canvas in Tealdraw is flat. It's like a flat list of things. Even when things are inside of the frame, believe it or not, the DOM is flat. We're not actually using HTML elements to clip. We're calculating a CSS mask for all of the children of that frame. So if something is like, kind of looks like it's like being clipped by the frame, as far as the browser is concerned, those things are not nested. Those are flat. It's just that we're dynamically creating a CSS mask around the clipped object, which is kind of what the browser does anyway. And it's just faster, believe it or not. So yeah, you just treat the whole page as like a flat list. And the result is the same. It's just like, because we're using those masks, we don't need to go into a frame. We just treat it just the same, top, front, from your perspective, front to back. And that just works.

I hope that made sense. It's a good question, by the way. And yes or no, last question from a guy who is afraid of QA engineers. Thank you for making my day. This is funny.

Is the infinite canvas infinite enough to hide the bugs from QA? No one's ever reached the end of the canvas. So there might be bugs still out there. Well, this person, journey to where no one has gone before, and to the end of the infinite canvas.

Thank you so much, Steve. Let's give them a round of applause.