Agents on the Canvas With tldraw

My name is Max Drake. I'm here to talk to you about Agents on the Canvas. I'm really excited to do that. But I guess this is a really demo-heavy thing, so I'm just going to kind of, like, try to breeze through the intro and the motivation.

So first, why am I up here talking about this? So I've been working on, like, Agents and LLMs on the Canvas for kind of as long as you could have been doing that. So in 2022, before ChatGPT, I was doing a bunch of Agents on the Canvas, or LLMs on the Canvas. This one's kind of cool. This is like using the primitives of the Canvas, so like moving and proximity and things like that, to prompt an LLM to do some sentiment analysis. Cool, like, you know, it's not the same kind of UX that you're used to of, like, typing something in and prompting it. So I've been working on this for a while. Also, more recently, I've been talking about it. Very excited to do so here. And as far as the Canvas stuff, I work for a little company called TLDraw. We're based out of London. Has anybody used TLDraw before? Okay, cool, some hands. That's awesome. So yeah, we basically build an infinite Canvas SDK. And what that means is that, like, if you want to build an app and you want the app to have some kind of Canvas component, right? So maybe it's, you know, a design app, a drawing app, a slide designer, ed tech. The Canvas is actually really difficult to get right. So we give you a, you know, a library, a JavaScript library, a React component, that is just the Canvas. And so you can focus on, like, working on all the things that makes your app work. This presentation is just one big TLDraw Canvas. We have lots of, you know, we covered all the annoying edge cases of moving and parenting and binding and rotating all this stuff. So, you know, it's really fun to work with. And all of the stuff that I'm going to be showing you has been built using the TLDraw SDK.

So for a while, so this talk is called Agents on the Canvas, right? And so for a while, we, you know, as a Canvas company, we've been figuring out how do we, you know, how do you bring agents to the Canvas? You know, there's a lot of reasons why we want to do that. You know, obviously financial ones, you know, people want to use their agents on the Canvas. But also, like, you know, a lot of the Canvas as a medium is, like, a really collaborative place. TLDraw has kind of like multiplayer enabled out of the box by default. And as we use agents more and more, you know, we're trying to figure out, like, you know, as a society, I guess, like how do we collaborate with agents? How do we work with them? And I think personally that, you know, the Canvas is like a really good place for that because you have all these kind of collaboration primitives built in, like cursors and proximity and things like that.

So yeah, we've been trying to, you know, I'm going to kind of take you on the journey of how we've been working on agents on the Canvas at TLDraw. And so just brief aside talking about agents not on the Canvas, tools like this, you know, Cloud Code, Codex, they're great. And they work really well partially because the way that we interact with them where you put in text and then you get out text is, like, exactly how they were designed. You know, I'm simplifying here but, like, they were trained by ingesting text and spitting out text. How they work, right? So then when you try to bring LLMs to the Canvas, it's actually, it's a little more difficult because they're not really, like, built to understand Canvas. There's a whole lot of things like, you know, spatial relations and the fact that, you know, it's a visual thing.

So you need to visualize it. And so you actually have to do a lot of kind of, like, engineering work in order to get agents to kind of, like, understand even, like, what the Canvas is. And so when we first started this, like, you know, research project a couple of years ago, we were, like, the first question was how do we get the agents to kind of just understand what it is when they're, quote-unquote, looking at the Canvas. And also how do we get them to act on the Canvas in a way that not only in a way that works and, you know, can actually do it but also in a way so that they understand what they've done.

Because that which, yeah, that's a whole thing. Yeah, so the first project, which was from, I think, 2025, it's called Teach. And this was basically Teach is, this is Teach. So I'm going to, Teach is kind of slow. So I'm going to prompt it right now. This might not be able to see it, but I'm saying, make the mouse blow out the candle. And so Teach is like this very, you can see in this frame here, Teach is basically, it's like a one-shot. You enter in your prompt and it basically manipulates the Canvas in a way that can best respond.

Oh, yeah, there we go. So yeah, the mouse is blowing out the candle. And so what this is doing basically is this, you know, this is taking all the shape information, which is kind of like, you know, we do a lot of manipulation to it to make it easier for the agent to understand. And so we send a screenshot and we basically teach it how to use the Canvas and also to act on the Canvas. So you see it made these little shapes. And this is all like, this isn't like a special mouse object. You know, this is just normal teal jar shapes, but it understood from the, you know, like this is just shapes. It understood from the prompt that I gave it and from the visuals that that was a mouse. And so it's kind of able to interpret it that way. And so now we have kind of this engine for the LLM, because I guess it's not an agent yet, the LLM to understand and act on the Canvas.

Our next project kind of like towards this was we built this like thing that we call the Agent Starter Kit, which is basically this is, we wrap that in a kind of like harness that lets the LLM kind of like drive and move around the Canvas in order to accomplish a goal. And so I'm going to tell the agent to find a friend for the cat and bring it over to her. Her favorite color is red. And I'll zoom out so you can kind of see there's some other, there's some other animals here, but basically what it's going to do if it's working, oh, it's decided not to. Maybe the model's not happy. Let's try that again. But basically what it can do is it can, did that, okay. It can basically set goals for itself. It sets to-dos for itself. It can manipulate its vision. It can move around.

Okay, well, this one just made a copy, an exact copy of, oh, okay. It's deleting it by way of teleportation, by reassembling it and deleting it instead of moving the actual shapes. That's not how I would have done the problem. But yeah, so you didn't see it move around, but basically it can just change its viewport in order to get more information about the canvas the same way that a coding agent might open different files or grep or try to figure out, in order to accomplish its goal. And you can imagine that that's quite a different process than greping because it needs to actually navigate spatially. So this was, we kind of like had our take on agents on the canvas. And the next thing that we built was fairies. So fairies is a really fun project. There's these little guys here.

You can hold them. They don't like it when you hold them, though. But you can throw them around. You can, obviously this is the first thing we do. You can configure their hats, their hair. This is really important. Yeah, you can, and this is, again, this is going to be hard to see, but you can, we added a leg slider, of course. So in agent, this was kind of, a lot of this was UX. So in agent, the fairy is that agent on the canvas, but it exists kind of in the same way. So I'm going to say it, I'll just do, I'm always telling it to draw a cat. But basically, you can tell it to do something and yeah, there we go. And it works on the canvas in order to kind of accomplish that. But its vision is always based on where it is on the canvas. So it has, it's actually embodied. And also, the animations are kind of silly, but you can actually tell what it's doing. So if you think about, there's this, it output this chat history, which is, again, I'm sure you guys have used Cloud Code and you're trying to read the commands that it's doing. It's really hard to parse. Even this too, it's a lot. But it's actually really easy to tell what it's doing just by looking at what, like its animation and its state on the canvas. And so also, what we did with this is we realized that, agents aren't just single agents. You also, people have been doing a lot of like collaboration stuff recently. And so we, fairies have friends basically. And so you can, what you can do is you can select a couple different fairies and you can make a group chat and you can tell them what to do. And so I'm going to say, I'll do something simple. I'll say like, give the cat two friends and put them in a scene. Again, this is a visual thing. So what's happening right here is that, you'll see in a second, so one of these agents has been chosen to be the orchestrator agent, and you'll see, see this text here, creating a plan. And so, yeah, basically the orchestrator agent is, it's creating tasks right now and it's like assigning, it's assigning tasks. And so, yeah, it's basically, okay, it's created one task to draw the scene background. And that agent, one of those agents is going to start working on it, but basically what's happening here is that like the orchestrator agent has no, does not have the ability to make shapes.

It doesn't have the ability to do actually any work on the canvas. But what it can do is it can, you know, it can create tasks and it can assign tasks and it breaks up the work. And then it assign tasks, it assigns tasks to the other agents and the other agents, they're only able to see what is inside their task bounds. So this is kind of a, something we realized through a lot of trial and error is that like you, and it's context management, but context management on a canvas looks really different. Because you know, if you have, I could have a whole, you know, presentation or a whole wireframe off to the side. The fairies can't know about that because it's going to confuse them, it's going to fill up their context.

Okay, yeah. And you see there's other, they're making some friends for the cat and the orchestrator is kind of just sitting there waiting. Now it's reviewing the first thing. But yeah, so when a fairy is working on something, it can't see anything else that, except for what it's working on. And that, you know, does a lot to, to like, you know, improve its, how it manages its context. And same with the orchestrator. It can't make things not because we thought it would be funny, but because if you remove the ability to create shapes, that's so much like schema and tool calling that it doesn't have to think of. And so yeah, this is fairies, this is basically, you know, multi-agent collaboration on the canvas.

And okay, this is now we're getting to the part where it's like, in my talk, at least, this is the, this is the, we're getting to the boring part, which is actually also the cool part, but, you know, all of this has been kind of fun, but like, it's not really like useful yet. And so there's a couple of different things. There's two different things that, you know, we've been working on recently that are kind of like trying to bring this, you know, what would this actually look like in a product? So we have like some demos of how this could work. The first thing that I want to demo is the tech tree. So when we do a launch at teal draw, you know, normally we use like our issue tracking software. It's, you know, linear or notion or whatever. But when we're getting close to a launch, we kind of abandon all of that because we really need this kind of like high level view of like what's going on. So we just make a teal draw board and we basically have, you know, it's, we make a big graph with arrows that shows how, you know, what dependencies, what tasks, what dependencies different tasks have. So, you know, I always thought that would be really cool if, you know, we could just assign like a coding agent to carry off those tasks, carry out those tasks when we do it. So if I assign this task to Claude here, I hit run. This is going to start a coding agent working on this task. You can see if I zoom out, this is like an entire, you know, project that has, you know, it's had some, you know, some work going on it before. The project itself is this experiments thing. This is just kind of like testing a bunch of different inputs for this work. Yeah. So this is kind of all this is just like, you know, silly little demos, right? But this was all created by the agents themselves.

I haven't looked at this repo at all. I haven't looked at any of the code. You can see that this one is working. We, this PR needs to be reviewed. So this agent has opened PR and now it's waiting. And so, yeah, that looks great. I'm going to merge that. Good job. And you'll see, you know, this will be, this will be finished in a second once the, you know, the polling, it takes a second. Well, we'll let the polling, did that actually merge? Yes, it did. Okay. Well, okay. Oh, there we go. Okay. Yeah, it's finished. It just took a second for the GitHub update. So now that this task is finished, this implementation is finished. So I'm just going to hit run on this and we'll see if it finishes before we, you know, before we go. But, so this is kind of like, you're kind of like managing a project and also kind of like prompting your coding agents here.

And I don't know about you guys, but like, I'm always when I have a project, I have like five different Claude tabs open at once. And they're all kind of like, they're like scattered. They're either in, like my cursor ID or they're just an actual terminal window or something. And, you know, looking at it, it's really hard, you know, they added that recap feature recently, but like, because it's so hard to tell what a Claude terminal thing actually represents when, you know, when you're coming back to it after a day. So this is kind of like, you get a really, for a project, you get a really high level of view of actually like what's going on, which I really like.

You can also, this is another thing I like about the canvas is that the, you know, you, sometimes, you know, I don't like, I'm often like drawing a diagram of what I want an agent to do and screenshotting it and pacing it into Teal draw. It's really, it's not great. So this one kind of lets you just like take some, take some, take a diagram or something, just kind of like draw a task around it. And then I can do the same thing. I can just assign it to Claude and I can run. And if I go and I open it, that's kind of like, this is now the, you know, this is the prompt, like, this isn't like just this thing is the prompt for the coding agent and that's going to be working.

Don't know if it'll finish. It seems like this one hasn't finished either. But yeah, so this is kind of like, how can we use this kind of like concept of agents on the canvas? How can we use the canvas primitives and like, you know, all of the things that the canvas gives you for free, which is like collaboration. This is also collaborative, which is really cool. So if I, you know, I close this window, my colleague joins, they can just inspect what because these agents aren't running on my computer. These are hosted agents. And so my colleague can inspect like, you know, what's going on as well. And we can all be collaborating on a project. And this is kind of like what I was talking about earlier with, you know, agents, like humans can collaborate on a canvas really easily. And we think that like agents is a great, is a great, like, I don't know, way for it to do that.

I'm going to see if this one is finished or if it's hung. It seems, okay, it's doing a lot of stuff, but yeah, we don't, we didn't make it very, we didn't make it very easy to understand what they're doing yet. But yeah, okay. So this is, this is like the kind of like tech tree thing. Another thing that's really cool. So, TealDRAW, we've been developing a desktop app that works on actual files, which I think is pretty interesting. And something you can do is you can drive the desktop app with Cloud Code. You don't need to like hook into our like special infrastructure or anything. So what I already did was I gave, I made this, I gave this like a physics simulator. If you've ever used TealDRAW you know that this is not a built in behavior. But what I did is I just kind of told Cloud Code to give it gravity and WASD controls. And it did that. I'm going to, I'm going to tell it to, to give, make this, so I drew this out but it doesn't do anything yet. I'm going to tell it to try to make this, this hat stuff. Hold on, I have dictation. Turn the ball hat selector into actual UI that changes what the ball, what the hat the ball has on. Awesome. Let's see if it does that. It can be, you know, that'll take a while. But there's just a lot of like really cool stuff you can do with this because this is like working on, it's Cloud Code so it has access to your entire computer.

So I'll do with this open code. Oh no, the ball. Okay, well the ball fell. But I'm going to tell open code, I'm going to say, look at my documents folder and for all of the directories, make a rectangle on the TLDR canvas that is size proportional to how big the folder size is in gigabytes. Yeah, sure. So this is kind of like, you'll see the, at some point an agent will spawn onto the canvas once it does this. But this is like, this is something that's really interesting because like you can make these kind of like interactive like UIs and these interactive experiences just like from the canvas.

It seems like none of them have actually finished yet. And we're, oh, there we go. Okay. So this is basically the Cloud Code agent here. And this is the open code agent. This is the Cloud Code agent here. And they're both kind of like, they're hooking in, they're using like the kind of like sub agent lifestyle functions to actually like render this UI on the canvas. So this is like, this is just like Cloud Code out of the box with, yeah, allow always. This is Cloud Code or open code out of the box with just like a sub agent installed. And okay, here we go. So yeah, we have all these, okay, I made a lot of these really small because I think there's text, but yeah. So these are, these are all of the folders on my, on my, on my desktop that it kind of like it created.

This is where in the unplanned part of the demo. I don't know what the agents are doing right now, but the, but yeah, so this is kind of like, again, this is where we are at TL Draw. This is like agents on the canvas. We're basically trying to get it so that, you know, people can kind of bring their own agents. And that's what this kind of like work at the, with the desktop app is, is again, you don't have to hook into like for fairies or the agent starter, you don't have to hook into your own, to our agent stuff. You can just kind of bring your own. And so, yeah, we're trying to make it really easy to, to work with agents on the canvas. I would love to talk about it more if anyone has any questions. I think maybe it's Q and a time. Thank you.

But can you maybe tell me more about the TL Draw SDK? So I guess you were showing demos on like the TL Draw app and then the desktop app, but I'm seeing all this stuff visual is really, I'm like, my, my mind's racing. It's like, how can I do this? And what I'm doing when I'm implementing. So does the SDK allow us to add it to our own apps? How does, how does that work? Yeah. So the, at the very like heart of it, the children SDK, basically just like your entry point is just a single react component, which is the Teal draw component. And we basically allow you to kind of override or customize any of the behavior you want. So like you had custom shapes, custom tools. And we have like a, you know, replet just recently launched their new kind of like agent canvas UI that's entirely built on TL draw. So they've kind of like really customized that for their own needs. You know, they have custom shapes for their own, like, you know, prompts and their own like app rendering systems. But yeah, they're like the, the simplest version of making your own, like Teal drop is just importing the component and putting the component in it. And from there, yeah, we have, that's kind of how you get started. And again, all of the besides the slideshow is just on the Teal drop app, but every single demo besides that was all just built on the SDK itself.

I see. So you have a custom app, you pulled in the SDK, and now you have your own canvas that isn't Teal draw, but you have all the same functionality. Yeah, yeah, exactly. And was there, and I might have missed it, but is there an MCP server that's giving cloud or open code access to it? So how are they getting direct access to that canvas? So we did a lot of experimentation with that. We actually, it's not really an MCP server. What it is, is it's just the sub agent. So the desktop app itself exposes just like a local host server with a bunch of endpoints that, and we just, the sub agent itself kind of like has all of these, has instructions and we're just like, hey, just curl it, you know? Like we don't- It's an API. Exactly. Yeah. So like, you can just, Hey, there's a server it's running on, you know, it's running on this port. Curl it and you can get all the, you know, and we have instructions in that as well. So we like, you know, you, if we wanted to standardize it, we could do it, but it's nice to just have the entry point of the sub agent is a lot nicer sometimes than the MCP app or the MCP. Sure. So we can like front load all of the tools. No, that makes a lot of sense.

So the agentic canvas, it's not part of the SDK itself. However, we do have the code. Tldraw is all source available, and you can find all the code on our GitHub. The agent starter is included in the repo under an MIT license, allowing you to use and get inspired by it. Additionally, there are various subsystems like a linting system to ensure code quality and prompt agent adjustments when needed.

This system helps in addressing layout challenges such as text overlapping issues. The system prompts for adjustments if necessary, aiming to provide a smooth user experience. These tricks and systems are available on TLDraw.dev or the TLDraw GitHub repository for developers to explore and integrate into their own projects. The focus is on offering adaptable patterns for developers to incorporate into their codebases rather than providing a complete SDK solution.

The development process is evolving rapidly. While certain features like fairies were initially included in the TLDraw.com app, ongoing experimentation and refinement are crucial. The team aims to perfect solutions before integrating them into the SDK. This cautious approach reflects the commitment to delivering the most effective tools and indicates the ongoing experimentation and optimization process to ensure the SDK's quality and relevance.

I know this is not the question asked. We have like a linting system. So if the, you know, the same way you would lint a code base or like lint a file, if the agent, cause layout is one of the really hard questions for this. Again, like I was talking, the spatial, you know, how does it understand like if, if text makes something overflowed, it's hard to make it understand that it's not going to be like, there's going to be overlap. So we have this system that like, once it's done generating, we look at all of the things that generated. And if there's two text shapes that are like overlapping each other, we prompt the agent again. We're like, Hey, like, was that on purpose? If not, or if it was ignore it, but if it's not, you might want to like fix it and stuff. So there's a lot of like really cool kind of like tricks and stuff in there that we made that we used to make the agent canvas work. So yeah, that's on TLDraw.dev or the TLDraw GitHub repo. That makes sense. So it's more so like patterns that you could adopt into your own code base rather than an SDK. Yeah.

The thing is that like this stuff is moving so fast. Like again, the fairies was, you know, fairies, we shipped in the TLDraw.com app as like a limited free time thing in December, but there are some of the things that we did there that are already like we wouldn't do again if we had to build it again. And so we're kind of like, we don't really want to put any of this stuff in the SDK just yet because like we really want to make sure that it's the right solution. And it's a lot of like, it's still lots of experimentation right now and we're trying to nail that down. That makes total sense.

A fun one here. Can you make the agents sabotage each other? You can actually. Something that I did, which was really fun is you're playing like turn-based games with it. So I made like, I made a maze once and I was like, I was like, okay, me and you are going to take turns completing this maze. There's like two characters or whatever. One for me, one for the agent and I would be cheating and I would like be skipping over walls or something. And it would be like, I don't understand how the other one is. Another funny thing you can do is you can say, tell two agents to both try to draw something in the same spot. And for the fairies, they know that there's other fairies on the canvas and they don't get confused. Which is again, part of the work that we did for the collaboration is they know where each of the other fairies are working. But for the agents, if we have two of them, they would make something and then they'd review their work and they'd get really confused why there were two things there and then they would delete all of that stuff. And then the other agent would review their work and be like, wait, somebody deleted all of my stuff. So they would do it again.

Fascinating. So they didn't know, like they didn't know who wrote it or who put that object there and were getting confused. Yeah. Yeah. Learning from that is actually, we also, we send all of the user actions that the user has done on the canvas since the last prompt we send to the agents. So you can be like, oh, wait, can you bring back that shape I just deleted? It knows what shape that you just deleted, for example. That's nice. It's like a built-in undo almost. Yeah.

Let's see. This one is, how do you manage the token usage when working with the canvas? Seems like it might require a lot of context to work. So what kind of data are we passing back and forth? Is it JSON objects? Is it, yeah, how are we, how's the token usage? It's a lot. No, it's a lot of tokens. Yes. I mean, one of the things, images, I feel like don't, aren't that bad now. In the, some of the earlier experimentation we were doing, images required a lot of tokens and were much more expensive to process. Now, not so much. Okay.

But yeah, we, something that we've been doing to manage that though, is we've kind of been adopting this, instead of just, we have this massive system prompt for fairies, as you can imagine, all the schemas for the shapes and everything. Something we've been doing with the more recent stuff, is we just let it read the docs. It's kind of like an RLM type thing where we say, hey, make a square or whatever, and it basically can query the docs for the schema for a square. And so it's only ever getting, and then it queries the docs for asking, how do I create a shape? And so it's only, we're allowing it to only get the information that it needs and add some latency, of course. But as far as token usage, it's a massive help. That's good.

This next question is related in terms of data usage. So are the LLMs using JSON schemas or XML div-like output? The latter might be efficient due to the large amount of HTML that LLMs have been trained on. Yeah, HTML is actually really interesting. We haven't tried that. We've actually, we've gone through a bunch of things. So teach, the first thing I showed, which was the single shot, that uses an HTML-like thing for the actions that it outputs. For the agent starter and for fairies, that's using JSON. And then for the desktop app stuff, what that's actually doing is it's writing JavaScript against the editor and just running it locally in the editor instance. So if anyone's used code mode or similar tools, it's basically, instead of giving the agent a set of actions that we allow it to choose from, we're basically like, hey, here's the entire TL draw editor SDK, and here's the docs. And you can just write JavaScript that we'll evaluate in there. And it's all running locally, so security-wise, it's fine, for the desktop app at least. But yeah, so that's another thing that is, if you wanted to make a thousand shapes, before it would be a thousand tool calls or whatever. But this one is just a for loop. So you can, as far as token usage and stuff, you can get a lot of gains from that as well. I like that.

That's a good strat. Only time for a couple more questions. Here's a fun one. Well, I lost it, but here's this one. So what AI library, if any, are you using for charts and TL draw? So are there any chart libraries or is it all built in shapes? It's all shapes. It's all shapes. Okay. Yeah, yeah, yeah. There's no, I mean, that would be something interesting. If you wanted to have the agents do data viz, you could kind of bring in your own chart library and have them fill them out. But yeah, these are all just shapes. I think I misread it. It's chats, not shapes. So are you using like AISDK or TanStack AI to like... Oh yeah. So for fairies and the agent starter, we're using the AI, the Vercel AISDK. But we're not hooking into that too much. The chat component and everything, that's all custom because of how we wanted to represent the UX for the agents. And for the agent starter, it is... Or for the desktop app, I think we're using, no, we're not using anything for that. We're just kind of like rolling it ourselves. Okay. That's all the time we have for questions, but thank you so much, Max. Thank you.