Agents on the Canvas With tldraw

My name is Max Drake. I'm here to talk to you about agents on the canvas, and I'm really excited to do that. But I guess this is a really demo-heavy thing, so I'm going to try to breeze through the intro and the motivation. So first, why am I up here talking about this? So I've been working on agents and LLMs on the canvas for kind of as long as you could have been doing that. So in 2022, before Chat GPT, I was doing a bunch of agents on the canvas or LLMs on the canvas. This one's kind of cool. This is like using the primitives of the canvas, so like moving and proximity and things like that to prompt an LLM to do some sentiment analysis. Cool, like, you know, not the same kind of UX that you're used to of typing something in and prompting it. So I've been working on this for a while. More recently, I've been talking about it. Very excited to do so here. And as far as the canvas stuff, I work for a little company called TLDraw. We're based out of London. Has anybody used TLDraw before? Okay, cool. Some hands. That's awesome. So yeah, we basically build an infinite canvas SDK, and what that means is that if you want to build an app, and you want the app to have some kind of canvas component, right? So basically, it's a design app, a drawing app, a slide designer ed tech. The canvas is actually really difficult to get right. So we give you a library, a JavaScript library, a React component that is just the canvas. And so you can focus on working on all the things that makes your app work. This presentation is just one big TLDraw canvas. We have lots of, you know, we covered all the annoying edge cases of moving and parenting and binding and rotating all this stuff. So, you know, it's really fun to work with. And all of the stuff that I'm going to be showing you has been built using the TLDraw SDK.

So, for a while, so this talk is called Agents on the Canvas, right? And so, for a while, we, you know, as a canvas company, we've been figuring out how do we, you know, how do you bring agents to the canvas? You know, there's a lot of reasons why we want to do that, you know. Obviously, financial ones, you know, people want to use their agents on the canvas. But also like, you know, a lot of the canvas as a medium is like a really collaborative place. TLDraw has kind of like multiplayer enabled out of the box by default. And as we use agents more and more, you know, we're trying to figure out like, you know, as a society, I guess, like how do we collaborate with agents? How do we work with them? And I think personally that, you know, the canvas is like a really good place for that because you have all these kind of collaboration primitives built in like cursors and proximity and things like that. So, yeah, we've been trying to, you know, I'm going to kind of take you on the journey of how we've been working on agents on the canvas at TLDraw.

And so just brief aside, talking about agents not on the canvas, tools like this, you know, Cloud Code, Codex, they're great and they work really well partially because the way that we interact with them where you put in text and then you get out text is like exactly how they were designed. You know, I'm simplifying here, but like they were trained by ingesting text and spitting out text and that's kind of like how they work, right?

So then when you try to bring LLMs to the canvas, it's actually a little more difficult because they're not really built to understand canvas. There's a whole lot of things like spatial relations and the fact that, you know, it's a visual thing so you need to visualize it. And so you actually have to do a lot of engineering work in order to get agents to kind of like understand even what the canvas is.

And so when we first started this research project a couple years ago, we were like, the first question was, how do we get the agents to kind of just understand what it is when they're quote unquote looking at the canvas? And also how do we get them to act on the canvas in a way that not only in a way that works and, you know, can actually do it, but also in a way so that they understand what they've done. Which, yeah, that's a whole thing.

The first project, which was from I think 2025, was called Teach. And this was basically, Teach is, this is Teach. So I'm going to, Teach is kind of slow so I'm going to prompt it right now. This might not be able to see it, but I'm saying, make the mouse blow out the candle. And so Teach is like this very, you can see in this frame here, Teach is basically, it's like a one shot, you enter your prompt and it basically manipulates the canvas in a way that can best, oh, yeah, there we go. So yeah, the mouse is blowing out the candle and so what this is doing basically is this, you know, this is taking all the shape information, which is kind of like, you know, we do a lot of manipulation to it to make it easier for the agent to understand. We do, we send a screenshot and we basically teach it how to use the canvas and also to act on the canvas. So, see, it made these little shapes. And this is all like, this isn't like a special mouse object. You know, this is just the, this is just normal teal jar shapes. But it understood from the, you know, like this is just shapes. It understood from the prompt that I gave it and from the visuals that that was a mouse. And so it's kind of able to interpret it that way. And so now we have kind of this engine for the LLM, I guess it's not an agent yet, the LLM to understand and act on the canvas.

Our next project kind of like towards this was we built this thing that we call the agent starter kit, which is basically this is we wrap that in a kind of like harness that lets the LLM kind of like drive and move around the canvas in order to accomplish a goal. So I'm going to tell the agent to find a friend for the cat and bring it over to her. Her favorite color is red. And I'll zoom out so you can kind of see there's some other, there's some other animals here. But basically what it's going to do if it's working, oh, it's decided not to. Maybe the model's not happy. Let's try that again. But basically what it can do is it can, did that? Okay. Okay. It can basically set goals for itself. It sets to do's for itself. It can manipulate its vision. It can move around. Okay, well, this one just made a copy, an exact copy of, oh, okay. It's deleting it by way of teleportation, by reassembling it and deleting instead of moving the actual shapes. It's not how I would have done the problem.

But yeah, so, you know, you didn't see it move around, but basically you can just change its viewport in order to get more information about the canvas. It's the same way that a coding agent might, you know, might open different files or grep or try to figure out, you know, in order to accomplish its goal. And you can imagine that that's like quite a different process than, you know, grepping because it needs to actually navigate spatially.

This was, we kind of like had our, you know, our take on agents on the canvas and the next thing that we built was fairies. So fairies is a really fun project. There's these little guys here, you can hold them. They don't like it when you hold them though. But you can, you know, you can throw them around. You can, obviously this is the first thing we do, you can configure their hats, their hair. This is really important. Yeah, you can, and this is again, this is going to be hard to see, but you can, we added a leg slider, of course.

But so in Asian, this was kind of, a lot of this was UX. So in Asian, the fairy is that agent on the canvas, but it exists kind of in the same way. So I'm going to say, I'll just do, I'm always telling it to draw a cat, but basically like you can, you know, tell it to do something and yeah, it'll, there we go. And it works on the canvas in order to kind of accomplish that. But its vision is always based on like where it is on the canvas.

So it has like, it's actually embodied. And also, you know, the animations are kind of silly, but you can actually tell what it's doing. So if you think about like, there's this output chat history, right? Which is like, you know, cloud code and you're trying to like read the commands that it's doing. It's really hard to parse even this too. It's a lot, but it's actually really easy to tell what it's doing just by looking at its animation and its state on the canvas.

Also, we realized that agents aren't just single agents; people have been doing a lot of collaboration stuff recently. Fairies have friends, basically. So you can select a couple of different fairies and make a group chat, telling them what to do. For instance, you can give the cat two friends and put them in a scene visually. The orchestrator agent is crucial in this process, creating tasks and assigning them to other agents based on their capabilities and task bounds.

Context management on a canvas plays a significant role in how fairies and the orchestrator agent operate. Fairies working on a task can't see anything outside of that task, enhancing their ability to manage their context effectively. Similarly, the orchestrator agent, although unable to create shapes on the canvas, excels in creating and assigning tasks to optimize work distribution. This approach minimizes confusion and streamlines the overall workflow for these collaborative agents.

And so, yeah, it's basically, okay, it's created one task to draw the scene background. And that agent, one of those agents is going to start working on it. But basically what's happening here is that like the orchestrator agent has no, does not have the ability to make shapes. It doesn't have the ability to do actually any work on the canvas. But what it can do is it can, you know, it can create tasks and it can assign tasks and it breaks up the work and then assign tasks, it assigns tasks to the other agents. And the other agents, they're only able to see what is inside their task bounds.

So this is kind of a something we realized through a lot of trial and error is that like you, and it's context management, but context management on a canvas looks really different. Because you know, if you have, I could have this whole, you know, presentation or a whole wireframe off to the side. The fairies can't know about that because it's going to confuse them. It's going to fill up their context.

And yeah, this is various, this is basically, you know, multi-agent collaboration on the canvas. This is now we're getting to the part where it's like in, in my talk, at least, this is the, this is the, the, we're getting to the boring part, which is actually also the cool part, but you know, all of this has been kind of fun, but like, it's, it's not really like useful yet. And so there's a couple of different things. There's two different things that, you know, we've been working on recently that are kind of like trying to bring this, you know, what would this actually look like in a, in a product.

And that does a lot to, to like, you know, improve its how it manages its context and same with the orchestrator. It can't make things not because we thought it would be funny, but because if you remove the ability to create shapes, that's so much like schema and tool calling that it doesn't have to think of. And so yeah, this is various, this is basically, you know, multi-agent collaboration on the canvas.

And, okay, this is now we're getting to the part where it's like in, in my talk, at least, this is the, this is the, the, we're getting to the boring part, which is actually also the cool part, but you know, all of this has been kind of fun, but like, it's, it's not really like useful yet. And so there's a couple of different things. There's two different things that, you know, we've been working on recently that are kind of like trying to bring this, you know, what would this actually look like in a, in a product.

So we have like some demos of how this could work. The first thing that I want to demo is the tech tree. So when we do a launch at TealDraw, you know, normally we use like our issue tracking software, it's, you know, linear or notion or whatever. But when we're getting close to a launch, we kind of abandoned all of that because we really need this kind of like high level view of like what's going on. So we just make a teal draw board and we basically have, you know, it's, we make a big graph with arrows that shows how, you know, what dependencies, what tasks, what dependencies different tasks have.

So, you know, I always thought that would be really cool if, you know, we could just assign like a coding agent to carry off those tasks, carry out those tasks when we do it. So if I assign this task to Claude here, I hit run. This is going to start a coding agent working on, on this task. You can see if I zoom out, this is like an entire, you know, project that has, you know, has had some, you know, some work going on it before. The project itself is this experiments thing. This is just kind of like testing a bunch of different inputs for, this is where it, yeah. So this is, this is kind of all, this is just like, you know, silly little demos, right? But this was all created by the, the agents themselves. I haven't looked at this repo at all. I haven't looked at any of the code.

You can see that this one is working. We, this PR needs to be reviewed. So this, this agent is open PR and now it's waiting. And so, yeah. That looks great. I'm going to merge that. Good, good job. And you'll see, you know, this will be, this will be finished in a second once the, you know, the polling takes a second. Well, we'll, we'll let the polling. Did that actually merge? Yes, it did. Okay. Well, okay. Oh, there we go. Okay. Yeah. It's finished. It just took a second for the GitHub update. So now that this task is finished, this implementation is finished. So I'm just going to hit run on this and we'll see if it finishes before we, you know, before we go. But, so this is kind of like, you're kind of like managing a project and also kind of like prompting your coding agents here. And I don't know about you guys, but like, I'm always, when I have a project, I have like five different clawed tabs open at once.

And they're all kind of like, they're like scattered. They're either in like my cursor ID or they're just an actual terminal window or something. And you know, looking at it, it's really hard, you know, they added that recap feature recently, but like, because it's so hard to tell what a clawed terminal thing actually represents when, you know, when you're coming back to it after a day.

So this is kind of like, you get a really, for a project, you get a really high level view of actually like what's going on, which I really like. You can also, this is another thing I like about the canvas is that the, you know, you, sometimes, you know, I don't like, I'm often like drawing a diagram of what I want an agent to do and screenshotting it and pasting it into tail draw. It's really, it's not great.

So this one kind of lets you just like take some, take some, take a diagram or something, just kind of like draw a task around it. And then I can do the same thing. I can just assign it to clawed and I can run. And if I go and I open it, that's kind of like, this is now the, you know, this is the prompt. Like this isn't, like, just this thing is the prompt for the coding agent and that's going to be working. Don't know if it'll finish. It seems like this one hasn't finished either, but yeah, so this is kind of like, how can we use this kind of like concept of agents on the canvas? How can we use the canvas primitives and like, you know, all of the things that the canvas gives you for free, which is like collaboration.

This is also collaborative, which is really cool. So if I, you know, I close this window, my colleague joins. They can just inspect what, because these, these agents aren't running on my computer. These are hosted agents. And so my colleague can inspect, like, you know, what's going on as well. And we can all be collaborating on a project. And this is kind of like what I was talking about earlier with, you know, agents like humans can collaborate on a canvas really easily. And we think that like agents is a great, is a great like, I don't know, way for it to do that.

So I'm trying to see if this one is finished or if it's hung. It seems, okay, it's doing a lot of stuff, but yeah, we don't, we didn't make it very, we didn't make it very easy to understand what they were doing yet. Yeah, okay. So this is, this is like the kind of like tech tree thing. Another thing that's really cool. So Tealdraw, we've been developing a desktop app that works on actual files, which I think is pretty interesting. And something you can do is you can drive the desktop app with cloud code.

So you don't need, you don't need to like hook into our like special infrastructure or anything. So what I already did was I gave, I made this, I gave this like a physics simulator. If you've ever used Tealdraw, you know that this is not a built-in behavior. But what I did is I just kind of told cloud code to give it gravity and WASD controls. And it did that. I'm going to, I'm going to tell it to to give, make this. I drew this out, but it doesn't do anything yet. I'm going to tell it to try to make this, this hat stuff. Hold on, wait, I have dictation. Turn the ball hat selector into an actual UI that changes what the ball, what the hat, the ball has on. Awesome. Let's see if it does that. It can be, you know, that'll take a while. But there's just a lot of like really cool stuff you can do with this. Because this is like working on, it's cloud code, so it has access to your entire computer.

So I'll do with this open code, oh, no, the ball, okay, well, the ball fell. But I'm going to tell open code, I'm going to say, look at my documents folder and for all of the directories, make a rectangle on the Tealdraw canvas that is size proportional to how big the folder size is in gigabytes. Yeah, sure. So this is kind of like, you'll see the, at some point an agent will spawn onto the canvas once it does this. But this is like, this is something that's really interesting because like you can make these kind of like interactive, like UIs and these interactive experiences just like from the canvas.

It seems like none of them have actually finished yet. And we're, oh, there we go. Okay. So this is basically the cloud code agent here. This is the open code agent, this is the cloud code agent here. And they're both kind of like, they're hooking in, they're using like the kind of like sub-agent lifestyle functions to actually render this UI on the canvas. So this is like, this is just like cloud code out of the box with, yeah, allow always. This is cloud code or open code out of the box with just like a sub-agent installed and, okay, here we go.

So yeah, we have all these, okay, I made a lot of these really small because I think there's text. But yeah, so these are all of the folders on my desktop that it kind of like, it created. This is, we're in the unplanned part of the demo. I don't know what the agents are doing right now. But the, yeah, so this is kind of like, again, this is where we are at TL draw. This is like agents on the canvas. We're basically trying to get it so that people can kind of bring their own agents. And that's what this kind of like work at the, with the desktop app is, is again, you don't have to hook into like for fairies or the agent starter, you don't have to hook into your own to our agent stuff. You can just kind of bring your own. And so yeah, we're trying to make it really easy to work with agents on the canvas. I would love to talk about it more. If anyone has any questions, I think maybe it's Q and A time. Thank you.

But can you maybe tell me more about the tail draw SDK? So I guess you were showing demos on like the teal draw app and then the desktop app, but I'm seeing all this stuff visual is really unlike my, my mind's racing. It's like, how can I do this and what I'm doing when I'm implementing? So does the SDK allow us to add it to our own apps? How does, how does that work? Yeah. So at the very, like heart of it, the children SDK basically just like your entry point is just a single react component, which is the teal draw component. And we basically allow you to kind of override or customize any of the behavior you want. So like you had custom shapes, custom tools. And we have like a, you know, replica just recently launched their new kind of like agent canvas UI that's entirely built on teal draw. So they've kind of like really customized that for their own needs. You know, they have custom shapes for their own, like, you know, prompts and their own like app rendering systems. But yeah, they're like the, the simplest version of making your own like teal drop is just importing the component and putting the component in it. And from there, yeah, we have a, that's kind of how you get started. And again, all of the besides the slideshow is just on the teal drop, but every single demo besides that was all just built on the SDK itself.

I see. So you have a custom app, you pulled in the SDK and now you have a, your own canvas that isn't teal draw, but you have all the same functionality. Yeah, yeah, exactly. And was there, and I might've missed it, but is there an MCP server that's giving cloud or open code access to it? So how are they getting direct access to that canvas? So we did a lot of experimentation with that. We actually, it's not really an MCP server. What it is is it's just the sub agent. So the, the desktop app itself exposes just like a local host server with a bunch of endpoints that and we just, the sub agent itself kind of like has all of these has instructions and we're just like, Hey, just curl it, you know, like we don't. Exactly. Yeah. So like, you can just, Hey, there's this server it's running on, you know starting on this port, curl it and you can get the, you can get all the, you know, and we have instructions in that as well. So we like, you know, you, if we wanted to standardize it, we could do it, but it's nice to just have the entry point of the sub agent is a lot nicer sometimes in the MCP app or the MCP. Sure. Cause you don't have to like front load all of the tools. No, that makes a lot of sense.

So we got some more questions coming in. Let's do this one. This is from Sam who says, is the agentic canvas available in the SDK? So the agentic canvas, it's not part of the SDK itself. However we do have the code, so a TLDraw is all source available. So you know, you can go on our GitHub and look at all the code for it.

The agent starter itself is part of the repo, however it's not part of the actual SDK. So you can go and take it as MIT licensed. You can copy as much of it as you want. And yeah, so it's available for you to, to look at and use and take inspiration from. There's a lot of like really cool subsystems in there. Like we have a linting system. I know this is not what the question asked. We have like a linting system. So if the, you know, the same way you would lint a code base or like lint a file, if the agent, because layout is one of the really hard questions for this. Again, like I was talking, the spatial, you know, how does it understand, like if, if text makes something overflowed, it's hard to make it understand that it's not going to be like, there's going to be overlap. So we have this system that like, once it's done generating, we look at all of the things that generated. And if there's two tax shapes that are like overlapping each other, we prompt the agent again. We're like, Hey, like, was that on purpose? If not, or if it was ignore it, but if it's not, you might want to like fix it and stuff. So there's a lot of like really cool kind of like tricks and stuff in there that we made that we use to make the agent canvas work. But yeah, that's on tldraw.dev or the tldraw GitHub repo. That makes sense. And there's also like patterns that you could adopt into your own code base rather than an SDK.

And the thing is that like this stuff is moving so fast. Like again, the fairies was, um, you know, fairies we, we shipped, uh, in the tldraw.com app is like a limited free time thing in December, but there are some of the things that we did there that are already, like we wouldn't do again if we had to build it again. And so we're kind of like, we don't really want to put all any of this stuff in the SDK just yet because like we really want to make sure that it's the right solution. And I know it's, it's a lot of like, it's still lots of experimentation right now and we're trying to nail that down. That makes total sense.

A fun one here. Can you make the agents sabotage each other? You can actually say something that, um, that, uh, I, I did, which was really fun is you're playing like turn based games with it. So I made like, I made a maze once and I was like, um, I was like, okay, me and you were going to take turns completing this maze. There's like two characters or whatever. One for me, one for the agent. And um, and I would, I would be cheating and I would like be skipping over walls or something. And it would be like, it would be like, I don't understand how the other one is.

Another funny thing you can do is you can say, uh, tell two agents to both try to draw something in the same spot. And um, for the fairies, they know that there's other fairies on the canvas and they, they, they don't get confused, which is again, part of the, the work that we did for the collaboration is they know where each of the other fairies are working. But for the agents, if we have two of them, um, they would make something and then they'd review their work and they get really confused why there were two things there and then they would delete all of that stuff. And then the other agent would review their work and be like, wait, somebody deleted all of my stuff. So they would do it again. So yes. So they didn't know, like they didn't know who wrote it or who put that object there and we're getting confused. Yeah. Yeah. A learning from that is actually, we, we also, we send all of the user actions that the user has done on the canvas since the last prompt we send to the agents. Um, so you can be like, Oh wait, can you bring back that shape? I just deleted it. Notice what shape that you just deleted, for example. It's like a built-in undo almost.

How do you manage the token usage when working with the canvas? It seems like it might require a lot of context to work. So what kind of data are we passing back and forth? Is it JSON objects? Is it, yeah. How are we, how's the token usage? Uh, it's a lot. Um, no, it's a lot of tokens. It's, um, I mean, one of the things images I feel like don't, aren't that bad now. Um, in the, some of the earlier experimentation we were doing, images required like a lot of tokens and were like much more expensive to process now. Not so much. Um, but yeah, we something that we've been doing to manage that though, is we've kind of been adopting this, like, uh, instead of just, we have this like massive system prompt for fairies, as you can imagine, all of it, like the schemas for the shapes and everything, something we've been doing with the more recent stuff is we just let it read the docs.

Um, but yeah, we something that we've been doing to manage that though, is we've kind of been adopting this, like, uh, instead of just, we have this like massive system prompt for fairies, as you can imagine, all of it, like the schemas for the shapes and everything, something we've been doing with the more recent stuff is we just let it read the docs. And so it kind of like an, you know, RLM type thing where we, um, we say, Hey, make us make a square or whatever. And it basically can like query the docs for the schema for a square. And so it's only ever getting, you know, and then it queries the docs for asking, how do I create a shape? And so it's only we're allowing it to only get the information that it needs and add some latency of course. But as far as token usage, it's a massive help. That's good.

I think this, this next question is related, um, in terms of, of data you should. So are the LLMs using JSON schemas or XML div like output? The latter might be efficient. Do the large amount of HTML, LLMs, the large amount of HTML that LLMs have been trained on. Yeah, we're HTML is actually really interesting. We haven't tried that. The, um, the, we've actually, we've gone through a bunch of things. So teach the first thing I showed, which was the single shot that uses an HTML like thing for the actions that it outputs, um, for the agent starter and for fairies that's using JSON.

And then for the desktop app stuff, um, what that's actually doing is it's writing, uh, uh, like JavaScript against the editor and just running it locally in the editor instance. So if anyone's used code mode, uh, or, or similar tools, it's basically, instead of giving the agent a set of actions that we, you know, allow it to choose from, we're basically like, Hey, here's the entire TL draw editor SDK. Um, and here's the docs and you can just like write, write JavaScript that we'll evaluate in there. And you know, that's, it's, it's all running locally. So like security wise, it's, it's fine for the desktop app at least. But, but yeah, so that's another thing that is, you know, if you're, if you want it to make a thousand shapes before it would be a thousand tool calls or whatever, but this one is just, you know, a four loop. So you can, as far as token usage and stuff, you can get a lot of gains from that as well. I like that. That's a good strat.

All right. Only time for a couple more questions. Uh, here's a fun one. Um, uh, well I lost it, uh, but here's this one. So what AI library, if any, are you using for charts and TL draw? So is it, are the, are there any chart libraries or is it all built in shapes? So we've got, it's all shapes, it's all shapes. Okay. Yeah. Yeah. Yeah. There's no, um, I mean that would be something interesting, you know, you could, if you wanted to have the agents do data viz, you could kind of bring in your own chart library and have them fill them out. Yeah. Um, but yeah, these are all just shapes. I think I misread it. It's chats, not, not shapes.

So are you using like AI SDK or Tan Stack AI to like, Oh yeah. In the, so for fairies and the agent starter, we're using the AI, the Vercel AI SDK. Um, and, but we're not, we're not like hooking into that too much, like the chat component and everything. That's all custom, um, because of how we wanted to, uh, like represent the, the UX for the agents and for the agent starter it is, or, or for the, the desktop app, I think we're using, no, we're, we're not using anything for that. We're just kind of like rolling it ourselves.

Okay. That's all the time we have for questions, but thank you so much, Max. Thank you. Yeah. Thank you. Bye.