Every API is a Tool for Agents with Code Mode

Welcome to this session on APIs, Codemode, and MCP. So I guess the thesis for this is that every API is going to be a tool for agents. And we'll talk about what that means a little bit in the future. But my name is Matt. I work on agents and MCP at Cloudflare. And welcome to this talk.

My job is really, really fun because I get to work out what does the future of this stuff look like? And I get to test it and play with it. And a lot of the times, it reminds me of this meme. So I thought I'd lighten the mood with the meme. So, if a dog wore pants, would he wear them like this or like this? I have no idea. But I guess it would be cool to experiment with one or the other. The right one looks like his legs might get cold. But you never know. But this is actually what I spend my day-to-day working on.

So, like, giving agents hands. How do we let AI, which has got so useful in the last few years, how do we let it access stuff outside of the environment that we've contained it in? How do we let it access stuff outside the glass box? This might be making, like, API calls. This might be using a computer. They're doing all of this stuff. But it basically boils down into what was called function calling and now a lot of people call tool calling, giving models tools. A user asks for something, the LLM replies and decides to call this, like, tool.

This is a tool that the user will then execute and then come back and give the LLM the result. Tool calling. And I'm sure you've seen a bunch of this stuff maybe, like, in chat GPT or whatever. But this is how, in general, we give agents hands to access stuff in the outside world. We've been doing this long enough now, a few years, that we have had some new things come up. We started with just giving agents tools inside their own applications. So, each developer would write their own tools, give it to their own agent, their own LLM in a loop, and they would execute the tools when it was needed.

And this is pretty useful for your own application. But as CloudFlare, as a company, a producer of things like APIs and a producer of a platform, we want to let our customers use our platform via these, like, LLMs, these agents. And for that, we need to, it would be really good to publish, like, more standardized set of tools that everyone else could use. And throughout some of this process, MCP was born in November of 2024, already, which is, like, a really long time ago, of what feels like yesterday. But the idea was that you could take tools and other primitives, like, resources and prompts, but mostly people have used tools, and you could host them in a server somewhere else.

And then any agent that wanted to use those tools could dynamically register with that authentication, use the tools, and say, thank you very much, really. And from a server developer point of view, it's, like, how do we share tools with, like, agents we've never met, with people we've never met? And this is not that normal on the internet. So, there was some new flow with new auth, DCR, dynamic client registration that had to be developed a little bit more. And now we're actually moving to a thing later called SIMD, Client ID Metadata Documents, like, just, like, ways that we allow agents to access stuff for things that they have no interaction with.

And it's a different type of contract from, like, an API to a client usage, because when you make that contract, you say that the API should never change. Because I hear, like, the MCP server could change their tools whenever they want. They're dynamically registered. So, it makes them very useful. But this is not really the point of the talk. This is a little bit of a background. We went from bundle tools in each application to a lot of tools being remote, shared via MCP, Model Context Protocol. And then, very quickly, we filled context windows. You've probably seen, like, memes, like, news articles or news in general about, like, there are some MCP servers that fill huge amounts of context.

For instance, the whole Cloudflare's open API spec is 2.3 million tokens. This is huge. This is way bigger than could ever fit in an LLM context window. They fluctuate somewhere between about 30 to 50,000 tokens, all the way up to, like, 1 or 2 million at the top end. But this is way bigger. And even if we stripped out a bunch of stuff and we converted it to the most bare-bones tool spec, it could still be about 1.1 million tokens. So, with this process giving agents tools, agents can't access the Cloudflare API. We end up having to do things like splitting it up, splitting these MCP servers up.

And actually, in May last year, we've added, or April last year, we added to it since then, we released about 13 MCP servers for different products. Customers can add them to their agent, use the products, then disable them or re-enable them as and when needed. But this, like, I think maybe this is a problem. And this is kind of the point of the talk. This context limit is not inherently an MCP problem. It's an agent problem. And we haven't learned the relevant techniques in order to shrink this enough to make it usable. We need progressive disclosure of tools.

So, you could accomplish this in a bunch of different ways. I'm sure if I do this talk in six months or a year's time, there will be another three ways at the bottom that we can accomplish this. But the main things are CLI bash, where you generate a CLI or you create a CLI for your API or service and let the agent install that CLI. CLIs normally have a dash dash help command. And this progressively discloses, like, different capabilities to the agent. They don't have to load it all into the context window initially. You can also do things like, oh, so the CLI is what OpenCore uses and other personal agents.

You can also use other stuff like tool search. So, tool search has been used in code, I'm pretty sure in cursor as well. And they do some sort of semantic search, BM25 search over tool descriptions. And then they load the tool that the customer needs on demand. That's cool as well. I'm going to propose a third method. And it's one we wrote a blog post about in the summer last year that caused a bit of a stir called CodeMode. And then Anthopic came out with something called programmatic tool execution, which is a similar thing. And recently I wrote another blog post about how we can do CodeMode with a MCP service. So, this is the point of the talk. But the main idea is that we think tools should be discovered on demand, not dumped into context.

Let's test it. Let's make an example. Let's build this project management API. So, I actually generated all of this, project management API. It's running on workers, using D1 as a little database. It's just CRUD operations. So, there's no frontend. So, we're going to basically build all the frontends. The agents don't need a frontend, they just need an API. And we can play with it. So, there's 20 tools here. Well, 20 endpoints, for instance. So, you have things to do with projects, sprints, tasks, maybe comments. And you can do get, post, patch, delete on pretty much all of these resources. Very restful. This is like a standard thing you might end up with. You can see how it can very quickly spiral into a lot of tools. So, 20 endpoints.

Let's try with bash. So, this is a little terminal that's connected to a sandbox. And I don't think there's anything here. But if we do a PM, we preloaded this. So, this is just generated CLI that can call endpoints on the SaaS service that we just created before. So, we can do something like PM projects, create, and then like test project. Oh, I need a name. And although I made that mistake there, agents are very unlikely to make that mistake. Because there was an example here which said it quite nicely. Okay. Cool. So, we create a project. And now if we do PM projects list, in just one second, we get the project back. So, like, you can see this works. And an agent could learn to use this quite nicely. You might have a skill or something like this in order to tell it to initially call PM. Because it was very hard to find that on the workspace itself. And that's pretty good, right? Like, each of these examples happens. If we do PM projects, so, we end up in projects, then we should be able to do help. Yeah. Cool.

It's like self-documenting. The agent can call all of this stuff, work through the process, and find out how to do stuff. We don't have the overhead of dumping all the tools into the content, which works well. OpenCore and various personal agents use this approach.

Tool search involves user intent put into a search, searching across it, and adding top results to the context. The model can call one of these tools, but the effectiveness depends on search quality and tool documentation. Tool search may pose challenges with variations in task wording and result relevance.

CodeMode combines CLI flexibility with tool search portability. It involves executing LLM generated code in a sandboxed VA isolate and dispatching tools securely. CodeMode generates a typed SDK from API specs, creating input schema types like create project input and output.

So, what you would do is the user intent would get put into some VM25 search. And then we would do some search across it. And then we would add, like, the top three. Like, the K equals three. We would add those to the context window. And then the model would have the ability to call one of these three tools. Or, like, or more, for instance.

Now, we've created a project. Now, maybe we want to list the projects. We would add, like, another couple, another few tools. But you see now, we're already at, like, 17,020 tokens. Like, this is quite a lot of tokens in the context window already. And we've only just added a few tools. And we've only, all we've done is create a project and then try and list projects. So, you're very beholden on how good your search is, how well your tools are documented.

For instance, maybe I want to create a task. Create a task, create a task in a project. But maybe I want to, like, start a task or something slightly different that the wording is not the same. Now it's not entirely obvious what I would do. So, the actual answer to how I would start a task or set a task as started is actually to get the task and then update the task. But you see these don't actually appear in the top three. So, you might have, like, worse results. And people often find this with tool search. It's like not the best for this type of thing. You have to have very good descriptions. Descriptions often including examples as well.

Cool. Yeah. So, that's tool search. And now, we're proposing something slightly different, which kind of combines the flexibility of a CLI with the portability of just a simple tool search. And this is code mode. So, this was from the blog post that came out in the summer. But where the agent executes LLM generated code in a sandboxed VA isolate and we dispatch tools back to the host. There's a lot done here to make it very secure. We'll talk about that in a little bit. But what is code mode? First, we generate a typed SDK, and this is cool.

So, from this, like, get API projects, we generate these types. They come straight from the open API spec and we write them in TypeScript. And if you want to create a project, you see this create project input, create project output. And these types act like an input schema you might see in a tool. Except they're a little bit less structured, potentially. But they basically act very, very similar. We can generate these from a JSON schema in an open API spec or from a schema and we can put this in a context window. So, that's kind of cool. We can generate these.

And then we can actually just call, so once we've generated the types, those types live in this SDK that we generate. And then we just call methods on the SDK and the model knows what methods it can call because it has the types and the types relate to the methods in a way that it can interpret. So, for instance, if we wanted to list all projects, the agent would generate this code, this project's code mode list projects. Maybe console log something, it doesn't really matter and then return the projects with name and ID. So, what happened here? We sent the code to this VA isolate, we called this list projects, we found one project, the project we just created earlier. This is all running on the API that we created right at the beginning. So, all of these processes, all of these things are running directly on top of this imaginary SaaS company we made at the beginning, this project management SaaS company. Think of it like a JIRA or a linear, for instance.

So, here we list projects. But now we want to create a new project. So, we ask the agent, can you create a project? And it generates code something similar to this. And then we run that code. And right now you're probably like, oh, my God, you're just running generated code. You're just running like generated untrusted code. This is super, super scary. And I agree with you. This is super, super scary. And we'll talk about how we make this less scary and why I think this is the future. But you can see here that we've just created a project.

And now maybe want to create a task. So, task, ship code mode, status critical. And it created a task. That's pretty nice. And we can assign a task. Let's assign it to me. Beautiful, beautiful, beautiful. And this is all editable. Say we wanted to console log like hello node or something like this. Oh, no. Remember not to use the arrow keys. If we wanted to run this. Oh, I'm missing. It has to be valid typescript. And this is just code that we can generate. And now we can run.

But how do we run it? Oh, sorry. Let's go back a little bit. First, why is this really, really cool? So, we've done all these different things here. But all of these could be mapped to an individual tool. And you'd say, Matt, we could have four tools, each one of these things. And sure, you could do this whole flow. And I'm like, yeah, sure, you could do that whole flow. But why if the intent of the user is to create a project, create a task in that project and then assign it to me? First of all, they don't need to list projects initially. The AI should know it could do that. And secondly, all of these three things, they don't have to be separate operations. They could be done in one. Oh, Matt, why don't we make a singular tool for this. Maybe I don't want to assign the last task. I want to assign it to someone else. Or I want to change the workflow slightly.

And now it becomes brittle. How do we make it less brittle? Code is a good representation of a control flow. Especially, I think, TypeScript looks pretty good. Or in this case, JavaScript looks pretty good. You can write Python in here. You can write what you want. But code has been built over the years to be a very natural, compact plan. So, instead of doing this very bulky tool calling where we have to call this one, call this one, call this one, call this one, and call this one, we can do one round trip and do the whole thing in one go.

The AI knows what it can do because it has all of the types. So, it has that ability already. And it can just do it all in one go. And you'll see like some like funky emergent properties of doing this. Like I was just playing with it earlier, asking it to delete some projects that I hadn't used in a while. And it looked up all my projects. It found the ones from over a few months ago. And it was like, do you want them over six months ago? I was like, yeah, over six months ago sounds good. It sorted, it ordered them, and then it just deleted them all in one shot. And like it can write a singular piece of code to do that. Which I think is really, really cool. And I guess the last thing about why is it's just hugely token efficient to do this.

So, like disregarding the fact it's so composable, it's like massively more token efficient. If we add a search and execute to the top of the code mode SDK. So, in search, it can search over the types or some sort of structured spec. And in execute, it can just write code to call the type SDK. In search, it also just writes code. There's no like static search function. It just writes code. Then we have the benefits of like Bash, of being able to be like composable and stuff and call different things and chain together operations. But we have the tool efficiency of code search, of tool search. And we also have some portability here as well. Because we don't actually need a full VM to do this. We're not running Bash. We're just running TypeScript. And in TypeScript, you could run things in very, very lightweight isolates. That's how we're running this. LLMs are better at writing code.

All of these slides are pretty generated. I think that came from one of our blog posts. I think that's the last one. And this is true. They're much more token efficient. We did some evals with brain trust I can't really share here. But much more token efficient. There's lots of people who have been testing it since our last blog post came out. But yeah, feel free, send me some results.

They're much better at writing code to call MCP than just calling MCP directly. That's because the idea of calling tools is a very bulky idea. You have to encapsulate one action into this tool. Why don't we just write code? Just write code. That's cool. But why haven't people written code before? Because generating untrusted code is super scary. It is mentally scary where you're like, okay, all of our customers can write whatever code they want and we're going to run it for them. We're not going to look at it. No one is going to look at it. We're just going to run it.

It might not be fine if you do that. And this is why people haven't done this in the past. It led to a lot of crazy situations. A lot of people have built their own domain-specific languages, their own query languages. You see most of the big SaaS companies have their own query language. Why don't they just let you write code against their internal APIs? Because they want to restrict what you can do. They want to have actions and have permissions policy and have all of this stuff be able to run it in some sort of engine that they control. And previously that was really, really tough. If you allowed them to just write straight JavaScript or just straight code and they executed that code, then what could you do?

You could grab .env variables. That would be bad. You could make network requests to Trojanhorse.com and then download all of that software and just run it in a sub-process. That would also be really bad. You could effectively denial of service attack them from inside their own compute environment. That would also be really bad. You could just run infinite loots. You could run crypto miners. You could run all this stuff. And you could also bring down their whole service by out of memory that process and any other process that they put you on. All of these things would be really, really bad. And people have tried lots of other things to try and restrict this. So DSL has been the popular one, but Docker in some way also does this. And these are also unlike VMs, like AWS Lambda is a VM. It uses Firecracker under the hood.

And that's literally what they're doing. They're preventing you from writing stuff that might break their system. And this is not super scalable, but human verification is also an option. And these are all just various versions of a sandbox. And I guess the reason why we're able to do code mode and we're very popular, we're very bullish on code mode, is we think we have a very, very good sandbox for running untrusted code. And this is dynamic worker loaders. Dynamic worker loaders, super catchy name, but basically you're just running a Cloudflare worker from a script. So actually, very similarly to how we run Cloudflare workers under the hood, you give us some sort of module or you could have multiple modules.

And then in this case, it's just generated code. And then you can just run that code in another worker that you spawned from the host worker. And so it's super isolated. We have almost like instant cold starts, like VA isolates start very, very quickly. Yeah. It's super isolated. There's no file system that it can access apart from its own code. And even that, it's only accessible in certain scenarios. You have full control over the network outbound. So you can block all fetch requests. You can make it completely isolated from the public internet. Or you can inspect each fetch request. Like this global outbound function here allows you to inspect each network request that goes out. Say it meets a certain criteria, hitting your SaaS platform, if not, or even just hitting certain endpoints, depending on the user. You have that full control there. Each isolates has its own memory.

Each isolates has its own memory. You can also give it external tools via bindings, via Cloudflare bindings, rather than just letting it call fetch. And so you have more control there as well. But essentially, we think these are a really, really good primitive for running untrusted on-demand generated code. And we can play with it a little bit now.

So this is just simple Fibonacci sequence. And we can run it. And this is it running code right now. I think this is cool. And just so, I'm not cheating or anything. We could just return Matt's. Hey. Oh, I need to remember not to use the arrow keys. It's a very hard habit to get out of. But if we run that now, we get Matt hello.

This is not executing in the browser. I'm running the slides in the browser. They're all React. This is not executing in the browser. This all goes to our server. It goes to our worker. And this code, as a string, is dispatched to a dynamic worker and ran in a dynamic worker. What does this mean? This means that if we try and do a fetch request, by default, it's blocked. And this isn't because we're patching global fetch or something. No, the actual host has no, we inspect all of the outgoing calls. And fetch here is just blocked. Completely blocked.

But maybe we're going to allow it, in this case. And now we can allow it. So this is like HTTP.org. I think this is a sample JSON people use. I don't know. Claude found it for me. But this is super powerful. Because now we can do things like put this in front of our Cloudflare API and have absolute knowledge. The only way people can use this service is to call the Cloudflare API. Previously, this would have been super, super tough. And this is an infrastructure primitive that has all this stuff out of the box. And there's a bunch of other stuff, you should have a play with dynamic worker loaders. I think they're really, really cool. VA isolates for untrusted code, super fast cold start. All of the good stuff. But with really good isolation. And just like a better sandbox.

So we're getting to the end of the talk now. So there's a little bit of a demo here. And what we're demoing here is back, when we went back to the beginning, we had this like the project management API. And we said that we could do code over the project management API, so we could generate a TypeScript SDK. And we could give that to the agent. So that's what we're doing here. And just like a very little chat interface. So we can be like, what projects do we have? And if we just inspect the tool calls here. We're still moving. We just called code mode.listprojects. And it's written code here. And it's returned these two projects that we have. So we have the test project that we created initially via the sandbox, via the CLI. And then we have the Node Congress project that we created via the dynamic worker earlier. And all these were created there. Yeah, cool. So you can show the tasks.

How about you show us tasks in the Node Congress project. Cool. So this is the task. So the task was to ship code mode. Cool. So how about we say something like this is done, delete it. But next step is we need to make this work for the Cloudflare API, not just some random SAS maker new ticket for that assigned map comment. Let's go. So previously this would have been a lot of tool calls. I mean, I kind of went a little bit overboard just to demonstrate that. But previously this would have been delete this task, make a new ticket, that's two already, then assign me maybe another one, and then comment. So that's like four. What is it now? It's one code mode tool where we write some code. And it does it all in one go. That's pretty cool.

That's pretty cool. So we updated this one, we created this one, and we commented on it. Sick, right? We marked the code mode as done. Okay. Marked it as done rather than deleting it. Thank you so much. That's very kind of you, saving all of my data. But like all this happened in one tool call. So not only are we massively more token efficient on stuff that goes into the LLM, we're more token efficient on the output. These steps, none of them had to get returned to the model. We saved a lot of tokens here just throughout the process. And we'll do some benchmarks on this later, but it's almost so overwhelming that it's kind of, once you try it, you're like, oh, my God, this is just better. But we will share some benchmarks around this for general use cases. Cool.

So this was doing code mode on that project management API. But we started this thing talking about MCP. How can we take what we've done here with code mode on just on an API that we own and tools that we own and make them remote. So say Cloudflare as a platform creates an API, well, we have an API, huge one, 2.3 million tokens. But we want an MCP server just for the whole of the Cloudflare API. But we don't want users, we don't want user agents, if I'm using that word, but we don't want user agents to have to understand how code mode works or implement it even to their agents because it is a lot of overhead. Building an MCP client is already outrageously difficult compared to building an MCP server. We have some expertise in this. We're going to try to bring some of this complexity to the server. And this is what our last blog post was about. And it went pretty successfully because people liked it. It means they can make very, very simple MCP clients or agents, just an LM in a loop that connects to an MCP server for its tools. And they don't have to worry about all of this stuff because we handle all of it on the server.

What do we do? The agent writes some code. It calls our MCP server, our Cloudflare MCP server has two tools, not just one code mode tool because our API is still actually too big. We still need two tools. We need to allow some progressive disclosure, similarly to tool search. So search and execute. And then the LM can call search or execute, call search to get an idea of what it needs to call for execute, calls execute to actually run the code. And then for both of these tools, this V8 isolate dynamic work loader runs the code. And in that it calls tools via RPC, using a protocol called CapMemWeb, which is super, super cool, but a story for a whole other talk. I think Kenton is speaking at this conference. So yeah, go to his talk to understand more about CapMemWeb. But that is actually some of the magic underlying all of this. Server controls the execution. Agents stay super simple. MCP protocol is preserved. We're not trying to reinvent MCP. We're just allowing it to work for massive APIs. And we're actually just allowing the tool paradigm to work for massive APIs full stop. So let's do our last little demo, our last little demo. Let's try and list some workers on my CloudFlare account. I use this MCP server. This is a very little demo. These ideas are not secrets or anything. But we're going to use map.

So we're going to see what workers are on my account. This is my private account. Or my non-work account. And so we list some workers. Oh, I should have closed that. Okay. These are all the workers on my account. Cool. Epic. They're mostly just demos. But let's make okay. So what do we do? We list some workers. It asks what account we use. And then it's written some code to call the it's written some code what's it down here? It's written some code and it's using this thing called spec.path. So in it's annoying it says tool here. This is actually calling the search tool. And in spec, it has access to the whole open API spec of the CloudFlare API. And so it's filtering through some paths. It's looking for the worker scripts end point. It's looking for it. It's found it. It's all the data about all my workers. And that's it. And then when it exits then it's actually calling that end point here. So it makes these two tools. Two tool calls. And it's called the end point. It's got all the data back of all my workers. Sweet. Right. That was cool. That's a very redone use case.

What about if we want agents to take action on CloudFlare on the CloudFlare platform? This is cool because now we can build infrastructure at a global scale if this works. But let's make a hello world worker. Cool. And we'll just like see what this does. Let's just call it hello world. So we're going to build a worker. And I'm going to start writing some stuff because this takes a second. I know we don't have that long left. But let's the next thing we're going to try and do is connect up to this worker. So we can have a visitor counter. So just using visitor counter. So just using another primitive in CloudFlare, kb is like a key value store. So we're going to try and connect that up to this worker. Just create a very, very basic visitor counter on the worker. Is it deployed? It deployed it. Give me a URL. So I'm just asking stuff in English. I'm not asking for very specific things.

Oh, we had an error. Let's go. Cool. Let's yeah. I don't think it read what I did before. So workers.dev domain. Give me one. Let's go. I want to show you all of this thing running. It's really good. It's really interesting here. It's a simple counter using kb. So concurrent traffic is not perfectly atomic. For a demo, it's fine. If you want an actual counter, I recommend durable objects instead. I like durable objects, too. Thank you. Cool.

That's rogue. It hasn't told me a thing. I know it's Matt Sedkeri. Let's put this out here. Demo, eh? Cool. We've got a visitor counter. Sick. That works. Hopefully, you can still see my screen. Nice. It hasn't deduped the favicon. Chrome always calls for a favicon. But we've got it. We're getting to each time. That's cool. Let's go a little bit more wild. Yes. Add the durable object. Allow me to write messages on this wall. Make it local first. And synci. And now let's just go a little bit more wild for the last demo left. Just do it. A lot of times I use cloud with this and it doesn't ask questions. It just goes and does it. But we'll see if this is going to work. Sorry, my UI is like freaking out. I haven't tested it with this much stuff. We might have to end it here. We'll give it one last little attempt to try to do this.

But I would, while this is running, I would urge you to have, if you are interested in code mode, in how you can give the API, a massive API to an agent, I would, yeah, have a little look at this last blog post we published. And if you want to try the MCP server yourself in an agent that's a little bit better than the demo one I just made now, maybe Cloud Code or Open Code or something actually really good, then try in mcp.Cloudflow.com. That's the URL for the server. If you want to play with code mode yourself and you want to put code mode on top of your own MCP servers, then we release an SDK for this. We're hiring for my team, agents team, really, really good fun. Or feel free to connect with me on Twitter as well. Yeah, we'll just double check to see how it's doing. I just asked it to do too much stuff in one go. That was silly me. My UI is so broken.

Yeah, all of these slides were generated. We have an internal slide generation tool called slides, I think, .Cloudflare.com, and it all uses Cloudflare sandboxes under the hood. It's very, very good fun. Yeah, some of the new models are really, really good at making slides. It's just React. But yeah, thank you so much. And yeah, feel free to reach out if you have any questions. Nice to meet you all. Bye.