Designing Sandboxed Dev Environments for Coding Agents

I am a big fan of demos, so let's start with a demo. Here it is a very silly pixel animated goose. And I want to make this goose follow my cursor. So let's ask an agent to do that. What's happening behind the scenes here is there is an environment that has spun up. The dev server has started. We have a preview URL. The agent is going through the code base that I had. It's thinking about how to go about implementing this change. And once it reasons with that change, ideally it should be making that change, and we should be able to see it. There we go. This talk is about all the different things that have happened to make this demo work. The most important thing to identify here is that there's no local host here, right? So all of this, it's a real demo. It's running somewhere, and the agent and this preview URL that we use to look at this work had to find its way from the internet, from my browser, to a very specific port in a specific environment. And if I close this tab and come back tomorrow, ideally the agent should be able to pick up where it left off. That's what we will try to understand how it works. So this preview URL is a computer. Of course, it's a server, but a computer is a nicer way to say it, and I like that. And every agent needs a computer. I think we have come to a point where models have become good enough to a point where long-running agents are now real. They are able to do a lot of very powerful things, and the computer that we give it also needs to be designed in a very specific way for it to be useful for the agent.

I'm Naresh. In my past life, I worked on coding agents that twice dropped FreeBench, and now I build the sandbox environments that they run in. Let's actually talk about what are all the things that just happened in that demo that we just saw and try to break it down into different layers. The first thing is the isolation boundary itself. There's a micro VM, its own kernel, own network stack, there's no shared kernel attack surface with anything else running on the same machine. And then there is a container that's running inside it. So, you know, it's running Ubuntu, OS we are very familiar with. Installed, or rather, there's a bunch of dev tools that we commonly use, pre-installed.

Then there are batch sessions and a file system. This is where the agent runs commands. It reads and writes files, installs packages, all the things that you and I would do whenever we had to work on a code base like this. And then there is the networking bit. The agent probably ran NPM start inside the container. It's running on an ephemeral port, but getting the traffic from the agent and our browser to that very particular port is slightly non-obvious. And finally, there is persistence. So, all the files that are installed, all the changes made by the agent, they need to survive all of these sessions.

We will actually go through some of the design decisions that go into doing two things. One, to build a sandbox environment like this. And two, how to use a sandbox environment like this to build whatever platforms or products that you are interested in building. Let's start with the runtime. What does the agent actually get to work with? Agents ideally need the same tools that developers need. You could technically just give it a very small repel and say, okay, run the code that you need, but it's super restricted. Ideally, we want to give it a full Linux instance, a real computer, so that it can actually perform non-trivial complex operations.

But the challenge is, how much of the heavy lifting to do a lot of these orchestration can the sandbox itself do? Because the less the sandbox does, the more your agent has to. The table stakes are very simple, right? This is the baseline. Run some commands, read and write files, clone repos, install packages, every sandbox, well, every container has this, and it's necessary. But what actually separates CodeRunner from a real-time environment is the next layer. The agent needs to be able to start background processes. In this demo, the dev server was an example of a background process. It needs to keep working in the background, streaming so that the agent can react to output as it happens, not after the command exits.

Continuity, when we work on a terminal, a bash session, basically makes sure that all the commands that we run are sequentially happening, depending on how you use bash, really. But the agent should have the same kind of capability. And it's not just a single session. Ideally, it should be able to manage multiple sessions within the same sandbox. When you want to build interactive experiences, when you want to give your users access to the sandbox environment so that they can also maybe look at what the agent has been up to, or even directly run commands of their own. You ideally want to maybe connect an XGEM instance.

The important thing to remember is that you are running code that you didn't write and executing instructions from a user who you don't control and you don't want to control. So that means that the attack surface isn't just the code, it's everything the agent can reach. And the paradox is that we ideally we want the agent to be able to do a lot and reach a lot of surfaces. But it still needs to have some guardrails around it. So the common problems basically that you're thinking about is, well, what packages did then agent install? Was it prompted to install it? Are there any possibilities of prompt injection that could impact what the agent actually does inside this environment?

And the paradox is that we ideally we want the agent to be able to do a lot and reach a lot of surfaces. But it still needs to have some guardrails around it. So the common problems basically that you're thinking about is, well, what packages did then agent install? Was it prompted to install it? Are there any possibilities of prompt injection that could impact what the agent actually does inside this environment? You don't know what the user actually asked your agent to build, and every message actually shapes the decisions that your agent is actually taking. You don't review it, so you don't control it. And finally, you also don't have full control over what the agent reads before it writes the code. So, you know, it's reading files in a repo, web pages, a previous turn in the conversation, all of it shapes its output. And all of this actually increases the attack vectors. And this is the reason why isolation isn't optional. And the way we do it matters. So at this point, I think we're all familiar with the idea of, well, actually, let's just run it in a Docker container. Sure. But what's wrapping that container itself? What's the boundary? The simplest way is to just name spaces, shared kernel with the host. It's fast, lightweight, but every syscall from every container hits the same kernel, which is fine for your own code, but not for arbitrary agent-generated code. And the other end of the spectrum is full VMs, wherein you get genuine isolation. But it comes with 100 MB of overhead, five to 30 seconds, cold boot times, all of which is actually really impractical at scale. Micro VMs are actually the middle ground, and I actually don't like calling it a middle ground because they are actually better than both the different directions. They come with very, very light overhead. They are extremely fast to start. They have their own kernel per sandbox. I think the most common implementation is Firecracker, which was built by AWS for Lambda. And it is specifically designed for this hardware level isolation without the full VM overhead. This highlighted column is ideally what you want to use for your sandboxes. The third bit is networking. How does the agent and the browser find the right sandbox? Let's look at this URL. The subdomain in this URL actually contains a couple of things. It contains a port. It contains a unique session token. But there's here that's actually pointing to a specific machine, right? And that's intentional because you can't pre-provision DNS per sandbox. You can't just assign a stable IP. So every time when the request comes in, the machine needs to be resolved by the platform.

So every time when the request comes in, the machine needs to be resolved by the platform. Extract the relevant information either directly from here or through an internal map. Do a fast lookup and open a TCP connection to it. All of this needs to happen behind the scenes.

And the second bit is even after the request reaches the right machine, you also want to make sure that it's connecting to the right port. And it's also established in such a way that, for instance, again, dev servers, they actually do hot reloading, which means there's a persistent WebSocket connection and changes are actually going back and forth all the time. This needs to just consistently work without any additional setup, right?

And the last bit is persistence. And I think this is actually the most important because persistence is not just about what you need to store and restore, but also the way you think about persistence impacts cold starts, right? The reason is, or rather, let's actually start with thinking about, well, what are the two things that dominate cold starts, right? One part of it is the infrastructure itself that needs to come up. How long does it take for the VM to exist? How long before you can actually connect to it? That's one part of it.

How long does it take for the VM to exist? How long before you can actually connect to it? That's one part of it. And the second is actually the application readiness, right? So when a user comes in, and they're starting a fresh session or they're continuing from an older session, you want your application to reach the state wherein your agent can actually immediately continue working. This is actually the gap that the developer is going to feel because this is where things can actually become really slow. And this is the part where you need to think about, okay, how do I optimize this? There are a few ways. The simplest, easiest to win is pre-built images, right? So you can actually build an image, make sure that it has all the code bases, dependencies, everything that you need. You can also maybe keep rebuilding it in the background if things keep changing. And this is the image that actually puts up within the VM. So that's one thing. The second is if you actually have slightly more dynamic needs, maybe you can't always bake everything into an image, then you use persistent volumes. So you can actually mount these volumes into these different ephemeral containers. And that way, you know, all the data survives, restarts as well. And you could actually take these volumes and attach them to multiple sandboxes as well.

The third one, which is actually things starting to get very powerful, is snapshots. You could potentially take or rather, you know, if the platform supports it, you should actually be able to take the entire disk or even a part of that disk and snapshot it. And even, you know, snapshot the state of the memory itself. And what this allows you to do is you could actually pause a VM and just like resume it at the exact same point, and the agent can actually just continue working on it. Or you could also, you know, snapshot an instance and start multiple sandboxes. This could enable things like forking the agent state and getting agents to actually, you know, pursue different options. And finally, you could also have warm pools. So this way, you know, even before you're, if you're able to predict what is the state of your sandbox before the agent needs to start working on it, you can actually just maintain a pool and get it going. And all of them have, you know, fairly significant impact on the amount of time it takes before the agent can actually start working. These numbers are just illustrative, but you can imagine why at every single step you actually drastically reduce the amount of time it takes for your application to be ready. And these are things to think about. And, of course, the way you optimize also differs on, depends on the kind of use case that you're covering, right? So for instance, code interpreters are just embedded in chat UIs, and, you know, the model's generating some code. You immediately want a response. Maybe you want a warm pool. Coding agents, you know, your users expect that these agents will run for a longer time. So, you start thinking about it differently. RL training environments, you are mainly thinking about aggregate throughput and so on and so forth. So, different kinds of optimizations depending on the kind of use case that you need.

And these are all the primitives that platforms need to build so that you can leverage it however you actually want. So, when all of this is actually done properly, you know, ideally you should just be able to say, okay, I want to just resume an older instance of something that I was working on, and here we go. The same state that we were actually in, we are able to get back to. So, let's look at under the hood and try to understand, well, how does this look from your point of view of you actually using some sandbox? So, it's actually this whole demo that we actually just looked at. In its simplest form, it's actually just, you know, eight lines of code. Let's break it down a bit.

The first one is isolation. So, every session, it gets its own VM. You can think about, you know, separate sandboxes per user, per org. There's all sorts of things that you could actually do. And then there are all the different methods that actually allows you to start processes, wait for a particular process to be ready, read and write files, you know, whenever you actually want to. Then the networking bit, which we spoke about, all the things that we addressed earlier in the networking section can actually just be one line. You get back a URL, and that's all you need to think about. And finally, you want to do some persistence. In this demo, frankly, all I'm doing is just storing my code in a bucket, but, you know, you could also opt into some of the more advanced strategies depending on the use cases like we discussed.

So, this is what, at least from a Cloudflare context where I work, the way this whole architecture looks like can be thought about as, you know, requests from the agent and the browser is reaching the edge, the worker. And from there, there's a durable object, which basically, the most important thing that it offers in this context is it's globally uniquely addressable, which is something that we wanted. We want to reach a very specific sandbox, right? So, the durable object makes sure that you're able to do the routing, and then you are able to reach the particular sandbox, and within the sandbox, you do all the things that you want to do.