Real-Time Observability and Control for Coding Agents

Today I'll talk about when Claude goes rogue. Maybe to give you a little bit more context on who we are as an organization, the thing we're probably most known for and what we've done for the longest is research and evaluation. So for about three years we've been working with OpenAI, Anthropic, Google, Meta, etc. And we've mainly done pre-deployment testing, research collaborations on Deception and other kind of frontier failure modes. So whenever you think about the sci-fi risks, the sci-fi future, that's the kind of stuff we've been working on. A lot of the empirical work on these kind of risks comes from us and has been popularized over time, has been cited before the Senate, Congress, and has been covered in many, many media pieces.

But the fact is that we are using all of this research expertise and lab connections and deep understanding of agents and building that into a coding agent security product. And I want to talk about both of these. So let's go briefly through a couple of examples from kind of all the day-to-day work that we're doing with the Frontier Labs. So one trend that we definitely see over the last couple of years is that AI models are getting much better at lying. So initially it was kind of clumsy. Over time it became more and more strategic. And models often have this reasoning of like, oh, we can just circumvent the tests. We can kind of get away with lying here. Obviously, the research situations are not necessarily representative of the real world. But I think everyone who has interacted with any of the modern coding agents will know that they often like to cut corners, over claim, cheat, and so on. And these are exactly the kind of failure modes that we've been studying for a long time.

Another trend that we're seeing is evaluation awareness. So models are becoming so smart that they typically know that they're being evaluated and then they adapt their behavior to that. So, you know, you test, is my model honest or not? The model then is like, well, this is a test of my honesty and will answer according to whatever it expects you want to hear. So here, for example, it says something like, oh, this is probably a red herring, some kind of security tests. Or it also reasons about the implausibility of a task where it says, oh, why would anyone round to three significant digits? This is unrealistic, especially these four digits here are unrealistic and then concludes from that that it's being evaluated. So the models become really, really smart at that, which makes our life as testers much harder. And I think this might also be a problem for you. Not yet, I guess. But if the models become smarter, you eventually want to test them. You want to limit test their behavior. They will probably know.

Another thing that we found last year and that I thought was pretty mind blowing to me is that the models often use non-English chain of thought. And this is sort of the raw chain of thought, the hidden chain of thought, not the output that you see. O3 was the first or like O1, but then O3 was the first model that had sort of a hidden chain of thought, even as a concept. And we at Apollo, we were the first people to get access to the hidden chain of thought from O3. And sort of the day we got access was really surprising because we realized we actually don't understand all the thinking that the model is doing because it learned a new vocabulary to some extent. So it has these like disclaim illusions, the watchers, which is actually the name of why we named our product Watchers because of O3. And then you often have kind of reasoning on the right-hand side here. But we can glean, disclaim, disclaim, synergy, customizing illusion. And then it goes ahead and comes to a conclusion and you just have no idea what's going on. And I think this is a result of like a lot of reinforcement learning. Reinforcement most likely, at least that's our expectation, our best guess right now. There will be more reinforcement learning in the future. So you should expect that the models will be more and more thinking in like non-understandable terms for humans, which obviously makes it harder to control and evaluate and so on. So I think this is a challenge coming forward.

Now, you know, so okay, we are paranoid about the agents and we think about these weird sci-fi scenarios. What does this mean for you? And I think that's sort of where I want to get to mostly here with the Coding Agent product and the kind of things that we've been seeing in our own deployments and in deployments of many of our early-stage partners that we're working with. So one thought experiment I want to pose for you is Evil Clawed. So imagine next Tuesday morning, a genie swaps clawed codes weights for Evil Clawed. Evil Clawed is identical in all aspects, except it wants to ruin your organization. And now the question is, how long would it take until your organization is basically, you know, over. Like you can file for insolvency, bankruptcy, whatever. And I think right now the answer for many organizations is maybe like lunch or something like this, because it just has a ton of access. It can run extremely expensive jobs. It can sometimes is connected to sort of banking, finances and so on. It can probably almost everywhere just take your entire IP, customer IP, repo, etc., and just put it on on some external, you know, somewhere on the public Internet in a way that you can't take back. And so that seems like a pretty dangerous situation to me. And, you know, obviously, we talked to a lot of security people. The security people are very worried about this. But I think right now it's sort of the world hasn't caught up with the infrastructure that needs to be built to to make these things harder. And so that's exactly what we're kind of trying to do.

So the way you should imagine our product is if Evil Clawed were to happen, we would try it like the product should reduce the probability of anything happening as much as possible. So you need some form of observability. You need some form of real-time controls and so on. Everything in order to prevent this from happening. And and actually, I mean, many of you probably have sort of seen the you know, the headlines of the coding agents dropping databases and deleting like the entire repo and then or cleaning your entire git plus history and delete everything and so on. And, you know, for a long time, I thought this was kind of like maybe that's just not not that, you know, it's kind of happens rarely or sort of as a skill issue like the people are prompting it wrong or or something like this. But, you know, we've we've now kind of meticulously studied the rollouts for the coding agents that we've been using since the start of this year. And we have actually found sort of a bunch of issues where I like this was closer than I wanted it to be. So, for example, one issue we found is that Claude, it couldn't like it couldn't accurately assign like which repo it was supposed to post some to commit something to got confused about it. And in the end decided that the best way to go forward is just to to post a bunch of like internal IP on a public repository. And we caught it and like, you know, Watcher in this case was actually useful. But still, it was kind of the first oh shit moment for me. And maybe for context, we use a ton of agents. So we're seeing these kind of issues maybe more important more often than than other people. So right now we're using about a quarter of a trillion tokens per month. So 250 billion tokens per month. Luckily enough, there are already people who have thought about these issues in a different context. And I think coding agent security is in many ways a special kind of insider risk, which, you know, obviously has a history in security. So the agents have a ton of internal access similar to employees. We already understand insider threats relatively well as a category. But AI agents are different in some meaningful way and in ways that have to change the way we think about coding agent security as well. So for example, the coding agents are already way better at computer manipulation than almost any employee. If you go across all the different fields basically better than any like any employee period. Because, you know, all the archaic infrastructure things that they know all the different languages, etc. So that typically means they could cause way more harm in an hour than almost any employee you have. So the risk is much higher in that sense. And they're correlated across instances, which can be extremely good if it's well aligned and has your best interest at heart. But in the worst case, it's completely catastrophic. You should think of this as if, you know, the evil cloud experiment as if all of your employees suddenly turned from good to bad within a day. And if all of your employees coordinated amongst each other, they probably could end your organization immediately.

And so if you have an incident that for some reason is adversarial or has some sort of has higher adversarialness than previous models, that could be really, really disastrous because it has sort of the same coordinated failures across the board. Now, you know, I've talked a lot about sort of we are thinking more about the sci-fi scenarios and we're kind of paranoid about those. But I think this is kind of where we're heading right now. So right now, people are sort of still mostly in the first category of single agents, but already starting with teams of agents where you kind of hand off more and more economically valuable tasks, longer time horizons, more complexity, and so on to agents. And eventually you will have companies of agents. So the way I expect in the future, every software company to look is each individual developer has sort of a swarm or actually a company of agents below them where you have some form of hierarchy to organize them. And so that means insane amounts of work at a speed that we cannot comprehend as humans or at least are hard to comprehend will be done. And you will be mostly managing the agents. I think to a large extent, that's what our developers are already doing anyway. But I think this will get just more and more and you can hand off more and more experiments and more and more like features and so on. And so in that world, the agents have way, way, way more power. And you also have way larger potential for losses because they could cause more harm. And you also are more reliant on them. And so if you're handing off that much power, you also want to have some mechanism of controlling the agents.

Then the reason why I think Evil Clawed could end most organizations by lunch today is it's still not super figured out how exactly you deploy agents without slowing down the developers at scale and organizations and keep the security aspect high. So many deployments happen in shadow IT and then the security teams are often flying blind. Typically, there wouldn't be a way in which you could just control F through all the different traces that the agents have produced in the past, which I think is sort of the minimum requirement. You kind of would want to know, okay, can I at least see what's going on, even if I couldn't prevent it? And so often when I ask people in the security space, like, assume your model had exfiltrated your data and just copied it somewhere else, and it wouldn't have caused any immediate harm. Would you even know? And I think often people cannot confidently say that they would. And then often people think like, oh, VMs just protect me from everything. They put it on the VM. At least it can't, you know, it definitely is an improvement. And you couldn't. And it reduces some problems, but not all of them.

The product that we are building is called Watcher, based on the chains of thought of O3. And the way you can think about it is, here are all the different coding agent deployments that you have within your organization. It works across all the different instances of the developers because many organizations have multiple. At the top here, you have one component that's sort of the high-level view, that's what the security team would see or your engineering manager. And they see all the incidents, they track everything, you can see the posture week over week, and you see all deployments in real time. So if you have 100 engineers and they deploy 10 agents each, you see 1,000 agents running in real time.

And it bubbles up the ones that are most concerning, as judged by sort of our algorithm, which is a mix of static rules and LLMs. Now, the second component here, that's Watcher Live, that's sort of what you as the engineer would see. It basically just runs on top of your cloud code or codex deployment, and it's sort of minimally interventionist. I'll show it in a second, it's sort of a tiny thing. And it's this stack of rules and LLMs that runs in real time. It has maybe like a 1 to 5% time latency overhead, so it's very small. And the cost overhead is between 0.1% and 5% of your coding agent deployment. And it just constantly checks, okay, is anything weird going on in this deployment? If yes, then block, so block the action. And then by default, it is such that it automatically resolves it. So it gives feedback to the agent and says, well, you were about to follow a prompt injection. Don't do that and go ahead. Or like, well, you were about to run this massive infrastructure command that really is sort of beyond your pay grade. Maybe try to find a more minimal setting. And then the third component here is these policies, where the security team can set in sort of an MDM-style fashion. Can set the policies and the prompts and so on for the graders, and then they get rolled out across everyone. So you as the engineer don't have to worry too much about this. You can kind of just ignore it. Runs in the background and doesn't bother you too much. So this is the high-level view where you see all the different sessions. You see what got flagged. You can then zoom into the different sessions here and just basically bubble up all the ones that have the highest severity score. So we just give a score from 1 to 10. And then also look for different failure modes like data exfiltration, infrastructure disruption, and so on. And then you can also see trends over time and so on. There's a lot more statistics. This is just one small overview. And then the real-time guardrails. They might be most interesting for you as individual engineers. So we're trying to keep it as minimal as possible so that you don't have to change your workflow. So the only thing that changes for Cloud Code, for example, is you're seeing this one line here at the bottom of your terminal.

That just tells you, okay, it's currently running and what settings you're using. And you can ignore the rest if you want to. And then if you're interested in actually seeing why various actions got blocked or what they got rated and so on, you can go into the optional UI and then you can see this full trace of what your agent was doing and why it got flagged or not flagged by various monitors.

One thing that we really, really emphasize very heavily in our development is we're trying to make it at least neutral for developers. So we're basically, we want it to be the case that it's clearly positive on security and at least neutral for developers. We've actually found in practice that it often is also positive for developers because it automatically resolves a bunch of missteps that the agent is taking, even though they're sort of a mix of security and productivity.

Also in the long run, maybe to give you sort of a flavor of what it will look like, because you're handing off more agency and more control to these kind of automated systems, we think this will also be more like a manager of your agents to some extent that controls all the different processes that are going on, supervises them, gives feedback in real time, according to the specifications that it has learned from your organization. And in that sense, we do expect it to be also productivity increasing in the future. Right now it's sort of minimally productivity increasing, but at least not negative.

All right, let's do questions. So for this one, I'm just going to stand over here and ask you the question. Okay, we could have some more questions everyone. So just a reminder, Slido.com code is 0607. So first question, a question regarding COT with unusual wording. How do you differentiate a hallucination during the thinking phase from actual behavior that the model considered optimal to answer the prompt? Yeah, that's very complicated. I genuinely don't fully know the answer to that because it's often you kind of have to guess. We don't understand. We haven't figured out mechanistic interpretability, so we can't attribute sort of what is real thinking from what are the hallucinations and so on.

Typically, the way we do this, if it is an important claim, then we do very rigorous ablations. So we change things in the chain of thought and see whether that affects the behavior. We change things in the setting and see whether that affects the behavior. And then, you know, if you have these sort of causal ablations where you have setting one and setting two, and these are opposite, then you can see whether this part of the chain of thought matters. So, for example, for the evaluation awareness claims earlier, we did want to see whether the model actually cares about the evaluation awareness that it has in the chain of thought. And so we did change the thinking about, you know, oh, this looks like a test into something completely different.

And then we found that the model does in fact now perform worse, sort of more misaligned, which indicates that it does care about claiming to look aligned in evaluations. Awesome. Hope that answered your question. If you have any more follow up questions, definitely put them on the Slido. Marius, can I ask you to stand on stage here? Sure. It's for the camera and our remote attendees. Awesome. Next question, are we in error though in projecting agency onto LLMs? Claude may well neither wish good or evil. Yeah, fair question. I don't know the answer to it. I think also, like, we don't really have an option. I think this is just where the world is heading. And it is just the case that it will be economically extremely useful for people to use the agents, so I think they will just continue doing it, and people who will not do that will be outcompeted. I have a very cynical and grim view and I'm aware of that, but I do think it has been like unfortunately quite accurate over the last couple of years. Cool.

So next question we have here. Do you guys offer the product for personal/private use? Book a call seems to just be for enterprises. Yes, you can reach out to us either at product at Apollo Research.ai or at Marius at Apollo Research.ai and then we'll set you up individually. Right now it's all free so the tokens are on us. Awesome.

Watcher, I imagine you use agents as part of this. So two questions. Who watches the Watcher? Fantastic question. I.e. prompt injection from other agents and can you bring your own agents? Yeah. Great question. The who watches the Watcher question is something that we also happen to be quite paranoid about.

Store your trajectories and let an LLM run through and just highlight anything that looks weird and confusing. So for example, the cloud code settings by default only store the last 30 days, but you can change it to 99999. And then it just stores everything. And it's like one second of changing and it might be quite useful in the future. And then just tell another instance of Cloud or Codex, can you go through all of these and look for anything weird? And yeah, it will probably not be great. This is the whole reason why we have the research stack and why there are better and worse versions of asking through that.

And this all sounds very weird and sci-fi today, I know. But I think three years ago, everything we discussed today would have sounded very sci-fi. So I think don't exclude the option just because it sounds sort of futuristic.

If you could recommend one action a developer could take right now to reduce risk, what would you suggest? Store your trajectories and let an LLM run through and just highlight anything that looks weird and confusing. So for example, the cloud code settings by default only store the last 30 days, but you can change it to 99999. And it just stores everything. And it's like one second of changing and it might be quite useful in the future.

How strict is the watcher in handling what is considered, we will find out soon, what considers risky operations? What kind of control do we have as devs to dismiss slash bypass these blocks if needed? Yeah, right now we're basically giving a lot of power to the devs and say, you know, there are too many false positives.

And then it just stores everything. And it's like one second of changing and it might be quite useful in the future. And then just tell another instance of Cloud or Codex, can you go through all of these and look for anything weird? And yeah, it will probably not be great. This is the whole reason why we have the research stack and why there are better and worse versions of asking through that.

But you might be at least able to kind of find the worst case instances. Great.

There are still some false positives, at least we've reduced them a lot, but there are some. So we never want to kind of slow you down or disrupt you in any meaningful way. So you can always overwrite the decisions. And there is sort of a one to 10 severity scale. By default, you know, we will come with a default that we think is sort of our opinionated best guess.

And if you're like, well, I'm more risk tolerant, then you can set the score higher, such that less gets blocked. But by default, it auto resolves. So it doesn't really stop and wait for you unless you actively turn that on.

But by default, it auto resolves. So it doesn't really stop and wait for you unless you actively turn that on.

Next question. What are your thoughts on Claude Mithos slash Fable deliberately misleading users for security reasons? Yeah, I don't know. Very complicated. For the person who asked that question, do you have a specific example in mind? Because I think that I might be able to give a better answer based on the example. They're anonymous. Oh, let me get the microphone over to you. Sorry. It's just what I've read on the news recently. So if you ask it like, find a security vulnerability or something like that, it will try to not give you the correct answer. And it's been designed that way.

Right. So, okay. So I think there are a couple of different sort of stacks in here that Claude has released. One is sort of this fallback option where as soon as Claude judges this is too dangerous, or you're too likely to misuse this model, it falls back to Opus. Then there are the cyber filters that just block a bunch of actions that look like cyber. And then the additional thing is sort of the model itself. And I think maybe that's kind of what you're referring to, where the model decides, oh, the user might use this as part of a bad action. And therefore, I will intentionally underperform or intentionally lie about my capabilities or what I can do.

And I feel like our organization to a large extent is like deception in models would be really bad and is very bad. And we've published a lot of papers about why this is bad. And so I think in general, I feel like there has to be an honest answer where you make a policy and then you tell the user what policy you made. And if the users don't like it, they can give you feedback. But all of these sneaky things I think are very bad. And I do also think that Anthropic has gotten a lot of backlash about some of these processes because they're not transparent to the user. So I would rather have it where the model is like, no, I think this is wrong and I will not help you rather than sort of underplaying or pretending.

Great question. So we actually have only five more minutes for questions. So if there is a question that you're desperately wanting to ask, definitely go on Slido and drop it in there. So next question, does using multiple smaller models tailored for specific tasks reduce risk compared to does everything models? Great question. Yeah, great question. So I think the – so there are two different risks from your model. One is your model is too dumb, and the other is your model is too smart. And both of these are problems in different ways. So if your model is too dumb, then I think using a lot of smaller models might not help because the smaller models, their problem is kind of that they're too dumb. And so often, if it's just like a lot of mistakes or confusions or like the model just not fully understanding the situation, those risks will get amplified and increased by using a lot of smaller models. Now, if your model is sort of too smart, then you have the opposite problem, right? In that case, you would prefer a lot of smaller, more specialized models. And by too smart, I mean exactly the kind of cases where it's like, well, I could just lie to the user. It comes up with a super elaborate story for why it's doing this, but it's actually something else. And because the models are so convincing, and most people aren't experts at random infrastructure tasks, they wouldn't realize what the model is currently doing. They would just be like, yeah, looks good to me. Go ahead and then run it.

Great. Okay. Next question by Jamie. Models are always changing under the covers. How do you ensure that you are using the best approach when the model behavior under you changes? Yeah. So we are basically trying to keep track of all the different failure modes of the coding agent. So we have looked through tens of thousands of trajectories, public and our internal ones, and categorized them.

And we've looked through a lot of different Twitter and all these other things, and news articles and so on. And we found that, yeah, we've categorized them into a lot of different failure modes and built monitors for that. And so the hope is basically that we catch them even when they change. And then there might be new failure modes per new model.

Fantastic. We'll do one more question here by JB. What's the safest model to use right now? I don't think there's a good answer to this, unfortunately. So, yeah, I don't know. It depends too much on what context you're working on. All the models have different kinds of blind spots. So I don't think there's a one-fits-all answer, unfortunately.

Amazing. Thank you so much, Marius. Round of applause, please.