AI Agents Drift: Identifying and Correcting Subtle Failures

Good afternoon, everyone. My name is Sam. I am a forward deployed engineer turned founder. My day-to-day work revolves around building production agents that are accomplishing a wide variety of tasks in agriculture. As I'm sure you've heard ad nauseum today, and as I'm sure you've seen online, the trend of companies building their own agents from the ground up is no longer a niche activity. This is something that a lot of organizations are investing in, and the reasons are fairly obvious. The technologies behind a lot of these libraries and platforms are open source. They're free. They're very robust. In addition, it's far cheaper. You know, if you're going to have a third party company come in and build these agents from the ground up, that's a cost you're going to have to incur.

Also, critically, all of the context that you and your org have about your products, your company's history, the competitive landscape, all of that, you're going to need to condense and somehow give that to this third party. It's not a lossless process. Often it's far better just to build these things on your own.

The flip side of that, though, is that now you're responsible for maintaining these agents, fixing them when they're broken, improving their efficiency, the time it takes for them to do things. You essentially have this new system that you have to debug. Agent debugging is this field that's, I think, growing more mature but still quite new.

The goal of this talk is to introduce just some of those technologies, what those feedback loops look like, and then I'll wrap it up with some of the learnings that I found in a few different industries which might be pertinent to yours as well. Again, I just want to begin with what an agent actually is. Again, I'm sure you've seen this plenty of times today, so I won't harp on it. At its core, an agent is some sort of reasoning model run in a loop.

It's run in an environment, so you can think of this as the architecture that it's actually running in. It has tools that it will use to interact with that environment. Critically, it's going to have context around, again, your business, what its task is. And then on top of all of that, you'll have some sort of evals or observability, maybe some permissions, security, things like that.

And then on top of all of that, you'll have some sort of evals or observability, maybe some permissions, security, things like that. And immediately looking at this, it's obvious why, potentially why, agent debugging is so difficult. If an agent's misbehaving, is it likely to be the model? Probably not. But could a tool be misconfigured? Could there be a piece of missing or contradictory context? Maybe there's some permission that's askew. All of these things could be what's causing the malfunction or the inefficiency. And it's up to you to trace from start to end what exactly is happening and why.

I often make the analogy that it's similar to this paradigm of compiled time versus run time errors, for those of us who are software engineers. When your code is compiled, if there are any errors with the syntax, if a variable's been defined incorrectly, you're going to be told very clearly, specifically, where the problem is and what it is. It's very easy to fix. If there's a run time error, if you input some sort of value into a form and that causes the entire application to crash, or if even something like the network request is taking way too long, there's too much latency, debugging why is a whole different beast in and of itself. And this is a lot more like what debugging agents is in reality. In fact, it's a lot closer to almost a psychological evaluation of what is happening, why the agent is behaving in a certain way, and what you can do about it.

Specifically, where are tokens going? Who is noticing this first? Is it your engineers? Is it some system you've set up? It's a different process on its own. To fix this, we need agent observability. We need to know what's going on, and we have to be able to fix it. Most agent observability libraries, well, I shouldn't say all, most of the ones you'll find online, are built on this protocol of open telemetry. This comes from systems engineering, and it maps quite cleanly to the paradigm of agents. Essentially, when you instantiate an agent and give it a task, all the way through to when it dies, that's considered one trace. And then all the atomic steps in between, whether it's a tool call, a reasoning loop, all of that, each one of those is considered a span.

Critically, each span is going to have, obviously, the model or the tool that was called. It'll also have the number of tokens that were used, so you can deduce the cost from that. It'll have things like latency, all these different, or even any tags that you've added manually. It'll have a plethora of metadata for you to look at.

So let's say you've instantiated an agent, given it a task, and let it run. You've now got all these spans in the single trace. What do you do now? Well, if the agent that was instantiated runs for maybe half an hour, and you only have a couple dozen spans, you can read it quite easily, linearly. You'll see the different reasoning steps, all the tools that were called. It's quite easy to deduce what's happening and why. Obviously, if there is a tool call that maybe shouldn't have been made, or there's some error, looking at the reasoning right before that, in the LM inference step, that'll often tell you a bit about why.

This is more or less the same when it comes to multi-agent architecture as well, although it's a bit more difficult. If you have multiple agents that are operating in the same environment, all editing the same artifacts, it can get quite messy, but a lot of platforms out there are actively working on this from a UX perspective. Now, the next logical question is, well, if we have a trace that, let's say, an agent's run for three or four hours, and we have tens of thousands of spans, are we going to read through that linearly? Probably not, right? Most platforms have this concept of a chat with your trace, some sort of LM overlay on top of the data.

Now, the next logical question is, well, if we have a trace that, let's say, an agent's run for three or four hours, and we have tens of thousands of spans, are we going to read through that linearly? Probably not, right? Most platforms have this concept of a chat with your trace, some sort of LM overlay on top of the data. This is done either by feeding the entire trace metadata in as part of the inference, or they'll do clever things like they'll vectorize the agent run, all the different spans, and then run inference on that. Either way, the interface for you is the exact same. You can ask very basic questions like, summarize this run, where was the concentration of tokens? And you can even ask questions like, what can I do to improve it? And very often, you'll get very detailed, good answers.

The next step is to imagine you have an agent in production, right? This isn't an agent that's working for you personally. Let's say you have one that's in the wild. Tens of thousands of them are running daily. And you want to find patterns in behavior. If you think about an agent and ultimately what its power is, it's extremely powerful in the sense that you can give it different bits of context, different inputs, different environments even, and it will still be able to operate to a certain degree of effectiveness. The issue though is that if you introduce some sort of change, let's say you add context, add a tool, and you want to see if there's a change in behavior, it's not statistically significant to just do it for one or a dozen agent runs. You have to test hundreds and hundreds. And going through those manually to deduce the effectiveness isn't feasible.

This concept of a cluster will take what I mentioned before about vectorizing an agent run. It'll take those vectors and then see if there are any clusters of behaviors, specific behaviors that you may look out for. These could be things like inefficient token usage. It could be something like a user getting aggravated in the course of a conversation. All of these behaviors will be picked up and then you can analyze them after the fact, see if any of them took place, and then drill into the specific runs that may be clustered there. These are some examples of common behaviors that you'll want to watch for. Very often the clusters you'll have to configure yourself, so they won't have an option just to cluster on any behavior.

It'll typically be things like, as I said, inefficient tool usage or token usage, if there's too much context or if there's too little. And then I also think an important one, as I mentioned, is if users are getting frustrated or if they generally like an agent.

Now, the flip side of this is that some behaviors that agents will undertake, things like high token usage or let's say mission critical behavior, you don't want to know after the fact. You don't want to have these clusters of hundreds of agent runs that fell victim to high token usage.

Instead, what you want to set up is some sort of signal that will detect if a particular agent run is exhibiting some of this target behavior and then will notify you via Slack or email or whatever. The process for setting these up is much the same as the clusters before. You define your behavior and as I said, you'll be notified in real time if there's a behavior that you want to know about.

These are some of the example signals I think are worth setting up. They're very similar to the clusters that you'll look for, except as I said, they'll be more mission critical. So, if it's something that you don't mind learning in two weeks after it happened, it's probably not worth setting up as a signal. You'll just get higher costs. If it is something that's going to be mission critical, then you do want to know about it as it happens.

And then the final piece that a lot of these platforms will offer is this concept of eval or testing changes. Let's say you've defined some behavior that an agent's doing that's incorrect and you have a potential fix, maybe there's some bit of context that's missing. Going back to our original point of making sure that fixes are demonstrated in a statistically significant way, how do you define that your fix, your additional context, will actually solve the problem? This is where this concept of evals comes in. You can define success criteria, which could be something like a string or Boolean or an integer, or very often it could be just plain text behavior.

Ultimately, it's just LM inference. The outcome is that you define the success criteria, you will launch a certain number of tests in parallel, and then you can clearly see if the change you made for a range of inputs is successful. This helps with things like debugging. This also helps with things like optimization. Let's say a new model is released and ultimately it's going to be five times the price of the state-of-the-art today. Do you want to integrate it? Maybe you do, but not if it doesn't have that big of an effect on the effectiveness. This is one way you can test that. Similarly, I think one trend that I'm actually quite excited for in the coming months or maybe next year is this concept of optimization for cost.

Similarly, I think one trend that I'm actually quite excited for in the coming months or maybe next year is this concept of optimization for cost. Can we use cheaper, maybe even open source models to do the work of proprietary models? And if you're working in an org that wants to optimize on that cost, you don't just want to throw an open-weight model into your agent and do some sketchy tests to determine if it's still effective. This is a good way to test with different models, see if it actually gets you the result that you still want.

Yes, a bit more about ebals. So, now onto some of the learnings that I've gotten. The first bit will be that most agent behavior issues are fundamentally issues of context. Typically, when you're debugging why an agent maybe didn't select a particular tool for the job or gave a user some incorrect advice, it's going to come down to something in its context window. Now, in a lot of the platforms I've highlighted, at each point in the inference loop, you can select to see what exactly is in context at any given moment. Typically, it's going to be too large to go through yourself, but you can just run inference on it and ask, is this particular bit of info in there?

And the last thing I'll mention is this concept of just returning to multi-agent workflows. The state of the art in terms of observability platforms is such that they don't have a great answer for multi-agent workflows. A, because I don't think the industry's adopted them quite widely. But B, it's just, it's naturally a difficult UX problem, right? You don't know, I mean, you have different agents spotting other ones. It's this weird linear or illinear problem. And I think the last part I guess I'd say about this is if you have a multi-agent architecture, don't be surprised if the many platforms you look at don't support this too well. There are some answers, but not many.

The state of the art in terms of observability platforms is such that they don't have a great answer for multi-agent workflows. A, because I don't think the industry's adopted them quite widely. But B, it's just, it's naturally a difficult UX problem, right? You don't know, I mean, you have different agents spotting other ones. It's this weird linear or illinear problem. And I think the last part I guess I'd say about this is if you have a multi-agent architecture, don't be surprised if the many platforms you look at don't support this too well. There are some answers, but not many.

So yeah, finally, I hope the goal of this is just to give you a bit of background about observability, hopefully convince you if you have agents operating in the wild to install a platform. I can't recommend specific ones. I think it's mostly just a case of asking an LLM. It'll give you a far better answer than I can. But it's extremely important. You get a lot of benefits from it. And a lot of these systems are open source and free. So it costs you nothing to try them.

Thank you for your time. Fantastic, amazing, amazing. Thank you so much, Sam. We have a couple of questions up on the screen. For Sam, how can success be evaluated for an agent whose output requires significant interpretation compared to an agent with a limited range of outputs or a clear it works or it does not work result? Yeah, it's a good question. So I think there are two options for you. The first is maybe a bit closer to what you're asking, which is a lot of these platforms allow you to define human in the loop evals. Essentially, they set up a web platform where a human will look at the different, I guess, the output of these different agents and judge them themselves. So if you have a task that can't really be easily defined by either booleans or strings or even just by a plain text description, this is a good option. I will say, though, that the LLM inference has gotten quite good, obviously, depending on the model you use, but you can define very vague behaviors, success criterias, and typically the platform will do a good job of judging those. But yes, human in the loop is also an option.

Fantastic. The next question I have is, reading context-widden window routes, there are a lot of observations on whether there is a hard threshold on token count to enter the dump zone or a passage. What's your experience with this generally? So sorry. Could you, so there's the percentage of, is it the, So is there sort of like a hard kind of threshold to reaching like token route? Hmm. Like getting your model to start becoming normal. Yeah. In terms of like what's in context. Yeah. I don't know. I mean, obviously different models are going to have different context window sizes. In terms of a percentage, to be honest, I don't think there's one particular for all models. I've used some, some open weight models, like there was a poolside or deep seek. And if their, if their context gets to like, you know, 50 or 60%, I've noticed a decent downgrade in performance. But then I think proprietary models, Sonic, Opus, I think they're quite good at, like you can pack the context full. But they also have quite good compaction strategies. I think that's the, maybe the thing that open weight models suffer from, you have to implement those yourself. So I wouldn't, I think it's model dependent, unfortunately. Yeah. Fantastic. Thank you so much.

Next one. How do you deal with observability in a situation where content of sessions need to be kept private? That's a great question. So many, many platforms will offer some sort of PII redacting service. They'll do this either. This is either a manual process that you can configure similar to the human in the loop evals, or you can set some, some private LLM to do it for you. Basically once the, once the, the trace comes in, like plain text right to the observability platform, PI is redacted and then it goes through the process of clustering, defining signals, all of these things. So yeah, most of the platforms will have that. Fantastic. Next one. How can you go about choosing what content, context to provide? Okay.

Like choosing what context to provide. I mean, yeah, I think. So I think in general, if it's not pertinent, well, it depends on first of all, what your agent's doing for something like software development. I have found that context around, you know, your, your competition landscape, your applications history. I found that to be quite useful when the agent's deciding what's structural changes to make to the code base. Then again, if it's an agent doing something very specific, I would argue the less context, the better. I think as a general rule, I would start with as little context as is necessary. And, and then using that, the, the eval loop to potentially add more context and hoping that the, the outcome will be better and seeing if it is and if it isn't going from Fantastic.

How do you make context available by reference? Can you expand on this? I think very similar questions. Context available by reference. Yeah. Sorry. What reference in what regard? So does anyone want to speak on this is anonymous right here. Reference in context available by reference, I guess maybe like you said, providing the less amount of context as possible as you need. Yeah. I mean, if it ref, if you mean reference in terms of like, if, if the, if something in the debugging loop indicates that like refers to a bit of context that you might need or, or is missing. Then I, I think again, again, starting with as little context as possible as, as possible, just to not pollute. And then if you, you typically won't see context that's missing reference, cause obviously the agent isn't going to know, you know what, what, what it's missing. But if you, if you notice that, you know, tool, there's a certain tool call that you've introduced and it's not using it or it's not using it correctly. Or if, you know if you're building a customer service agent and there's some database that should be pinged and it's not, those, then you, then you can test adding those to context. And if it, if it has an update, I hope that was helpful.

How could context be injected efficiently so that the agent can continue longer without hitting limits? Yeah. It's a good question. I think it's important to this paradigm of like making it to what we're talking to our point about trying to not pollute context. A lot of people will advise similar to how MCP works, where agents are not given this giant list of tools and all the inputs that it takes, what's helpful for they're given a very brief description, maybe even just the title. And then as they need, they will query. So the last question, just before we leave is how do you evolve orchestrating agents? Any ends are observing such complexity and direct plus indirect side effects. Seems like we should be avoiding this as much as possible.

So the last question, just before we leave is how do you evolve orchestrating agents? Any ends are observing such complexity and direct plus indirect side effects. Seems like we should be avoiding this as much as possible. So if by orchestrate agents, you mean agents with a lot of complex infrastructure around them, I do think in general that avoiding that is best. Obviously, as I said, if you have a lot of different moving parts, trying to figure out where in this big ecosystem, something went wrong is going to be more challenging than I think it needs to be. I do think typically how I structure agents is to give them an environment that is packed full of tools that they can use if they want. You don't need to orchestrate things like I mean, I've seen people put like web browser tools on Lambdas that agents can query themselves. It's just this insane AWS architecture that I can't even wrap my head around. When in reality, I think just keeping it very simple, even just a VPS where your agents running, give it a browser, give it maybe a browser use tool and then letting it use that. I think that's that'll lend itself better to the debugging process than it would otherwise. Fantastic. Thank you so much, Sam. Give it up for Sam.