Building Blocks of an Agentic Engineering Platform: What SRE Taught Us About Running Agents

So first, about me. My name is Ilja. I live here for the last 12 years in Amsterdam. I spent 20 years doing hardcore engineering, replatforming, refactoring, so platforms like legacy platforms, working for companies like Ikea, Nike, and others.

So what I'm doing right now is like I started a year ago, more than a year ago, an agency called Endgame. And what we're trying to do is basically the same, but we're building systems which are building other systems or doing any kind of legacy modernization.

And what we also do and try to do, we try to scale these practices, like agentic engineering practices to organizations of 500 engineers to sometimes we'll have clients of 3000 engineers. This is the challenge. When you have a team of two or three people, it's one challenge. The other challenge is how do we do it on the scale of 3000 people? And what do we do with all this spend? How do we make sure that we have return of investment? How do we make sure that people are actually using it in a secure way? And this is what we're trying to reflect on for the last year.

And what we learned is, it's basically a difficult situation on the market right now. So people don't trust agents. People think that still you need to go and verify each line of code produced by agent, even if you do a simple Next.js application. Question to you guys, how many of you are using Cloud Code daily or any kind of agentic and don't use IDE at all? No. All right. I think like five people, including the first row here and myself. That's fine. Because what we learned is agentic system is still a system, still distributed system, and it still needs to have the same approach, the same type of patterns as if we're building and scaling distributed systems.

But the difficulty is it fails silently. You can get 200, but task is sloppy. Code produced is very much bad quality and so on and so forth. So the other problem is that this nuclear energy you can contain and you can put into any kind of pipelines, any kind of enterprise environment, and you need to make sure that you still have salary money for the next month and you still have your environment keys secure. Right? And this was going to be the talk about. So what we see right now, a lot of stuff, what we see happening with agentic engineering and building agentic systems, it's basically very much derived from the patterns of SRE development experience and what we were doing for the last 20 years with cloud movement and cloud transformations.

So if we look into, for example, DevOx, and this is the most interesting topic for me, it shifts to agentic experience. What does it mean in reality? You need to have environments provisioned in seconds of time. You need to understand how to test your systems. You need to be able to sustain the load of those environments against the agents and fleets of agents. And this is something I can talk for hours and hours afterwards. I tried to prepare this talk to be more high level, but if you want to go into detail, I will be still here, around, just let's catch up.

So how we look into the problem itself. So you have one part of the problem, which is agentic engineering. And this is what we're doing, I assume, the majority of us every day. This is how do we make sure that we have a proper harness on the code base level or multiple code bases. How do we make sure that agents have safety net, verification points? How do we make sure that we're able to single shot some small things, and it's going to work out and the code will be produced on the level of patterns we use in our software, on the level of the code standards. And I can continue on and on.

And this is one part, this is more individual team level. If we scale it to the organization of 3000 people, we have totally different problems. And the problems are being solved in the past, as I said, with the cloud transformation. And what we call right now, this type of setup as SRE, or any kind of team that responsible for making sure that best practices, best pattern, secure way of doing things is scaled across the organization, we call it agentic enablement. And this is the difference in terms of how to do it locally with three people, and how to scale it to a whole organization. So how I structured the talk itself is three areas, I will basically go through a list of questions we typically being asked from our clients. And those questions are sometimes technical, those questions are operational to the most extent and a lot of questions and organizational. Again, this is not a full list, I tried to condense it and distill it into the most important ones. Let's go.

So in terms of technical patterns, I will cover just a couple, because you definitely know majority of them. So the first one is, the most difficult part to convince any developer to do, is sandbox their agents. You have many ways to do it. But the problem if you don't do it, you can have security leakages, you can have the majority of things that are going out of your agent being actually not secure as well. It can access network you don't want to access. And this is one of the biggest problems we see in the bigger organizations that they just give away flawed team subscriptions. People start using it on the level of their computers or their local environments. And actually what we have, and it's going to be a snowball in the future, we've heard right now in some talks of new ShyHulud attacks through CloudQuot recently, is going to happen more and more. And make sure that you secure and make your agents of least privilege access to whatever they're doing.

Next, what we started to explore even a year ago is I knew that the coding on the individual level on the one repo will be solved. But what's not going to be solved and is going to be hard to do is context management on the scale of a system. And system typically, what I call a system, is going to be multiple repositories, sometimes separate repo for front end, five different services, some different configurations, Terraform end-to-end tasks, and so on and so forth. And this is where you think, okay, how do we do it? How do we do it in the scale of multiple repositories? And we started to experiment with something we call context hubs. So it's a separate repo that has all the decisions, structured way of recording what ADRs across the system are. And then we tried to shift towards building up on the top of it, building different context graphs. And there are multiple things on the market you can try away right away, from ByteProver to do context on team level to actually code base level, understanding like graphified. And then of course we started to develop our own tools. And I think this is where it all goes, is you will start developing your own custom tools to understand and build better systems and report records like ADRs, report scope and any kind of product decisions.

Funny, fun fact, what we actually do, we don't have the ideal state for us. Even if we go to clients and work on some complex environments, we don't like to use GRS, linears or any kind of this stuff, because we don't think it's actually suitable for agents. What we usually do, we have JSON L files, for example, in the separate context hub repo, and we sync it back and forth just for reporting for product managers and stuff. Another pattern we've seen in the wild is if you have handovers from one activity to another activity, for example, you have a PRD or EPIC or any kind of specification, you need to set up a very much standardized format. And more to that, you need to verify and validate that the format is actually perceived and filled out and worked well in between of those stages. Which means you need to start consolidating and understand how do we standardize all the handover artifacts across the company.

Next, the most interesting part is when people ask me, what is, how do we do with the autonomous agents? Where do we deploy? This seems like a big capital investment we need to do. But my answer usually is if you have GitHub actions, this is the best starting point for you to go into autonomous agents. And this is true because you have something you can embed into your pipeline right away. You have events that are going from your CI pipeline. Also what you have is sandbox environment right away. So this is the easiest way to do it. And what we started to do quite a lot is going from simple ways of course, everyone is doing code reviews here, doing PR reviews, probably have been doing a bit of quote, quote actions and tweaking that. But what we're doing to the next level is we, a good example of very advanced pipeline can be a bug hunting pipeline where you do understand the most complex areas of your code base. You go in with any change to run to an understanding, semantic understanding what can break from business standpoint, also from the technical standpoint and try to auto-fix it. So I think if you will be able to do it in your CI pipeline, that's going to be, I think, one of the big wins for the whole team, which will require a lot of context, a lot of context management suppression and working on this.

If we go up next, I think the first interesting point is like, of course, GitHub actions can cover a lot. You can embed agents into your CI. It's going to be slow. And then we're going to more advanced patterns like, okay, we can wrap it up, wrap the for example, plot SDK or deep agents SDK into a Python application, for example, right? Post it in Docker somewhere, provide some pull mechanics or push mechanics on the web hooks. And then it's going to be always on fast replying type of agent. Very simple way of how to start rebuilding your open flow is thinking that way. That's all about technical pattern. Very easy. You can embed your agents into your CI pipeline. You need to start doing this. You need to start thinking about the context and context is not stale. The problem with context is also it's another layer of activities your team should be doing. Suppression, stale context, review of the context, understanding what should be put, how to put, how to access it properly, how to make it accessible, fast, reliable to the agent.

Let's go into operational. So the first question we're usually asked is like, okay, if we go into Claude and Claude is the massive infrastructure, we in the market right now, it's a huge vendor lock in. And apart from this, there are multiple other things that are asked about the Claude and how do we protect ourselves going single vendor? Of course, the whole harness, the Claude pod, the installable package you install on your environment and actually interacting the terminal or through SDK is the most powerful one in terms of functionality and features. No one wants to lose these features. For example, workflows recently were announced. It's quite hard to imagine my life without workflows right now. So I don't want to lose this functionality.

But what do I do is like I don't have money for this. I cannot afford it in the scale of 3000 people. So what do I do? Simple way what we advise to do is like look into compatible models and compatible API. And also there is a way to implement something called LLM gateway, which provides an ability to also translate any kind of, for example, bedrock hosted models, cheaper open weight models like kidney, for example, and make them quote quote compatible so you can easily switch. You can easily track what the whole organization do it. But apart from this, LLM gateway provides a lot of security features. Allow these for MCPs, any kind of settings, prompt, injection mechanism and stuff.

Next is how do we make sure that we spend some money on something which is valuable? And also there is DX and many other big platforms. You pay probably hundreds of thousands of euros every year. We think that is going to be very easy to just connect to basically LLM hotel and create a sync to your Prometheus and put Grafana on the top. And then what you can get from it, you can get quite a lot. You can get not only the whole span, but also price per commit. What was the model used? If you connect every type of PR to any story or record, what is the value? You can think more. So how much did they actually spend on delivering some value? And then if you go next and next, you can connect it to any kind of product management tools or analytical tools for customer insights. And you can understand quite a lot in terms of where you spend this going. Simple way. Don't overthink, don't over-engineer. You already have it. Just create the sync of Hotel from your Cloud Code or any kind of other tool to your environment.

Another question, how do we keep costs in check? And there are multiple ways. I don't want to speak about the individual level much. But I think I want to open up with a bit of, try out some tools that are doing rationing of the output of tools. What it does, usually you have a batch of tools being run from the Cloud Code or any kind of agent. And what you can do with it, you can say, okay, I don't need the whole output. And those tools like RTK or Headroom allow you to actually ration. Make sure that your agent gets only what it needs. But apart from this, of course, cost caps and any kind of progressive autonomy, just to make sure that you don't allow your agents to do more than they need to do. If we go up in terms of the patterns, what we also see helps if you have any kind of blueprints of your services available from your DevOps team, for example, how to provision a fast API service or how do we provision a microservice connected to Kafka right away and start building fast. You need to make sure that there is a baked-in harness into it already in place. Means skills to do the important stuff, hooks, any kind of workflows that are needed, any kind of additional verification points that should be in place.

So any kind of distributed package within the organization, outside the organization, I assume right now should have some harness embedded. Internal marketplaces, again, the fastest way to start it is just Git repl. Nothing more. It provides a governance mechanism and many more to actually make it scalable to the whole organization and contributable to the whole organization. And many companies really think that context and information within the company is shared and everyone knows everything, which is not true in many cases. And use Confluence and any kind of that tools. We think that Confluence and human focused tools are not good for agents. What we usually do is we create something called context distribution and we use standard imagery or artifactories to do it. So you can actually package a piece of context from one team, push it there, version it and another team can actually pull it or another agent can pull it.

Organizational patterns. And this is where the fun starts. So if you look into everything I discussed, it becomes a massive actual job and probably a separate function within the organization. Why? Because it's not very much even. We have different priorities. Everyone has different experience and you need to make sure that the best practices are distributed across the organization. But also you need to have a very reliable platform to run your agents. And for us, how we see the whole platform itself, it doesn't matter whether you do agentic engineering or you do any kind of business process automation with agents, it can be run on the same platform. And on the screen, you see a bunch of layers that you need to introduce to make it happen, including orchestration, including governance, including security layers, including the micro VMs for sandboxing and so on and so forth.

In terms of teams, and this is a big question I can talk for years about it. And it starts with like magical number three. Why three? Because there is something called coordination, communication, in the organizational design. And three people is where there is a golden number where your communication paths are not growing exponentially. After this, they are. It means if you add another person to the team, another person to the team, another person to the team, you spend more and more time on collaboration, communication. And right now with agents, what it allows us to do to actually go back to this small teams, it opens a lot of questions. What do we do with other people? How do we relocate capacity? I don't want to talk about this. What it means you still need to enable all of those ports and teams to have best practices and how we think what should be done is exactly this agentic enablement. And this is a separate team that does a job of making sure that the agentic experience is on the high level, the development experience as well. And how to scale it practically, because there should be also dynamics and movements how to do it.

And how to scale it practically, because there should be also dynamics and movements how to do it. So how do we actually structure it? We try to find the pockets of adoption within the company. We try to find those people, Pathfinder teams that are doing already quite a lot of what I've listed and try to institutionalize and make sure that this knowledge and this patterns and those practices are qualified. And this is exactly where enablement team plays well and distribute through champion network of other teams to other, the whole organization. So all in all, just to conclude what I was talking about, it's quite a lot we can derive from practices and disciplines we've had in the past already. And this is Esri, development experience, cloud engineering, and how we went from knowing nothing about the cloud and distributed systems in the cloud to where we are at right now. But some stuff is very new and you can get a lot from this past, but you need to learn quite a lot. And what I want to show you and what I want to kind of produce a bit of a spring of interest is this is still a distributed system. You can embed your agent and put structured output out of them and connect to any kind of business processes and business workflows, including your SDLC. It's totally the same. But you need to embrace this entropy and you need to understand how to move on with this nuclear power that is actually can be for good and for bad. Thank you.

We have a first question for you. Part of our problem with agentic AI is difficulty finding best practices. We're all struggling with this because it's changing so fast. What's your approach on building a foundational framework, prompts, skills, rules, etc., versus compound engineering and that type of thing? Good question. My take on this would be there are some things you can already... So there are multiple directions how to answer this question. One would be is you already have things that are more or less set, which are like skills will be here and standardized. It's still not very much. So what you do with the context engineering, to be honest, in the simplest form, it just marks down files and records. But if you go up and up in terms of how do we structure it, it's basically the same as doing a specific data model for how you do stuff as a process.

For example, you need to record architectural decisions. Every each of them should have an ID. It's connected to a specific system and so on and so forth. So basically underlying execution layer can change how you do it, but this knowledge won't be lost. So how to start with this is like start thinking about what I do, how I do, what I need to report and what type of tacit knowledge should start being more explicit within my team or organization and then only think about how we do it, what it is. But how we do it, what it is, is basically some stuff which is standardized already.

That leads really well into the next question, which is how do you maintain those context jobs with architectural changes, decision logs, etc., automatically as you go through this? Everyone thinks that it's going to be just plug and play and maintained automatically and then people should not correct. Unfortunately, it's not the reality. As I said, it's going to be more of an additional level of activity that the team and the individuals should do, but there are a couple of ways to do it. So the simplest way, again, I think I'm back to foundational engineering here, is like you can connect gate books to making sure that some stuff after your change is updated and then create a review with and put together with the PR, for example.

Additionally, of course, you can do a weekly automations to do it. But what I like is some stuff which is agents to agents can be done automatically and there's a lot of this. But some stuff, which is again, as I mentioned, there is product decision, architectural decisions, a specific functionality and intent what we want to do should be reviewed by human beings.

All right. Next question we have. Do you have any advice for teams or organizations that create high-risk code, for example, software as a medical device, and employ AI agents or do the same rules apply? Wonderful question. We have two big healthcare clients. We recently were doing grave digging in the application of 45 years old with the C++ and Delphi. Technically, same rules from working on the code base will apply same, totally the same.

There is a lot of regulatory mechanics that you need to comply to. For example, you cannot do fully automated mergers. You cannot do quite a lot of things that you would be imagining doing automatically, including the maintenance of the documentation, because it still should be signed off by humans. But in general, it goes back to safety net and testing. How do you test it? But you will be surprised that if medical devices, everyone thinks that it's covered with 100% coverage of tests and stuff. No, it's not the case.

It's not the case. I've seen it all. No, it's not the case. So my take on this is there are multiple ways to do it, but also multiple ways how to contain the change and how to understand how we can touch all of this with the eye, because the speed of change would be fast. I will just put one interesting direction, try to understand what characterization tests are. It's basically regression tests without understanding the output. And what we usually do when we're stepping into brownfield, we put together a safety net of characterization tests, at least around some of the most business critical or mission critical modules. Then you kind of understand, OK, this is black box, this is how it worked before. Finally enough, you will find a lot of bugs in the first place. But then you also will understand this is how the state of it should be before we touch on the agents. Yeah, it's a good reminder. Humans make bugs, too. We're also non-deterministic entities. Exactly.

Next one up, talking about code review. So usually we have a pair of developers for front end and back end in a team to review work effectively. How would that work in a three-person team? Difficult question to answer. First, we started with a bit of a history and learning from myself and doing this for some time already. We started with, like, we don't care about the programming languages anymore. And we need just fundamental engineering experience and expertise, like system design and understanding how stuff works. And we still keep it as a basis, but we still think that some specialization will be required and needed. And it works well, especially if you touch some front end, if you touch some databases, if you touch some legacy. So there's still a lot of differentiation in terms of this specialization. And it will be kept. That means you still need those in the team. But that means we need to assemble the higher bar for making sure that those people are effective is set. Right. But I don't have an answer, like, complete to to this question.

Right. But I don't have an answer, like, complete to to this question. But I think the point there is still that you still need specialization. You still will need specialization. But to majority extent with the AI you can learn fast. And if you have this fundamental knowledge, you can learn quite a lot. For example, I started doing quite a lot of Rust, which is hard to learn. I learned it the agentic way instead of just doing .NET as well. So I didn't haven't done any .NET in the past. Right now I'm pretty confident we can pass the interview for senior developer, .NET developer.

One last question that we'll get to, and I think it ties into this. If you have those expertises, you can embed them in skills or other things. So your front end developer can do a front end review skill that looks for those particular things and similar that can be reused. And this gets to this last question we have time for, which is, do you have a perspective on running PR review locally through a skill versus a CI agent, dedicated agent, things like that? And that's complimentary to be honest. It's complimentary because you need to make sure that, first of all, there's still your poll. Right? It doesn't matter who someone should be responsible for the quote and accountable for the whole system to be reliable, right in the production stuff. So it's still you need to know the base.

The shift is like, how you do it and how you make sure that this is actual, this whole base is reliable to the standards and it's not doing any harm. And that's the whole difference. Right? There are multiple ways of doing this agentic way, of course. But to some extent, you can use the agents to be a your navigator and kind of do your legwork for you. But you just make sure that it's on the proper level. All right. That's all we have time for. Let's get one more. Thank you for, Ilya. Thank you.