From One Repo to Hundreds: Building an AI Agent Fleet for Large-Scale Code Migrations

Hello, hello everyone. I'm really glad and great that it's actually I'm here today with you. I'm here today to share with you a migration story. I work in a platform behind 500-plus micro frontends and one day we had to change them all in a way that a new node version was actually introduced. And how could I do that? So basically the first idea was actually we're going to actually throw a small army at it, doing everything by hand. But we have LLMs, we have basically all these great tools out there. Instead, we built a fleet of AI agents to do that for us. And this talk is actually how we did that. Let me show you where this actually ended up.

So this was actually how many repos we had to migrate. This is 527 repos and 87% of them run without any human in the loop. And this is by design, not the corner we cut. Most are already merged, but the rollout is still ongoing. So a quick intro about me. I'm Konstantinos, I'm a tech lead at Monday. I lead technical initiatives over there and I own the platform and the frameworks behind those 500-plus micro frontends. This is a machine that we built to migrate all these repos, mostly without anyone touching these repos.

But first, why not just do that in the normal way? Because the normal way breaks at scale. And when I say normal, this is basically what I mean. We have two options here. Basically we can migrate with manual PRs and also with code mods. Doing it by hand, 527 repos, multiplied by fixing them, reviewing them, and babysit all the CIs is not possible. It doesn't scale. Writing the code mod, this is a script that rewrites the code, but code mods only fix basically what you predicted. They're not very good at the blind spots where the software is actually breaking on the runtime. And worst case, it does nothing and leaves the repo in a state where you think that the software and the migration is done, but it actually isn't. And this is actually quite industry-wide. Google's own migration team found scripts break down in the complex non-uniform code. And every big change is a boring majority plus a very messy tail. And that tail basically decides what is the tooling that you're going to use for these kinds of migrations.

And it's very easy to underestimate them. So this is actually how everything starts. It's just a node bump. A day, most. Then reality kicks in. We have five node versions across all these repos that are live. We have a native don that actually builds only on a specific laptop. A log file that has actually been upgraded since lockdown, and no one updated that since then. And tests? Nowhere. We never found any tests. So revised estimate, Q3 from Q1. And Dave has since left the company, so good luck with that.

So we didn't start with a whole fleet. We started with one repo. And the way that we did that was one agent first, one repo, and five different steps. The agent looks at your repo and checks basically what already passes and creates the baseline. This is our baseline. This is our build. This is our linting. This is our tests. Then the agent will actually migrate this repo. It runs again all the quality gates, makes sure that they're passing, and finally open the PR. Get this one thing boringly reliable first, before moving to the agent fleet.

Now we're going to see basically how we can actually make all these different steps quite reliable. The interesting part is the migration. So every migration splits into two different steps. One is the deterministic one, and the other one is the non-deterministic one. The deterministic has the same exact result every time. We bump the version, we rebuild the whole repo, we run the tests, we run the log files, we update the log files. A script can do that.

And the other side actually needs a judgment. And the script is not a good job for that place. Fixing whatever actually broke, fixing whatever actually broke, it's very hard to every different repo, because all the repos are actually breaking into different things. And that side actually needs a brain behind that.

Starting with the cheap side, the deterministic lane is basically pure scripts. Pure scripts cost us zero dollars, because LLMs don't need any brain around them. They can actually just run, they can upgrade the log files, they can update the node versions, the node engines. Honest caveat is that just a script can fail.

A log file rebuild depends on the registry and on the native code. When it does fail, it doesn't guess. It stops and asks for a human. And this is how we add a human in the loop. So where exactly is the line between the script and thinking about it? So basically the scripts will only touch the version files and the log file.

And when we actually start needing the judgment, this is where the LLMs kick in. We don't want to actually inflate this kind of migration with other irrelevant steps, like we don't run auto formatting on the whole repo, we don't run fixing the lending, or other irrelevant changes. So we can actually keep the PRs very small and the humans can very easily review them. That line also decides where we actually spend the money. What we need to do is basically match the model to the work that we want to implement. The scripted lane, as we said, is free. We only pay for the fixes.

A cheap model can triage every fix and handle the easy ones, which is the majority. And when this first triage actually fails, then we can delegate to the stronger models and retry for fixing them. So this can run a few more triages and the bulk case is going to be sold from the cheap ones, and then we are going to be paying only for the stronger ones to fix these few issues that we might have in these repos. Now, one thing that makes this whole loop trustworthy is knowing what broken exactly means. So before even we touch the repo, we run the build, we run the tests, we write down what already passes and what doesn't. That's our baseline. And then the agent only changes failures that know that they have changed because of this migration. It never wastes time on things that were broken before we even did this migration.

Here there's a small catch, because if we have some flaky tests, this actually poisons the baseline. So we run a few times and set the flaky ones aside if we need to. But a baseline is worthless if the repo doesn't have an eyesight on itself, which is a real like first gate. Tests are the agent's eyesight. So if we don't have tests, there's no baseline, there's no signals. So verification comes before the risks, always. Does it build? Is there an SCI? Is there a real coverage? If not, you invest there first or you hand it to a person. And to be honest about what coverage buys you is not the correctness of the migration, it's actually the autonomy, being able to be autonomous into this kind of migration with the agents. That's one solid agent.

Now how we turn it to hundreds. And this is like the important part. Here's the trick that made scaling simple. The agent never calls each other. Each agent is stateless. It reads the board, does one job, write the result back to the board and stops. If something breaks, it lands as stuck. So we are visible of these issues. We are visible of this situation. We are not swallowing it. The handoff through the board never directly with each other.

So if they never talk to each other, how do they agree on anything? So the message isn't the agreement is not the message format, but the board itself. Each agent has a rule, which rows it will pick up and which columns it will write. The risk agent grabs the rows that are marked as PR opened. As you can see over there, it scores them from one to five and writes the score back to the board. That right is exactly what makes the agent rule match. And notice this score is deterministic. We have plain rules over there. No model needed. And the column is the actual contract.

Because that's the only contract, adding extra steps is very easy. So as you can see, we can just add a step over here. What it picks up, what it writes, what it runs, and nothing else in the fleet knows it exists. In your stage, you can just add it in this list and it can run basically in this order. No rewiring between the agents. The agents are just talking to the board. So what runs all of them? Less than you'd think. This is the whole orchestrator. Very simple. No messages, no bus, no queue, no agents to agents call.

Just a loop over the board, reading the items, run once every few hours. And for each stage, we actually grab these matching rows, do the job, then we write back to the board. Stateless, safe to run. That's our runtime. That's how we can run multiple steps simultaneously.

One piece handles the heavy lifting. There are two loops in this workflow. The first loop addresses failures in CI gates, reapplies fixes, and opens the PR when everything is green. The second loop manages PR level checks, retries until green, and auto-merges on success. To prevent endless retries, we set limits, escalate cautiously, mark issues as stuck if needed, and involve human intervention when necessary.

A loop without bounds can lead to runaway scenarios. The loop launching the PR safely is crucial. Everything relies on this shared approach. Establishing boundaries and handling failures effectively are key. This strategy is considered vital for future projects, emphasizing the importance of controlled autonomy.

But a loop that just retries until it's green is dangerous, right? So what we do? We need to lease it. And every retry until green hides a real bug and burns money. What we do? We bound it. Try the cheap model first. Escalate the strong one only on failure and only twice. Still failing. Mark it as stuck, move it away, and a human will actually get into the loop. This isn't just my paranoia. Karpathy just put in a few words. Keep AI on a lease. A loop with no bound is an autonomy. It's basically a runaway. The agent that opens the PR is easy but the loop that launches it safely is the actual product. Everything so far leans on the shared thing and let me name it. This is the part I'd steal for my next own project.

The board isn't just the storage. It's the control plane between humans and a fleet of agents. Every column is a signal. The agents read and write. A human overrides just by editing a cell and when the reality drifts, basically a human can get into the loop and merge the PR by hand. One agent can reconcile when external users have actually updated the PRs and it always converges on the reality. What is the current state of the PR? We used Monday. You can use anything. You can use a database, you can use a spreadsheet, you can use pretty much anything. This is what the board looks like. This is a list of my MFs. All the MFs that I'm actually migrating into. You can see that we have different statuses about the merge state, the risk, the PRs, a link to the PR so agents can actually go through. This is basically how they communicate with each other.

So let me put everything, every piece back together on the screen.

This is what we built. We have the orchestrator that walks through the different steps. First we upgrade, then we run the CI gates, then we assess the risk, and finally we auto-merge. Everything is done through the board. That's the third bus, as you can see here in the middle. Then we have a few more agents that keep the state of both the board and the PRs in a good state.

The refresh which actually will run when there are conflicts on the PR level and the sync which will bring the board back to the real-life situation when humans also get into this process. Finally, if the risk is low we can auto-merge these PRs. If the risk is high then we actually move the state to a stack state and human can get into the loop. So what did this all buy for us? You don't ship a production 527 PRs all at once. You rate the risk from 1 to 5 and then you ship the safe ones first. Low risk is the majority, up front and then the risky tail in the end.

About 87% run end-to-end with no human intervention and that's by design. Agents do the volume, humans own the judgment tail. The 13% is where a bad auto-merge can actually turn into a very memorable post-mortem. So humans here own those on purpose. Here what we got wrong before we got it right. Green CI doesn't mean correctness. As Simon Willison puts it, a passing test suite doesn't mean the server even boots. So not bump changes runtime behavior. Be aware of that. The test might never check it so we need to test it with the baseline diff.

A flaky test that finally passes on retry is a regression in disguise. We require stable green green twice in the CI before we auto-merge. Finally, the repo with no tests is the one that is going to bite you. So verification is very very important before autonomy and there's no exception to it. We didn't get here by luck, we got here by knowing exactly where it breaks. So here's the real payoff and isn't just one migration.

The machine isn't a single use. You can use it in your next node bump, you can use it in a lean rollout, you can use it in a framework upgrade. They all start near the bottom of this curve. Same machine, a different rule. You turn months long program into a couple of weeks. The win isn't the migration, it's the capability of doing that. So here's what you can take home. Six lines. Split the deterministic work versus non-deterministic and route the cost by step.

Verifications come first before any steps and any agent's implementation. If we have no tests, we're not going to have autonomy and this is what is going to be biting us in the future. Build one very reliable atom before you start parallelizing with multiple fleet agents. Decouple through the board, get a board and actually make it the directorship for your own agents and the way that you communicate with your agents between human and agents and then land the loop through this board and finally order by risk in order to be able to auto merge these kind of changes and you don't have to take my word for it.

There are two ideas that the field is actually converging into. First, the machine is the asset. There's a growing consensus that the model is swappable. The harness around it, the loops, the tools and the guardrails is where the product lives. My whole talk is one world example on that. Second, the human owns the tail. Autonomy clears the mechanical majority. Judgment, knowing what good looks like, is the part that you can't automate away. Build the factory. Keep the human on the judgment tail. This is also what Gergely Oros came up with and this is where the value as experience of our engineers come in. We need to own this judgment tail. So we didn't migrate 527 repos. We built a machine that migrated these repos for us and is running right now. The migration was just a demo. The machine that we built is the actual asset.

Thank you very much. Fantastic, thank you so much. All right so I see that we've got a few questions here. Reminder for everyone on the stream or in the room, if you want to ask a question put it on Slido 0607. Okay, first question. You showed some slash commands but also pseudo code. Are you using an agent sdk approach or a daemon run agent? Yeah, so basically we used both approaches. So the first approach was actually by implementing our own skills. So we can actually implement this agent workflow. Now we're actually working into implementing that with the Anthropic SDK. Sorry for that, and basically it's quite abstracted in the way that can actually be migrated also to different kind of providers as well and element integrations. So yeah, I think that answers the question.

Amazing and next question we have here. What was the most memorable issue caused by auto merge? Are the risks worth the benefits? Yeah, so we have had some issues of course like into this kind of migration.

One of them was actually, I mean they didn't actually affect the end users in a big kind of capacity. But the thing is that we learned basically that for example the green ci doesn't necessarily mean that the migration is done properly because as I mentioned this migration will actually touch runtime execution as well. So this is very important and if you don't have like enough verifications in place then it's going to be very hard for you to test and make sure that everything is working fine. So yes, that's why invest more and more into the verification part which is adding more tests as many as you can.

Has this increased the amount of time spent reviewing PRs by your team members? How do you deal with this so they aren't locked up and unable to do their own work? Yeah, basically as I mentioned the majority of them are actually auto merged. So basically the 87 percent of them were actually auto merged. We didn't have to add the human in the loop. That's why we're actually investing first into the testing. So we can actually also based on these risk factors evaluate the risk of each of the PRs. If the PR is risk level is very low then we cannot merge this without nagging any other kind of colleagues. All of these changes also have an owner from the teams, from their relevant teams. So if something goes bad in production they also like the owners of this kind of part.

How long did the factory take you to build? Any lessons learned from the orchestration side? So basically this whole thing took us a couple of months to implement and make sure basically that it is running in a good scale. We didn't actually were able to run it across like 500 repos at the same time, but we actually incrementally started adding basically this factory along. What was the second part of the question? Any lessons learned from the orchestration side? Yeah, from the orchestration side we learned basically that you definitely need to have an owner as part of any of these changes because otherwise the teams themselves might not be as aware of what is going on and basically something that might go wrong when this is released to production.

So basically this whole thing took us a couple of months to implement and make sure basically that it is running in a good scale. We didn't actually were able to run it across like 500 repos at the same time, but we actually incrementally started adding basically this factory along. What was the second part of the question? Any lessons learned from the orchestration side? Yeah, from the orchestration side we learned basically that you definitely need to have an owner as part of any of these changes because otherwise the teams themselves might not be as aware of what is going on and basically something that might go wrong when this is released to production.

The methodology you propose seems beneficial not only for migrations but also as a general approach to any task requiring scalable AI automation that remains on a leash. How have you used this approach in other areas? Yeah, so basically this is actually another reason to invest more into verification. So there are teams that are actually not investing that much time, unfortunately, into adding extra steps, extra tests like end-to-end tests or like storybook testing or anything like that. So it's a very good opportunity to actually raise the awareness of that and basically make sure that everything is verifiable and can actually better deliver basically new updates in the future.

So because like of our microphone implementation. So basically we have like, we give the teams basically autonomy in the way that they want to implement new features. So our microphone implementation actually allows them to create new features by implementing basically new micro frontends and basically, each micro frontend can be as part of a mono repo or a repo. So they are actually part of their own kind of code base. Great. Okay, this is a really cool question. So if you have applied this to product development, what was your input? How much effort are you placing in human and or AI spec development? Yeah, this would be actually a bit harder to implement in the full feature life cycle because of the unexpectancy of what the feature will actually be. Then we actually need to go back to the spec driven development and actually start discussing basically how we write a good spec, what is a good workflow. This kind of migration was actually like a very small and very like determinist kind of migration, but we plan to apply it into more and more kind of migrations in the future. Feature wise, we are not there yet. Okay, there was a really good question that has just disappeared. Okay, Jamie has asked, with auto merging code based on risk, there's always things that could go wrong. How are you protecting production at the same time? We always have a rollback mechanism. So this is like very important.

Okay, this is a really cool question. So if you have applied this to product development, what was your input? How much effort are you placing in human and or AI spec development? Yeah, this would be actually a bit harder to implement in the full feature life cycle because of the unexpectancy of what the feature will actually be. Then we actually need to go back to the spec-driven development and actually start discussing basically how we write a good spec, what is a good workflow. This kind of migration was actually like a very small and very like deterministic kind of migration, but we plan to apply it into more and more kind of migrations in the future. Feature-wise, we are not there yet. Okay, there was a really good question that has just disappeared.

Okay, Jamie has asked, with auto-merging code based on risk, there's always things that could go wrong. How are you protecting production at the same time? We always have a rollback mechanism. So this is like very important. So when everything like keeping alerted, having observability, if something goes wrong, either use your error tracking tools, log in tools. When you get alerted, be able to roll back. This is very important. The scale of it can vary, but having basically like the, if it is like business critical, having a human in the loop is very important into this. I really like this next question on observability. So what's your observability for debugging a stuck item? When something lands in stuck, how do you trace that back through which agent did it? We have a, what is the, about the agent, who agent did that? Yeah, which agent did that? Sorry, my slide keeps jumping. No, no, no. So basically like this is like the agent, we have multiple agents, like in the process. So basically, depending on the status, we can actually identify where exactly we are in this kind of process. So basically by referencing the board, we can understand basically which is the exact step, which agent will actually run last. So we can understand basically where it got stuck. So then a human can actually get into the loop and get it from there, like unblock the next step and then assign it back to the, to the agent. We'll take one more question because we've got one more minute left here. I'm going to ask you metrics. I love questions about metrics. Do you have metrics for the time from idea to production before and after AI? Idea to production before and after AI? Specifically for this kind of, for this kind of migration, I would say that this kind of migration would roughly take, for all these repos, would roughly take six months more or less. But with this kind of like approach, even like by building also like this kind of like agent fleet, it took us like three, three months, let's say. So, and you can also like basically depend on this kind of like fleet to for the next kind of migration as well. So basically this is where the value lies.

Incredible. I know there were so many questions left on the slide though. I don't know if you're staying around for the rest of the conference. Yeah, I'll be around. So please feel free to find me around and ask any questions. Thank you very much. Amazing. Thank you, everyone.