AI-accelerated Legacy Modernisation

So yes, I want to talk to you today about our learnings in adopting AI in legacy modernizations. And first, why this is actually a very important problem is that our world is running on legacy IT systems built more than 20 years ago. As software has eaten our world and taken over planes, cars, homes, and healthcare, this has actually been built as layers and layers on top of the legacy IT running our world, the energy, and transport, finance, communication infrastructures. And this is actually starting to show some quite big issues because it's creating three types of risks.

First, the business risk. For example, Nike Capital, a trading company, rushed to release a new feature and they reactivated by mistake that code in the systems that they hadn't touched for nine years, but they had forgotten to delete it. And this created weird training activities and they lost $440 million in just one day. Then there's a security risk. Another example is that the Scala system controlling Ukraine's power grid that dated from the Soviet era and allowed attackers to remotely cut power twice in the country. And finally, a compliance risk.

You know, when Citigroup is transferring $81 trillion to one of their clients by mistake, which makes you want to become a Citigroup client, of course. And it's actually the 10th time that they've transferred more than $1 billion by mistake. Well, every time they managed to undo the transfer, but at some point they get fined $136 million for these very clunky systems. So investing in modernizing IT legacy is becoming a global problem. But of course, all organizations across the world are procrastinating because modernizations are incredibly difficult problems. We kind of categorize the challenges in four categories.

The first one is legacy ecology. Once you start trying to understand the system, you realize nobody in the organization remembers the business rules. Then you've got reliability rupture. Everybody's scared to touch something that kind of works because touching it will probably create more issues at least at first. Then there's a feature freeze. You know, business is very annoyed when you tell them that you are going to stop building new features for a very long time while modernizing the underlying infrastructure. And finally, migration mutiny, because teams working on the legacy systems are often not that excited to help the team migrating to new systems, despite being actually quite critical in making the modernization success.

So this is where AI is coming, especially since I would say Cloud4, GPT4, when AI started being able to actually code decent tasks on their own. So that's a unique opportunity. And at Theodo, when AI arrived, they were wondering, OK, it's not super clear how to adopt AI and problem solving and building new products because these are very difficult, very complex issues. But probably when it comes to rewriting technologies, modernizing stacks, that should be a great place to start. The problem is that AI brings speed, but also variance. And so the combination of the two, when untamed, leads to chaos. And this is, of course, what we all know as engineers, what we're all afraid of as engineers is like crazy vibe coding going into production.

So how do we accelerate with AI, but avoid the chaos? So I'm going to show it very concretely on an example, very much inspired by a real-life example. But I've kind of merged some stuff, some multiple examples together. That example was a leading player in the telemedicine space who wanted to merge their stack. So the backend needed a full migration because it had been built on Loopback 2, which meant they were stuck with Node 10. That was unacceptable. So the idea was to migrate to Node 20 and out of Loopback. And with 114 API endpoints migrated, this seemed like a doable project in about 26 weeks.

Of course, as soon as they started looking into the migration, as soon as they started executing and they started also using like properly analyzing the system with static analysis tools, they realized that the system was a lot, lot more complicated than expected with a lot, a lot of dependencies and cross dependencies. And worse, even the static analysis tool had missed whole chunks of dependencies hidden behind hooks. So the new estimate went from 26 weeks to 70 weeks. If you've done a migration project for the moment, everything is normal. So poor CTO has to find a solution and by chance we actually meet and I tell him about our experience of migrating an app in four months instead of 12 using AI.

So this was really our first big kind of AI accelerated monetization going from iOS and Android to React Native. So I don't know if that convinced him or if he just had no other option. But anyway, he accepted to work together. And so that's how we started. So of course, to come back to the initial question, which is how do we accelerate with AI but avoid chaos? We bring our secret sauce at Theodos, which is Lean Tech. So what is Lean Tech? Lean Tech is the system that we've built at Theodos over the last 12 years, growing from 10 people to 700 people. And so bringing all the knowledge from Lean Thinking. So from basically what makes Toyota an amazing company to software engineering, to maintain business and technical agility while scaling.

And what we've realized is that Lean Tech was not only super relevant for us to grow as a company, but it's actually even more relevant in this age of AI. So the first principle of Lean Tech is value for the customer. And that is a very interesting guiding principle because what we realized is that when we're monetizing, the first thing we need to do is to break down the legacy system into touch points, that's what we call them, that make sense for a business user. And so for that, we've got a discovery agent that goes through the system and starts mapping like individual endpoints and the dependencies. And then we use another agent, a reverse engineer agent to explain every touch point, but at the business level, so we can actually understand what is happening. So proper documentation, quite technical, but actually explaining what the value is.

So I hope this is going to work. This is demo time. Right. How does this work? Yes, it's semi-demo time. It's a video, it's not me, like, please. Maybe I'm missing it. Okay, it's working. So this is a tool we built to visualize all the endpoints that we have in a system, and we can select the endpoint that we want to migrate. Probably time to go a bit faster up. Here you can see for the selected endpoints, all its dependencies, you can navigate across that. You already get instantly like a feel for the complexity of this. So here you see first a lot of dependencies, but also cross dependencies between classes and functions. So this is not looking like the best, easiest example, and then you can click on a particular class, for example, function, and generate documentation in here. This will explain, there's actually quite a lot of work going on in the background, and it will explain that this, for example, is responsible for uploading CSVs to S3. Then you have schemas and explanations.

That was step one, better understanding what the legacy system does, and breaking down the problem into chunks that make sense for the business users that we call touchpoints. Why do we call them touchpoints? Because it's either API endpoints, when we're talking about backend usually, or when we're talking about frontend, it will be screens or interactions, that anyway, things that a user can then validate and test. So we go the other way around, test and validate. The next principle of Lean Tech is tech-enabled network of teams. The idea is if you want agility at scale, you need the right modular architecture to support the right team topology. What we want when modernizing a system is to heavily improve the architecture. How do we do that? This is where we bring our architect agent that combines the understanding of the legacy system with the codes and our architecture standards to write down the migration technical strategy. Time to write the code now, and that's where the right first-time principle of Lean Tech is super important.

We're not vibe coding here. We're using what we call the Jidoka approach to AI, to use a fancy Lean word. The idea is that we iterate on our agent coders until they achieve a very high accuracy on the first few examples. This means we're writing tests, the code is tested, and then we still pair program with the agent coders. We validate each line of code that is produced. This approach ensures industrialization and maintains high quality by shifting left quality and stopping early when detecting quality issues. By following this principle, we ensure that the code is thoroughly tested and validated in collaboration with agent coders, emphasizing the importance of high accuracy and quality in the coding process.

Another big aspect is that we've realized if we want very high accuracy, depending on the size of problem, we often have to break down the touch point and smaller components to get much more reliable results. This is where there's a big, let's say, architecture effort from the tech lead.

Finally, we have a QA agent that then runs a battery of tests on the system to check that everything's working. Demo time. Just like a magic trick, this is just to confirm that there's no rabbit in the hat. Basically, that this endpoint doesn't work. Okay, 500 doesn't work. Perfect. Now we're going to execute the agent coder. You'll be able to see it code on the right side, and on the left side, VS Code shows you the output.

This is accelerated. You see the time on the top right corner. And so eight minutes later, the agent is done. So it's time to run the QA agent. That takes it under four minutes. And we're done. All tests are green. Now it's time to check that if it worked. Yes, the API endpoint is now releasing, outputting the right results. This example, of course, for the demo effect, is not one where we needed to pair program. The task was simple enough that it actually worked right first time. We estimate that if we had needed to code it on our own, it would have taken us about two hours. So 12 to 120 minutes. That's interesting because it's a consistent kind of order of magnitude we realize. On specific tasks, we have approximately a 10x acceleration. All right. No please. Not again. Then there's a critical step, and clearly informed by our lean tech culture, which is leveraging the building and learning organization principle by doing Kaizen. Kaizen is another fancy lean word.

There's a lot of them. To capture the idea of continuous improvement. And here what we're doing, which is interesting, is we're checking all of the modifications that we made during the pair programming session. So things we didn't agree with, et cetera. And the agent collects all that data, analyzes what could have been improved. So here, for example, a comment about being unhappy about something. And that is then used to write suggestions of what should be improved at the system level. So how the agents should be modified or the standards should be modified to avoid that mistake in the future. So that's something that, of course, should be done by the humans. But again, we use an agent to accelerate some of that work.

This means, of course, that we are then able to industrialize. Because the more the system becomes robust and resilient, the more we can start accelerating the whole migration. So this is time for the last principle of this LeanTech system, which is adjust in time. So adjust in time is really principles that are quite famous in manufacturing. Inspired by how Toyota produces millions of cars every year at super high quality and super high flexibility. So in short, we industrialize the whole migration around these trust in time principles, making sure that if there's any issue at any stage, there's a red light that shows up, stops the thing, validate the issues, and always a human to check and assemble all these things together to make sure it works.

The results on that particular example, as I said, 10x acceleration per endpoint. Of course, we're not just migrating the endpoints. We have to invest in the AI tuning. We have to invest in the breaking down of the problem, et cetera, et cetera, et cetera. And then we have to reassemble everything. So what we've actually done in practice is finished the migration in 21 weeks, instead of the 70 weeks that had been estimated after the first step. The first try. And what is very important when we're talking about AI, because of the way we use AI, which is definitely not vibe coding, we actually end up with a much higher quality architecture. The two learnings I want to share from all this. First, that Toyota's 70 years of learnings on how to automate car manufacturing, so a very, very complex delivery issue, and doing that while delivering superior quality. Toyota is very famous for having four times less reliability issues in the cars than most of the competitors. All these learnings, which are usually captured under the body of knowledge called Lean Thinking, are incredibly relevant for our AI age. And that's very good for us, because we've invested 12 years, so we're quite happy. And the second learning is that AI can contribute to rebuilding healthy infrastructures if done right or if used the right way.

That's it. Thank you. All right. Cool. So we have loads of questions for you. We'll get started with this one, actually. How do you tackle the sheer amount of code generated by AI? Doesn't it get overwhelming quite quickly? I mean, that's definitely one of the reasons we're only going three times faster at this stage, because it is accelerating a lot of the boilerplate you have to write, but since we're still validating it, we're not getting a 10x or 100x kind of acceleration. That's one answer. And the other answer is, of course, in our architecture standards and in the guidelines that we're writing, we're trying to bring all our knowledge of how to create better systems, so typically creating the right levels of abstraction, so you don't create thousands of useless lines of code. Exactly. Thank you.

And that actually links nicely onto how do you know that the QA agent is testing it properly? So things like scope, true equals true, and things like that. We do measure code coverage, clearly. Well, we designed a QA agent to test the features, that's why we also separate the understanding of what the feature does from the tech migration plan, that's part of the answer. But to give a more transparent answer, we've also tried generating tests on top of code just to increase the code coverage. So very interesting exercise where, clearly, you're completely biased because you're generating the test from the code. But one interesting learning from doing that is we actually identified bugs in the code because the AI has this weird intelligence where it didn't test what the code did, but it tested what the intention of the function was, and so it did identify this edge case that hadn't been captured before. Thank you for that.

And someone has a question about wouldn't your multi-agent approach and somewhat specific driven development also be suitable for any new feature in an enterprise setting? So that's very interesting. I mean, so Theodore, I'm the co-founder of Theodore, we're a tech consultancy. And so our job is to build tech products for a wide range of clients. And clearly, the reason I'm speaking today about modernizations is, as soon as AI became this kind of existential, not threat, but existential question for us, we also realized it was unclear how we would leverage it in product building. So that's really an important thing. So we knew that if we invested in modernizations, we would learn a lot while creating value very quickly. And some of these learnings could then be captured in product building. So yes, on some projects now, we're able to go, I'd say, yeah, almost 2, 3x faster on like new builds. But still with a lot of constraints, you need to have basically the tech leads, harnessing the AI needs to have a lot of autonomy on the product. Because they don't have, like if you're dependent on a product owner, and if that product owner is dependent on a lot of stakeholders to make decisions, then the AI is basically going too fast. And the bottleneck is not the AI anymore, not the coding anymore. It's really like the product decisions.

So if the product decisions are not the bottleneck, now we are reaching a stage where we're actually going better on product building. So that's most products that I've seen in my life. The product decisions are the bottleneck. Yes, agreed.

And which AI models have you used for the migration of that for the example? And how did you evaluate which model to use? Yes, so we evaluate all of them on a regular basis, of course. The one that still stands out, and we need to try Gemini 3 seriously, but Claude has been our preference, because it's much more, I don't want to say obedient, but that's the intention, it follows rules a lot better. And since our approach is that every time there's a mistake, we iterate and add like a rule to avoid that mistake in the future, it's much better to have a Claude that will actually take into account each of the rules, rather than a GPT that is quite creative and decides to sometimes not follow the rules we've given it. Thank you.

And this actually is nice to this one. Do you have any examples, as it looks like you're using Claude, command files that you're using in a GitHub repo, somewhere to help us implement this lean tech approach? Oh, you mean open source examples? Yeah. Huh. No, not yet. It's a good question how much we want to share. I mean, some of it is like, let's say IP. No, I'm very happy to share a lot of our learnings, because I do think it's a global problem. And we won't tackle it on our own. But yeah, no, no, I don't think we've open sourced anything yet. So not yet is the answer. Okay, cool. There are a few more questions. Okay. Sorry, it's just that I'm just having a few. Here we are. So a question about all teams lose their knowledge of their codebases and become dependent on AI for even basic troubleshooting. How do you see that manifesting? I mean, clearly, we're right now, again, you know, the use cases I'm talking about, the team has completely forgotten what the business even does. Like, they don't even understand the features. So we're not making the situation worse. We're making it better. That's the actual answer. But of course, yes, but that's another whole debate, which is that when you use AI on current products, there's a trade off between the speed at which you're going and the mastery of the features you're building. Fabrice, thank you so much. We can't answer all the questions, but this is time. Thank you very much. Thank you, Fabrice.