Scaling AI Adoption: The Real Challenges of Transforming 300 Engineers

Hi, everyone. Today we will talk about how we are scaling AI adoption in a company where we have more than 300 engineers. So we'll talk about what are the real challenges, how we are doing it, what worked, what didn't work, and maybe just share a few, like, tips and tricks. So first of all, sorry, I lied. Actually, we are more around 400 engineers at the moment. So the number changed a little bit. But the principles, of course, are still pretty much the same. Just a couple of words about myself before we start. I'm Alfonso. I'm from Italy. I'm a tech lead in a company called NearForm. NearForm is basically a company selling services to highly regulated industries. We are, as I was mentioning, like more than 400 engineers while we speak. I am building AI agents. So I'm in one of our AI projects. I am supporting multiple teams adopting AI native engineering. And I'm also writing a book called Learning AI Native Software Engineering. But just I will give you a couple more informations later on.

Today we will see a little bit of a journey. So we can imagine a team on the bottom left, a team like usually a cross-functional team with software engineers, PMs, designers, every type of professional that you might need. And we will see how this team can be transformed to become an AI team. We will see some pitfalls, some issues, you know, something that they will learn along the journey. What are the techniques that we are teaching those engineers and those teams? And you know, hopefully, something that is going to be useful and reusable across your teams as well.

There is a challenge, a very common challenge that I'm seeing across multiple teams, across multiple of our clients, which is, okay, we got, we bought the copilot licenses or cloud code, whatever your team are using, but nothing really changed. I mean, yeah, we can see that something changed in terms of like velocity and like what the team is producing, but we're not sure whether it's for the price, right? So we have seen that, yes, something is happening, but we're not sure about like what's really changed once we got like the copilot licenses or whatever. And so actually what we found out is that like moving in engineering department of a lot of engineers from just experimentation with AI tools to like really AI-native development, it's way more than just tooling, right? It's going to take mindset shifts, tooling, of course, upscaling quite a lot, a lot of peer learning. So today we will answer a few questions. So we will answer why buying tools is not enough. First of all, what actually shifts when teams go really AI-native, right? When things start to work and how do you scale? How do you scale everything to an entire organization without breaking people, processes, teams, and all this good stuff? All right, first of all, let's talk about a maturity gap. So I would expect that in your organization you will have like multiple teams, and I'm almost sure that you can take every team and point them into one of those quadrants.

So this is basically a very simple diagram. And as we can see, we have individually AI fluency as x-axis. So like people learning skills, implementing the right tooling, using SDD, using verification, building harnesses. And then on the y-axis, we have the team standardization, because what we noticed is that it doesn't matter too much whether an individual has AI fluency if the entire team doesn't get all these practices. Starting from the bottom left, everything is untouched, right? So the team doesn't have standardization. People don't have AI fluency. They didn't do up-skilling. So basically, everything is coding in 2022, pretty much untouched. So nothing changes.

Then, in some cases, we have seen teams where there are like possibly rules, there is team standardization, but really people didn't do up-skilling properly. So even though the rules in theory are in place, we cannot see a huge amount of benefits. On the other side, on the bottom right, we can see that there are like champions. So there are very skilled individuals, but there are like no shared rules, no team standards. So everyone is more or less doing their own thing. And so, even though from an individual level, people are smart and they're working well, there is no compounding at the team level and then at the organizational level.

What we're looking for is to move almost everyone at the top right, which is the AI native team. So we have skilled people, the right tools, the shared practices, and those are the teams that we really want to scale. Those are the teams that we are looking for. Of course, it's not just, as I was mentioning, it's not just about different teams at different levels in the maturity scale. It's also about within the same team. Within the same team, we're going to find the same gap, pretty much. So we will see, we will have someone which we call an AI team engineer, which is a person that knows how to leverage properly, the tooling, the practices and everything in an AI native way. And then we have inside the same team, we might have product manager, engineer, slidge, platform engineer.

So everything that you really need into your team. And every person within the team is going to be at a different level in maturity scale. So someone might be starting to experiment with AI, someone might be very mature in their journey. Someone might be a bit reluctant. So even though you are in the same team, you actually have different starting points.

And of course, what you're going to need to do is first of all, upskill people at the individual level, but then also upskill them at the team level. Of course, this doesn't happen in void, right? So the team itself doesn't happen in the void. You have surrounding teams that will interact with this team, you will have like custom standards at the company level, you will interact with customers, with leadership, with the engineering department, the design department, you know.

So it's not just about building the team, but it's also about at the wider scale, building the right context for this team to thrive and to learn. One thing that we notice, especially from working with very senior engineers, is that there is an aspect of resistance. We notice that at least initially, right, maybe right now it's happening less and less. But we are at a different level in the maturity scale. What I would expect is that if you start right now to introduce AI native engineering into your company, I would expect that there might be some resistance from skilled engineers.

And there are like common objections that actually make sense, right? So the first one is, okay, but I tried like six months ago and this all doesn't really work for my use case, right? I'm doing like this very narrow thing, which is specific, which is important, which is mission critical. And those AI tools, yeah, they can make like a fancy React application, but they cannot do well. They cannot work well on my use case. And this is the fear of losing precision and quality in complex systems. And that's completely normal. As we will see, that's not really the case. So we have evidence now that AI systems are good enough also for very complex use cases if we can guide them correctly and we build right harness.

The second thing, the second common objection that we have seen multiple times is, okay, I will stop thinking, I will just prompt every day, I will just tell the agent what to do and the agent is going to do it. In reality, it doesn't work like this, right? So when you learn how to use this tooling properly, what you find out is that you are really thinking, you are designing the system, you are validating the system. So you're doing basically a lot of stuff, like your thinking is still there, you're just missing the implementation bit in some cases. In some other cases, you might just want to intervene and implement it yourself, right? So it's not, you will never stop thinking, right? If you're using those tools correctly, if you're using those practices correctly, you will think quite a bit, right? It just changes the level of obstruction in this case.

And then the third one is, I became an engineer to write code because I was passionate about code. I think we all heard stories of engineers starting writing code at when they were like 13. That's my story, for example, I started coding when I was very young and then became an engineer because I love writing code. What I found out actually, like after using those tools for a while, is that I really don't like writing code for the sake of writing code. But I love writing code because it helps me to solve a problem, right? And so if we can build the specs, if we can build the architecture, if we can build the verification layer, we are still solving the same problem, right? It's just in a slightly different way, just with a slightly different level of obstruction. So a lot of it, a lot of working with this resistance from skilled engineers is about reframing what we are doing and what's our job and where we can create the real value.

Because the value is moving from like implementation, like picking a card, picking a story from Jira, to high-level judgment, to, for example, designing the system, being almost like an architect. And then, of course, decomposing problems, building specs, the verification layer, as I was mentioning, like moving from like writing the code to reviewing the actual code to understanding what's doing to ensuring that everything is stable. And then, of course, a lot of very skilled engineers, they say, oh, I don't want to write code, right? And that's, of course, something that makes a lot of sense. I feel there is a very important difference between like by coding or by prototyping, however we want to call it, and like AI native engineering, where we want to ship production ready code, right? Production quality code.

There is an individual contribution journey, and then there is like teams journey. Let's get started with the individual contributor itself. So AI native engineering changes the unit of work from code to intent. And we will talk about the intent in a minute. And then when we try to rephrase a little bit and think about how we want to change our mindset, we should think that software engineers are moving from implementers, right, to orchestrators, to people building an environment that can be optimized not just for humans, but also for AI agents. Let me give you an example. When you are onboarding a new colleague, you are usually writing documentation, creating scripts, doing pair programming sessions so that you can onboard this person. Now, let's imagine that you have to onboard an AI agent, right? You want to give this AI agent all the tooling, all the context, make the AI agent able to explore everything so that it can perform as it must. And that's like building the harness.

So the new center of gravity, as it was mentioning, is now defining the intent, setting the constraints, and building the verification layer. So for the intent, it's basically like deciding what we want to build before we build it, giving clear goals, working with our team, right, working with product managers, working with designers. In the first phase, defining the intent is super important, and that's one of the first parts of the Spectre development. Then there is building constraints, which is what I want the system to do, what I want the system to not do, right? We want to create boundaries so that we can keep our AI on track, basically. And then there is the verification layer, which is we need ways for the agent, ideally, to self-validate what's building so that we ensure that the final, so that we can trust the final result, and then we can properly review it. I just want to give you a very quick understanding of Spectre development. I know that this is not a talk about SDD, but just very quickly, because it's one of the important aspects that we are teaching to our teams. So SDD is basically a methodology to work with agents where we, first of all, define the, our intent, right? So what we want to implement. We create the spec, we create a full spec file. Then we have the human in the loop. So we review the spec, we ensure that it's aligned to what we are looking for, and then we create an implementation plan together with the agent. So we are going to tell the agent, OK, let's create an implementation plan together. I can steer you a little bit, I can review what you build. And so we iterate until the implementation plan is ready. Once the implementation is ready, the tasks have been decomposed and so on, we can ask to the agent, we implement every task and sub-tasks, of course. And then we can review the implementation, iterate on it, create PRs, you know, and then we go back to the next task and we create the new spec. As we can see, there are multiple places where there is the human in the loop, because, of course, we want to be in control and we want to steer our agent.

And then there is a little bit of a substrate, which is very important, super important, which is, first of all, the project context, right? So our architecture file, our PRD file, so what we are building, our skills, everything that can be useful for the agent to steer it into the right direction. And then on the flip side, we have everything around verification and guardrails. So just to give you very simple examples, our unit tasks, our lynching, formatting rules, our, I don't know, application guardrails, like everything that can give feedback to the system while it's creating the plan, while it's implementing, right? So that it can almost figure it out what it did wrong and then fix the implementation before the implementation is sent back to us for review.

Now, what we are doing and what we did, which I think has been very successful in upskilling all our engineers, we basically found multiple ways to do co-learning. So instead of just giving them, hey, that's what you have to learn, go for it, we designed co-learning opportunities, which are basically moments where engineers teach to each other, speak to each other, share what's working, what's not working. We are building together. We are having a lot of sessions, especially because this world is moving so fast and there is nothing set in stone. So we want engineers to learn from other peers, which are doing really, and which are sharing what works and what doesn't.

So the first thing that we have is, well, actually multiple training support channels where people can share asynchronously news, blog articles, what they're working on, whether they have questions, whether they found something interesting and then they just share it. So this asynchronous way to share information has been proven to be very, very helpful because people can just literally learn from each other. It's like having Stack Overflow, but for learning this type of AI and native engineering practices and tools. So that has been very helpful. Apart from the asynchronous part of it, we do also have a synchronous part of it, which is the community sessions. So we basically host multiple community sessions, usually a few per month, but like in some cases, we have also multiple community sessions per week, which are completely free, of course, non-mandatory, and recorded in case you missed something that you were interested in.

And so basically, we have multiple live spectrum development sessions, but AI native engineering sessions in general. And of course, it's not just like a show and tell. It's an open discussion about things that worked, things that didn't work. And of course, like everything which is going under the hood, everything that our teams are doing.

On top of that, we are also engaging and we are asking people to do practical stuff. So one example of this is an app that we organized, where we asked people, I think it was like over Christmas, we asked people, OK, folks, let's have some fun. Let's build crazy side projects with AI agents and AI native engineering. We share what we learned. We share the projects. And this has been the opportunity for a lot of people to try it out for the first time on a non-trivial problem, on something that was real.

Of course, all this stuff goes together, goes along with a properly built upskilling path, an internal upskilling path that we have. This upskilling path is mandatory for our engineers. Of course, if you already know some of the stuff, you can skip what you already know and you can just go straight into what you're looking for, for the upskilling. But the main idea is that it's divided into multiple modules. So the first module is, of course, what is AI native engineering, right? So we basically collected a lot of different existing material, which is useful. And so what is AI native engineering? How it works? What are the tools, the practices?

So the first module is, of course, what is AI native engineering, right? So we basically collected a lot of different existing material, which is useful. And so what is AI native engineering? How it works? What are the tools, the practices? And then we go to basics, which is like, what is an LLM? What is an agent? How do things work? What is memory? What is the context? What are the limitations? What are the failure modes? What is an hallucination, for example? Why the human in the loop is important? So we want our engineers to be skilled on the basics first, and then we go on to more advanced concepts. Then, of course, we have to know what is the AI agent landscape, right? So how things are evolving. What are the tools that we can leverage into different projects? So it's just about exploring the current space and what we can use. And then, of course, everything related to context engineering, how we can build right context for the agents, that's very helpful. It's a big part of our job, right? So skills, tools, everything around that. So context engineering is a very big part of it.

The next big part of it is spectrum development, which is the methodology that we are now using in the vast majority of our projects to build production-ready code. And so for SDD, we are investing quite a lot of time into spectrum development. And during this upskilling part, something that we are asking to our colleagues is to build a project a bit similar to the hackathon, but building a project, a real one, with spectrum development. And they can choose whatever project they want to build, like a side project, and they just build it, and the result is amazing. Of course, then we review the project, we score it. If there is something that didn't really work well in the adoption, in the usage of SDD, we give a feedback, and then everything else. So of course, this upskilling roadmap is in continuous evolution. And we also released an open-source version of it, an expanded version of it, on this GitHub URL. So in case you're interested, you can just take a look. This is not the internal one that we have, but it's a little bit more expanded and wide.

There is one thing that we noticed over time, which is some of our very strong engineers are starting to orchestrate multiple agents at the same time. And there is a cognitive load into orchestrating multiple agents at the same time. So you create different work trees, you can work with the agents on different problems. There is the cognitive load issue to manage, which is after five hours of doing that, your brain just melts, because you have a lot of context switch. So you have to manage that. I'm just mentioning this because I've seen it in a couple of contexts, and I think it's important to just keep in mind that sometimes we just have to slow down and manage priorities correctly. Now, we have seen the individual contributor journey, right? But the question is, okay, we talked about three things, right? So we talked about the individual contributor, the single team, and the entire organization. Now the question is, okay, how we are scaling AI and IT engineering across multiple projects. And the way we're doing it is with this iterative approach of trying something, learning, and then scaling it. So the first thing that we do is we choose a small set of pilot projects, right? So we try something, we say, okay, this project is starting. And we identify some light out projects, right, which might be a good fit for what we're trying to understand. We staff them with a very small and upscale team with like some of the AI champions. We run the project with this methodology.

We run the project with this methodology. So we run the project with everything around AI and engineering. We execute the pilot in real condition. We collect as much feedback as we can. Ideally, I don't know, weekly or biweekly, we collect as much feedback as possible via, I don't know, recordings, qualitative feedback, quantitative metrics, like we just got as much data and as much context as we can. We try to get what worked and what didn't work, something that surprised us, something that the team started to do, but now it's not doing it more. So we just try to learn as much as we can.

Then, of course, the team is reflecting. So we are just looking at what worked, what didn't. We are updating our tools, our practices, our guidelines. We are just getting all this information and generate signal for the next teams, starting from all this information, right, starting from all this data. We basically summarize all this data in all the learnings. And then we take all the learnings and then we expand our project base. So we generate, basically, an updated playbook and we give all this information. And, of course, there are some moments of knowledge exchange. We give all this to the other teams. And then the pitch, and then, sorry, the loop repeats.

So basically, we expand, we create, we target new projects, we try to test new ideas, we try to test new assumptions, we look at what's going on in the market. And then we say, okay, let's learn on this new wave of projects. And we are doing that multiple times now. And it's giving a real competitive advantage because every time you learn something new, every time you learn what's working now that wasn't working three months ago, right, and so you can stay very at the edge of what's possible right now with AI and inter-engineering, what's working, what's not working yet, what you can foresee that will work in a few months. And, of course, everything around how we structure the team, what are the rules that we want to put in place for the team, what's the ideal team structure, and how we can optimize the velocity of the team, of course, while enhancing at the same time the quality of the outputs, right? And how we improve the communication with the stakeholders. So we're trying to optimize multiple variables at the same time while using this methodology.

Now, the question is, okay, but let's assume that our team has managed to learn how AI and inter-engineering works. Now we are shipping code way faster than before. What's next? So, where is now the bottleneck? Because now, assuming that your team is very good at using AI and inter-engineering, they can build code very fast, right? They can clarify requirements. They can build great products. Right now, the bottleneck then shifts from the actual implementation to reviewing the code on one end because you will end up with a lot of PRs, of course. And on the other hand, the bottleneck is understanding what we have to build. So, given the tier of constraints, the bottleneck didn't disappear. It just moved in different places. So, the question, what's next, is, let's understand, based on our use case and based on our context, where the bottleneck moved. Whether it's review, business decisions, or anything else. And let's try to work on that. So, for example, when we are working on the review bottleneck, we might want to introduce a better harness for our agent. We want to introduce PR automated code reviews. We want to integrate a set of layers stacked on top of each other so that we can trust the AI generated code more and more. And we need to spend less time on the review. But, of course, that's really at the edge of what's possible right now. And I'm sure that over the next few months, we will see a lot of this stuff changing.

Now, in case you want to learn more, as I was mentioning initially, I'm writing a book called Learning AI and Native Software Engineering, which is going to be published by Uralis soon. And in case you want to take a look, it's 100% free. You can either scan this QR code or go at this URL to get 30 days of free reading. You can read the early release. At the moment, we already have a few chapters out about what we mentioned. So like spectrum development, about context engineering, the first chapter about what really is AI and native software engineering and how it changes our job. So in case you're interested, just give it a shot. It's 100% free, and I hope that you will find some value in it. Thanks a lot for listening so far. I hope this has been interesting. I'm sure that things will just change quite a lot. In this talk, I've just tried to share what we learned so far, but I'm sure that a year from now, a lot of the things will be different. Thanks a lot for now and have a great day. Cheers.