From Experiment to Enterprise: Scaling AI Coding Assistants Across Engineering Teams

Yes, my great pleasure to share what I know, what I learned about scaling AI coding assistance and more general, adopting AI native development tools in engineering teams of the organizations. Of course, like, mainly relevant for medium to large ones, but the techniques are applicable because I see there is a huge gap between companies buying licenses for this or that tool and companies seeing return on investment on this movement. And in Microsoft, I help engineering leaders to actually find the way for this adoption. And what I'm going to share in next, maybe like, six, seven minutes is some pieces of framework called Well-Architected, created by my dear colleagues from GitHub and also, of course, my personal learnings from meeting with engineering leaders.

And day job, I meet these folks and their developer teams almost every day. So there is a kind of adoption paradox and let's uncover this uncomfortable truth. We often treat adoption of AI assistance and more general AI native dev tools as technology problem, while in reality, this is a change management problem. And it's not about buying licenses and organizing some technical scaffolds on how to use that. No, it's actually about rewiring how developers work, how whole developer teams work. And I'm here to share what allows in our timing this, let's say, tactical framework. What are your next steps and where to put main attention if you feel this gap as well? And three stages, three phases, and it could be your mental model. Onboard, adopt, succeed. And we'll briefly go through these three pillars.

First of all, onboarding. Just to make sure that your developers have access to these licenses, to these seats, subscriptions, whatever. And for sure, you want to come up with different policies for vetted versus unwetted developer tools, especially if you're talking about AI. You definitely don't want to end up in a situation what we call shadow AI. It's much worse than shadow IT. If your developers are not using AI tools you suggested them to use, make sure they use something else, starting from old gold copy pasting to child GPT or whatever. It's a nightmare. It's a disaster for every IT manager. You don't want to kill yourself by manually distributing licenses to the developers. It heavily depends on a particular tool, particular vendor you work with. But try the ones with capabilities where developers can self-serve themselves. And to sleep good at night, make sure that all security guardrails are in place so that works both for you and for developers that you offer these developers tools to. Starting from, I don't know, maybe today's modern example, which MCP servers they can connect to their AI coding assistants. In many cases, I mean, also depending on a particular vendor, some of them either all or nothing. Some of them provide more granular control. So make sure you choose the right one. Then if I ask you to remember one particular term from my presentation, it's human infrastructure.

And you cannot succeed with adoption of AI dev tools without building human infrastructure. Three pillars of this concept. AI advocates. For sure, there are always people in dev teams that are a bit ahead of, let's say, average developer in adopting these tools. And you are in the best situation if the same people are keen sharing their experiences, both successes and failures.

Communities of practice. Yeah. It takes a bit more effort than just organizing Slack channel or another team's channel. Yeah. Make sure that you nurture these communities. And executive sponsorship. Here we go back to change management. And if you folks went through change management training from ProSci Institute, if you not, my best recommendations invest some money and time in that. You definitely remember that reason number one for failed change management projects is lack of executive sponsorship.

And it's absolutely the case if you talk about adoption of AI dev tools. Maybe it's even more important. And please understand me right. This sponsorship is not buying pizza for next evening's event. It's much more demanding from leadership side. But it must have. And on this phase you measure monthly active users and ephemeral metric engaged users. Because don't be trapped in this simplicity of monthly active users metric. Maybe you are not that interested in average number for all your team or all organization because you better segment. You find people who are dedicated users, occasional users, or just tire kickers.

Because adoption strategies could be different for these groups of people. For example, identify dedicated ones and convert them to AI advocates, champions, ambassadors from human infrastructure slide.

And finally, how we measure success. Well, I'd say there is no silver bullet. There is no one particular metric. It's interconnected zones. And for sure you don't want to sacrifice one of metrics in favor of others.

It's super important for you to drive these metrics as a system, not to get trapped. For example, if you over focus on one particular metric, let's say speed of code reviews, you optimize that. Now they're super fast. But in reality, as a system, it failed because if you look a bit broader, a bit longer term, OK, you now have super fast code reviews. Can you guarantee the same code quality, same attention that you had with longer ones?

So your code quality metric sinks. And what about developer happiness metrics? Most likely tanks as well because developers now feel pushed and pressed under that. So principle is simple. All these four pillars simultaneously. And again, unfortunately, there is no simple way to track, to measure some. Some of these metrics are more or less automatically measured.

Some of the metrics can be only measured by serving, chatting with people. But it's the goal for you as a leader. Checklist for you to more or less group everything what I mentioned into strategic versus tactical points for you to better understand how to structure that.

And my last checklist for you, your path forward, what you do after this conference. Start with guardrails. Measure depth. You remember these four pillars. It's system of interconnected metrics. Think about these building systems, not particular metrics. And again, human infrastructure.

Folks, let's stay connected. I'm already in WhatsApp chat that we shared with you recently. So feel free to ping me there. And this is my link in profile. I'm super happy to accept all incoming connection requests. Just follow me, no pressure. And ask me for this deck or extended version of the deck. I have this deck for full scale session. Not talking like now.

And of course, as I mentioned, it's heavily based on amazing piece of content, well-architected framework by GitHub. And also engineering system success playbook. How GitHub implemented same like adoption in its own company. Like customer zero. Thank you so much. Thank you, Maxim. Thank you, Maxim. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.