Beyond the Hype Cycle: Driving real ROI with AI in Your Organization

Your AI-assisted engineering program is working beautifully. The dashboards are green, co-pilot usage is up to the right, leadership's thrilled. And it's producing nothing. Not because your engineers don't use the tools they do. Not because the models aren't good enough. They are. It's producing nothing for the same reason the agile transformation produced nothing. And the DevOps initiative before that, we're measuring the ritual, not the result.

I'm HD. I spent the last two years leading engineering teams, building the teams behind IKEA's global supply chain, one of the largest physical logistics operations on the planet. And when we rolled out GitHub Co-Pilot across those teams, I built one of those beautiful useless dashboards myself. This talk is about why it was a nothing-burger.

So a little warm-up. By show of hands, who here has an AI dashboard somewhere in their company? Okay, way less than I expected. Keep your hand up if that dashboard has usage, adoption, active users, acceptance suggestions or tokens consumed. Okay. Still, very low, way lower than I expected. Now, keep your hand up if that's some, if the same dashboard can clearly answer what shipped faster. Wow, zero.

So I'll show you a dashboard that actually fooled us. We rolled Co-Pilot out to a few hundred engineers three months in. The dashboard says 94%, weekly active usage. Wow. Tens of thousands of acceptance suggestions a day. Almost every single engineer is actively using Co-Pilot in their daily work. We presented, people nod, what a celebratory moment, right? Oh, no, hell no. Man, we are just burning tokens. Yep, the output's faster, PRs are open more quickly. Everything feels faster, that's more efficient. Engineers share how they use and build skills, how faster they are now, but I'm feeling something's missing, something's off. When we start to ask right questions, the truth starts to unfold. Is AI helping us save on cloud costs? Did it help make our environments more secure? Are we testing better now? Do we have lower escape defects? We ask ourselves, silence again, genuinely, long silence. So cycle time unchanged, release cadence, the same two-week sprint rhythm we've had since before GPT 3.5. And we've had baggage that we needed to shake off. Our workflows did not change. The mindset is still the same. So 94% adoption, zero change in delivery. So remember this question, we'll get to that later? What ships faster? We'll come back to it. So it was just a nothing burger, we're shipping fast, but what? So most of the dashboards we build or ask AI to build are not outliers, they are industry standards. McKinsey state of AI survey says 88% of organizations now use AI in at least one function. Only 39 can attribute to any impact on earnings to it. So basically more than half of the companies actually are not meaningfully benefiting from using AI. They're not making money. Worse yet, they're not saving money either. So why doesn't it convert to value? That's a bigger question. So where does the ROI go to die then? We use AI where it's easy, write code, write docs, write tests, those docs who nobody reads. Over 60% of teams don't use for the messy parts, planning, reviews, team retros maybe, deployments, monitoring and even better, they don't plan to. If you look at this tech overflow surveys, then actually I could have, I'll share this slide later, but 60% don't even plan to.

If you look at this tech overflow surveys, then actually I could have, I'll share this slide later, but 60% don't even plan to. So we're basically saying AI is transforming engineering while carefully keeping it away from the engineering system itself. That's not transformation. It's autocomplete with a marketing budget. I want to make a note here, Deloitte survey says in 2026, I'll go back one more, companies are still focused on AI fluency instead of redesigning work around AI. This is terrifying. I know nobody will like what I will say now, but if your team has had access to these tools for months and the workflow is exactly the same, don't start by questioning the model, the tools you use. AI just happens to be coincidental litmus test here. It's about the willingness to change in your organization, in your teams, in your people.

So we find ourselves in a waiting room, okay? We've arrived here, we have arrived in the AI, but we're in a waiting room. So there are three lies or metrics tell. Why are we stuck in that waiting room? The metrics we chase are telling us three lies. Each one comforting enough to stop the investigation, we're burning a lot tokens, we're using AI, we're shipping faster, the output is bigger, yay. So lie number one, usage equals adoption. This had been the biggest lie that fooled us. We had three months in 73% adoption, 73% of all of our GitHub users on the organization are using Copilot. We're utilizing it. Then we've seen 93, 94, but usage does not equal adoption. The 94% was usage, logins, completions, acceptance rates. Usage tells you engineers press tab, enter prompting. It tells you nothing about whether the workflow changed.

So don't ask are people using AI. Ask a harder question. Where would the workflow break if we stop using AI this way? Not because we should turn AI off. Please don't. That would be ridiculous. Of course, developers would slow down. But that's not the interesting part. The interesting part is if AI disappeared tomorrow, would your release process change? Would your review process change? Would your testing strategy change? If the only thing that changes is that people type code more slowly, then AI is still sitting at the edge of the workflow. Your workflow is still the same. Useful? Yes. Adopted? Maybe. Transformative? No. It should and leaves fingerprints on the system.

And lie two. We're not using AI only. We have great adoption. But it doesn't mean integration. You have adoption on an individual level. Everyone's using it. And everybody's actually benefiting from AI. Yep. But you are still not integrated yet. Suppose your engineers do rely on the tool. Are they relying on it inside the work? That matters. Ours genuinely adopted Co-Pilot for boilerplate, unit tests, scaffolding, commit messages. The parts on the job that were already easy. It just got faster. The hard part, system design. Debugging 20-year-old warehouse integrations. Code review.

The release process itself. Never touched it. Here's the principle underneath. Typing was never the bottleneck once we had AI. Okay. We have great autocompletions. We're faster. Adoption of the periphery is the easiest thing in the world. Because the periphery doesn't resist. Nobody's career depends on how their commit messages get written. Or how they document stuff.

So the core workflow resists. The core workflow has owners, politics, audit trails, and 20 years of muscle memory. Which is exactly why the core workflow is where the money is. And we are coming to the money now. We have adoption. We started integrating. Actually, in my organization, let's say, we are lucky and mature enough that we are integrated as well. We are using AI at its best until the end. We're power users.

Also, at the team level, we have adopted it. Yay. But integration does not equal value. Lie number three is this. So even when AI is wired into real engineering work, you can accelerate a process that was never the constraint. Our engineers really did write code faster. That faster code flowed into the same review queue. The same test environments. The same two-week sprints. We're still running two-week sprints.

We're still releasing every four weeks. We are writing faster code. But are all teams like that? No. We, of course, have teams that have integrated CoPilot deeply. And use the time saved to generate beautiful documentation around that. But we eventually established no human had read in two years.

More code. More docs. Same product. Same release cycles. So we were producing more code, but not more product. That's not a technology failure. That's the old truth wearing a new costume. Most AI problems are organizational problems that learn to type. Sometimes meetings, trainings, et cetera, you just cannot solve it by the conventional ways.

Most of the problems, this is not new that came with AI. This is not a three-year-old problem. This is an age, a millennial-old problem. Sometimes — not sometimes, but most of the times, almost always — the problems we have are actually people problems masquerading as technical problems. Better CI, better testing, better models, better tools, better agents is not going to solve your problem. Sometimes the people problems are actually masquerading as technical problems that you believe that you can solve. No.

I have a bonus lie. We are integrated. We are bringing value. Yay. But saved hours does not also mean saved dollars. No, they don't. Saved hours like hotel shampoo, okay? So everybody takes it, brings it home, but nobody actually uses it. So this one's my favorite because it sounds so CFO-friendly. That saved us 10,000 developer hours. Amazing. But where are they? I mean, are they in the bank? Did we put them in a warehouse? You know, IKEA terminology. Are they sitting in a little jar labeled AI savings? Of course not. Saved time is not value. Saved time is potential energy. If you don't decide where it goes, the organization will decide for you. And the organization usually chooses meetings.

So AI saved us time, so we scheduled a meeting. So if usage, adoption, and integration all lie, what do you measure? You measure at three levels. And if you refuse to let one substitute for another, individual, team, business or organizational level. Individual level. Did this engineer's throughput quality actually change? Not do they like the tool, measure output, PR throughput, defect rates, review turnaround, before and after. Same person, same code base. Two, team level. Did the system get faster? Level time, review velocity, time from commit to production. This level matters because individual gains routinely have a price here. An engineer who codes 30% faster becomes a 10x engineer now. Feeding a release process with a six week queue has changed nothing except the length of the queue. You're still shipping every two weeks. We made individual contributors faster, the ICs faster, and the system didn't care. Level three, business level, which is the most important here.

Time to market, cost, incident impact, revenue, the level your CFO lives on. The level my finance colleague asks about. Slow, noisy, confounded. Most organizations report level one with the vocabulary of level three. So we say $10,000. It didn't save you money yet. At least yet. That's the whole scam. Usually an innocent one. Co-pilots say $10,000 developer hours is a level one claim in a level three suit. Saved into what? If those hours became more meetings, you bought nothing.

So three traps that eat your data while you're measuring. Now even measuring right things, those three traps will eat your data. I stepped in all three so you don't have to. Probably you might have as well. One is survivorship bias. Your case studies come from teams that volunteer, succeeded and love telling the story. The team that quietly turned co-pilot off in a week, in week three, doesn't present at the engineering showcase.

We surveyed our AI champions about AI value. You can guess the methodology score on that one. Attribution error. We celebrate a 10% drop in PR merge time for a quarter. Until someone pointed out we'd simplified the branch policy and upgraded the CI runners in the same quarter. Was it the AI or the plumbing? We had no idea. Because we changed everything at once. If you can't answer compared to what, you don't have a measurement. You have a press release. And the Hawthorne effect, this may be a bit familiar to some, engineers perform differently when they know the pilot is being watched. And pilots are always being watched. Your pilot results are the best numbers you will ever see.

Even for the regression, not for the peak. So, what to take away from this? Monday morning, trace one metric to one thing that shipped differently. What did we ship faster? That's the most important thing. Run a workflow dependency test. Pick one real change that is going to production this week. Measure how many suggestions they have produced. Trace the work end to end. Design implementation, review testing, release and support. At each step, ask did AI materially change the decision, the speed, the quality or the handoff to the next step? If the answer is only we wrote the code faster, you have local acceleration. That's nice, but it's not transformation, again.

The test is not can developers live without AI. That ship has sailed. That was 2025. Write your kill list before the next steering meeting, not during it. Decide now what evidence shuts a pilot down. If no evidence, good. It isn't a pilot. It's a belief with a budget. You need to be killing things, not starting more, but you should also be killing things. You don't need a better model first. You don't need a better question. So, when you go back on Monday, take your proudest AI metric and ask, so what ships faster? And if the room goes quite good, that's not failure. That's the first honest measurement you have had.

Thank you. I'm HD. I left three real stories, if I had more time. I'm only 45 seconds over time, so I'll quickly go through them. One, we were shipping faster. We shipped features nobody asked for.

We shipped features nobody asked for. Thanks Google Analytics told us that, but we were three months late.

We identified cloud cost savings. I used cloud code to use my gcloud CLI to go and identify our stale resources. VMs that were open, disks that we haven't touched since 2024. Python notebooks from a data analytics lead who left the organization in 2025. It's caught stale VMs, disks, clusters, et cetera, that I could not list and figure out as a human because there's just too many. And yeah, great.

We deprecated costly third-party dependencies. We were using it all. It was very expensive. We built an in-house version of it because we could do it in such a short time with filling in the gaps so that we could confidently deploy saving multi-million euros per year. Thank you very much.

But it was again useless because we did not fix them. I did identify the savings, but Tim was confident enough. Tim wasn't confident enough to kill those VMs or shut down those disks. So those were two failures that we had. We can still save the second one. But three, we deprecated costly third-party dependencies. We were using it all. It was very expensive. We built an in-house version of it because we could do it in such short time with filling in the gaps so that we could confidently deploy saving multi-million euros per year. Thank you very much.

Yes. So far, we have only one question, but I'm pretty sure that's just because you watch, you didn't want to spend time on typing because the content you shared is gold. I hope that you, by the way, work closely with my colleagues from the software solution engineering department at Microsoft Sweden. If not, I will introduce you to them because what you shared, we want to share with our customers as well.

But, yeah, let's take this question we have so far. And yeah, I think you highlighted all the problems, all the challenges, all the issues, but only started to give solution. You teased a bit people, and I believe many of you folks feel the same pains, the same kind of... This is really in the air, but you didn't give us too many details on what's the solution. And I believe the question is exactly about that. Can you give an example of how you changed the workflows in your projects which made impact of generative AI, like for coding, for development, let's be more specific.

And what was the metric, preferably an example of a brownfield project? What you can share without oversharing about your real projects? We actually quite haven't. I'll be honest here, because especially when it comes to brownfield projects, the engineers are not quite confident. And for those who are confident, they will only get disappointed by getting shut down by business. So looking at the question again, Janaya, what was the metric? Right now, we are still at the individual to team level. We haven't been able to come up to the third level of bringing real business value at least. So I have to give an honest answer here, we're not there yet. But we've figured out, I can tell, we've figured out coming up to the second level from individual to team, and that was that. Thanks for your very honest and open answer. Let's mark this as answered.

Let's take... Yeah, basically, I think you more or less mentioned on that about how you measure the impact of AI. You gave one example, and of course your whole session is about metrics, so all questions around that. Besides developer experience metrics, what would you recommend to measure when evaluating AI output? That depends on what output we're talking about. But what I can tell, one of the... So we can talk about the teams, we can talk about the business. I'll take that question again in two levels. For teams, you can solve the pain points that the teams have. I think CI, but CI takes an hour to run. Can you use AI to drop the CI running time from 60 minutes to 25 minutes to 50 minutes? Then that's a win. Not because we're saving compute for 45 minutes, but we're saving so much frustration and interrupted time of the valuable time of senior engineers in that matter. And when it comes to business, as an engineering manager, I have a budget, but I'm also accountable for it. So I need to ask the question, okay, why is our cloud cost so high? So I gave that example of going through our G cloud resources. For example, as a human, you need to go list all the VMs, list all the disks, the Python notebooks, your clusters, et cetera. As a human, this is a very difficult process. So you can answer either, does it help me save money or does it help me make more money? I think that will be the ultimate question that we can answer. Yeah. Fair enough. Yeah, I believe here in the majority of cases we talk about enterprise software development, when the final goal is some revenue, some profits. Thank you. And if we take this one, how do you measure the impact of AI if you start measuring when the teams are already using AI? So maybe this is a fear from someone, that you are a bit too late.

You didn't make a snapshot of the baseline before AI, because we are with AI for some years already. Give us hope that it's not too late to still start measuring the impact of AI, even for the teams that already use AI. That was a question about junior engineers and how leads can help them. I want to bring a very different perspective to this, because I don't want to give an answer here that you can easily find on Reddit, LinkedIn, or Google. So another different perspective that I would give is, not the impact of AI on your releases, process, business metrics, your Dora metrics, etc. But you can look at it from a people growth perspective. So there was a question about juniors, for example. After using AI, did my engineers get more competent? Did it help them learn new frameworks, new libraries, new technologies? Is my front-end engineer already started contributing to our infrastructure code so that my senior back-end engineer wouldn't be bothered so he can focus. And also my junior engineer will also pick up new skills. Did it result in new promotions in the past year? So how is it upskilling our people and how is it transforming the way they work? This is one of the things that I would measure in addition to how fast we are releasing, the escape defect rate, cloud costs, the business impact, etc. So if you're sitting on that list already, when you Google it, you go and ask your GPT, you will already have this answer. So I don't want to give you this answer. But you can look at it from a completely different perspective on seeing how people use their daily workflow so that it helps your people, your team, to upskill. So that would be one of the different things that I think you could measure. Thank you so much, Ht, for your session and for your answers. My pleasure. And yeah, we are moving on with our program. Let's give a huge round of applause to Ht.