Forging the Agentic Engine Room: A Human-Centric Playbook

Gregor's talk was brilliant, and I was smiling and nodding all the way through it because he's very much focused on the why and the what, and I was really happy because hopefully I can bring the how to some of what Gregor said. So it was back in the end of summer this year, we'd just finished productionizing our first agentic AI product, and I was buzzing off it. And this talk came up, there was the opportunity to put it in and apply, and I applied. And I have to admit, the title is generated from GemIIni, purely from that.

Yeah, I generated that and went, right, what do I want to talk about? Well, I want to really focus on the projects that we launched this year. And part of the research that I did is that only less than two in 10 AI projects are making it into production. And I don't think that's good enough. I think, you know, us as leaders, we have to do something about that. We have to help that and increase the chances of AI products making it into production.

Just a little bit about me, I'm a software engineering manager at a company called CDL. It's based in Stockport. We focus on insurance technology. I've been in the tech industry for just about 25 years now. Like probably most of you, taking a really diverse path. There's no straight path in technology. I do flitter between the technology and the design side. I love it just as much. But I've been in engineering roles previous to my current one, and I'm part of a software engineering department of 250 people. And I have one of the sub-departments in that, which is called the digital department. And I have four teams currently in that one.

Just a little bit about CDL, because the likelihood is that when you looked at the website for the tech leaders, you were like, you know, you recognise METI, you recognise all those. You did recognise who CDL were. But if you live in the UK and you've got home insurance or car insurance, and you've purchased that through an insurance comparison website, we've probably processed your data at some point, because we power a lot of the retail insurance. We're actually nearly 50 years old, and we've been helping clients with technology solutions in the industry for that long. And then more recently, it's that real focus drive on how we can use AI to unlock the next level. And in 2024, we handled over a trillion service calls, which I don't think is a mean feat for a company the size of 600 people. We've been described as the veins of the insurance retail industry from that point of view, all our ecosystem. So we do handle big data from that. But my department is very much about the digital side, so more about the web applications, which is really important part of I am telling you why. So back in March this year, the architects presented a proof of concept of an agentic AI solution of how we could use AWS Bedrock to connect to our ecosystem.

So the first thing we focus on was some of the simpler services that are easier to unlock. Insurance is quite complex. There's lots of policies, and PI data is very critical to what we do. But we found four functions that we could unlock with AI and do that as a proof of concept. And they showed us that demo of AWS Bedrock with a simple interface, and it was then communicating with our ecosystem. Brilliant, we all thought this is it. We can see this is where we're going. But we had no one to build it.

Every department was already had their product backlogs full for the next two or three years. But I stupidly put my hand up and said, digital will build it. And the reason is for the last six months previous to that, I've been told by from the CEO downwards that user interfaces are all going to go. They're all going to change. You know, digital is dead from that point of view. So I knew that my department had to be part of the change. We couldn't hide away from it and go, oh, that's not us. We had to change. We had to upscale.

We were predominantly front end at that point. Yes, we did do some APIs into our ecosystem, but a lot of our people were more front end and UX from that point of view. And I did get a puzzled look of why I was doing it. And I was very passionate that my people, we'd already give them the skills and the competencies that they could handle any challenge that was put in front of them. So skip along a little bit. We got the team together just after the March time, a month or so in, and we got this LinkedIn post by CDL. We're already launching something that we hadn't built. And we've probably all been part of this and seen this happen. But it did worry us. You know, the panic did set on. It's now the end of summer isn't just a kind of target. They've gone out wild. They put that out.

We're going to have to launch something. And to be fair to the team, they focus. We got down. We discussed this. We worked with product to de-scope a lot of things. And we communicated with the business about, this is what we're launching. There's nothing extra. But it did rot the boat.

Throughout the middle of the project, we started with Node because that was our technology of choice in digital. A lot of the devs already knew that language or already using it, thought it was the best way. And we could probably do a talk on why we switched. But we did switch halfway through the decided Python was a better way of doing it. We're predominantly an AWS consumer. And a lot of our technologies and some of the libraries, et cetera, were more capable in the Python place. So we decided to switch.

But none of them knew Python. But within a week, by using AI tools, we were able to make that switch pretty seamlessly and move over. And they were able to upskill. We did then, we were able to then launch in the beginning of September, which I just call about end of summer for me. And it only has four capabilities. I'm not saying this is a product at the moment that's going to change the world yet. But we're getting there. And we were able to get something into production that our clients can consume. Over to commercial for that one.

But I wanted to talk to you about some of the things that I think worked well. And I think could be used in other businesses. And actually, I come back to what Greg was saying about the culture is really critical. And I heard this talk about a year ago. Culture is what people do when nobody's watching. And I thought about that. About what does it actually mean? And it's, you know, us as leaders, yes, we've got to lead people. But we've also got to trust people in what they do. And we've got to put those behaviors in them so that when we're not around, they can carry on. And I think that's kind of the theme of my topic really for today. It's what do you want them to do when you're not around?

And one of the first things you have to do is set the expectations. Four years ago, when the current software engineering manager leadership kind of formed, there was a few changes about four years ago, we decided we were missing one critical component. We had a values framework that said how to, you know, how to be a good CDL employee, how to work. But the engineer's staff didn't have any direction about what that meant for their role. So we set about creating a career framework. And if you haven't got one, I'd highly recommend getting that down. It allows, it gives the direction to engineers and all the other roles within that about what to expect.

The first most important one is the scope and impact. And I've included, hopefully this one does work, a QR code to our career framework. And you can, if it doesn't, just go and Google progression.cdl.co.uk and you should find it. We define for each role and each level of those roles, the scope and impact we expect from the engineer. We define what the technical execution, what they're expected to own, and how they're meant to collaborate. As you go up the roles, your collaboration changes from just being in your team to be wider across the departments. We focus very much on leadership as being something that we expect in our senior members of the team. When you come through, yes, we want you to be an independent contributor. We want you to write good code and do all that. But actually, as you go up through the levels, no matter if you choose a people pathway or a technical leadership pathway, leadership is a critical component.

And that's how we label our levels as you can see on the screen. The key thing is not just setting the expectations, it's creating the opportunities for people to flourish. And this is probably when it ties back into the AI. Now that you have those expectations, you've got to create those learning things. It's not just about giving them a personal development plan and letting them go on their way and expect them to learn. It comes back to that culture thing. Create some opportunities that you can cultivate that.

What I think we do well at CDL is that we've created a learning course alongside our learning and development department that focuses on key three strengths. Authenticity, this is about the brand of leadership you have. Everyone will bring it. Everyone's got different personalities. There isn't one way to do it. You've got to be authentic as a leader because that helps get the trust. So whether you're a technical or whether you're a people leader from that point of view, you've got to be authentic in your role.

We wanted to give them coaching skills so that they can give back to the rest of the business and the juniors within that. Just because you've got the experience doesn't necessarily mean that you know how to then share that knowledge. So it was a clear one that we wanted to focus on. The last one, probably the most important one is having honest conversations. No matter whether you're a people leader or a technical leader, you're going to have to have difficult conversations and you can't shy away from that. And that is a skill that you have to practice just like your code reviews. We've created a culture of experimentation and we do that through consistency of running regular days where people get the opportunity.

So once a year, we run a two-day hackathon, which has prizes for the winners. There's food on site. We try and get everybody in. We encourage people to work in different teams and come up with ideas that are themes. We also run quarterly innovation days, where that's a day where they can choose what they want to work on an experiment as a team.

And I think those have been critical to why we were able to get that agentic product out there. People had that mindset of experimentation. They weren't worried about looking at new technology, that they'd embrace it from that point of view. We also run regular enablement days. AI has been a massive shift since, was it three years ago, since ChatGPT kind of launched and came out into the wild. There's been a clamor around these tools.

But it's all right having the tools, but do you know how to use them? Don't just throw them at the engineer and expect them to take it. Run days. If you've got partners such as your cloud provider, we're very, very fortunate in that we have a really, really good relationship with AWS, and we make sure we get the most out of that. We get them in to do enablement days with us, and we give our people the time and space to do that. And the benefit to that, I think, happened back in March this year.

We invested last year in enablement days for Amazon Q Developer, and this year, I think in March, we were the top contributor in the UK. We wrote the most lines of code through that tool, and we're only a small engineering department comparison to the bigger organizations. Clear one as well, and it was mentioned by Greg, was that communication. You need to communicate well. What we did in this project is we made the team own that communications. It wasn't just written as me as a leader. I didn't put loads of spin on it. They contributed back to what they wanted to say to the business about where we were up to. Yes, I helped, and I helped make sure it was clear to the stakeholders. I removed some of the technology things, but they owned the communication, and that meant that they had the commitment to deliver, which I think's really, really important, and a really easy one to try with your team, especially if you get your company putting out LinkedIn posts about when you're going to deliver it.

The other one is documentation. I think AI has massively helped in that way. I'm dyslexic, for example, always probably wrote really rubbish documentation, so being able to use those tools to do it. What we actually found that worked on this project, though, is that we create the glossary. Terminology in the AI space is changing frequently, and one thing that we had was a collaborator agent. That was the name one week, and then the week after was a sub-agent, action group agent, and so on. So we made a glossary for those and made it really easy that people could go to that when they're in meetings and stuff. Just bring it up. Oh, yeah, right. I know we're all on the same page. So it's really important that that's a clear line of your communication that you're all talking about the same things.

Adding reflection points, now call it a retro, they normally happen for us every two weeks, but for this project, things were moving fast, finding things, getting a pulse of the team as well. Were they feeling the pressure from that LinkedIn post? What did they think about it? Give them opportunity to reflect. What we did with this project is that we reduced that down to every Friday, so every Friday afternoon, we'd stop, just have a brew together, just talk about how the week's gone, how are we going to set up for the next week? That was important that they had that safe space where they could tell us stuff and that we could then help and action on things. My last kind of segment is user experience, and this is, again, why I put my hand up to do that, to do the agentic AI project. Even though we're using technology and technology's the main enabler, we have to think about the user. That is what we're trying to do with AI. We're trying to make it better for everyone so you can get on with other things. So with this project, we used the Google Heart Framework, and it's the first time we used it, but what it did is it helped with communication. It simplified it for everybody, not just the team, but for the stakeholders, and it made us really, really think about what we're trying to deliver and why it's going to be successful.

Now, this could be a talk in itself, and it wouldn't be me doing it justice. It would 100% be the senior product designers that implemented it, so what I've done is I've just put a few links to their Medium blog post, which I'd strongly encourage to read. They go into a bit more detail about how we used it from the project and what effects, but just having those clear goals and agreeing those metrics are really critical, especially between products and stakeholders. You are working to the same thing, and it gives that ability then to look at operational efficiency and that user satisfaction.

One thing we learned is that you can't test early enough. When you're building that agentic AI solution, no matter the state of the interface or whether it's API, get it out there. Start testing early. You're going to pivot loads anyway because of technology, and you have to do all that, but make your users the most important thing. Get tested early. What we were able to do after a month, we user tested, and we learned a hell of a lot more and that able to make the right decisions, especially when it was coupled with the heart framework. We were able to then learn a lot more We'd waited a little bit till we got a polished product.

In conclusion, for me, the key thing is set those expectations of what you expect from people. Give them learning opportunities that they can then take those expectations and make them into tangible things. Communicate well. Bring your team into that communication, and lastly, focus on the end user is critical and their experience and not just the end user. If you're handing it over to an operations team and they're the ones implementing your product, make sure that you treat them as a user and giving them a really, really good experience, and lastly, from me, just enjoy the ride. It's going to be really bumpy with the AI stuff. Things change regularly. Take some of our learnings and just enjoy it. Thank you very much.

I'd highly recommend that you have a mechanism, a way that you can evaluate those conversations and you can tweak those easily and then test again quite frequently.

In conclusion, for me, the key thing is set those expectations of what you expect from people. Give them learning opportunities that they can then take those expectations and make them into tangible things. Communicate well. Bring your team into that communication, and lastly, focus on the end user is critical and their experience and not just the end user.

If you're handing it over to an operations team and they're the ones implementing your product, make sure that you treat them as a user and giving them a really, really good experience, and lastly, from me, just enjoy the ride. It's going to be really bumpy with the AI stuff. Things change regularly. Take some of our learnings and just enjoy it. Thank you very much.

There's one question, and I had the same one. As you switched from Node to Python and reskilled the team with AI assistance, did the code quality suffer, and in the end, did that matter? Tough question. I think, at first, it would have done. However, we were quite fortunate in that there was other engineers and principal engineers that did Node Python, so we were able to call on them to look at it. Now, at CDO, we do have lots of tracking metrics, so we use the things in GitLab for monitoring code quality. We use Orca as well for testing vulnerabilities, and we have a very hard stance in that we don't want any, or nobody wants any vulnerabilities in it from that point of view, but we've already read into the teams to check those. I would say, no, it didn't from that point of view, but I can only go off the dashboard from those.

How big was the team that actually worked on it and executed in the end? So, the team at the start was, I think, eight people, including the tech lead. We did purposely have it small because they had to move fast, and they had to make decisions, and I think sometimes when you have multiple teams, you can spend more time discussing things, and we knew this was very much, we had a forced deadline on us, and we would agree the kind of requirements that they wanted, and we wanted to move fast, so we kept that team pretty nimble from that point, but that's not stayed that, now. Now, it's scaled to four teams off the back of it, but I think starting off small and then growing it and taking that learnings rather than just trying to throw it out, the whole business really, really, really worked, and especially when now you've got the kind of AIDLC kind of concepts and being able to trial that in one team first really works and see how it goes.

Nice, very nice. Thanks for the talk, Tim. It was great. That's always nice. You mentioned Amazon Q. Did the team use any other AI tools? Are you prescriptive about what tools slash models engineers can use? So, in insurance and generally CDL, we have a lot of policies that kind of prevent us using things, so we do have an AI group that me and anybody within the business can submit a tool of their choice, but it has to go through compliance. Yes, AWS things are easy because they're already signed up and sealed, but we have things like Google Gemini, which I love for my job and find that one more useful. They do have access at some level to chat GPT and other ones, but we have a list of approved things. The designers, for example, they have Figma, so they can do all the AI stuff in that, but, yes, we have a list of approved tools that they can use. It's just we like to, to be blunt, rinse the partnership with AWS, so we get everything they offer, we kind of take and bring in, so that's probably why it's more mature in that AWS space. And then in the hackathons and innovation days, it's the same for the tool set or do people get to experiment in those days specifically? So, yes, when it comes to tools, you've still got to put it through the process, if you like, and what you can get is like a status of you're investigating and then you just got to be careful about the data you can put in, so we do allow that to happen, it's just that you've got to do a bit of paperwork to get it finished. Everyone's really supportive and it's becoming a lot easier, asking the questions around which foundational model it is and if it's using Claude or Gemini, then it becomes a lot easier for us to do. So it's just picking those, but the company is receptive to it, it's just that we have to do our due diligence. Yeah, that makes sense. All right, how do you give the team space to learn and adopt these practices versus the pressure that you had to deliver? Really great question and I think it's changed recently, so we do have things like the hackathons and the enablement days. We also give everybody, it might not sound like a lot now, but we give people 5% to do what they want to do and the reason that might feel low is that we've recently moved to a product operator model where we use initiatives to drive the work and if you're doing learning as part of that initiative, it just counts to that, so really, it's unlimited. It's like, if you need to do that learning to achieve that goal, you can use that time and you don't have to log it elsewhere, you can just put it as part of that. It's just a discovery, right? It's how you're going to build that and everything, so it did scare people at first when we dropped it from their 10% time or whatever, but once we explained, look, no, just articulate it as part of the initiative and actually that had a secondary help. It helps with our R&D tax credits, so if you're in the UK in technology, you're probably claiming those, so it made it really easy to log that because it's research, right? You're researching the tools to do, so yeah, that's how we do it. We don't have a limit, but obviously, we're very try to be focused on the things that we try to achieve, so it's those tools that you think will help.

Nice. I like how you basically allow people to be responsible with their time and take that time if they need it. Yeah, it took some education because obviously they thought we were taking it away at first. It's like, no, it comes back to this communication point, right? Tell us why you want to do it and no one will stand in your way. We have quite an active CEO who's always on chat GPT, you know, like whenever a new feature comes out and he's presenting it back and it's like you feel the pressure yourself as someone in technology. It's like, well, we've got to keep... if he's doing that, we've got to do it. So yeah, it's just making sure people have those goals.

Gotcha. Nice. Let me just locate this one. All right. How did you actually build the agent into your tech stack? How did you ensure it only had access to the right info and processes and didn't expose data that it shouldn't? Ah, really good one. So I've probably got the... another talk that our architect did. It could create a lot more detail, but it essentially... or the original one, because it is evolving, is when we started back in March, it had three kind of components to it. It had the... what we term the AI front door. So that's all your API gateways, etc. from that, how you access it, how you authenticate, how you know which policy is going to... you know, which policy is going to be attached. We had the AI engine room, which was the supervisor agent within that, and then that had those sub-agents. I can't remember where we were up to with the naming convention of that. We've had collaborator agents and worker agents, but yes, it had those in. And then the third layer that we had was the... we had a Lambda stack. So each kind of intent function, that's what it handled, and that made sure it was going to the right service at the backend, and that would protect it from that point of view.

Interesting. So the data protection was mostly on the latter part that you mentioned.

Yeah, and the AI engine room. Yeah, yeah. From that. Very cool. Let's see. This person wants to see results. It's awesome that the team made all these things, but what was the customer impact in the end? Did you find them using it and liking it, or did you make something that was too advanced for the user base that you had? Really, really good question. So I could throw this one over to commercial, really. So when it comes to the clients, yes. Look, insurance and the kind of stereotypical company, they're risk-averse. Right, insurance, right? So there is a lot of getting them on board with understanding AI. So our CEO, and I won't mention the company they went into, but he went into a big insurance firm, and he was talking with his counterpart, and they weren't even using chat GPT or anything from that point of view, so there's a massive educational point of us. Yes, from our production-ready thing, we haven't got it live with the client at that point, but that's happening, and it's just going through the due diligence because insurance has things like FCA and everything, and they're really, really home in it. So we have to be patient when it comes to that. But our mindset as an engineering department is we're always ready. We have to be like that. It's like just because the commercial side takes a long time, we have to be ready to implement. And a really good example was last week when the CEO did have a big presentation to one of our clients. We were able to spin up one of those AI instances in an afternoon and then integrate it into our customer self-service portal so we can see it. So for me, that's how we're showing that we can do it. We're not really in charge of the other bit. It just takes time in our industry, so...

How much did you pivot? Because you mentioned user testing is very important. I'd do it as early as you can, but then I'm wondering how much did you pivot through that user testing or how much did you actually pivot after it went live? A lot. Like technology-wise, that didn't affect it. It was more around the prompting and doing that. Look, we haven't got any like experts in that. We're all learning as part of that. But yes, every test meant we iterated over that and still are now. What we need is big data to test it and understand.

First, just having 600 employees kind of test it, gives us some insight, but that's not 60 million people in the UK, right? And it's going to be a continuous evolving thing. And what we try to do is allow our clients to be able to then add their own guide rails to it and everything from that. But yeah, it's going to evolve and pivot and pivot and pivot.

Yeah, it makes sense. All right then, final question. What strategies can organizations use to foster a psychologically safe culture where engineers actually feel comfortable adopting AI tools and developing new AI competencies? Get rid of the carrot and stick approach. So obviously, every shareholder, everyone wants us to adopt AI from that point because we want to see it. But the first thing you have to do is have a conversation about it. You know, like what do they see the future of it? And if they're bringing up concerns, don't ignore them. Don't try and I suppose gaslight it. Like, we'll work out how we can overcome those.

So a big one for me is that we talk about AI making improvements, but you've got to think about what technology is there to do, is to make a human experience better. So that's why I'm really keen on focusing on the user experience of what we're doing. But you've got to bring them on the journey. You've got to talk to them. We ran a survey recently, you know, in our kind of quarterlies where we ask questions around it and try and take those ideas. I think about communication and how just, you know, go and spend some time with the team, go and talk to them and actually understand what they're concerned of. You know, lots of jobs, right? It's a big one.