Shipping AI Under Constraints: Build, Buy, or Kill

I'm Dave. I'm an Engineering Manager at Capital Untap, and thank you for coming to my talk on Shipping AI Under Constraints. When I pitched the talk, I did not have the subtitle of how to waste 200k on AI. So as you can tell, it's been a fun few months for me.

Capital Untap is a medium-sized fintech. We do credit and savings for small and medium businesses, UK and US, and we only have about 100 or so developers, quite small for the amount of revenue that we generate. Our senior execs are very into AI, very enthusiastic. When we saw the email from the Shopify CEO earlier, it made me realize where my CEO got a lot of his ideas. And we have a culture of pilots and experimentation within the company.

This is just a brief story of failure, and how we wasted a load of money, and what we're doing differently now. The key lesson that I took from it was that we gave AI special treatment, and we moved too fast. What does it mean to move too fast? Well, the AI team actually grew out of the business. They did not start in engineering. Head of AI originally came from the business. And if you think that's a baffling decision for a company to make, I'm not here to argue with you on that. The engineering department treated them kind of as an internal security threat, and tried to limit the amount of customer data going to them.

Which I think is actually quite sensible. But here we are. Not the strongest start. If you do any kind of customer service, it has probably occurred to you that you could have an AI chat bot. It's a very obvious use case. And it's the first thing that the AI team set about doing. Trying to build something that would chat to customers, answer simple questions, and so on.

So, as I say, the team started outside engineering, outside engineering governance and standards. And after eight months of hard work on this chatbot, amongst other projects, engineering won the argument, and the team was moved to work for me. I started off with the assessment of projects, taking on a new team, assess what they're doing. There were more than just the chatbot, but it was the big one. It was their flagship project. It was the big thing that we were working on.

And as it came to me, they were telling me, we're weeks away from go live. We're very close. Let's go. So this is where we come in with the evaluation using stage gates. I mean, the step one is just evals. I'm sure everyone in the audience knows what that is, but LLM integration tests, essentially, get to a certain level, move to shadow mode for our chatbot. So it would comment on customer queries and production. And it would just say, this is what I would have said if I was talking to the customer.

And that gave an opportunity for operations agents to compare what they were saying with the chatbot and give us any feedback. And of course, we need to have a high level of confidence with that stage gate before we go to customer contact. This is where the problem started to emerge. If you ask the chatbot a question that's not in its knowledge base, most of the time it will escalate to a human, but a few times it will just make something up, which I'm sure comes as no surprise to anyone, his work with LLMs. It was pretty good, but 90 to 95% accuracy was not good enough for something that's going to talk to customers thousands of times a week. And here was where the problem started to emerge, right? Complexity. We had a lot of this papering over the cracks of a small LLM problem. It's lying to you, okay, how do we fix that? We put in some validation after it makes its answer to just check, is this factually correct? And maybe that will work pretty well, but ultimately we have to build the validation and test it before we know. So not only have you got complexity of kind of workaround upon workaround, but it's very hard to estimate because we just don't really know how well it's going to work.

To give a counter example, we have built other AI projects within Capital OnTap. We have one called Blaze, which is essentially AI transcription and summarization of customer calls. It's a bit more complex than that, but it's pretty straightforward. And that was built by the operations engineering team. So it started off in the correct team. It started off with a very product focus, and we started off with tracking operational metrics. So after a phone call, operations agent does the wrap up, and we've reduced that time from three minutes to two, which is a massive impact for 600 calls a day. The decision came, the decision time came for Merlin, our chatbot. We had to decide what to do with it. We'd spent at least 200K in people's salaries, as those involved, and LLM costs. And we had to say to the business, look, this thing that you thought was weeks away is actually going to be paused and months away now. But the CEO understands sunk cost fallacy, so we'd killed it and we decided to look at third-party vendors.

Ultimately, we decided this was not our core competency for such a complex project, and we should stick to the fintech side of things and simpler AI projects that we can own within product teams, but go with third parties for something as complicated as this. So we had two vendors that we liked. We essentially just found other businesses that had integrated with them and went to chat with them just to see how they performed. One was much faster, and one of them lied to us about balance amount, which was very interesting to see in the live real world. And as we were killing the main AI project, we decided to reset how we do AI at CapitalTap, so we no longer have a separate AI team. We have AI engineers who will embed in an existing product team, help them deliver their project, and then step away.

One was much faster, and one of them lied to us about balance amount, which was very interesting to see in the live real world. And as we were killing the main AI project, we decided to reset how we do AI at CapitalTap, so we no longer have a separate AI team. We have AI engineers who will embed in an existing product team, help them deliver their project, and then step away.

Ultimately, we decided that a team built around a technology can't own domains and products not in the same way that an actual product team can. And then, again, back to the comparison between the two. Blaze almost started accidentally in the correct team, correct ownership. And if you read through these just very briefly, what you see is not really AI problems. It's just standard engineering problems, and that's really the lesson that we took from this.

Again, outside our core competency to deliver a very complicated AI project, so we didn't. And we used the same intake process for AI as any other project, any other feature. So we have a named product owner, clear success metrics, stage gates that I've shown you. Each gate has a killer criteria, so you can just back off a project if it's not good enough. And this is kind of what it boils down to for me, can a product team own it? Do you need permanent AI specialists to own it? For us, again, that's a red flag. We're not an AI company. We're trying to adopt it, but we are ultimately not an AI company.