Democratising AI in Engineering: Lessons from Typeform's Journey

So, hi everyone. Welcome to my talk about democratizing AI in engineering, which is a sort of summary of some of the learnings we've had from our journey at Typeform this year. I thought I would start with some introductions. So, my name is Andy. I'm a staff engineer at Typeform. In case you haven't heard of Typeform, Typeform is an online forms builder, a bit like Google Forms. You can sign up and you can create a form for lots of information you want to receive. You can send that form out to your customers and they can fill it in and you can analyze your results and send it to third-party integrations and things like that. Are there any Typeform users here? Has anyone used Typeform or heard of Typeform? Okay, good. Right, a few friendly faces, good, that's good to know. Okay, other thing to know about me is I'm a Star Trek fan. Are there any Star Trek fans here? Okay, a few people. Good, all right, live long and prosper. There are some Star Trek memes, so you'll probably find them funny. I hope so anyway. If you're not a Star Trek fan, which is the vast majority of you, when you see a Star Trek meme, just laugh. That's just a sort of cue to laugh. Other thing to know about me is I'm an Emacs user, big Emacs fan, thanks for the intro. I wrote the presentation in Emacs. I'm actually presenting in Emacs. Those of you at the front will see my mode line, which is very exciting. Are there any Emacs users here? Okay, there's one. Okay, it's quite a familiar experience to me to be the only Emacs user in the room. Maybe not a room with so many people in, but okay, that's fine. Don't worry, we'll just forget the whole Emacs thing.

So, what am I going to talk about today? I'm going to talk about the journey we've been on this year at Typeform to provide our engineers with access to AI, and I'm mainly going to talk about that inside our engineering organization. I'm going to spend a bit of time on that. I'm also going to share a few anecdotes about the impact AI has had outside of our engineering organization, and I'm going to close with a few sort of learnings or lessons from that journey. Now, before I talk about our journey inside engineering, I thought I would just frame this conversation with an illustration. Now, this, I just want to… fair disclosure, this chart is entirely fabricated.

No real data was used. I actually asked an LLM to generate this diagram, and those at the front, you might get bonus points for spotting some clues that this diagram was AI generated. So, a complete fabrication. But I do think it illustrates the point I want to make. Which is that this year, there's been a huge amount of hype about AI. An exponential amount of hype. The green line is an exponential amount of hype this year about AI coding tools. And the kind of coding tools I'm talking about are agentic coding tools like Claude Code, Cursor, Klein, that kind of thing. My presentation is about the journey we've been on at Typeform, but I think it's probably a familiar experience for many people in many organizations. And the other line trailing behind the hype line is adoption. Basically, I'm trying to illustrate that whilst I think there's been a lot of hype in the industry, I think actually people's adoption has been somewhat slower than that hype might indicate. Especially, and that might be especially true in larger organizations, you know, because for an engineering team to really adopt AI tools, it's not just a case of individuals picking it up. You know, the company needs to buy licenses, pay for LLM access, choose tools, consider security, all of this kind of thing, which slows down that adoption curve. So, when I'm talking about the adoption of AI tools, it's that adoption curve that trails behind the hype. That's the curve I'm talking about, the adoption of tools in engineering teams.

So, let me tell you a bit about our journey at Typeform. So, this year, I think I could probably summarize the entire experience we've had with this one phrase. Adopting AI in our engineering organization has all been about democratizing access to AI and federating usage of AI. And I'm going to unpack what I mean by those two terms a little. But that's really been the guiding principle throughout the whole year, democratizing access and federating usage. So, let's start with democratizing. What do I mean by democratize? So, I asked an LLM to write a summary of the word democratize, and we didn't so much write a summary as an essay. So, I said, okay, this is far too long, summarize your summary. And then it wrote this paragraph, which I thought was actually pretty good, so I kept it in. So, what do I mean by the word democratize? Democratizing is all about making something accessible to ordinary people, everybody, not just a specific group or a set of elites, making something accessible to everybody by removing barriers to entry, simplifying how they can use it, and distributing that usage as widely as possible. Ultimately, so that it's available to everyone and not just a small group. So, when I say democratizing access to AI, I'm talking about making access to AI available to everybody in your team, reducing the barriers to entry for everybody, so it's as easy as possible for people to use it, and distributing that access as widely as possible, so everyone in your team has access to AI. So, you might be thinking, that sounds great. How did you do it? This is the first Star Trek meme, so it's... Captain Picard.

So, you might think, that sounds great. How did you do it? Well, don't worry, that's what the presentation's about, so I'm going to tell you. So, one of the main enabling technologies we've used this year is AWS Bedrock. Some of you may not be familiar with AWS Bedrock, so I'll tell you a little bit about it. This is not AI-generated. I actually copied this from the AWS Developer Guide, so it's not fabricated. So, AWS Bedrock is a managed service in AWS that gives you access to LLMs from a wide range of vendors.

So, for example, you'd find Anthropic models in there, Claude Sonic, Claude Opus. There's some Amazon models, there's models from Meta, there's models from all sorts of vendors. And you access those models through a single, unified API. So, you use the Bedrock API to make inference requests, no matter which model you're using. And you control access to those models using familiar AWS primitives, so like IAM roles and policies and that kind of thing. And you pay for your usage through your Amazon account.

So, basically, it's a way of accessing LLMs without leaving the AWS estate. So, that works quite well for applications. So, if you're building an application that uses Bedrock for LLMs, it's pretty simple. You deploy your application, your application makes inference requests to Bedrock, and it probably authenticates with its IAM role or something like that. And under the hood, it's able to use these foundation models. So, like Claude Sonic or Opus or whatever. So, that's pretty straightforward. But I'm talking about democratizing access to AI for engineers, for people, for members of our teams.

So, these are not applications running in AWS, they're people at keyboards. And that's a little bit more complicated. A user would need to authenticate with AWS somehow, maybe they've got an access key or something like that. There might be some special security requirements that they might have to follow. So, for example, we would need our engineers to join a VPN in order to access Bedrock in this way. And only then can they use Bedrock to make inference requests.

And you remember one of the principles behind democratization was lowering the barrier to entry. This is quite a high barrier to entry. It's quite complex to sort of settle this stuff up. We did start here earlier in the year. But the point is that it was quite cumbersome. We'd say to people, if you want to use AI for coding, just follow this massive piece of documentation and that will get you set up. Luckily, AWS released a new feature for Bedrock earlier in the year called Bedrock API keys, which massively simplifies this process.

So, with Bedrock API keys, you can generate a single API key, which is effectively a sort of credential for accessing Bedrock. It's got very tightly scoped permissions, so you can just use it to make inference requests. And you don't need any of the other stuff. You don't need to authenticate with AWS, you don't necessarily need to be connected to a VPN or anything like that. Just with your Bedrock API key, you can then make inference requests. This is what we're using. We're giving all of our engineers a Bedrock API key. And then they're empowered to use that Bedrock API key however they like with whichever tools they like to make inference requests to LLMs.

And you might be wondering quite how we did that, and this is kind of how we did it. We have an internal command line tool called TF, which stands for Typeform. And all of our engineers have access to that command line tool. It's got sort of a Swiss army knife, lots of utilities on there for our engineers to use. And we wrote a new command called generateBedrockToken. And when you run that command, it generates a Bedrock token. All of our engineers can use that, and that grants them access to LLMs.

There's a bit of plumbing behind the scenes, but basically this is the distribution model. Now, I mentioned earlier that you can use models from many vendors in Bedrock, and most of our engineers are using Anthropic models like Claude Sonnet. And you might be wondering why we went through this whole process of using Bedrock when we could have just used Anthropic. Second Star Trek meme. Okay. Anyway, so why did we go through this whole process when we could have just used Anthropic? We could have just signed up for an Anthropic account, generated an Anthropic API key for our engineers, and they wouldn't have had to use Bedrock at all.

Or we could have done the same with OpenAI or something like this. And in fact, we did try that as well. We tried lots of different approaches. And although it technically works, it's a lot less convenient because we use AWS to host our platform, so they're an existing vendor for us. Anthropic would be a new vendor for us. We need to sign up for an agreement. We need to pay a bill. We'd need to generate API keys for everybody. We need to distribute those API keys for people. When a new engineer joins, we need to remember, oh, generate that person an API key. When someone leaves, we've got to remember, oh, deactivate that API key.

So there's a whole load of admin we'd need to do to do it this way. Whereas all of our engineers already have an AWS account. When an engineer joins, they just get an AWS account by default. When they leave, we've got a process to deactivate that account. And now they have access to LLMs as well because they can generate a Bedrock API key. And we've got a big bill with AWS already for our platform. So what's a few Bedrock API keys? We just pay for our LLM usage for engineers through that existing billing process. So we could have done what data suggested. We could have used Anthropic, but it turns out it's just much more convenient for us to use an existing vendor like AWS.

And ultimately, what this means is that every engineer in our organization has unlimited access to LLMs for software development. And that that might be through using a tool like Claude Code or Klein. It might be for writing scripts. I mean, however they want to use it. They just use their Bedrock API key and they've got unlimited access and they don't need to worry about, you know, our Anthropic account running out of credits or anything like that. Okay, so that's the first part of this big idea, democratizing access. Coming up with a way with a very low barrier to entry and distributing it widely to everyone so everyone in our team can access AI.

What do I mean by this second part? Federating usage? What do I mean by the word federation? I don't mean the United Federation of Planets. That's another Star Trek thing. No. Okay. What I mean is that everyone's got the same way of accessing LLMs through Bedrock. But when it comes to choosing which tools people use, they might be using Claude Code. They might be using Klein. They might be using Emacs. I know one person using Emacs. That decision is entirely up to them. The decision about which tool to use is federated down to the level of the individual. So we're not prescribing which tools people are using. We're giving people a sort of single paved road for accessing LLMs, but then federating the decision about tooling to individuals. And ultimately, it means that everyone has access, the same way to access, but they can use whichever tools they like.

You know, that's a sort of a matter of personal preference and we're federating that decision down to individuals. Okay, so that's a summary of the journey we've been on this year to democratize access to AI inside our engineering organization. I'm just going to share now a few anecdotes about the impact AI has had outside of our engineering organization as well.

One of the main areas AI has had an impact outside our engineering organization has been on hiring. Now, before we really started adopting AI properly, this is what our hiring process looked like. Maybe it's a familiar experience to some of you, either as an interviewer or as a candidate. We basically say to people, listen, you need to code for an hour and we're going to watch you do it.

It was a bit nicer than that. We didn't ask them to traverse a binary tree or like right now algorithm or something. We gave them a nice practical task and sort of a pair programming exercise, so we would pair with the interviewee and assess their coding skills, their problem-solving skills, their ability to communicate. So, it was a sort of a collaborative exercise.

But critically, we did say to people, you're not allowed to use AI coding tools because we wanted to see what people could make with their own hands. But the trouble with that approach is, as we started adopting AI more and more inside our engineering organization, we're asking people to do one thing in the interview, but then asking them to work a completely different way once they started working with us. And furthermore, we're expecting all of our engineers to sort of, you know, pick up AI coding tools and we haven't checked for that in the interview process at all. So, now, we've changed our interview process to this. We say to people, actually, we want you to use AI for an hour instead, and we're going to watch you do that now. And it's still a sort of friendly approach. We give them a different task, a much bigger task, way bigger than you could possibly do in an hour by yourself. There's no way you could complete this task in an hour by yourself. But we say, listen, we'd like you to use AI to work on this task, and we're still going to assess your programming skills, your problem solving skills, your ability to communicate. But now we're also assessing people's AI proficiency, because we want them to join the team with AI literacy, you know. We want them to join the team and share their experiences with their team members and, you know, enrich everybody's usage of AI as well. And that's proving far more effective as a hiring process.

Okay, and the other thing I want to talk about outside our engineering organization is the impact it's had at the boundary between our engineering organization and adjacent technical disciplines like design and product management. Now, the eagle-eyed amongst you may notice this is not a Star Trek meme. I sort of stepped into an adjacent genre here. So one of the main ways our design and product management teams would interact with our engineering team is via our design system. So we've got a design system with a load of front-end components in and corresponding components in Figma. So, for example, a designer can, you know, create a mock in Figma and then hand it to a front-end engineer, and they can recreate that with real code, and it sort of looks and feels the same. So we built an MCP server for our design system so that a front-end engineer can rapidly transform a Figma mock into a working sort of prototype, working piece of code, which isn't the finished product. Maybe it gets them, I don't know, 60 or 70 percent of the way there. But then it basically has massively streamlined that interface between design and front-end engineering. And it's also proved incredibly useful for product managers because product managers would previously try and articulate an idea by maybe sort of putting together a demo or a sort of mock experience, either by relying heavily on the designer, or increasingly by using vibe coding tools like Lovable or Bolt. But the trouble with those vibe coding tools is you get something that sort of works, but it looks completely different from our actual product. They can now plug in this MCP server for the design system, and they get a living, breathing prototype that looks just like the real thing. And of course it's not production-ready code, but it means they can rapidly iterate and test new ideas without those ideas looking wildly different from the main product.

Okay, so that's just a few anecdotes about the impact of AI outside of our engineering organisation. And the last thing I thought I would share is just a few sort of learnings or principles that we think are powerful, that have come from this journey this year. So the first is that I think AI adoption in a team is all about empowerment, but not about prescription. And for me that comes out in this idea of democratising access. It's about empowering everybody to access AI, without being too prescriptive about what they use it for, or how they use it. So we've really tried to empower everyone to access AI by giving them simple tooling, but not prescribe what they use it for, or how they use it, by saying listen, you can go and use whichever tools you like, as long as they can use Bedrock API keys, you're good to go. Go and try lots of tooling.

The other thing I've certainly learned this year is that AI enablement is really a people thing. It sounds like a technical thing, and there's technical aspects to it, you know, we had to do the Bedrock API thing, and write that command line tool and so on. There's a technical aspect to it. But really, it's a people thing. If you want your team to adopt AI, you have to go and, you know, speak to your team members, find out where they're at. There'll be some people in your team who are early adopters, who are power users. There'll be some people in your team who are very sceptical, who aren't really interested in using AI. There'll be people who want to get involved, but they need further training. There'll be people with weird use cases that you need to understand, you know, how you can support them. So, it's really a people thing. Adopting AI is about talking with people in your team and understanding where they're at.

And the final thing I'd say is a sort of tongue-in-cheek thing, almost, right? So, AI adoption isn't over until it's over, and it's not over yet. It's not like choosing an IDE, and, you know, where you say, oh, we're going to buy a JetBrains site license, and anyone else can use VS Code or Emacs. It's not like that at all. It's still a very rapidly shifting landscape. So, I can stand here today and say, oh, the journey we've been on this year is X, and we've chosen to use Bedrock API keys and so on. But I think it's really important to keep an open mind for the future and be ready to adjust the way you're doing things, change your approach, and just be ready to move with the industry, because I don't think an AI enablement is a sort of static target. I think it's a constantly changing landscape that you're going to have to keep up with over time.

So, yeah, those are my learnings from this year. If you're interested in finding out more, I have written a blog post on this subject. And so, a lot of the stuff I've talked about is in this blog post. There's also some examples, sort of metrics or reports about the impact it's had on our team, as well as some code samples. So, if you're interested in how we wrote that command line tool and distributed Bedrock API keys to individuals, if you go to the blog post, there's some code examples that will show you exactly how we did that.

Okay, and that's the end of the presentation. So, thanks for listening. I love the Star Trek memes, by the way. It's truly iconic. We have a flood of questions. So, let's see how many we can get through in the time that we have. First question, okay. First question, based on the hiring approach and the pro-Vibe coding stance, it sounds like Typeform is quite hands-off on the final output. Do you think that's quite sustainable in the long term? I mean, it's an interesting question. But I would say that Typeform is definitely not hands-off on the final output. I mean, when you talk about Vibe coding a prototype from a product manager's point of view, that's right at the beginning of the product and design life cycle, right at the beginning, you know.

And for that to go from a prototype to a sort of working product in production requires lots of hands-on. And actually, what it's really done is enable us to increase the bandwidth with which we're able to experiment. So, rather than trying out, you know, a couple of ideas at the beginning, we can try out, you know, 10 ideas, we can do research, user research on lots of ideas. So, I think if anything, it's probably improving the quality of the final user experience, because it means we're able to test ideas much earlier on, and test more ideas much earlier on. And I guess that also links to everyone's definition of Vibe coding is completely different. So, when you talk about Vibe coding, what exactly do you mean?

Well, I guess in the context of a product manager building a prototype, I think maybe the term Vibe coding is appropriate, because they're not software engineers. They're building a product using a sort of business language or product language. But at the same time, they're not building a production ready, you know, piece of software. They're just interested in the sort of look, and feel, and user experience, which I think is quite different from how a software engineer might use, you know, agentic coding tools to accelerate them in their day-to-day work. Yeah, love that definition. So, thank you for that. Can you share your thoughts on the impact on engineers' authenticity and loss of technical knowledge when using AI coding tools as we all can learn to become managers of agents?

It's very topical, I suppose. This came up in a panel talk earlier, and I think my personal point of view is if I use AI to help me generate code, write code, build something, when I eventually open a pull request and ask a colleague to review it, I don't really say, oh, I used AI to help me write this code. I don't advertise the fact that I used AI to write this code. In fact, I sort of present it as my own work. And that's because, you know, AI might have helped me try a few ideas or accelerate my progress, but ultimately, it's my responsibility as a software engineer to make sure that the code is of a high quality and, you know, does what it says on the tin.

So, I'm not really sure. Maybe it's too early to say if it's having an impact on people's skills in the long term, but right now, I think it's essential that operators of AI coding tools fully understand what they're doing and are able to operate the tools, you know, as with as much subject matter knowledge as if they were writing the code themselves. Thank you for that, Andy. Also, we have here, what were the results you saw in the hiring process by introducing AI in the interview?

Yeah, I guess there's a couple of things I've been looking for. One is the way you might instruct or use AI, you know. So for example, spending time writing thoughtful prompts, giving the AI sufficient information in advance, sufficient context, sufficient guidance, compared to writing, you know, like one line questions or instructions. And also, being critical of the AI output. So for example, if a candidate in an interview just sort of accepted as given everything that was generated by the AI, that would be a red flag for me. Whereas if someone was being, you know, thoughtful and critical in inspecting the output of the AI and making sure that they were happy with that before moving forward, that would be, you know, something I was looking for.

And I don't think it's a big stretch for all of us as software engineers, to be asking ourselves questions like, you know, where am I at on the AI proficiency curve just now? Because I don't think these tools are going away. And I think it's going to be a skill we all need to develop in addition to technical knowledge, you know, the ability to operate AI proficiently, I think, is going to be important for the future.

Yeah, I guess there's a couple of things I've been looking for. One is the way you might instruct or use AI, you know. So for example, spending time writing thoughtful prompts, giving the AI sufficient information in advance, sufficient context, sufficient guidance, compared to writing, you know, like one line questions or instructions. And also, being critical of the AI output. So for example, if a candidate in an interview just sort of accepted as given everything that was generated by the AI, that would be a red flag for me. Whereas if someone was being, you know, thoughtful and critical in inspecting the output of the AI and making sure that they were happy with that before moving forward, that would be, you know, something I was looking for.

Yeah, I mean, they're told about it in advance as part of the process, so they know they're walking into an interview where they're expected to use AI. And I even tell people at the beginning, like we're giving you a task that's going to be too big for you to do manually. And I'm really interested in seeing how you're going to use AI. So, you know, we're very upfront with people about that, that we're expecting to see AI usage as part of the interview.

And I even tell people at the beginning, like we're giving you a task that's going to be too big for you to do manually. And I'm really interested in seeing how you're going to use AI. So, you know, we're very upfront with people about that, that we're expecting to see AI usage as part of the interview.

Okay, thank you. And so moving beyond the interview, so now we're actually talking about people internally, how do you assess AI's impact on productivity within your engineering teams? I think it's really hard to assess the impact on productivity. And in fact, I haven't even tried to assess, I mean, how do I, I don't, how do you even measure engineering productivity in the first place? I don't know. So, the things I've been, I've been focusing on, let me step back a bit. My personal experience of AI has been that it's been a good thing, that it's improved my productivity.

So I'm working on the assumption that it's going to have positive impacts on everybody else. So I've really been looking at metrics like, you know, the number of people using it, how much they're using it, how, you know, what proportion of our team are using it on a regular basis, whether or not there are some teams with, you know, loads of token usage, but only one user, which tells me that there's like a power user buried in the team. Or if there's like moderate usage, but many users in that team, which tells me it's sort of embedded in the way that the team's working. I've just been focusing on metrics like that, without really worrying about the end results, because my feeling is that the end result is likely to be positive.

And I think, I don't know how you would measure the impact on productivity. If anyone knows, I'd love to know, you know. I think that comes up every year, like beyond AI, how do you measure engineers' productivity? Does anyone have an answer to that? We might talk about that another day. But I'm actually curious as well.

So we're talking about embedding that learning, how do you actually share that knowledge internally? So if someone has like best practices and new tools or ideas and things like that, how do you stay up to date and share that knowledge? Yeah, I mean, we've got a Slack channel where there's a lot of active talk about this. And we've got some shared documentation places. So, you know, if someone new joins, or if they're an AI novice, there are some baseline resources to help get up to speed. So I think there's like a baseline there and a community that you can get involved with if you want to. And beyond that, you know, we're not really sort of really pushing adoption. We're just empowering adoption.

So we're just letting people adopt these tools organically. And what we're seeing is that, you know, people are adopting these tools because they want to. So for me, it's all just about removing those barriers, and just sort of greasing the wheels, and ensuring it's as easy as possible for people to get up and running. Thank you. And so do you practice TDD with AI at Typeform? I mean, I can't speak for every engineer at Typeform. I can answer on my part. And what I can say is that I think one of the one essential aspects of using AI for coding is to give the agent a way to determine whether it's sort of done the job or not, a way for it to get feedback.

So, for example, if you're asking it to fix, you know, linting errors, you need to tell it what command to run, to run the linter, so it knows when it's done. And so when it comes to TDD, I mean, I think TDD is essential to make sure it knows when it's met your requirements, because otherwise, it's just sort of, you know, throwing tokens in the wind, and some of them might stick and some might not. I like that. Okay, cool.

Another one here. So if individual devs use different models when developing in the same code base, do you find that that leads to any inconsistencies in the code? I mean, I'm sure it does. And we don't have any systematic ways to assess that. You know, I said earlier, when I'm using coding tools, when I open a pull request, regardless of which model I used, I'm presenting the changes as my own work. Not because I'm trying to hide the fact that I'm using AI, but because you know, I'm taking responsibility for the fact that, you know, I'm in charge of the one, I'm making these changes.

So in the same way, I would expect a code reviewer to assess that change as if it had come from any other person in the team, you know. So if there are inconsistencies coming through, you know, they passed code review, so they could have come from any individual having different sort of, you know, standards or preferences, you know. So thanks.

And how do you manage and limit costs using the bedrock approach? I don't know if there's many senior leaders from Typhoon here. We basically don't do either manage or limit. You know, so I've run a few reports, so I know like roughly how much it's costing. So I can tell you, we've got about 100 engineers, and it's costing us about $150 a month per engineer in the bedrock expenses. And I've shared those numbers with our senior leaders, and they're like, okay, yeah, that's fine. I mean, I think if it was like $1,000 a user, they'd probably be ringing alarm bells. But right now, we're just not, we're not limiting it.