Democratising AI in Engineering: Lessons from Typeform's Journey

So, hi everyone. Welcome to my talk about democratizing AI in engineering, which is a sort of a summary of some of the learnings we've had from our journey at Typeform this year. I thought I would start with some introductions. So, my name is Andy. I'm a staff engineer at Typeform. In case you haven't heard of Typeform, Typeform is an online forms builder, a bit like Google Forms, so you can sign up and you can create a form, for lots of information you want to receive, you can send that form out to your customers and they can fill it in and you can analyze your results and send it to third party integrations and things like that. Are there any Typeform users here? Has anyone used Typeform or heard of Typeform? Okay, good, right, a few friendly faces, that's good to know. Other thing to know about me is I'm a Star Trek fan. Are there any Star Trek fans here? Okay, a few people. Good, all right, live long and prosper. There are some Star Trek memes, so you'll probably find them funny. I hope so anyway. If you're not a Star Trek fan, which is the vast majority of you, when you see a Star Trek meme, just laugh, that's just sort of cute. Other thing to know about me is I'm an Emacs user, big Emacs fan, thanks for the intro. I wrote the presentation in Emacs. I'm actually presenting in Emacs. Those of you at the front will see my mode line, which is very exciting. Are there any Emacs users here? Okay, there's one. It's quite a familiar experience to me to be the only Emacs user in the room, maybe not a room with so many people in, but okay, that's fine. Don't worry, we'll just forget the whole Emacs thing.

Okay, so what am I going to talk about today? I'm going to talk about the journey we've been on this year at Typeform to provide our engineers with access to AI. I'm mainly going to talk about that inside our engineering organization. I'm going to spend a bit of time on that. I'm also going to share a few anecdotes about the impact AI has had outside of our engineering organization, and I'm going to close with a few sort of learnings or lessons from that journey. Now, before I talk about our journey inside engineering, I thought I would just frame this conversation with an illustration. Now, this, I just want to, fair disclosure, this chart is entirely fabricated. No real data was used. I actually asked an LLM to generate this diagram, and those at the front, you might get bonus points for spotting some clues that this diagram was AI generated. So, a complete fabrication. But I do think it illustrates the point I want to make, which is that this year there's been a huge amount of hype about AI, an exponential amount of hype.

The green line is an exponential amount of hype this year about AI coding tools. And the kind of coding tools I'm talking about are agentic coding tools like Claude Code, Cursor, Klein, that kind of thing. My presentation is about the journey we've been on at Typeform, but I think it's probably a familiar experience for many people in many organizations. And the other line trailing behind the hype line is adoption. Basically, I'm trying to illustrate that whilst I think there's been a lot of hype in the industry, I think actually people's adoption has been somewhat slower than that hype might indicate. And that might be especially true in larger organizations, because for an engineering team to really adopt AI tools, it's not just a case of individuals picking it up. The company needs to buy licenses, pay for LLM access, choose tools, consider security, all of this kind of thing, which slows down that adoption curve.

So, when I'm talking about the adoption of AI tools, it's that adoption curve that trails behind the hype. That's the curve I'm talking about, the adoption of tools in engineering teams. Let me tell you a bit about our journey at Typeform. This year, I think I could probably summarize the entire experience we've had with this one phrase. Adopting AI in our engineering organization has all been about democratizing access to AI and federating usage of AI. And I'm going to unpack what I mean by those two terms a little, but that's really been the guiding principle throughout the whole year, democratizing access and federating usage.

Let's start with democratizing. What do I mean by democratize? So, I asked an LLM to write a summary of the word democratize, and we didn't so much write a summary as an essay. So, I said, okay, it's far too long. Summarize your summary. And then it wrote this paragraph, which I thought was actually pretty good. So, what do I mean by the word democratize? Democratizing is all about making something accessible to ordinary people, everybody, not just a specific group or a set of elites, making something accessible to everybody by removing barriers to entry, simplifying how they can use it, and distributing that usage as widely as possible, ultimately so that it's available to everyone and not just a small group. So, when I say democratizing access to AI, I'm talking about making access to AI available to everybody in your team, reducing the barriers to entry for everybody so it's as easy as possible for people to use it, and distributing that access as widely as possible so everyone in your team has access to AI.

So, when I say democratizing access to AI, I'm talking about making access to AI available to everybody in your team, reducing the barriers to entry for everybody so it's as easy as possible for people to use it, and distributing that access as widely as possible so everyone in your team has access to AI.

Well, don't worry. That's what the presentation's about, so I'm going to tell you. One of the main enabling technologies we've used this year is AWS Bedrock. Some of you may not be familiar with AWS Bedrock, so I'll tell you a little bit about it. AWS Bedrock is a managed service in AWS that gives you access to LLMs from a wide range of vendors.

For example, you'd find Anthropic models in there, Claude Sonnet, Claude Opus, Amazon models, Meta models, and models from various vendors. You use the Bedrock API to make inference requests regardless of the model you're using, and you control access with familiar AWS primitives like IAM roles and policies. This approach enables accessing LLMs without leaving the AWS estate, simplifying usage for applications utilizing Bedrock for LLMs.

This is not AI-generated. I actually copied this from the AWS developer guide, so it's not fabricated. AWS Bedrock is a managed service in AWS that gives you access to LLMs from a wide range of vendors. For example, you'd find Anthropic models in there, Claude Sonnet, Claude Opus, Amazon models, Meta models, and models from various vendors. You use the Bedrock API to make inference requests no matter which model you're using, and you control access to those models using familiar AWS primitives like IAM roles and policies. You pay for your usage through your Amazon account.

Accessing LLMs without leaving the AWS estate simplifies usage for applications. When building an application using Bedrock for LLMs, the process is straightforward. Your application makes inference requests to Bedrock, authenticates with its IAM role, and can utilize foundation models like Claude Sonnet or Opus. However, democratizing access to AI for engineers and team members introduces complexities. Users need to authenticate with AWS, possibly requiring special security measures such as joining a VPN to access Bedrock.

Lowering the barrier to entry for AI usage was a key principle behind democratization efforts. Initially, the setup process for accessing AI tools through AWS Bedrock was cumbersome and complex. However, AWS introduced a new feature called Bedrock API keys to streamline the process significantly. With Bedrock API keys, users can generate a single API key with tightly scoped permissions, eliminating the need for extensive authentication processes like connecting to a VPN or authenticating with AWS.

They're people at keyboards, and that's a little bit more complicated. A user would need to authenticate with AWS somehow. Maybe they've got an access key or something like that. There might be some special security requirements that they might have to follow. So, for example, we would need our engineers to join a VPN in order to access Bedrock in this way, and only then can they use Bedrock to make inference requests.

And you remember one of the principles behind democratization was lowering the barrier to entry. This is quite a high barrier to entry. It's quite complex to sort of set all this stuff up. And we did start here earlier in the year, but the point is that it was quite cumbersome. You know, we'd say to people, if you want to use AI for coding, just follow this massive, you know, the piece of documentation, and that will get you set up.

Now, luckily, AWS released a new feature for Bedrock earlier in the year called Bedrock API keys, which massively simplifies this process. So, with Bedrock API keys, you can generate a single API key, which is effectively a sort of credential for accessing Bedrock. It's got very tightly scoped permissions, so you could just use it to make inference requests. And you don't need any of the other stuff, so you don't need to authenticate with AWS. You don't necessarily need to be connected to a VPN or anything like that. Just with your Bedrock API key, you can then make inference requests.

So, this is what we're using. We're giving all of our engineers a Bedrock API key, and then they're empowered to use that Bedrock API key however they like with whichever tools they like to make inference requests to LLMs. And you might be wondering quite how we did that. And this is kind of how we did it. We have an internal command line tool called TF, which stands for type form. All of our engineers have access to that command line tool. It's got sort of a Swiss army knife, lots of utilities on there for our engineers to use. And we wrote a new command called generate Bedrock token. And when you run that command, it generates a Bedrock token. All of our engineers can use that, and that grants them access to LLMs. There's a bit of plumbing behind the scenes, but basically this is the distribution model.

Now, I mentioned earlier that you can use models from many vendors in Bedrock, and most of our engineers are using anthropic models like Claude Sonnet. And you might be wondering why we went through this whole process of using Bedrock when we could have just used anthropic. Second Star Trek meme. Okay. Anyway, so why did we go through this whole process when we could have just used anthropic? We could have just signed up for an anthropic account, generated an anthropic API key for our engineers and they wouldn't have had to use Bedrock at all. Or we could have done the same with openAI or something like this. And in fact, we did try that as well. We tried lots of different approaches. And although it technically works, it's a lot less convenient because we use AWS to host our platform, so they're an existing vendor for us. Anthropic would be a new vendor for us. We need to sign up for an agreement. We need to pay a bill. We'd need to generate API keys for everybody. We'd need to distribute those API keys people. When a new engineer joins, we need to remember, oh, generate that person an API key. When someone leaves, we've got to remember, oh, deactivate that API key. So there's like a whole load of admin we'd need to do, to do it this way. Whereas, all of our engineers already have an AWS account. When an engineer joins, they just get an AWS account by default.

When they leave, we've got a process to deactivate that account. And now they have access to LLMs as well because they can generate a Bedrock API key. And we've got a big bill with AWS already for our platform. So what's a few Bedrock API keys? We just pay for our LLM usage for engineers through that existing billing process. So we could have done what data suggested. We could have used Anthropic. But it turns out, it's just much more convenient for us to use an existing vendor like AWS.

And ultimately, what this means is that every engineer in our organization has unlimited access to LLMs for software development. And that might be through using a tool like Claude Code or Cline. It might be for writing scripts. I mean, however they want to use it, they just use their Bedrock API key and they've got unlimited access. And they don't need to worry about our Anthropic account running out of credits or anything like that. OK, so that's the first part of this big idea, democratizing access. Coming up with a way with a very low barrier to entry and distributing it widely to everyone.

So everyone in our team can access AI. What do I mean by this second part, federating usage? What do I mean by the word federation? I don't mean the United Federation of Planets. That's not a Star Trek thing. No, OK. What I mean is that everyone's got the same way of accessing LLMs through Bedrock. But when it comes to choosing which tools people use, they might be using Claude Code. They might be using Cline. They might be using Emacs. I know one person using Emacs. That decision is entirely up to them. The decision about which tool to use is federated down to the level of the individual.

So we're not prescribing which tools people are using. We're giving people a sort of single paved road for accessing LLMs, but then federating the decision about tooling to individuals. And ultimately, it means that everyone has access, the same way to access, but they can use whichever tools they like. That's a matter of personal preference. And we're federating that decision down to individuals.

OK, so that's a summary of the journey we've been on this year to democratize access to AI inside our engineering organization. I'm just going to share now a few anecdotes about the impact AI has had outside of our engineering organization as well. One of the main areas AI has had an impact outside our engineering organization has been on hiring. Before we really started adopting AI properly, this is what our hiring process looked like. Maybe it's a familiar experience to some of you, either as an interviewer or as a candidate.

We basically say to people, listen, you need to code for an hour and we're going to watch you do it. We gave them a nice practical task and sort of a pair programming exercise. But critically, we did say to people, you're not allowed to use AI coding tools because we wanted to see what people could make with their own hands. But as we started adopting AI more inside our engineering organization, we changed our interview process to have candidates use AI for an hour. Now we're assessing people's AI proficiency during interviews to ensure they join the team with AI literacy.

So it was a sort of a collaborative exercise. But critically, we did say to people, you're not allowed to use AI coding tools because we wanted to see what people could make with their own hands. But the trouble with that approach is as we started adopting AI more and more inside our engineering organization, we're asking people to do one thing in the interview, but then asking them to work a completely different way once they started working with us. And furthermore, we're expecting all of our engineers to pick up AI coding tools, and we haven't checked for that in the interview process at all. So now we've changed our interview process to this. We say to people, actually, we want you to use AI for an hour instead and we're going to watch you do that now. And it's still a sort of friendly approach. We give them a different task, a much bigger task, way bigger than you could possibly do in an hour by yourself. There's no way you could complete this task in an hour by yourself. But we say, listen, we'd like you to use AI to work on this task. And we're still going to assess your programming skills, your problem-solving skills, your ability to communicate. But now we're also assessing people's AI proficiency because we want them to join the team with AI literacy. We want them to join the team and share their experiences with their team members and enrich everybody's usage of AI as well. And that's proving far more effective as a hiring process.

OK, and the other thing I want to talk about outside our engineering organization is the impact it's had at the boundary between our engineering organization and adjacent technical disciplines like design and product management. Now, the eagle-eyed amongst you may notice this is not a Star Trek meme. I sort of stepped into an adjacent genre here. So one of the main ways our design and product management teams would interact with our engineering team is via our design system. So we've got a design system with a load of front-end components in and corresponding components in Figma. So, for example, a designer can create a mock in Figma and then hand it to a front-end engineer, and they can recreate that with real code and it sort of looks and feels the same. So we built an MCP server for our design system so that a front-end engineer can rapidly transform a Figma mock into a working sort of prototype, working piece of code, which isn't the finished product. Maybe it gets them, I don't know, 60 or 70 percent of the way there, but then it basically has massively streamlined that interface between design and front-end engineering. And it's also proved incredibly useful for product managers because product managers would previously try and articulate an idea by maybe sort of putting together a demo or a sort of mock experience, either by relying heavily on the designer or increasingly by using vibe coding tools, like Lovable or Bolt. But the trouble with those vibe coding tools is you get something that sort of works, but it looks completely different from our actual product. They can now plug in this MCP server for the design system, and they get a living, breathing prototype that looks just like the real thing. And, of course, it's not production-ready code, but it means they can rapidly iterate and test new ideas without those ideas looking, you know, wildly different from the main product.

OK, so that's just a few anecdotes about the impact of AI outside of our engineering organization. And the last thing I thought I would share is just a few sort of learnings or principles that we think are powerful that have come from this journey this year. So, the first is that I think AI adoption in a team is all about empowerment but not about prescription. And for me, that comes out in this idea of democratizing access.

It's about empowering everybody to access AI without being too prescriptive about what they use it for or how they use it. So, we've really tried to empower everyone to access AI by giving them simple tooling, but not prescribe what they use it for, how they use it by saying, listen, you can go and use whichever tools you like. As long as they can use Bedrock API keys, you're good to go, go and try lots of tooling.

The other thing I've certainly learned this year is that AI enablement is really a people thing. It sounds like a technical thing, and there's technical aspects to it. If you want your team to adopt AI, you have to go and speak to your team members, find out where they're at. There'll be some people in your team who are early adopters, who are power users, there'll be some people in your team who are very skeptical, who aren't really interested in using AI. There'll be people who want to get involved, but they need further training. There'll be people with weird use cases that you need to understand how you can support them. So, it's really a people thing.

The final thing I'd say is a tongue-in-cheek thing, almost. AI adoption isn't over until it's over, and it's not over yet. It's still a very rapidly shifting landscape. It's really important to keep an open mind for the future and be ready to adjust the way you're doing things, change your approach, and just be ready to move with the industry because I don't think an AI enablement is a sort of static target. I think it's a constantly changing landscape that you're going to have to keep up with over time. Those are my learnings from this year.

It's still a very rapidly shifting landscape. So, I can stand here today and say, oh, the journey we've been on this year is X, and we've chosen to use Bedrock API Keys and so on, but I think it's really important to keep an open mind for the future and be ready to adjust the way you're doing things, change your approach, and just be ready to move with the industry, because I don't think an AI enablement is a sort of static target. I think it's a constantly changing landscape that you're going to have to keep up with over time. Those are my learnings from this year. If you're interested in finding out more, I have written a blog post on this subject, and so a lot of the stuff I've talked about is in this blog post. There's also some examples, sort of metrics or reports about the impact it's had on our team, as well as some code samples. So, if you're interested in how we wrote that command line tool and distributed Bedrock API Keys to individuals, if you go to the blog post, there's some code examples that will show you exactly how we did that.

Okay, and that's the end of the presentation. So, thanks for listening. Full crowd. That was a massive run. I love the Star Trek memes, by the way. It's truly iconic.

Good. We have a flood of questions, so let's see how many we can get through in the time that we have. Okay. First question, based on the hiring approach and the Pro-Vibe coding stance, it sounds like Typeform is quite hands-off on the final output. Do you think that's quite sustainable in the long term? I mean, it's an interesting question, but I would say that Typeform is definitely not hands-off on the final output. I mean, when you talk about Vibe coding a prototype from a product manager's point of view, that's right at the beginning of the product and design lifecycle, right at the beginning, and for that to go from a prototype to a sort of working product in production requires lots of hands-on.

And actually, what it's really done is enable us to increase the bandwidth with which we're able to experiment. So, rather than trying out a couple of ideas at the beginning, we can try out 10 ideas. We can do user research on lots of ideas. So, I think, if anything, it's probably improving the quality of the final user experience because it means we're able to test ideas much earlier on and test more ideas much earlier on. And I guess that also links to everyone's definition of Vibe coding is completely different. So, when you talk about Vibe coding, what exactly do you mean?

Well, I guess in the context of a product manager building a prototype, I think maybe the term Vibe coding is appropriate because they're not software engineers. They're building a product using a sort of business language or product language. But at the same time, they're not building a production-ready piece of software. They're just interested in the sort of look and feel and user experience, which I think is quite different from how a software engineer might use agentic coding tools to accelerate them in their day-to-day work.

Yeah. Love that definition. So, thank you for that. Can you share your thoughts and impact on engineers' authenticity and loss of technical knowledge when using AI coding tools as we all can learn to become managers of agents? That's a good question. It's very topical, I suppose. This came up in a panel talk earlier, and I think my personal point of view is if I use AI to help me generate code, write code, build something, when I eventually open a pull request and ask a colleague to review it, I don't really say, oh, I used AI to help me write this code. I don't advertise the fact that I used AI to write this code. In fact, I sort of present it as my own work. And that's because AI might have helped me try a few ideas or accelerate my progress, but ultimately, it's my responsibility as a software engineer to make sure that the code is of a high quality and does what it says on the tin. So, I'm not really sure. Maybe it's too early to say if it's having an impact on people's skills in the long term, but right now, I think it's essential that operators of AI coding tools fully understand what they're doing and are able to operate the tools with as much subject matter knowledge as if they were writing the code themselves. Thank you for that, Andy.

Also, we have here, what were the results you saw in the hiring process by introducing AI in the interview? So, I can't speak for results in terms of the number of candidates we've hired and stuff because I don't really have visibility of that, but I can tell you from the point of view of an interviewer, I certainly have seen lots of candidates who I would otherwise maybe have sort of passed through the hiring process, who when it came to using AI tools, didn't show the level of literacy I was expecting. One of the things I said was, we're now assessing people's AI proficiency. I've seen lots of people who seemed like, as far as you can tell, they seemed like very competent, experienced engineers, but simply weren't as proficient as I was expecting with AI coding tools. And I think that's a reasonable criteria to assess someone on, especially if they're joining a team that's already working on their AI proficiency. And I don't think it's a big stretch for all of us as software engineers to be asking ourselves questions like, you know, where am I at on the AI proficiency curve just now, because I don't think these tools are going away. And I think it's going to be a skill we all need to develop in addition to technical knowledge, you know, the ability to operate AI proficiently, I think is going to be important for the future.

Yeah, I guess there's a couple of things I've been looking for. One is the way you might instruct or use AI, you know. So, for example, spending time writing thoughtful prompts, giving the AI sufficient information in advance, sufficient context, sufficient guidance, compared to writing, you know, like one-line questions or instructions. And also being critical of the AI output. So, for example, if a candidate in an interview just sort of accepted as given everything that was generated by the AI, that would be a red flag for me. Whereas if someone was being, you know, thoughtful and critical in inspecting the output of the AI and making sure that they were happy with that before moving forward, that would be, you know, something I was looking for. And have you seen that in your interviews? Yeah, I've definitely seen both, definitely seen both, yeah, from like, you know, people using, incidentally using AI because they thought that was important in the interview, to being clear that it's a core part of the way that somebody works. And have you included that in terms of like guiding people how to be successful in an interview? Have you shown them that it's important for them to use these AI tools? Or how do they know that they can use it, basically? Yeah, I mean, they're told about it in advance as part of the process, so they know they're walking into an interview where they're expected to use AI. And I even tell people at the beginning, like we're giving you a task, that's going to be too big for you to do manually. And I'm really interested in seeing how you're going to use AI. So, you know, we're very upfront with people about that, that we're expecting to see AI usage as part of the interview. Okay, thank you.

And so moving beyond the interview, so now we're actually talking about people internally, how do you assess AI's impact on productivity within your engineering teams? I think it's really hard to assess the impact on productivity. And in fact, I haven't even tried to assess, I mean, how do you even measure engineering productivity in the first place? I don't know. So the things I've been focusing on, let me step back a bit. My personal experience of AI has been that it's been a good thing, and that it's improved my productivity. So I'm working on the assumption that it's going to have positive impacts on everybody else. So I've really been looking at metrics like, you know, the number of people using it, how much they're using it, what proportion of our team are using it on a regular basis, whether or not there are some teams with, you know, loads of token usage, but only one user, which tells me that there's like a power user buried in the team. Or if there's like moderate usage, but many users in that team, which tells me it's sort of embedded in the way that the team's working. I've just been focusing on metrics like that, without really worrying about the end results. Because my feeling is that the end result is likely to be positive. And I think, I don't know how you would measure the impact on productivity. If anyone knows, I'd love to know. I think that comes up every year, like beyond AI, how do you measure engineers productivity? Does anyone have an answer to that? We might talk about that another day.

So we're talking about embedding that learning, how do you actually share that knowledge internally? So if someone has like best practices or new tools or ideas, and things like that? How do you stay up to date and share that knowledge? Yeah, I mean, we've got a Slack channel, where there's a lot of active talk about this. We've got some shared documentation places. So you know, if someone new joins, or if they're an AI novice, there are some baseline resources to help get up to speed. So I think there's like a baseline there, and a community that you can get involved with if you want to. And beyond that, you know, we're not really sort of really pushing adoption. We're just empowering adoption. So we're just letting people adopt these tools organically. And what we're seeing is that, you know, people are adopting these tools because they want to. So for me, it's all just about removing those barriers, and just sort of greasing the wheels, and ensuring it's as easy as possible for people to get up and running. Thank you. And so do you practice TDD with AI at Typeform? I mean, I can't speak for every engineer at Typeform. I can answer on my part. And what I can say is that I think one of the one essential aspect of using AI for coding is to give the agent a way to determine whether it's sort of done the job or not, a way, you know, for it to get feedback. So for example, if you're asking it to fix, you know, linting errors, you need to tell it what command to run, to run the linter, so it knows when it's done. And so when it comes to TDD, I think TDD is essential to make sure it knows when it's met your requirements, because otherwise it's just sort of, you know, throwing tokens in the wind, and some of them might stick and some of them might not. I like that. Okay, cool.

Okay, cool. Another one here. So if individual devs use different models when developing in the same code base, do you find that that leads to any inconsistencies in the code? I mean, I'm sure it does. And we don't have any systematic ways to assess that. You know, I said earlier, when I'm using coding tools, when I open a pull request, regardless of which model I used, I'm presenting the changes as my own work, not because I'm trying to hide the fact that I'm using AI, but because, you know, I'm taking responsibility for the fact that, you know, I'm in charge of the one making these changes. So in the same way, I would expect a code reviewer to assess that change as if it had come from any other person in the team, you know? So if there are inconsistencies coming through, you know, they passed code review, so they could have come from any individual having different sort of, you know, standards or preferences, you know. Thanks.

And how do you manage and limit costs using the breadrock approach? I don't know if there's many senior leaders from Typeform here. We basically don't do either, manage or limit. You know, so I've run a few reports, I know, like roughly how much it's costing. So I can tell you, we've got about 100 engineers, and it's costing us about $150 a month per engineer in the bedrock expenses. And I've shared those numbers with our senior leaders. And they're like, okay, yeah, that's fine. I mean, I think if it was like $1000 a user, they'd probably be ringing alarm bells. But right now, we're just not, we're not limiting it. We're doing exactly what I said on the slide, which is we're giving everyone unlimited access, you know? If and when it becomes too expensive, I'm sure we'll have a conversation about it. But right now, I think we're of the mind that we just want to encourage people to adopt these tools in whatever way they feel comfortable.

We're at time. Thank you so much, Andy. Round of applause for Andy, please.