Ensuring Quality with AI

So, my name's Richard Rodenkemper. I'm a senior software engineer at Sentry and I'm here to talk a little bit about how we can ensure quality with AI. So you might be familiar with this chart. If you haven't seen this, this is some data from GitHub about the number of pull requests, commits, and new repos per month over the last couple years. And you can see at the late part of 2025 and early 2026, growth has essentially gone exponential. And we shouldn't really be surprised by this, right? Coding agents have really taken off and it's not a surprise that we're producing a lot more code than we were in the past.

You might also be familiar with this chart though, right? This is the unofficial GitHub status page where they're not even going for, you know, four nines anymore. There's zero nines in that uptime, right? They're below 90%, well below 90%. Yes, some of this can be attributed to the previous chart I showed where we're just writing a lot more code and their systems are struggling to handle that. But you have to wonder how much of this is also related to coding agents being less trustworthy than a senior software engineer, a staff software engineer, or even a more junior software engineer.

Cloud is a similar example. Yes, growth has gone up a lot over recent years, but, you know, they're definitely using Cloud Code Ad Anthropic to create their app and you have to question what the relationship is. We've also seen a lot of studies about the fact that even though we're writing a lot more code before, that's not really translating to too much, right? App releases are up, but app reviews and app usage aren't. And overall lines of code is significantly up, but the number of releases isn't up by nearly as much.

So I'm here to talk a little bit about how we can use AI not just to produce a lot more code but actually as a quality tool. Tech dep and cleanup are more important now than in the past because of the code being written, using AI as a quality tool lets engineers focus on what they like doing and what they're good at, and reliability is still important. If you want to create a good product that makes money, it needs to be reliable. Every time GitHub goes down, engineers around the world notice and it really annoys them. You can't have your app be as unreliable as that. We still need to strive to create quality products.

So to use AI as a quality tool, we have to ask ourselves, what is AI good at? What are these coding agents good at? They're good at handling relatively low level tangible information, less good at making things up on their own. They're very good at searching, better than a normal engineer at searching a code base and finding things. They're also really good at handling large amounts of data, right? You can introduce an AI coding agent into a new code base and very easily it will understand what's going on. You can't say the same about an engineer. And it's also pretty good at only looking at the short term, short term thinking. Longer term, the consequences of decisions and things like that, it's less good at, right? Design and all those things too.

Here's some examples from Sentry from across the organization of where we've used AI to push quality, not new features. We've been using a lot of AI code reviews. We built one ourselves. We've been using BugBot and other ones as well. It basically adds another layer to these PRs that are going out. People are creating more PRs, but how can we ensure that they're high quality and they're not breaking our services? It's not a replacement for humans. At the end of the day, every PR still needs to be reviewed by a human and merged by a human. The engineer is still responsible for their code, but it has helped us find critical issues, incident causing issues before we've merged PRs.

That's really, really useful. A project I worked on is we had to do a lot of endpoint deprecation for this new project we were doing, and I thought this would be a great task for AI to do. Compared to a week where the engineer wasn't using cloud code to do this, I was able to do 20X the number of deprecations in that week. Now this isn't a particularly fun task. No one's going to work every day excited about deprecating endpoints. Engineers want to work on more important things, but I was able to do this in a lot less time and do a lot more than if an engineer was doing it.

This was a project that had to be done, and AI was able to do it significantly more efficiently than an engineer would be able to. We've been able to update a lot of our systems too. We moved an internal shipped blog from some old version of Remix over to Vercel. It loads faster, fewer lines of code, significantly smaller bundle size, and we've moved off of some pretty legacy frameworks and packages. We helped move one of our design systems in our front end for Sentry over using a bunch of code mods, which saves a bunch of time.

That's another thing that engineers aren't really excited about doing, migrating design systems. But AI was able to help out, put up a number of really, really massive, tens of thousands of lines of PRs, and we used a code mod so that, especially for a design system, it's editing files in a bunch of different areas. We have code owners set up in GitHub, so it pinged a lot of teams. We created a code mod using Cloud Code that helped reduce the number of notifications that people were getting when we were doing these systems.