Your AI Code Reviews Are Missing the Point (And How to Fix It)

Hi, everyone, my name is Ishai. I'm the CTO at LearnrB. And today I'm going to talk about AI code reviews. Maybe you have them, maybe you don't. I'm going to talk about why they're probably not doing everything they can for you and how to fix this.

So, I'm going to cover some intro about AI code reviews and why we think this is the first step that AI really takes into the SDLC. I'll show some anti-patterns that cause AI code reviews to miss the point. And then I'll focus on how to make them better.

What really matters in your AI code review system and how to get from mediocre results to great results that really transform how you work. So, AI code reviews, that's a growing success story for AI within the coding workspace. And in contrast to how AI typically helps developers in the IDE or in CLI by helping them code, code reviews actually begin AI's journey into the process itself because they now leave the scope of a single developer and are really a place where AI becomes or augments the process. If we look at the different ways in which AI helps developers and helps engineering teams deliver value through changes in a code base, we have three major modes.

And in this slide, I'm going to describe the modes, but also their maturity model and how they advance over time. So, the first row is the coding assistants. These are the famous cursors, cloud code, client copilots and so on, typically living inside my IDE or on my laptop. In a CLI, maybe I have a swarm of agents, but they're in the scope of helping me write code. So, I'm the developer. I own what I'm doing. These agents help me write more code, help me write it faster, maybe help me write it end-to-end. When it's ready, I will submit the code as a pull request or as a merge request, and it will get through the rest of the SDLC.

This mode obviously exploded, began experimentation in 2024, and last year basically was the year when this mode matured. So, large organizations already deploy agents and helpers in this mode, and in maturity, I'm talking about RFPs, about baking this into budgets, about having measurements around how is it helping me, about having training programs, becoming part of even resumes and how we evaluate candidates. All of these things have happened, and now this basically has become another mature tool in the hands of developers. It's not over in terms of evolution, but it has gone through a quick maturity cycle to become something everyone needs.

The second mode, assistant agents, are typically ones that are a little more autonomous. They're no longer within my private yard or my laptop as a developer, and they typically react to new code. So, these could be obviously AI code review agents, but also agents that help me update my documents, describe my pull request. They're responding to some changes or to steps in the SDLC. They're not taking work end-to-end, and these are now going through late experimentation and beginning to be mature.

With AI code reviews being the, I would say, the avant-garde of maturity, many organizations, including large ones, are now looking and deploying and doing bake-offs with AI code review and similar flows that help the developer process with augmenting some steps, and particularly reacting to new code.

Finally, the third row is the more autonomous agents, like the devins, the co-pilot coding agents, and others, that are designed to take a task end-to-end. These typically run in some loop in the cloud or in my pipeline. They look at a JIRA ticket or a similar description of what needs to be done, maybe take something off the backlog, maybe find a bug. They create a fix or they create a new feature. They will put up the pull request or the change request, and they're much more autonomous in their nature. These are still going through experimentation. These are still early. Organizations are trying these out, understanding the challenges they bring, such as ownership of the code and so on.

They're obviously happening. They're adding value, but they're not in the mature phase yet. In this talk, I'm going to focus on AI code reviews, which is in the second row and is now beginning to move from experimentation to maturity. A lot of what we'll talk about is what is required, if you like, to become a mature solution. In Leonard B., we talk with a lot of engineering organizations, and we ran a survey to understand where people are seeing AI getting involved in the SDLC.

I'm not going to walk through all of the areas here, but you can see the area that is under the coding category has a lot of AI and mixed modes where AI is doing more work and AI plus humans is taking over. So that could be writing the code. That could be creating a pull request and so on. If you look at the review and release, the box I highlighted, these are one of like a couple of the smaller areas. Specifically, if I'm looking at who reviewed the PR, this is where you see a lot of mixed mode and you see some nice AI activity.

So AI code reviews are starting to help organizations with reviewing the code suggestions. But if we look at who made the decision to merge, who actually approved the pull request and merged the code, that is still very, very much human. That is the smallest bar of all the table with regards to AI involvement. We see organizations leaning into AI code reviews, but not so much, at least at this point, leaning into AI to make a decision about accepting the code. Remember that. We're going to come back to that later.

So one of the key challenges coming from using AI to help me write my code is that it creates new types of bottlenecks. We now have much faster ways to code and developers who have learned to use the AI tools and really leaning into having agents like Cloud Code on their laptop, running a few of them in parallel, spec-driven development. There's a bunch of new practices that all result in a lot of new code. So there's much more new code. AI is happy to create a lot of code, very verbose. But who's going to review all that code?

If the review process, and you've just seen it, the decision process is still manual, we now have a new bottleneck about how do we handle all of this, you know, 10x amount of code, if our delivery pipeline, the reviews, acceptance, decisions are all still human. In addition, having so much more code and the additional increased velocity in code creation means that quality issues are now becoming more expensive. I have 10x more code, there's going to be 10x more quality issues, even if I keep the same quality standards, which means I will need to respond to quality issues in a much higher rate.

So having more codes actually means we have to increase our quality rates and have even fewer bug density than or smaller bug density than we had before. Just to get everyone aligned on AI code reviews, if you don't already have those in your organization, these are code reviews happening in the pull requests, very similar to what you would expect from a human reviewer, whereas there is a bunch of comments in the pull request, and in some cases, there is a suggestion highlighting a specific line of code or area in the code where there's a problem and a suggestion on how to fix it. In some of the Git provider UIs, you can directly commit such suggestions.

So whatever you expect from a human to provide you when reviewing your code, you should expect from AI code review bots to look at your code, look at the changes you've made, understand their meaning, look for problems and give you a result. The real reasons why this helps to drive productivity is first and foremost, a short feedback loop. If I typically have to wait for hours or maybe a day, maybe two, maybe two weeks to get someone to review my code, when this is a bot, when this is AI, you get feedback in minutes. So I'm still in the zone, I'm still very much thinking about the code that I just changed. Getting early feedback means I don't have to context switch, I don't have to remember what I did here. I can quickly fix or address the feedback and improve my code. When this actually reaches a human reviewer, maybe after I fix the stuff that AI found for me, the human reviewer gets a much better pull request. It's already fixed of all the easier issues to find. I now have much better quality code to look at and I can focus my human brainpower on the difficult things and waste less time on dealing with either simple things or just things that AI found before I had to. And the third bullet here is important. It's an opportunity to add another automation step. So we just talked about this new kind of bottleneck and the fact that organizations still need a human decision to decide to accept code. When I have a good AI code review, I can start thinking about, maybe I can automatically merge the code if it passes the code review. At least in some cases, maybe some PRs in some parts of the codebase, under some rules, if they pass, everything is clean, AI has inspected the code, it's determined there's no law for bugs, maybe I can do a merge, maybe I can skip a human completely. And that is where you gain huge productivity because humans will probably always be the bottleneck. So why does the AI code review make so much sense? Why is that a great use case for AI? So first of all, it's an existing human process with a lot of friction. I have to wait for others to provide me with feedback. They have to context switch out of what they're doing. There's a lot of back and forth between humans. So there's a lot of friction in an existing process that is a prime target for AI to optimize. It's a well-defined task. The input and outputs are well-defined. Here is my diff to the codebase and the output is a list of comments and suggestions to fix the code. So it's very well-defined, another thing that AI loves. AI code review typically can work with limited context. I don't need to know everything about everything to provide a solid code review. I can look at the codebase, but I don't need to put all of it in my memory at once. Much like a human knows some of the codebase, knows how to search for more information if they need to understand something they haven't learned before. There is a well-defined context that I can look at and it doesn't have to be huge. And this rides the wave of using AI to assist with code.

We've all seen that AI is really good. The LLMs are really good with reading, understanding, and writing new code. That's already been proven by the first wave of AI coding assistance. So it's natural to extend that to do code review in addition to code generation.

Now I'm going to talk about why AI is missing the point in many cases when doing code reviews. And I'm going to highlight some anti-patterns we've seen with organizations that are starting to experiment or to roll out AI code reviews and talk about why these are basically blockers from getting the most value from AI code review.

Finally, I'm going to talk about how to improve it. The first pattern is ad hoc adoption. Just spend $29 or $20 a month, buy a tool, provide your team with access, maybe roll it out in some repos. You now have random pockets of use. Maybe it's not reviewing all code. Maybe it's not reviewing all code all the time. Maybe it's reviewing just the first version of a pull request but doesn't review it again after changes are made.

Maybe it's getting ignored. We know about a lot of tools that are running in PRs, spinning out output, but they're not really getting used. No one is actually looking at them. No one is relying on them as part of the process. So just putting a tool out there is not enough. Typically, you are blind as an organization. Who's using? What is it finding from you? Is it really finding bugs? What kind of bugs? Where are the bugs happening? What's the density? All of this information, typically, at least in a naive AI code review rollout, is not there, and you are flying blind. If there's any learning here, it's going to be lucky.

AI code review, does it learn? Does it improve from the feedback that it's getting from users? Does it understand the way we want to see code and code changes happening in our codebase, the differences between teams and all of that? So if you're just slapping on a tool, these are the walls and the limits you're going to hit.

Second pattern is over-reliance. So, oh, there's a code review. AI has reviewed my code. I don't have to review my code. I can write some sloppy code and hope that AI will catch it. So that's a pattern where, by having a tool with some quality label around it, some teams, some developers will over-rely and just say, okay, there's someone else that's going to find the problems. And then there's also a case where you have, you know, some people use basically the same model, like cursor and then bug bot, the same model to write the code and then to review it. That in itself could be a problem. You want to at least look at a variety of models between writing code, reviewing it. So you have more chances of finding things and more angles here.

And then the third pattern we're seeing is sticking to the same old rigid process, right? We have one step, which is code review. We can augment that, augment the human at just that point, maybe save some time for getting the first review and early feedback. That's a lot of value. But we're not changing the process. There's still going to be a review and a human decision. And it's the same one process fits all, for all pull requests at all times.

This is what we see happening at like 95% of the industry. Everyone has to go through a human approval, and the process is extremely rigid. Ignoring the opportunity to make some changes to the process to instill some flexibility and automation is like a glass ceiling on what you can get from AI code reviews. If you can somehow leverage them to make automatic decisions about merging, that becomes a game changer. So keep that in mind as we talk about how to make things really, really better.

So what really matters? I'm going to start talking about, like, first of all, how to think about evaluating a code review rollout or solution. So this is about the tools you're choosing, also about how you're using them. And there's going to be a progression here. I'm going to start with a simple table stakes and move up the ladder with each rung, adding more requirements and more value that really makes AI code review solutions shine. So the first part, it has to be, you know, not noisy.

So the first part, it has to be, you know, not noisy. It has to not spam me, not provide me with wrong comments or with a lot of noise. If it provides me with noise, it will get ignored. The second rung, it has to find problems and errors. If it's silent or if it's missing a lot of the things that I needed to find, again, I'm not getting a lot of value, so I'm looking at some completeness. I need to be able to rely on this to find issues.

Third rung is having multiple types of findings. Not everything is a bug. Some things are security issues. Some things are around maintainability or readability of the source code. Some things would be performance issues. So having, like, multifaceted code reviews, which can highlight different types of findings and hopefully categorize them so it's clear what it is finding. That is going to be the next level up. You want not just a single track or a single-minded code review that looks just for logical bugs, because the real problem is not just the logic. Because the real problem is never just one thing.

The next rung up is it has to know why. It has to be aware, at least with some awareness, of why am I trying to make this change? Why am I building? For example, can it look at the ticket, the JIRA ticket, or the issue that you're working on and compare the implementation with what the product requirements really are? Maybe it's a great implementation of the wrong thing. So having the ability to talk about scope, about did you actually implement what was needed? That's the next level up. It goes beyond just looking at the code. It's also understanding why we're doing this. Going up the ladder to the higher-up parts of AI code review solutions.

Not all teams work the same way. Not all code bases are the same. An AI code review tool has to know how the team works, has to be able to learn and conform to what the best practices are for that organization, for that team, so using guideline files, skills, other inputs that basically fine-tune it and customize it to become a team player and not just a generic thing that looks at code. It has to understand this is how we review and accept code in this code base in this team. It has to have some learning capability and get smarter, get better, so improving based on feedback it's getting from users, leveraging existing knowledge, looking at previous pull requests and comments.

The next level is, can I actually talk about is it making me more productive? So we talked about all the value it's giving me, but do I have a way to measure this? Do I have a measurable way baked into the solution that lets me understand this is actually helping me move faster? This is finding more things, improving quality, reducing cycle time, helping us move faster? And can I prove it? Can I show it? Because there's gut feeling and then there's data. And finally, can I use it to transform how we work? Can I use it to change the process, to automate some things away, skip humans where possible? It's a great building block for that. But am I able, do I have the surrounding support and tooling that lets me leverage the findings and the fact that I have the AI Code Review to really transform the way we work, the way we accept code? That would be the top of the ladder.

So by looking at all of these, this also hints at what makes a great AI Code Review solution and rollout. It has to touch all of these. And I'm going to give a few examples or show a little how we address this with Linux B tooling. But again, you can apply this evaluation ladder to any solution that you want to deploy out there and see where you have good matches for each of the stages. And maybe one question you should keep in mind and look at whenever you're rolling out the solution, do you have a way to prove the value you're getting? Because you're going to get asked if you're a team lead or someone in charge of rolling this out. This will be the first question.

This will be the first question. Can I prove the value that I'm getting from that tool? So looking at some examples, this is an example of like a ticker that shows the actual findings, the actual PRs that LinDB has scanned for. This is dog food. This is our own code. You can see the numbers and the types. We have bugs, performance, maintainability, different findings, different types. And it gives me at a glance an understanding of the action. What's actually happening with AI Code Review across my teams?

We talked about being able to measure and understand. So here are some examples of how to talk about measurements. I'm going to focus on just two or three of these. I can look at coverage. So how much of my new PRs are actually getting code reviews by AI? This depends on the way I rolled it out and whether it's enabled everywhere. You can see coverage is not always 100%. And this is maybe I started rolling out by having just half of my code base, maybe part of my repos, and then I want to push this up to have AI review all of my code and see this go to 100%.

If I'm looking at issues resolved, so a good AI review solution tracks the changes that are made after the comments to understand whether the comments and the issues raised were actually fixed by the developer. Did they commit or push another commit that fixed those problems? So you can see the number of potential issues identified and then the number of issues resolved. I can understand whether the comments and the issues that are found actually get addressed. And then there is the direct commits from the AI suggestions. So if the AI code review actually gave me a snippet to fix the problem and someone committed it directly from that snippet, we can track that as well. And this is like a great signal telling me this comment was directly accepted. AI has actually contributed codelines here through the review. That is a great signal of success.

Another interesting example is looking at scope issues. So remember when I mentioned AI has to be able to understand what you're trying to build. So here is a drill down looking at specific PRs across different repos where AI code review found a deviation between the implementation and the ticket. This really helps me understand how I can double-click and understand not just the overall numbers, but go into specific examples. Double-click all the way to a pull request to understand exactly where and how AI has helped me. Over time, you can see it's finding about 22 or 23 scope issues every week for the team.

This is another example of what the scope finding looks like, discussing handling users in our database, contributors, and so on. The implementation differs from the requirements, necessitating the developer to fix it before meeting the necessary criteria. AI code reviews serve as a signal or sensor for comprehending code quality. By reviewing incoming code, AI can indicate the density of various problem types like bugs, security issues in PRs created through different flows. This allows comparison between manually written code and AI-generated code, providing insight into code quality and potential issues.

AI code reviews offer an interesting angle beyond PR fixes by measuring incoming code quality multidimensionally. This approach allows evaluation of code quality metrics like review time, time to merge code, pull request size, and others. Comparing metrics between AI-reviewed and non-reviewed code areas aids in understanding the impact on cycle time, code quality improvements, and team efficiency. Demonstrating how a new solution enhances productivity through such comparisons is crucial. In summary, AI code review presents a significant opportunity, albeit with challenges. To maximize AI code review benefits, focusing on observability is essential.

I can look and say, am I getting good quality from my AI tools? Or maybe I have junior developers that are creating code that is not up to par and AI code review is telling me that. So there's another very interesting angle of using AI code review, not just to fix the PR but also to help me measure incoming code quality across the board and multidimensional. I'm going to skip a couple of these. I talked about proving the value and proving the impact.

Here's an example where we look at various engineering productivity metrics such as review time, how long does it take me to merge a piece of code, or the pull request size and others. And I can start comparing, compare the behavior or the metrics for when agency code review was applied to other places where it wasn't applied or rolled out yet. Now I can understand, is it helping me to make my cycle time shorter? Is it helping me with eventual code quality? Is it helping me move faster as a team, basically.

So remember the question is always, can I prove, can I show that this new solution that I rolled out, this new practice, actually helps me deliver more. And having those comparisons is a great step in that direction. To summarize, code review is a great opportunity for AI. It's almost like a task that is perfect for AI. But there's a bunch of challenges between you and getting AI code review to really, really help you move faster.