AI Reviews AI – Closing the Loop in Agentic Development

OK, thanks for having me. On the next talk about closing the loop in the agent development, first I want to introduce myself. My name is Daniel, I'm from Germany, working as a consultant for a company called Thinktexture, and I'm also the MVP for web and developer technologies, since I'm talking a lot about Microsoft tech for my customers.

My focus was always how to get my colleagues and developer teams more productive. And so, of course, with the rise of AI, I'm also talking about this kind of topic. And first of all, I think we can all say that more and more code is written by AI. I think we don't need to discuss that. So it's just one more number of the Sonar survey, for example, that 42% of the code we are currently shipping is AI-generated. Personally, when working with clients, I would say it's more like 90% already. But yeah, that's those numbers you can find online.

And what we can also see is that we are merging more and more code into production. Maybe this is also one of those problems GitHub currently has, like they're constantly down because more and more code is shipped to their platforms. So we're shipping more code faster. Is it good or is it bad? This is the main question. And I want to give you a few of my learnings and perspectives, how you can solve those problems we have. So we can now ship more code, and we don't have time to review it. I think this is the current biggest bottleneck. And when you Google around, you'll find lots of studies and numbers.

So this is just one example of a company called Pharoes AI. So we also should take those numbers with a grain of salt because all of those companies with those numbers are also trying to sell you some products. But I still want to come back to those numbers because I think they are real when I'm taking a look into the customers I'm working with.

So we can see that, OK, we're shipping more code to production, but with a price. So in this case, in those numbers, we can see that, for example, the time for pull request reviews are skyrocketing. They are tremendously engaging. We can also see that 31% of code is now merged without a review.

Because, for example, one of my customers, they switched completely to agentic developing. So you can think if it's good or it's bad. But when I'm talking to developers, they told me that, OK, we cannot review the code. So we try to create smaller pull requests. And when the pull requests are small and the CI is green, then we merge it anyway. Because their code quality does not matter anymore. Pipeline is green, so we can ship it.

But we can also see, for example, the incidence per pull request are also increasing. This is a really, really big problem. So there's a number, a term called a code churn. Code churn means you're creating a pull request, you're merging something, and directly after merging, you need to touch the code you merged before. So you are adding code to your code base that maybe works, but then you realize, OK, it isn't working. Or the code was not good, so we need to touch it. And we need to change the line of code or fix a bug.

This is a really, really big problem. For example, you can also take a look into the latest Dora report from Google. So they're also releasing this report once per year. And you can also see that huge companies like Google are also struggling with this problem. For example, the last report, the number of incidents increased by 5% or 6%. So they are shipping now more features, but the stability of the products, the internal products we are using as a customer, are declining, going down to 5% or 6%.

So this is a really, really big issue. Not only I have with my smaller customers, but also big companies like Google. So how can we deal with these problems? So the main problem often is that the code looks plausible. So it looks like the code is working. Like, OK, we have a CI, we have tests, tests are green, build is green, the linter is green. So why should we review the code? It's working. When we're clicking through the UI, it's maybe also working. But the problem is those hidden issues. Again, one example from my clients. Maybe you know this tool called Sonar. So I also showed you some numbers from them called Sonar Cube or Sonar Cloud. It's a software you can use for static code analysis to analyze, for example, if you're introducing duplicated lines of code.

So I also worked with a customer. They wanted to introduce AI-assisted coding to their team. So they had like 50 developers. And then we spinned up Sonar Cloud, and the existing code base had already 35% of duplicated lines of code. So it's a really big problem. Now think about you adding more AI-generated code to an already related code base. So it's a really, really big problem. And they have a lot of unit tests, like it was at around 80% of coverage. But when you're taking a look in their backlog, it was really filled up with bugs. So a really, really big problem.

So we need or we still need quality gates. This is the biggest problem I can see when talking to developers that are, I would say they know how to use cloud code or any other harness. It does not really matter. But most of those developers are not thinking about quality gates. So I want to give you a few examples how to solve those issues. First, there are deterministic ways to solve these problems and probabilistic ways to do it. First of all, you should always start with deterministic solutions. So tests, linters, typechecks, hooks, and more. I will come back to that afterwards. And on the other side, we still can use, or in some cases, AI to review our code with the help of different models or tools.

So the layer of defense, it's really small. So when developers talk to me, and because I talk a lot on conferences about AI-assisted coding, they always think that my setup might be really complicated. I should use a lot of sub-agents and different kinds of tools and harnesses. No, my setup itself is really, really simple. So first on the left side, because I also talk a lot about Spectrum development, is I'm as a developer still writing the spec. I'm not writing the code anymore, but I'm writing the spec. So this is, of course, my first challenge. Then I'm handing over the spec to any kind of harness. First of all, when I start my own personal projects or when joining developer teams, I first set up some deterministic gates. You can assume, I would say I'm a developer now since 10 years, and I've worked with, I don't know, 100, 200 customers, and there was, I don't know, I would say 5% of my customers used linters and formatters correctly in their code base.

So for me personally, when I started developing with Angular and TypeScript, so 10 years ago, then there was, for example, the strict mode introduced. I don't know when it was, but a lot of my customers still don't use the strict mode, for example. So first things first, before you spin up any harness, get your basics done, define the linter, define code formatter, start the strict mode, whatsoever. This is the first step. This is really important. After that, then I try or start adding AI sub-agents into my loop. More to that also later. So this is the second stage.

And then the third stage, I also want to talk about it later, are some pull request review services. So there are some tools you can use. And after that, there's still a human review needed. In some cases, maybe not. But at the end of this loop you can see it starts with my personal input and it also ends with my personal input. So this is my personal preferred loop.

First of all, one of those features when working with harnesses are hooks. Hooks are so simple. They are really simple. But again, when I'm talking with my clients, I don't know, nobody of them are using it. The cool thing with hooks in any kind of harness, in this case it's Cloud Code but it doesn't really matter, is that they are deterministic. So it does not make sense, for example, one of my favorite examples is that someone is writing something like in their Agents MD, please after each task run the unit tests or please do a code review. This is complete nonsense. It's non-deterministic.

You should not write something like this into your Agents MD or please don't do any kind of commits, please don't delete any line of code. This does not really work. You need a deterministic way to handle these cases. And hooks are the simplest way. There are different kinds of hooks. This is one example. This is, for example, the post tool hook.

So this hook is always called, in this case, again by Cloud Code, when Cloud Code calls a tool and then you can write a measure. For example, in this case I'm filtering the edit and the write tool. So anytime Cloud Code touched a file and edit the file, the hook will execute and then you can run any kind of scripts. So in this case, so it's fitting on the screen here, I'm running ESLint, for example.

A cool thing here is that the hook can then return the result to the agent again as a prompt. Like, okay, the function you added is introducing some lint issues or some typing issues or the application is not compiling or some unit tests are not running. This gives the agent direct feedback. So the agent does not wait until the task is completed and then it tries to fix something, but while writing code, it fixes something for you. Hooks are awesome.

Hooks are awesome. There are more hooks like, for example, there's one example of the stop hook. In this case, I'm running a TypeScript compiler, but you can also spin up sub-agents as a hook. So you can, at the end, define a stop hook and a stop hook is then, for example, spinning up your code review agent you defined. And then automatically you don't need to prompt something, you don't need to hope that your harness is calling the sub-agent. The agent will always run. Really, really simple. So you should start adding hooks.

Also, one thing to note. Also, there's a hook, for example, the pre-tool use hook. You can use it to make sure your agent in the auto mode or YOLO mode is not deleting something for you. So you should definitely try it out. Really important. Also, part of layer one is test-driven and behavior-driven development. There's also, for example, when I'm hanging out on LinkedIn, there are so many posts about, oh, I use cloud code to create a coverage rate of 99% by generating unit tests for my existing code base. Yeah, congratulations.

This is complete nonsense. This makes no sense at all because the problem is like you have a function with a bug. You ask cloud code, please write a unit test. And then the unit test just confirms the existing code. And then you have a unit test that is always green and not helping you to find a bug or to fix bugs. So don't use AI this way. I was not a huge fan of test-driven development when I started software developing, but I was always a big fan of behavior-driven development because the good thing of behavior-driven development is that, for example, those acceptance criterias in those files can be written by also non-technical persons in your team.

So we are moving from software engineers more to maybe product developers. Writing code is less important now with more team members joining the software development pipeline. Behavior-driven development becomes crucial to ensure file readability and maintenance by anyone, regardless of coding expertise. The focus shifts from code aesthetics to functionality, aligning with defined specifications upfront. This shift is essential given the increasing volume of code added without sufficient review time.

For front-end developers, tools like PlayWrite simplify setting up workflows for behavior-driven and test-driven development. Integration with platforms like Jira streamlines the process of generating tests from user story requirements. The review agent, utilizing AI for code review, presents challenges with noise generation and simultaneous code generation and review sessions, emphasizing the need for separate workflows. Avoiding these pitfalls is crucial for an effective development process.

Considering the transition towards product development and the importance of behavior-driven and test-driven development, ensuring code functionality aligns with defined specifications becomes paramount. Tools like PlayWrite and integration with platforms like Jira facilitate the generation of tests from user story requirements. However, challenges arise with review agents using AI, necessitating careful management to avoid noise generation and overlapping code generation and review sessions.

So you should definitely think about behavior-driven and test-driven development. For front-end developers, for example, I'm using PlayWrite all the time. So with PlayWrite, it's really simple to set up such workflows. There are multiple libraries out there. It doesn't matter if it's PlayWrite, VTest, or anything like that. And what's also cool, you can combine it then with platforms like Jira, Atlassian, Tools, it does not matter.

So your product owner, for example, defines your user stories, acceptance criterias, and then your agent, if you like to use an agent to generate those files, can pull the requirements and generates your tests out of those. So really simple. And the layer two is now the review agent in the loop. So you can use AI to review AI. But there's one problem. Those agents are adding often too much noise. This is one problem. And also a lot of developers using, for example, they're on the same session, so they're using Cloud Code to generate code. And in the same session, they are starting their review agent. So in the same context window. So this is also a big problem. So you should never do that. So when you're reviewing your code, make sure you always start a new session with a clean context. So this is the first part. It's really important. I will come back to that later after this slide.

But you can also use some services. Just one example, GitHub Copilot, GrabTile, CodeRabbit. There are so many tools out there. I'm a fan of those tools because they are kind of cheap and they're working really good. But my main issue still with those tools is the noise. For example, GrabTile adds a lot of code comments. This is maybe not a problem because there's a reason why there are some comments. But for example, I often have this problem.

Okay, there's a review. I change the line of code, rerun the review, and then the agent is telling me the complete difference. Okay, now I'll change the code back to the previous one. This is a bit frustrating. And there's one learning with that. So who reviews the reviewer? Is that you should not use the same model. There are some studies you can also find and read through if you like to. And you can see that there's something called rubber ducking.

That's also on the next slide. It's called rubber ducking. With rubber ducking... What's the issue? I did nothing. Okay, sorry. Okay, now it's back. Really simple. Instead of using SONET to review SONET, OPUS reviewing SONET, you're using SONET to generate code and then you use any GPT model as one example to judge and review the output. You can also use it for when you generate a plan, for example. You can use it for code reviewing. It does not really matter.

There's also another benefit. When you combine two cheaper models to judge each other, you come also closer to the more expensive models. You will also find some numbers and studies out there. So you can also use it not to only make sure the code is better or the requirements are better defined, but you can also use it to save some tokens because tokens are getting more and more expensive. But at the end, you can see there's still me, as a human, reviewing the code. Okay, so coming to an end, the takeaways. Self-correcting loops you still control. Sorry. So reviewing is a bottleneck. Of course, there are more requirements. It's also a bottleneck.

But when working with my customers, the code reviewing part is currently the biggest bottleneck. And you should always start with deterministic tools and loops. And for example, using hooks. This is really important. And also, you should think about how you're generating tests. So please don't use AI to generate tests of existing code. This is complete nonsense. Use it to write test-driven development tests or behavior-driven development tests.

And if you like to use AI to review AI, try to combine it with different models. So don't use the same model. Use different models. And at the end, still, you should be in the loop. Maybe this will change in the future. But currently, I still want to understand what the code is doing. Maybe not all details. But I still want to make sure that if someone is asking me, what is the code doing, I'm still able to explain it.

That's it for my side. Here's the QR code. You can scan it to find the slides. You can also follow me on LinkedIn if you like to. So I give you a few more seconds to scan the QR code. So feel free to connect. And afterwards, it's time for your questions. Fantastic. Thank you so much, Daniel. Round of applause for Daniel. Amazing. So please keep your questions coming on Slido. The code is, again, 0607. I'll give you a few more seconds.

One, two. Is it working? Okay, great. So I can also switch over. Fantastic. Thank you so much for that again, Daniel. So the first question here. You mentioned that for test, one should use deterministic processes first before LLMs. Is this the mentality we should apply to other tasks or problems? Are there cases where AI first is more beneficial? Yeah, so this is my personal preference and learning that you should always start or should use deterministic ways to review. Yeah, because it's cheap. It's also cheap. It's fast. It's reliable. Because my personal experience is with a lot of developer teams is that they are now trying to add AI to everything. But I don't think you need AI for everything. So start with the basics. A lot of teams don't have the basics. It is also a good thing because a lot of C-levels are coming to me and asking, okay, we want to enroll now AI to our team. And then you will find all those problems from the past they need to fix. So it's also a good thing to start with the old things first. That's amazing. Awesome.

So with hooks, another question. We run the lintar type check-in after every file, which seems a lot compared to running it once after some files have been touched. Does cloud get the feedback from the hooks too? Yeah, sure. So of course, you should not run maybe everything on all file change and edit. So you don't want to run, for example, the build or compiler each time. This will be too slow. And what's the second part? Does cloud get the feedback from the hooks too? Yes, this is the good thing that you can say, okay, the hook run through without finding an issue, then you are not reporting anything back. But if you find a compile error, lint error, then you're sending the feedback back to the agent as an input prompt, and then the agent can work on this. Okay.

Amazing. Thank you so much. The next question we have, keep your questions coming, by the way. They're streaming online as well. The next question we have is, in your opinion, which models are better for reviewing the code? I can give you an answer for what is maybe the best model. So from my perspective, of course, all those new models are really good. So as I showed you or talked about with this rapid tracking, for me, it makes more sense to use different models. So personally, I use cloud code a lot to generate code. But in my code review pipeline, I'm using the GPT models. Just my personal preference, it could be vice versa, it does not really matter from my perspective currently. Okay, that makes sense. Awesome. The next one, if we're using AI reviews at PR time, if you could rub it, et cetera, you mentioned this was L3. What do you see in the next stage of the evolution, basically? That's a good question. So my personal perspective is that there might be a point, maybe in the next couple of months, I don't know, that code quality is so good we don't need to review it. This only works when your architecture is well defined, so you have to think about your skills and everything about that. And then you still need tests. So this is why I'm enforcing my customers to start finally writing end-to-end tests and integration tests to make sure that nothing breaks, your code is working, your stability stays high, your speed stays high of your application, so then you can ship code without taking a closer look because you still have these observability tools to make sure everything is still working. So I think this is important, but not yet. Amazing. So the next one we have is, what do you think about having the LLM find and write missing unit tests with human reviewing as a way to highlight unexpected edge cases? Yeah, I think this is fine. But then, of course, you still need to think about the code. So you should take a look into code and then you can judge the unit test the LLM wants you to write. I think this is fine, but you should not use it blindly, just generating tests for having a higher coverage rate. Awesome. Fantastic. We have more questions coming in. So the next one is, can you explain more on behavior-driven development? Yeah, so it's really simple. So, for example, we can use BDD. Maybe I can open it.

I have a repo locally, but I'm not sure if I can show something from my customers. So where is it? Maybe we can find something like this. So this is a behavior you can define. So it's human-readable. So we have a feature, then you have scenarios, and then you always have this given-when-then syntax. And based on this text, tests are generated. So not the full test, of course, but the inputs and the outputs a test should have.

So the arrange part and the assert part is generated. And the good thing is that, as I said in my talk, anyone can read this text, you as a developer, but also your product owner or someone from the support can read those texts. And this is the benefit out of it. And then, of course, the test still needs to be written by you or by the AI later. But yeah, that's it.

This is a very interesting one. So in the end, who to blame if code review means a bug, which causes an outage? Is it the developer or maybe someone else? I would say still the developer, of course. There is something called extreme ownership, for example, so you should be the owner. So this is why I also say I still want to understand what I'm shipping. So maybe I cannot explain everything in detail, but I still want to be able to explain what the code is doing I'm shipping.

I want to understand the APIs I'm using. So if something breaks, it's still my fault, of course, at the end. So my personal ownership. So yeah. Fantastic. Thank you. Next question on metrics. Which metrics would you advise to use to measure if the approach is working? Good question, because it's not easy which KPIs we can use to measure. So maybe you should take a look into Dora, for example, or Space or DevX. With those metrics, you can, for example, measure the stability of your software. Because numbers like pull requests, lines of code, they make no sense.

With those metrics, you can, for example, measure the stability of your software. Because numbers like pull requests, lines of code, they make no sense. Maybe you can measure how often you release something, but I'm also not a big fan of those numbers, but I like to measure the stability performance of my application. And you can also measure, for example, a customer or also developer's satisfaction to make sure if they are happy with the end results. Those kind of numbers are making sense to me.

Amazing. We have another one. We have a couple more questions. So keep them coming. So do you find that us being swarmed with AI-generated PLs, as I like to add, our developers don't follow good practices? Adding compressive tests for your features? I need to check your work before putting it into a PL. What do you think?

Let me think about it shortly. So first, often this is a problem for open-source projects, so I'm not maintaining them anymore. But there's often still a problem that many open-source repositories are closing pull requests because we get overwhelmed by those pull requests. So, yeah, again, then you should define some quality gates as a team before. So you should define your definition of ready. When is something ready for review? Those quality gates should be defined by the team are really important. So someone wants to ship something, but then you need to enforce them to make sure to think about before shipping something like that. Amazing. Thank you so much.

The next one we have is, can AI reviews identify contract drifts? Yeah, it's a good one, because often those code review tools often only taking a look into your code, but not taking a look, for example, in your G-Route ticket, something like that. So this is why I like Spectrum development. I'm not a fan of the tools for Spectrum development, but I like it that you have a single source of truth. And then you can combine it. For example, you can build your own review agent using GitHub Copilot or whatever, it does not really matter. And just connect it using an MCP tool to your Atlassian products and also the requirements into the code review process. This is also important.

But again, coming back to behavior-driven development, thanks to BDD, the requirements from the ticket are also translated into tests. So you can take a look into your tests in the behavior-driven or in this feature file to make sure that there is no drift. Amazing. Sounds good. We have two minutes, so we're just going to take maybe one question.

So this one is interesting. Are there any frameworks that basically scaffold what you've talked about today and put the process together? Because I'm a front-end developer, I work a lot with, for example, for GitHooks I'm using Lefthook instead of Husky. So Lefthook is a more lightweight solution. ESLint, Prettier, Sonar, as I said. Sonar is really cool because you can host it, so you don't need to pay for the cloud solution. You can run your own Docker container and then you can also install, for example, when you're still using IDEs like Visual Studio Code, you can sync the Sonar rules from the server with the plugin in your IDE, so you as a developer directly see, okay, if I'm missing some of the rules we defined as a team, for example.

So nothing spectacular, so no fancy tools. I like to keep my setup clean. So no one tool to rule them all. Yeah, correct. Amazing. Yeah, we are on time. Thank you so much, Daniel. Round of applause for Daniel. Thank you.