For Agents, By Agents: Building AI Tools That Maintain Themselves

Hi, everyone. My name is Rudhankariyam and today I'm going to talk about how I used AI-driven coding to build AI tools that used to build and maintain themselves. So, I built a developer tool for agents where I watched the agent fail, improve itself, added analytics so I could stop guessing why it is failing and have these loops running where the agents will report the issues, fix them, audit them, triage them, create the releases afterwards, and also verify the releases autonomously. Because I feel like in 2026, the developers are not just the only one using the CLIs or AI tools at the terminal, but the agents are using them as well and improving upon them by maintaining them as well.

So, a brief introduction about myself. I have been doing Apple platforms development for the past seven years and I also worked with Apple Cupertino with the Team Voice Control team, and previously I worked with Alex Sidebar, which was cursor for Xcode and later acquired by OpenAI. And currently I work at Rourke, where I manage the App Store publishing as well as the Google Play Store publishing. So, this January, it's been a long time, when I talk about AI months, so I created an open source project called App Store Connect CLI. The idea was that this project is something that I will use as an example throughout the talk and as a case study on how I approached AI-driven coding.

So, I wanted a way to automate the App Store publishing because if you have worked with the App Store Connect website, you will understand how difficult that is, and I wanted to have a cloud code or codex or any agent that you are working with to go through all the builds, the metadata, the screenshots, everything that the App Store supports should be in a CLI so that the agents can do the grant work for me. And I realized that it was possible because the App Store Connect has its own API, even though there are a lot of them, and this is the idea of giving you this context is that how I approach building the tools in three stages where it is for the agents, it is built with the agents using the coding agents and then used by the agents in the form of CLI and then now maintained by the agents now that we have automations with loops and scheduling.

So, I wrote this CLI in Go because I wanted an open source inspiration which is the GitHub CLI. GitHub CLI is something that we have all been using and it feels like the models are themselves trained to use this CLI very well and because the code was open source, if for any architecture, I could just point my agent to that repo and it was written pre-AI so I know that it is very well done as well and this is how I wanted the CLI to be that you drop the CLI in any coding terminal and the agent will be able to figure it out. And for that, I need to understand how the agent would also think when working with the CLI and this turned out to be very well for stress testing it the AI driven approach because it has 1300 endpoints and there is no way I'm going to manually type the commands myself or create them myself. So, this from the first day itself, it was an AI native project that was built by the agents designed to be used by them and then maintained by them as well because I feel like agents or agents or the large language models have a much better understanding of working with each other. It also changed how I would think about developer tools because earlier when I was designing a command, it would be for humans who would type that out but now it's like I would brainstorm with the coding agent first and then ask it, would it make sense to you? Would you know which command to call? And if the help description that we have given is enough or not, can you pass the output very well and does the error that it gives make sense for you to understand what to fix next? And I realized that the agent can tell me where the agent itself is being confused and me figuring it out for them. And that's where the four agents parts come in.

So, the only way I could cover this much surface area was relying on cloud agents and splitting the work. I'm not affiliated to any other products, but for this particular case, I use Cursus Cloud Agent because it offered me setting up an environment where I can have my App Store Connect API key, give it to it for the particular use case and also create a throwaway app for the exact purpose of giving it a verifiable loop where instead of just the agent spitting out code that looks good, I have to manually try it out. So in the agent or the environment itself, it will try to use my App Store Connect API key, get the live data based on the commands that it has to work on, create a verifiable, like verify the output and then also mention all of that in the PR that it creates later on. So, one agent would get each of the area, one on build, one on test flight, one on subscriptions, metadata screenshots, and each of them will have a current idea of how the shapes are built for the commands and it does not memorize it. It starts from scratch. So, it has an idea of the whole codebase and the whole CLI itself and then use the open API schema to implement the endpoints, test them, build the binary and also use it and we then a third party agent or some other agent would use it. And if the mocks were not enough, it would then live test using my API on the throwaway app. And then before going to sleep, I would run six, seven agents and then in the morning review the PRs. And because another thing I want to mention is that all the commands for the cloud agents was not written by me. I would have cursor itself, write the issues, split the work and then write the whole issue for me because it was much better than what I could have written myself, the whole issue description. And this CLI got popular in the iOS community and tens of thousands of developers using it and there are hundreds of, more than hundreds of contributors as well who are opening issues daily. And I started getting job opportunities and acquisition offer. And later it brought, acquired it where I work right now. And it made sense for them because they were using the CLI in their App Store publishing pipeline. But that changed the problem because I went, I am now in a place, I was in a place where I was, when from creating something to maintaining it and now I have the users, the contributors, all the edge cases that I could have not imagined out there. And I do not want to have it as my main quest anymore because I have a full-time job. So maintaining this CLI is a side quest now. And that's where the part on maintaining comes in. And so, and just to do some humble bragging, in the last 60 days since the acquisition, I have shipped 26 versions. It feels like an impostor saying I shipped, Codex has shipped and I have merged, Codex has merged 183 pull requests. And this is an output I would expect roughly from a small team. But my direct time that I invested on maintaining the CLI, especially all the cases and everything has been under one hour a week. And I realized my role kept shifting working on this. I went from writing the prompts, then becoming the reviewer of the agent code, then I became a manager of the agents and the rules I created for the agents.

And the rules changes where I talk about more on the maintenance part now. So for a while, I realized that the agents are doing most of the work. I was kind of a scheduler. So I would say, look at this PR, look at this issue, try it, do you think it makes sense, reproduce it, please fix, and decide if we need a release or not. And then run the release. And I realized that this is not an autonomous system at all. It is just that I'm manually doing a lot of the grunt work. And I felt like a cron job. And what I wanted was a loop that can the agents are smart enough to run on its own when required. And then update the issues of PRs. If it is a P0, fix it on spot and then create a new release. Otherwise, wait a whole day and do daily or every few days do a new release if it makes sense. So I wanted to have a more smarter system that I had earlier compared to what I had earlier. So now we are at a point where the automations are much better in the respective apps provided by the Frontier Labs. And I realized that I do not want to create my whole system around a custom solution. Just use whatever is there and just have a minimal scheduling around it. So what I would have is that instead of, because the CLI is much more manageable now, maybe less than 10 PRs or issues a day. So I would have rather managed them in bulk. So on my Mac mini on the side, I would have, I would run codex automations as well as cursor cloud automations that would have specific prompts and rely on specific skills that I have created over the months on working on this and understanding what would be the best approach on handling the automation. So each run would get a fresh isolated work tree, have a shared memory file where it talks about the previous run, what went wrong, and then I have a change log at the end of it so I can understand what happened at the end of the day. And one of the things that is what I have automations for is called a wall of apps where the people who have used the CLI would proudly showcase the apps that they have worked on.

And there's a command for it, but what I wanted to check was that the app exists, the icon that they submitted is the latest one or not, and there's no malicious code in any sense. So there's a small JSON entry in the CLI and one automation that runs half a day apart will check and then merge it, approve it, put an emoji that is relevant to the app itself, and then just merge it and approve it as well. So I was comfortable giving full autonomy here because the blast radius is small, the policy or the rules that I have is very clear, and the verification is deterministic. Either the app exists or it does not. Another one was on about issue hygiene. The idea is that it picks up some incomplete issues that are there, tries to reproduce them against the current CLI because my Mac Mini already has my credentials, so I'm already logged in, and then it will classify them by the priority and difficulty, label them. If it's a P0, it starts working on its own. When I say that it starts working on its own, it would also create a release so that thing is shipped ASAP to the user so that they can get unblocked. And I realized that I'm very grateful to my contributors because I feel like almost all of them use agents to create the issues or the PRs.

So I separated having a specific contributor guide for the agents so when my agents talk to the issues that their agent created, it's much nicer for them for both of the agents to talk well, and I understand that it will have the same content that it will want it and then work on it. So I created this very funny thing called as AAC snitch where the agents have a free space to rant about the work that did not work out. So if the agent is using the CLI and gets confused or the command feels like it should not be that way, the agent will snitch and create a perfect, beautiful issue where it says what happened, what was expected, and what could have been done better. And these issues then will be picked up by the scheduler to work on it.

The idea of this snitch was to get much better feedback from the users directly who are using it without relying on any analytics at that time. And it really worked out very well for me because just talking to people that it's not working did not help, but now I have the whole transcript of how the agent worked out and create an issue out of it that's much more actionable to work on. And another thing I realized is creating another feedback loop, which is with analytics. The reason I added analytics wasn't because I wanted invasive. I do not collect anything, raw arguments or the account ID, app ID. The idea was just to get the command level signal because a lot of the developers that I talked to in real life and on socials mentioned that cloud code would get confused working with the CLI, but Codex works very well.

And I had no idea how that is possible. I know that I have used cloud models to work on this, but for me, I really needed that raw data to understand what happened. And in the end, it turns out to be that cloud code when working with the model was making up the commands. It was taking some assumptions and then it wasn't using help as much as Codex was using. Codex was using triple times the help option on each of the commands that would use more tokens, but in the end, Codex was much better to work with because of such a less error rate. So what I realized that in this case, if instead of having the models or the commands that does not exist, we meet halfway.

So I would add aliases for the commands that cloud code hallucinated and I got analytics for it and then create those made of command shapes. And in the other cases, I tried to improve the command description and the example so that the next time the agent use that, it's much better. And based on these, I also created a daily automation. So this is a very nice one because it would create a PR based on the CLI's analytics every day, see what went wrong compared to how it was before, and then reproduce any issues that it thinks the agent.

In this case, I specifically used cloud models because I know that that is one data point, a very strong data point that where it's not doing well. So I know that if the other one is doing well, I would rather focus on the one that's not. So for these exact cases, I would use cloud models and with this, when this PR is created, the other scheduler that runs on the PR would look into it and try to reproduce it again. And when it's happy, it will merge it and then create a new release.

So every day the CLI is getting better because of the agent keep leaving these traces where the tool called or anything was confusing. I would do not want to call it a self-improving agents, but on a surface level, the idea is that it is improving every day based on the feedback I get from the agents and is maintained by agents. I'm tired of seeing the word agents again and again. And I also have specific skills for it. So every time there's a new release, I would tag it and then run a scheduled agent that would go through the new code base, the release that happened, all the commits that happened from the previous release.

And then cross verify if the skills files, there are I think a dozen of the skill files and go through them and update them automatically and then create a PR and then another agent would verify it again, just so I can be, I rather be safe than sorry. And then it does another release as well. So at the end, how do I trust any of these to merge it to something that is being used by tens of thousands of people? So the rule that I landed on is that the agent that builds the fix do not get to create its own work because we have realized that the worker agent that finished the task has much more confidence in its work than an independent other agent that has the fresh context. So there's a separate auditor agent that the idea is that it basically attacks what the work the worker agent did. It gets the issue, the repo policy, the exact commit, the diff, and then use code simplify, be ruthless about it, the changes that it made and then update the command doc, see everything is updated or not. And then check all the exit codes, check on my throwaway app, it didn't leak any secrets, anything. And then when it is happy, then it would go and directly merge that PR.

Because at this point, the code base is half a million lines of code. And all of it is agent written in the past few months. I relied on test driven development from day one, because I realized that I would want it to have much better test cases than not having any. And the test code actually is much more than the actual code itself. And I realized that I do not want to refactor it very often and just let these agents go rogue when it comes to refactoring and burning all my tokens. So what I would have is in the commands or the files that I have, I would just mention that focus on small bug fixes here and there. If there's a duplicated helper that it added, just refactor that or audit some commands that were handling it differently, you have much more one place to do it. Because for a CLI, it's not much hard to have a lot of code compared to let's say something that is an iOS app or an Android app or a web app.

One thing I've realized, if you look at the screenshots, I realized that this one day I woke up in the morning and saw the release notes at the same time as others were looking at it, because it had went through all my PRs that it worked on and then did a whole bunch, like in bulk, merged all the PRs and then it would create a new release for me after doing all the audit that I wanted it to do. And then I realized that we are at a point where I am very happy to have it maintain the whole process. And with that I have realized that this autonomous releases as much as it is cool to do and have it maintain itself, I realized I feel much more alienated to the product now in a sense that, as I mentioned, I do not want to see the release notes at the same time as others, I need to be more in the process. But for now, I am enjoying what I call the factory of agents.

So if I have to advise you, do not start from automating the whole release part, start with something that is just ask the agent that you're using and ask him what can I automate in the first place, where it can be some rules that can be created into a skill, something that you're asking it again and again, create it into a skill, and then run a scheduler around it or an automation, and then have different... Another agent that I have is on security, so one is for the triage, one is auditor, one is analytics improver, which goes the self-improving one, one is for the release runner and the public verify is where I have a security audit, which checks for the SHA, the CI labels to make sure that there's no way that there's a prompt injection that happened and that can be released into a public release that others will download as well. So in the end, I realized that having zero issues and zero PRs is something that I'm very happy with and it does not trigger my OCD at all. And my conclusion is that the humans should decide the direction of the product and write the initial rules and policies, but you do not have to be the scheduler or work on monotonous jobs that can be just taken by all these agents. And a lot of them can be now taken by the agents to the point where you can trust them to maintain your work as well. And with that, thank you for listening to my talk and you can find me on X at Rudrank. I think on most of the socials, you can find me by my first name, thanks to my unique SEO, the name that my parents has given me. And I share a lot of my experiments over there. Thank you. Thank you Rudrank for your insightful session. Let's have a conversation. I see that we have a couple of questions so far from our dear attendees and for sure we'll have more, because the topic is definitely resonating with many developers. I personally build quite a few developer tools, of course AI-powered. And I personally do my best to automate everything what's ever possible to automate. So I call this kind of meta AI, when AI helps to build and maintain AI. But let's take the first one. Alejandro is asking, how do you decide the model and effort for each task? So on a non-serious note, it depends on how much of usage I have remaining. So if I don't have a lot, I would just use a medium model. Otherwise, I would just go on extra high. But on a serious note, I have realized over the past few months of building this CLI, is that or any other tools that for codecs models, it's much better to use on low or high or medium for most of the tasks and go for high and extra high, only when that it cannot solve on the lower reasoning levels. Because the reason is that I feel like on the high and extra high levels, they kind of overthink. But on the other side for cloud models, I'd rather use the high reasoning and not medium and low because on low and medium, I feel like they take shortcuts. And I don't have eval to prove this, it's just from the experience of building these all of my open source tools over the past five months. And yeah, this is my idea. For most of the tasks, for cloud models, I would use high and for codecs models, I would use medium and or high based on how much token I have left. Yeah, thank you so much. That's quite concrete recommendations. And I personally hope that sooner or later, developers ourselves will be kind of freed from ever thinking about that because I feel that we, like vendors of the models and harnesses, should come up with just very smart routers, right? We don't need to think about, hey, like, which model of the week or which kind of reasoning case is to use, it should all happen automatically because we have, I mean, we can have quite powerful intelligence level layer just locally, right? So just based on your prompt, smart router should define that. I hope that this is the future. And let's take the next one, also from Alejandro.

By the way, thanks, folks, for sending your questions. Yeah, I really want to have this event as like bidirectional conversation. Do we have evals for guaranteeing the quality of the outputs? Yeah, maybe let's generalize. What's your approach on setting the bar and following the bar of quality in model responses? So the first thing is that the model should have given me the response that I want, that is the quality that I would expect without any errors. And previously, I did not have evals. But slowly, what I'm trying to build is based on the analytics, I'm trying to understand how these models perform and understand more on the, not on how well they perform, but more on how many errors that I'm getting. So, for example, in the CLI itself, the problem that I was facing was the model would not call the exact command that I would want and try to assume things. So that was the error that I was trying to chase, and that's how I came up with the conclusion that codecs models versus cloud models, cloud was a 2x, the error, giving 2x errors than what codecs had based on a thousandth or ten thousandths of output. So this is for me, I realized, okay, codecs models are much better, but because half of my users or developers use cloud, still use cloud, so I would have to improvise the CLI on this. And then on the basis of that, I had an automation that specifically focuses or asks cloud models to automate or create an automation every day to go through the analytics and then improve. So that is my eval set that I built.

Cool. So, also kind of meta AI loop, right? Yes. So AI follows and improves AI. And, I mean, my personal take on the same question is, if there is a chance to provide any deterministic check, evolve, you name it, go for that. I mean, AI-based evaluation is the last resort, right? Of course, in many cases, it only can be powered by AI, but if you can just plug in old gold deterministic algorithmic, if you wish, evaluation, it's the simplest option. And the same for approach, how I use AI coding tools. I try to instrument as much as possible deterministic gates, because that helps with saving tokens, cost money, time efforts and everything. Yes, and let's take the next one. That's very interesting, by the way, and also raises a big open question. Let me also generalize this a bit. Are there any tasks that you maybe tried to automate with AI, but failed or maybe even didn't even try to automate with AI? So where our judgment is still the only way to go forward. Any tasks, any experience like that? I think not. I cannot think of something on the CLI side, but at least on work, what I realized, because I cannot write code a lot on how I worked on the CLI, because a lot of money is involved. So, I work on the app store publishing, so any small mistake I make will affect thousands of developers in an hour, so I have to be very careful with that, and that's where the human judgment is very irreplaceable. So, I do have an automation where I would see the app store errors, and then run our RORC bot. So, I work at RORC, so I would have a bot that would run on the issues, but sometimes, even after giving it as much as context as possible, the code base, and even my local codex charts, I have found that it would not be able to pinpoint to the exact problem where I would want it to be. Maybe, right now, the latest GVT models are not at that point where I think maybe it's a skill issue on my side, where I have not been able to write enough skills, pun intended. But the main idea is that I feel like at this point, there are a lot of places where I tried automation, and especially this pipeline where it would take the errors that I get from the users, and try to automate it to a point where I can say confidently that this PR that has been generated is something I can autonomously merge into the production code base. So, I'm still a human in the loop right now.

Of course, judgment is one of the last points that really differentiates us humans from all kinds of machines. So, maybe judgment, based on experience, based on skills, based on years of working within a particular domain or with a particular product, is the most precious part of our work as humans today. So, that's my approach. I hear this word judgment more and more and more often today in context, how to stay relevant in software development, how to stay relevant with such a pressure from UI, from AI, of course. I think that actually you answered the last question that we had about the negative experience. You elaborated a bit what's not possible to really automate today.

But maybe, as a very last comment, any interesting cases of failures, of complete disaster, something that you can share with us? So, on the negative experience side, I have been very cautious of including the automation. So, I only started automations like two months ago, even though I've worked on this CLI for five months. Because in the first three months, I was very cautious, like, things can go wrong. But fortunately, it did not. And because we are in this era that you get a lot of slobbed PRs or issues, or even prompt injected issues or PRs, that's where I had a lot of gates.

I still invest a lot in the skills or the automation that, for an e-step, would guard them, so that I can sleep well at night. This is the idea. So, for this thing, I haven't had any negative experience where I had to roll back or revert any changes, but at work, I have to roll back a lot of things, because there's a lot of edge cases that come when you're working at a bigger scale. That models, even if they try with sub-agents and you try to have a lot of sub-agents working on different aspects, something can still be missed from an angle that you did not thought of. So, those are the things where I am very considerate about.