Autonomous AI Agents in Action With the Ralph Wiggum Method

Hey everyone, today we're going to be talking about the Ralph Wiggum method with autonomous AI agents. I was already introduced, but my name is Eddie Vink. I'm from the Netherlands and I work at a company called FrontValue.

Let me quickly get a pulse on the audience. Who here has tried the Ralph Wiggum loop? I want to start by talking about the basics of a Ralph loop. Why is it even called a Ralph loop? It's a method coined by someone named Geoffrey Huntley. The concept is similar to Ralph Wiggum from The Simpsons, which is that he's very stubborn and just keeps doing things until they work or don't. And that concept is translated to Ralph loops as well. So you just have consistent loops over and over until you get what you want. That's the core idea.

At the core, that comes down to spec-driven development. And spec-driven development, at least in the context of AI coding agents, is just telling your coding agent, hey, I have this spec here in the file or somewhere online. And you just tell it, hey, go read it. Go read maybe one of the tasks and go implement it. And that's spec-driven developments in a nutshell. And why do we even do that? Well, there's a problem with long AI chats. And that basically all comes down to context management. And if you've heard of this, you probably know, but AI agents, they don't really stay as smart as they do in the beginning of a context window, the longer the context window grows.

So you have these models, it's like a million size context window, which, by the end of like 700,000 tokens, it's not going to be as good as in the beginning. It basically gets a little dumber over time. And that brings me to compacting. So you have this concept of compacting, which is basically summarizing a context window, which is not lossless. So eventually, when your agent runs into a limitation on the context window size, it will try to summarize it so it can keep going. But that can lose some details that you had specified that might or might not be important.

So that's why we have Ralf loops, because Ralf loops start with a fresh context window every time based on a specification. And this is the most basic version of a Ralf loop. It's just a bash script that pushes the same prompt into Claude or another coding agent. And that is the most basic version that you can do. This would just run endlessly. But in a nutshell, this is a more thorough implementation. So you have a loop that basically checks some specification file and maybe an implementation plan. And then you tell Claude, hey, go pick a task, implement it, do some feedback loops like running some tests or linting, make a commit, and then continue on with the next task.

And then, in the end, you can have a script that kind of looks like this, where you just pass in the prompt. And it'll run in a loop with an additional stopping condition of an iteration count, so you don't completely go through your usage whenever you kick off that script. So let's basically look at the prompt, because that's kind of the magic sauce maybe. So it kind of looks like this. So you just tell, it's really simple in essence. So you just tell it, hey, go read some specification and an implementation plan. I want you to decide whatever to work on next. So you tell Ralph, pick a task that seems the most important, because maybe you update your specification or implementation plan as you go, and some tasks get mixed up or moved around. And so you can make some smart decisions there. Check any feedback loops, and then update your progress in this implementation plan, and then make a commit of that feature. And this is also important, only work on one thing at a time, because again, we do not want to get into the problems of big context windows.

Now, you might be thinking optimistically, wow, these Ralph loops, they're amazing. They can do whatever I want. I don't have to code ever again. Or maybe you're a bit more pessimistic. But this is just a tool. This is not like the new way of working. I'm just showing you an idea, and it's up to you as an engineer to kind of decide, like, okay, this is where I can apply this, and this is another area where it might not be as useful.

Again, so about feedback loops, they're also called back pressure, if you've heard that term. It basically just means telling the LLM to run some checks, like tests, type checking, or running a build, or even writing new tests to ensure that it works, what it has made actually works, so it can kind of self-correct if it doesn't work yet. Because of course the point of this is to have an autonomous coding agent, and you don't want to have to babysit this, like, oh, no, it didn't work, go fix it, the tests aren't passing, why didn't you just run the tests? You have to specify it, otherwise it might not do it.

And also, another thing is permissions, because if you've ever used a coding agent, you will have probably noticed that it asks you, like, can I run this command, can I install this, whatever. What some people do is they run their agent with like dangerously skipped permissions as a flag, and what can go wrong? I probably don't have to explain it, but there's like this post on Reddit that I saw of someone saying, like, oh, yeah, I ran into an issue where, like, my coding agent was running, and it needed some more disk space to do the task, so the coding agent figured, well, I'm not done yet, so I'm just going to go clean up some files so I have some more space to work on, which is pretty bad. So a solution to that is called sandboxing, which is basically running your script in a container, and there's a Docker command that you can run in your Relf loop as well, or some other approaches to that problem.

Okay. So I'm kind of not on pace, so we're going to try to speed things up. So let's talk about creating your specification. So basically, in essence, you can just have a conversation with Claude or another coding to talk about your idea, and it will keep at you the important part here is that you ask it, hey, I want you to have a discussion with me and interview me until we have clarity on what we want to implement, and that will just keep asking you questions over and over until you have, like, you know, a shared agreement, basically, and in the end, you tell it, hey, I want you to write some specification files, and these can just be Markdown files on your repo, or an implementation plan, which is just a list of checkboxes, again, in a Markdown file, or it can be something completely different.

So how do we actually know that this works, and how do you actually know that your plan is good enough, because, well, you know, how do you actually do that in practice? So what I would recommend you do is actually not there's an anthropic plugin for Ralph that I actually do not recommend you use, because it doesn't it kind of goes against the basic principle of a Ralph loop which is that it doesn't clear the context window, so it just runs into the compaction issue eventually, which is kind of bad, so what I recommend you do is to just use skills, and if you're unfamiliar with skills, skills are just repeatable instructions for your coding agents, and that basically comes down to a Markdown file that you tell it to read, and there can be some instructions in that file. There are some tools around this approach as well, called OpenSpec, which is kind of a nice free tool that applies some structure to this way of working with some common skills to ideate on your ideas, so there's one called Explore, and then there are some other ones to take that exploration of an idea, turn it into a specification, and then some other skills actually go around that task and do the work for me. Then there's another set of skills that I really enjoy working with lately, those are the skills by someone called Matt Pocock, it's a Git repository of just Markdown files and has over 100,000 stars on GitHub, it's kind of wild, so these are the core skills that he has, so there's GrillWithDocs, which basically just grills you on an idea, then you have another skill that turns that into a specification, and another skill that turns that into GitHub issues, and then there's SlashTDD, which you use to just take one of those issues and run a TDD agent to implement it. And then in the end, you get a list of tasks on GitHub that are ready for the agent to implement, which is kind of nice because it's contained instead of one big Markdown file, but again, this is not set in stone, you can do whatever you want.

So let's talk about how to actually automate this. A problem with automation recently is that three weeks before my talk, Anthropic decided to, hey, programmatic use of plot code is going to change at least the pricing and how you can use it, which is kind of a shame because I really love working with their tools. But there are other solutions and other tools out there, one of them that I mentioned briefly was Docker Sandboxes, which actually allow you to use other coding agents and other harnesses as well, so you can use something like CodeX or Py if you've used that before. And then there's another tool by Matt Pocok, it's called Sandcastle, and Sandcastle is basically a TypeScript library that lets you run these AI coding agents in sandboxes that are triggered by a TypeScript script, which is pretty neat. And the reason that I love this is because it just gives you so much flexibility. The bare bones version is so simple, you just have the run script, you pass it in a coding agent, you pass it in a specific sandbox, and this doesn't have to be Docker, it can be something like Potman or your own solution, whatever you want to do. And then you just pass it in the prompt. And I actually recently just published an article on my own website on how to set it up with your CodeX subscription, because the guys at OpenAI, they are fine with you just using your normal subscription when you work like this, which is kind of nice because it saves you some money. So let's kind of look at a workflow example of something that I did recently for a side project at my job while I was working on other things. I wanted to have Ralph do something in the background and see what it would come up with. So we had this discussion about a tool that could help us write better documents in Confluence. And I already had an existing solution that was made by another Ralph loop, which was really hideous. I just focused on the functionality because that's what I asked it to do. And I wanted to experiment with some UX prototypes and bring that back into another Ralph loop to iterate on and build. So what I did is I invoked the grill with Doc skill, which again, just interviews me, it keeps asking me a bunch of questions. And this is, well, I said interview, but it's more like a conversation because it's not like, okay, he asked you, he is, is Claude a he? I don't know. Anyway. Or codex in this case. But it's asked you a bunch of questions and you can push back, you can steer it into a different direction because that's kind of, you know, on you, you're in charge. So if it asks you something that completely goes off rails, you can just steer it back into a different direction. And then after a while, after like 10, 20 questions, you get to some point where you're like, okay, I'm done. I think we're on some sort of shared understanding for the requirements. But I wanted to experiment with some UX prototypes before actually handing that back to a Ralph loop. So what I did is I used another skill called handoff, which is kind of like a smarter version of compact where you can decide the direction you want to compact in. So I told it, like, hey, I want to hand off this conversation to another coding agent, a fresh context window, where I can use a prototype skill to work on some UX prototypes, which the whole point of this is to save or keep the context window in my grilling session smaller. So the agent is able to do some smarter work. And then I have the separate thread where I can do all the iteration that I want and then do another handoff back to the grilling session with the useful takeaways of the UX prototype. So that's what I did. I just told it, like, hey, I just did a prototyping session. Here are the results.

You know, include this and make use of it when we're going to be doing a specification and turn that into issues as well. So then after that, I used the two PRD skill and the two issues skill and that ended up with this list of issues on GitHub that I could run in my RALF loop. And that's exactly what I did. So in the end, you just have a script that's called Sandcastle. You do npm run Sandcastle and here you can see, okay, it ran twice because I told it I only want to do a few iterations just so I can kind of check, you know, is it going in the right direction or is it going off rails? That's something that I like to do before I completely let it go and do its own thing. And then on the bottom, you can see the logs as well. So you can kind of still follow along what it's doing. And of course, this is running in a sandboxed environment. So even if it's not able to ask me permissions because I don't want it to do, I want it to run autonomously, it is able to just do things safely and bring the desired results back into my codebase in the end.

Then when it's done, you just have a pull request with a bunch of commits made by RALF that says, okay, I implemented this specific issue of this PRD that we discussed. And this is just the example. So this is like the before. It's a very ugly application. And then I had the UX prototype and it ended up looking still not amazing, but it's good enough for something that I was doing on the side while I was doing my core work at my client. So I can take that back and show them, hey, I made this thing or RALF made this thing, what do you think? And let's talk about it. We already kind of went over this.

But I want to talk about some more advanced feedback loops to keep RALF producing quality code. So, of course, you can have it run a bunch of unit tests, which is pretty fast, but you can also do end-to-end tests, which is of course probably going to be more useful, at least in my experience, because if you let the agents write its own tests, it tends to sometimes just write some kind of garbage tests, not really testing the actual feature, but more like the implementation details, which you can kind of steer. But in my experience, the slow tests also are some pretty valuable. Of course, there's static analysis tools, so your linting rules, but there's also maybe some other ones that you haven't heard of, like knip.dev, which kind of helps you flag some unused code in your project, unused exports, unused dependencies that the AI agent might have added in the process while it was running, or some other stuff.

And then there's a new one that I recently learned about called Fallow, which is something that is even able to measure the complexity of the code that is produced by the agent. So if it has too many if-else branches, and you look at the code and it's like, oh, what is this? This looks like garbage. Then you have like a static analysis option feeding that kind of information in a more nicer way back to the agent. So go fix it, make it more readable. Because in the end, we're not vibe coding here. We want to produce some code that is actually production-worthy at least, or kind of, so you can read it instead of a bunch of slop that you're not going to read.

There's also some other tools that are pretty cool. So there's a Chrome DevTools MCP server that you could add to your agent that allows it to basically look at the output in your browser console or your network tab to kind of do some more advanced debugging. So you don't have to do that yourself. Like, hey, Claude or Ralph, I checked out your work, but I get this massive error in my console. Now, instead of that being the workflow, it just gets fed into the process while you have it running in the background while you do some other stuff. And then there's, of course, the Playwright MCP, which you might have heard of. And another one is called the Agent Browser CLI by Furcell, which also kind of gives the AI agents the ability to use a browser and check some things in your projects.

Running out of time, so let me speed this up. And of course, you can have agents also check the work in your workflow. So let's get to the conclusion. So what are some of the downsides? Because again, this is not like the end-all be-all. So this approach obviously requires a lot of planning up front, and you shouldn't make the tasks too big. Otherwise, it'll probably still run into issues. You need to have pretty good scope and specification of what should be done. And it's not always the fastest approach, and you can also pretty easily use up all your tokens in your subscription if you're not careful.

But some of the upsides, it can actually run autonomously while you do other stuff. It's great really for exploring and proof-of-concept things that you do on the side, maybe. It's great for tasks where you have the intuition, like, OK, the LLM will probably do this correctly. You could set it up to read your backlog for simple problems like that. And it follows your plan, and it doesn't compact. And if you're not someone that is always running into your subscription limits, then a great thing is also that you will actually use your plan instead of only using 10%, 20% of what you're paying for. So that can also be a benefit.

But again, the key takeaway here is not me trying to sell you on this concept of Ralph Loops, but what I want you to really take away from this talk is, hey, start thinking about these new approaches and new methods of using these AI coding agents. It doesn't have to be a Ralph Loop. You can try experimenting with a bunch of tools like Sandcastle or not, but just try things and see what works for you. Because in the end, if you have a bigger tool belt, then maybe you're able to leverage these tools better and just produce better results for your employer or client or whatever, or your own projects. So yeah, that's kind of the big takeaway. I'm not trying to sell you Ralph, but it is another tool in your arsenal. And even if you don't want to run it in a fully automated way, you can still utilize the core principles and skills for tasks that would be more human in the loop kind of work. So yeah.

So again, try it if you want to. Chat with your LLM about the specifications. Definitely be sure to sandbox if you're using it. And use a bash loop. Don't use the thing from Anthropic. And then as a final closing words, this is a tweet that I saw a while ago. You can outsource your thinking, but you cannot outsource your understanding. Which basically comes back to, we're engineers. We're trying to do good work. So don't just trust the LLM output. You need to set up smart systems and think critically about how you're applying them to produce good results. Otherwise, you're just going to end up with something that is completely vibe coded in essence. So yeah. That is the conclusion of my talk. Thank you everyone.

A good feedback loop seems to be the key to autonomous agents. Good feedback loops seem to be the key to autonomous agents, but unit tests can be useless if they're hallucinated or made up to pass broken code. How do you avoid it or how do you maintain a high bar of quality? The key point kind of goes back to reviewing the work. So if you're having something running in a loop, that can be tricky, but that kind of goes back to instructing the LLM in your agents.md or cloud.md. Like this is the way that I want you to write tests. Do not test a bunch of implementation details.

So that kind of goes back to how you're using the tool. If you're not using any specification for what kind of tests that you like to write, then you're probably going to run into some problems. But if you're using something like Sandcastle, like I mentioned, if you have the type script, you could just have your initial coding agent running and then have another coding agent like that is very narrow, specialized in reviewing like, hey, are the tests that this produced actually good quality tests based on some criteria? So you can kind of arrange it however you like. But yeah, that is a very valid concern and something that I ran into early when trying an approach like this. So that's a very good question.

I've definitely encountered a lot of like fake tests that say they're attesting the thing. They're not. You shared a lot of tools with us and a lot of different things you'd recommend. But somebody has asked what has been your most impactful skill.md so far? Definitely the Matt Pocock one. So if I had to call out one, it's probably the grill with dogs skill, because that it's like if you've seen the grill me skill before online, which is a pretty big one, it's kind of that. You can use some additional instructions to save a shared understanding or shared language in a context.md in your repository, which is really great because it increases shared terminology that is then used by your slash TDD, for example, when it is implementing. So you have like, it's basically a way of adding kind of domain driven development or design to your coding agents. So you have like a shared phrasing for everything. That's really cool. I'm really looking forward to having a go at using that.

You can use some additional instructions to save a shared understanding or shared language in a context.md in your repository, which is really great because it increases shared terminology that is then used by your slash TDD, for example, when it is implementing. So you have like, it's basically a way of adding kind of domain driven development or design to your coding agents. So you have like a shared phrasing for everything. That's really cool. I'm really looking forward to having a go at using that. Do you use any CLI compression tools to reduce your token consumption in order to minimize costs? Not currently, no. I've seen the, what's it called? The like the caveman skill, but I haven't really tried it yet. No, it can be hit and miss with the caveman skill. I should have done that because before this talk, I had to upgrade my codec subscription to another higher tier, which is maybe it would have saved me some money if I did use that.

How do you convince leadership that the Ralph method is viable to use? By showcasing that it can produce some results. Of course, if you're going to be using that instead of you doing your normal work, then maybe there could be issues. But the example that I gave, for example, was doing something on the side that we kind of discussed as like a nice to have. You could take some sort of initiative that historically would have been deprioritized over like the core work at your job, but now with these AI agents, you can just have them running in the background and maybe you can achieve some good results and some nice like internal tools that could be valuable to the company. I mean, you could then take those results to your leadership and show them like, hey, this is how I made it. This is what it is. And do you like it? Yes or no? You go from there, right? But yeah, that's kind of my answer. Yeah. Yeah, great. Somebody has asked, Vincent has asked, what are the best use cases for the loop? Where would you use it? So after a while of using these coding agents, you get sort of intuition of like, this is something that I am expecting it to do right. And the best use cases for the loop are things where so you have like, if you would classify the type of tasks that you do with an LLM is like, okay, there are some tasks that are like human in the loop required. So you have to babysit it, you have to check the results. Maybe it's like very, you know, taste dependent if it's something UX related, for example, that's why I did like the prototyping session in my grilling session that I talked about. That is something that I wouldn't do in the loop. But once you have something that is pretty crystal clear, so the UX prototype, for example, I included the code for that in the project as a reference for Ralph to like, use in the proper implementation. At that point, I did trust it to do that work correctly. So it kind of goes into, you know, your intuition of like, what kind of tasks can I do well, whatnot, and also how much effort you put into the planning of these types of tasks. Yeah.