Autonomous AI Agents in Action With the Ralph Wiggum Method

♪♪ Hey, everyone. Today we're gonna be talking about the Ralf Wiggum method with autonomous AI agents. So I was already introduced, but yeah, my name's Eddie Vink. I'm from the Netherlands, and I work at a company called FrontValue.

So let me just quickly get a pulse on the audience. Who here has tried the Ralf Wiggum loop? Okay, that's great. So not many people have used it yet. That's good. So I want to start by talking about the basics of a Ralf loop. So why is it even called a Ralf loop? Well, it's a method that was coined by someone named Jeffrey Huntley, and it's basically because the concept is similar to Ralf Wiggum from The Simpsons, which is he's very stubborn and just keeps doing things until they work or don't. And that concept is kind of translated to Ralf loops as well.

So you just have consistent loops over and over until you get what you want, basically. That's the core idea. But at the core, that comes down to spec-driven development, at least in the context of AI coding agents. It's just telling your coding agent, hey, I have this spec here in the file or somewhere online, and you just tell it, hey, go read it. Go read maybe one of the tasks and go implement it, and that's spec-driven development in a nutshell.

And why do we even do that? Well, there's a problem with long AI chats, and that basically all comes down to context management. And if you've heard of this, you probably know, but AI agents, they don't really stay as smart as they do in the beginning of a context window the longer the context window grows. So you have these models, it's like a million-size context window, which by the end of like 700,000 tokens, it's not going to be as good as in the beginning. It basically gets a little dumber over time.

And that brings me to compacting. So you have this concept of compacting, which is basically summarizing a context window, which is not lossless. So eventually when your agent runs into a limitation on the context window size, it will try to summarize it so it can keep going, but that can lose some details that you had specified that might or might not be important. So that's why we have ralph loops, because ralph loops start with a fresh context window every time based on a specification. And this is the most basic version of a ralph loop. It's just a bash script that pushes the same prompt into Claude or another coding agent.

But in a nutshell, this is a more thorough implementation. So you have a loop that basically checks some specification file and maybe an implementation plan. And then you tell Claude, hey, go pick a task, implement it, do some feedback loops like running some tests or linting, make a commit, and then continue on with the next task. And then in the end, you can have a script that kind of looks like this, where you just pass in the prompt and it'll run in a loop with an additional stopping condition of like an iteration count so you don't completely go through your usage whenever you kick off that script.

And then you tell Claude, hey, go pick a task, implement it, do some feedback loops like running some tests or linting, make a commit, and then continue on with the next task. And then in the end, you can have a script that kind of looks like this, where you just pass in the prompt and it'll run in a loop with an additional stopping condition of like an iteration count so you don't completely go through your usage whenever you kick off that script. So let's basically look at the prompt, because that's kind of the magic sauce maybe. So it kind of looks like this. So you just tell it's really simple in essence. So you just tell it, hey, go read some specification and an implementation plan. I want you to decide whatever to work on next. So you tell Ralph, pick a task that seems the most important, because maybe you update your specification or implementation plan as you go and some tasks get mixed up or moved around, so you can make some smart decisions there. Check any feedback loops, and then update your progress in this implementation plan, and then make a commit of that feature. And this is also important, only work on one thing at a time, because, again, we do not want to get into the problems of big context windows.

Now you might be thinking optimistically, wow, these Ralph loops, they're amazing. They can do whatever I want. I don't have to code ever again. Or maybe you're a bit more pessimistic. But this is just a tool. This is not like the new way of working. I'm just showing you an idea, and it's up to you as an engineer to kind of decide, like, okay, this is where I can apply this, and this is another area where it might not be as useful. Again, so about feedback loops, they're also called back pressure, if you've heard that term. It basically just means telling the LLM to run some checks like tests, type checking or running a build or even writing new tests to ensure that it works, what it has made actually works, so it can kind of self-correct if it doesn't work yet. Because, of course, the point of this is to have an autonomous coding agent, and you don't want to have to babysit this, like, oh, no, it didn't work. Go fix it. The tests aren't passing. Why didn't you just run the test? No, you have to specify it, or otherwise it might not do it. And also another thing is permissions, because if you've ever used a coding agent, you will have probably noticed that they ask you, like, oh, can I run this command? Can I install this? Whatever. So what some people do is they run their agent with, like, dangerously skipped permissions as a flag, and, you know, what can go wrong?

I probably don't have to explain it, but there's, like, this post on Reddit that I saw of someone saying, like, oh, yeah, I ran this issue where, like, my coding agent was running, and it needed some more disk space to do the task, so the coding agent figured, well, I'm not done yet, so I'm just going to go clean up some files so I have some more space to work on, which is pretty bad. So a solution to that is called sandboxing, which is basically running your script in a container, and there's a Docker command that you can run in your RELFLOOP as well or some other approaches to that problem. Okay, so I'm kind of not on pace, so we're going to try to speed things up. So let's talk about creating your specification. So basically, in essence, you can just have a conversation with Claude or another coding agent to talk about your idea, and it will keep asking you. The important part here is that you ask it, hey, I want you to have a discussion with me and interview me until we have clarity on what we want to implement, and that will just keep asking you questions over and over until you have, like, a shared agreement, basically.

And in the end, you tell it, hey, I want you to write some specification files, and these can just be markdown files on your repo or an implementation plan, which is just a list of checkboxes, again, in a markdown file, or it can be something completely different. So how do we actually know that this works, and how do you actually know that your plan is good enough? Because, well, you know, how do you actually do that in practice? So what I would recommend you do is actually not there's an Anthropic plugin for RELF that I actually do not recommend you use, because it doesn't goes against the basic principle of a RELF loop, which is that it doesn't clear the context window, so it just runs into the compaction issue eventually, which is kind of bad. So what I recommend you do is to just use skills. And if you're unfamiliar with skills, skills are just repeatable instructions for your coding agents, and that basically comes down to a markdown file that you tell it to read, and there can be some instructions in that file. There are some tools around this approach as well called OpenSpec, which is kind of a nice, free tool that applies some structure to this way of working with some common skills to ideate on your ideas.

There's one called Explore, and then there are some other ones to take that exploration of an idea, turn it into a specification, and then some other skills actually go around that task and do the work for me. Then there's another set of skills that I really enjoy working with lately. Those are the skills by someone called Matt Pocock. It's a Git repository of just markdown files and has over 100,000 stars on GitHub. It's kind of wild. So these are the core skills that he has. So there's Grill with Docs, which basically just grills you on an idea. Then you have another skill that turns that into a specification, and another skill that turns that into GitHub issues, and then there's slash TDD, which you use to just take one of those issues and run a TDD agent to implement it. And then at the end, you get a list of tasks on GitHub that are ready for the agent to implement, which is kind of nice because it's contained instead of one big markdown file. But again, this is not set in stone. You can do whatever you want.

So let's talk about how to actually automate this. A problem with automation recently is that three weeks before my talk, Anthropic decided to, hey, programmatic use of plot code is going to change at least the pricing and how you can use it, which is kind of a shame because I really love working with their tools. But there are other solutions and other tools out there. One of them that I mentioned briefly was Docker sandboxes, which actually allow you to use other coding agents and other harnesses as well. So you can use something like Codex or Pi if you've used that before. And then there's another tool by Matt Pocock. It's called Sandcastle. Sandcastle is basically a TypeScript library that lets you run these AI coding agents in sandboxes that are triggered by a TypeScript script, which is pretty neat. And the reason that I love this is because it just gives you so much flexibility. The bare-bones version is so simple. You just have the run script. You pass it in a coding agent. You pass it in a specific sandbox. And this doesn't have to be Docker.

It can be something like Potman or your own solution, whatever you want to do. And then you just pass it in the prompt. And I actually recently just published an article on my own website on how to set it up with your Codex subscription, because the guys at OpenAI, they are fine with you just using your normal subscription when you work like this, which is kind of nice because it saves you some money.

So let's kind of look at a workflow example. Something that I did recently for a side project at my job while I was working on other things, I wanted to have Ralph do something in the background and see what it would come up with. So we had this discussion about like a tool that could help us write better documents in Confluence. And I already had an existing solution that was made by another Ralph loop, which was really hideous. I just focused on the functionality because that's what I asked it to do.

What I did is I invoked the grill with doc skill, which again just interviews me and keeps asking me a bunch of questions. And this is, well, I said interview, but it's more like a conversation because it's not like, okay, he asked you, or he, is Claudia he? I don't know. Anyway, or Codex in this case, but it asks you a bunch of questions and you can push back, you can steer it into a different direction because that's kind of, you know, on you, you're in charge. So if it asks you something that completely goes off rails, you can just steer it back into a different direction. And then after a while, after like 10, 20 questions, you get to some point where you're like, okay, I'm done. I think we're on some sort of shared understanding for the requirements, but I wanted to experiment with some UX prototypes before actually handing that back to a Ralph loop.

I think we're on some sort of shared understanding for the requirements, but I wanted to experiment with some UX prototypes before actually handing that back to a Ralph loop. So what I did is I used another skill called handoff, which is kind of like a smarter version of compact, where you can decide the direction you want to compact. And so I told it like, hey, I want to hand off this conversation to another coding agent, a fresh context window where I can use a prototype skill to work on some UX prototypes, which the whole point of this is to save or keep the context window in my grilling session smaller. So the agent is able to do some smarter work. And then I have the separate thread where I can do all the iteration that I want and then do another handoff back to the grilling session with the useful takeaways of the UX prototype. So that's what I did. I just told it like, hey, I just did a prototyping session. Here are the results. You know, include this and make use of it when we're going to be doing a specification and turn that into issues as well.

Then after that, I used the to PRD skill and the two issue skill, and that ended up with this list of issues on GitHub that I could run in my RALF loop. And that's exactly what I did. So in the end, you just have a script that's called sandcastle. You do NPM run sandcastle. And here you can see, okay, it ran twice because I told it, I only want to do a few iterations just so I can kind of check. You know, is it going in the right direction or is it going off rails? That's something that I like to do before I completely let it go and do something. And then on the bottom, you can see the logs as well. So you can kind of still follow along what it's doing. And of course, this is running in a sandbox environment. So even if it's not able to ask me permissions because I don't want it to do, I want it to run autonomously. It is able to just do things safely and bring the desired results back into my code base in the end.

And then when it's done, you just have a pull request with a bunch of commits made by RALF that says, like, okay, I implemented this specific issue of this PRD that we discussed. And this is just the example. So this is like the before. It's a very ugly application. And then I had the UX prototype and it ended up looking still not amazing, but it's like good enough for something that I was doing on the side while I was doing my core work at my clients. So I can take that back and like show them like, hey, I made this thing or RALF made this thing. What do you think? And, you know, let's talk about it. Is it useful or not?

So, of course, you can have it run a bunch of unit tests, which is pretty fast. But you can also do end to end tests, which is, of course, probably going to be more useful, at least in my experience, because if you let the agents write its own tests, it tends to sometimes just write some kind of garbage. That's not really testing the actual feature, but more like the implementation details, which you can kind of steer. But in my experience, the slow tests also are some pretty valuable. Of course, there are static analysis tools, your linting rules. But there's also maybe some other ones that you haven't heard of, like Knip.dev, which kind of helps you flag some unused code in your project, unused exports, unused dependencies that the AI agent might have added in the process while it was running, or some other stuff.

And then there's a new one that I recently learned about called Fallow, which is something that is even able to measure the complexity of the code that is produced by the agents. So if it has too many if else branches and it just like you look at the code and it's like, oh, what is this? This looks like garbage. Then you have like a static analysis option, feeding that kind of information in a more nicer way back to the agents. So go fix it, make it more readable. Because in the end, we're not vibe coding here. We want to produce some code that is actually production worthy, at least, or kind of, so you can read it instead of a bunch of slop that you're not going to read.

There's also some other tools that are pretty cool. So there's a Chrome DevTools MCP server that you could add to your agent that allows it to basically look at the output in your browser console or your network tab to kind of do some more advanced debugging. So you don't have to, you know, do that yourself. Oh, like, hey, Claude or Ralph, I checked out your work, but I get this massive error in my console. Now, instead of that being the workflow, it just gets fed into the process while you have it running in the background while you do some other stuff. And then there's, of course, the Playwright MCP, which you might have heard of. And another one is called the Agent Browser CLI by FurCell, which also kind of gives the AI agents the ability to use a browser and check some things in your project. Running out of time, so let me speed this up. And, of course, you can have agents also check the work in your workflow. So let's get to the conclusion. So what are some of the downsides? Because, again, this is not like the end all be all. So this approach obviously requires a lot of planning up front. You shouldn't make the task too big. Otherwise, it'll probably still run into issues. You need to have pretty good scope and specification of what should be done. And it's not always the fastest approach. And you can also pretty easily use up all your tokens in your subscription if you're not careful. But some of the upsides, it can actually run autonomously while you do other stuff.

But some of the upsides, it can actually run autonomously while you do other stuff. It's great, really, for exploring and proof of concept, things that you do on the side, maybe. It's great for tasks where you have the intuition, like, OK, the LLM will probably do this correctly. You could set it up to read your backlog for simple problems like that. And it follows your plan and it doesn't compact.

And if you're not someone that is always running into your subscription limits, a great thing is also that you will actually use your plan instead of only using 10-20% of what you're paying for. So that can also be a benefit. But, again, the key takeaway here is not me trying to sell you on this concept of RELF loops. But what I want you to really take away from this talk is, hey, start thinking about these new approaches, new methods of using these AI coding agents. It doesn't have to be a RELF loop. You can try experimenting with a bunch of tools like Sandcastle or not. But just try things and see what works for you.

Because in the end, if you have a bigger tool belt, then maybe you're able to leverage these tools better and just produce better results for your employer or client or whatever or your own projects. So, yeah, that's kind of the big takeaway. I'm not trying to sell you RELF, but it is another tool in your arsenal. Even if you don't want to run it in a fully automated way, you can still utilize the core principles and skills for tasks that would be more human-in-the-loop kind of work. So, again, try it if you want to. Chat with your LLM about the specifications. Definitely be sure to sandbox if you're using it. And use a bash loop, don't use the thing from Anthropic.

And then, as a final closing word, this is a tweet that I saw a while ago. You can outsource your thinking, but you cannot outsource your understanding. Which basically comes back to, you know, we're engineers. We're trying to do good work. So don't just trust the LLM output. You need to set up smart systems and think critically about how you're applying them to produce good results. Otherwise you're just going to end up with something that is completely vibe-coded, in essence.

That is the conclusion of my talk. Thank you, everyone. Thank you. Oh, yeah. A good feedback loop seems to be the key to autonomous agents. Sorry, can I just ask for people filtering out if you could do it quietly so that we can hear Eddie's answer. Thank you very much. Good feedback loops seem to be the key to autonomous agents, but unit tests can be useless if they're hallucinated or made up to pass broken code. How do you avoid it or how do you maintain a high bar of quality? Well, the key point kind of goes back to reviewing the work. So, if you're having something running in a loop, that can be tricky, but that kind of goes back to instructing the LLM and your agents.md or cloud.md.

Like, this is the way that I want you to write tests. Do not test a bunch of implementation details. So that kind of goes back to how you're using the tool. If you're not using any specification for what kind of tests that you like to write, then, yeah, you're probably going to run into some problems. But if you're using something like Sandcastle, like I mentioned, if you have the TypeScript script, you could just have your initial coding agent running and then have another coding agent. That is very narrow, specialized in reviewing, like, hey, are the tests that is produced actually good quality tests based on some criteria? So, you can kind of arrange it however you like, but yeah, that is a very valid concern and something that I ran into early when trying an approach like this. So that's a very good question.

So that's a very good question. Yeah, I've definitely encountered a lot of fake tests that say they're attesting the thing, but they're not. So, yeah, it was great advice. You shared a lot of tools with us and a lot of different things you'd recommend, but somebody has asked, what has been your most impactful skill.md so far? Definitely the Matt Pocock one. So, if I had to call out one, it's probably the grill with dogs skill, because that it's like if you've seen the grill me skill before online, which is a pretty big one. It's kind of that, but with some additional instructions to save a shared understanding or shared language in a context.md in your repository, which is really great, because it increases shared terminology that is then used by your slash TDD, for example, when it is implementing. So you have like, it's basically a way of adding kind of domain driven development or design to your coding agents. So you have like a shared phrasing for everything. That's really cool. I'm really looking forward to having a go at using that.

Do you use any CLI compression tools to reduce your token consumption in order to minimize costs? Not currently, no. I've seen the, what's it called, like the caveman skill, but I haven't really tried it yet, no. It can be hit and miss with the caveman skill. Yeah, I should have done that because before this talk, I had to upgrade my codec subscription to another higher tier, which is, maybe it would have saved me some money if I did use that. How do you convince leadership that the Ralph method is viable to use? By showcasing that it can produce some results. Of course, if you're going to be using that instead of you doing your normal work, then, you know, maybe there could be issues. But the example that I gave, for example, was doing something on the side that we kind of discussed as like a nice to have. You could take some sort of initiative that historically would have been deprioritized over the core work at your job, but now with these AI agents, you can just have them running in the background. And maybe you can achieve some good results and some nice internal tools that could be valuable to the company. I mean, you could then take those results to your leadership and show them like, hey, this is how I made it. This is what it is. And do you like it? Yes or no? And you go from there. Right. But, yeah, that's that's kind of my answer. Yeah. Great. Somebody has asked, Vincent has asked, what are the best use cases for the loop? Where would you use it? So after a while you of using these coding agents, you get sort of intuition of like this is something that I am expecting it to do right. And the best use cases for the loop are things where you have like if you would classify the type of tasks that you do with an LLM, it's like, OK, there are some tasks that are like human in the loop required. So you have to babysit it. You have to check the results.

Maybe it's like very, you know, taste dependent if it's something UX related, for example. That's why I did like the prototyping session in my grilling session that I talked about. That is something that I wouldn't do in the loop. But once you have something that is pretty crystal clear. So the UX prototype, for example, I included the code for that in the project as a reference for Ralph to like use in the proper implementation. So then at that point, I did trust it to do that work correctly. So it kind of goes into, you know, your intuition of like what kind of task can I do well, what not, and also how much effort you put into the planning of these types of tasks. Yeah, that's great.

I think we've got time for one final question. Even with external tools that use clause code, would you not find that they also hit the limits? I mean, you do run into limits, but I mean, you can go as hard and as fast as you like with these types of tools. So when I started looking into this, my reason was that I wasn't actually using my entire subscription, like 50 percent left by the end of the week. So I figured, well, OK, let me just try seeing what else I can do in the background. And before Anthropic made things like the slash goal skill or slash loop themselves, this was like the earliest version of that. And yeah, I found it pretty useful for like things that I want to do kind of, you know, while I'm working on my other main tasks in the background. Awesome.

And I love the idea as well that you mentioned about having things run outside of the hours in which you would do it yourself. So you've always got something to come back to when you do get back to it. Awesome. Thank you so much, Eddie. You will be able to ask Eddie some more questions in the deep dive section behind this room.