From Prompt Engineering to Loop Engineering

Thank you. I think it's my second public speaking ever. So I work as a software engineer at Google, I'm with GitNation since 2016, helping with conferences. At work, I work on AI data pipelines and AI developer workflows.

So today's lightning talk is about loop engineering. So everybody knows this quote from Boris Chorny and I was thinking about it a lot as well. I've had some thoughts that maybe prompt engineering is dead and new products like Biotropic are now developed with magical loops like CloudTag and so on. And what is that?

Apparently it was not. And loop engineering was basically designed to replace these moments where human judgment becomes a bottleneck. Models became super, super fast, cloud got fast mode, codecs got fast mode. They're coming back to you very quickly, waiting for the next decision, and you are basically being chained to a machine with the model waiting for the next prompt. So the solution to this is writing loops to automate decisions which a human is not necessarily needed, like writing tests or splitting your task into parts and so on.

So in order to start doing loops, we need to ask ourselves which decisions we can automate. These decisions are what to do next, how to split work, what context to load. There was research by Geoffrey Huntley, which is by the way very famous for inviting a ralph looping model, and he concluded that only first 146 000 tokens are meaningful for context, and then the model starts to drift and hallucinate after that.

So what can loops build? With loops we can obviously maintain the repository, do migrations, build small product features, we can even build full applications and languages from scratch, and I'll go through a couple of real-world cases. So first would be a programming language Kerst. It was developed using loop engineering. A loop was running in three months, and the result was a fully created language. It worked in a way. Also had a prom.md file, which was checking a fixed plan.md and executing it, updating plan for itself, and executing it again for the whole three months. In order to do that, the author pre-created a directory with specs, put basic information, how language should work, grammar, memory management, concurrency, how llvm should be structured for it, created an agents md with some basic guidance on if model learns something, how it should proceed, and as a result, after three months, he's got a fully working programming language with a funny jenzy syntax. A few seconds on the slide.

Important note about this loop. A language was developed by a system constantly updating fixed plan.md. It was comparing specs and source files, it was writing what's missing as a prioritized list, it was constantly picking what's most important and forming a new to-do list for the next iteration. So, an important takeaway from here would be that the repo state always tries to loop. A second research was actually ended up at the top of Hacker News for a while. Guys from Y Combinator created six projects overnight. Three of them were just translating code from one language to another, and some experiments just creating projects from spec files. They achieved their goals. I think the budget was $50,000. They spent $800 on this project. The QR codes for this research and deck will be shared later.

So what about it? After this obvious success in the morning, they've got repositories translated. Obviously Claude was good at that. They decided to refactor the code base in a loop. They ended up with a huge CL, a huge PR, which had 20,000 lines of code. And that loop ended up with creating 1.6 thousand to-do items, which were basically unverifiable. And next morning when they woke up to this PR, they checked it. The rumors were they said it was nice, but they ended up not merging it, because no human can verify that. And another takeaway was that actually the speed of a loop moves the bottleneck. So apparently we are creating loops to help us move on projects faster by eliminating the need of making some decisions. But in the end we ended up with how fast we can be to make certain decisions for a loop to proceed. And this PR was a good example on that.

So general conclusion from both cases. We should be having enough repetition to loops, but still for some cases there will be human judgment. So what we should automate? Obviously a task should be recurring, checkable, and bounded. So recurring, obviously there is no point of writing loops for something like one-off tasks. Checkable, obviously we don't want LLMs to be too creative. As first speaker said from Apollo, models became good at lying, so verification is a must now. And bounded, again example with PR showed that if we're creating loop for something, you should be able to do it in small verifiable steps. And now I think the biggest question with loop engineering is is it suitable for creating full-blown apps? Language example was good, but quite synthetic.

As a last part of this presentation, I want to guide you through creating a simple project yourself using loop engineering. Every project implemented using loop engineering has five moving parts: four .md files and a bash file. The .md files should include a spec, a plan, agent.md, and program.md. The spec file serves as a design doc guiding the desired end product, not created in a loop but with model assistance. The plan outlines the implementation steps created with the spec, acting as the core of the loop. Agent.md dictates project execution details, while program.md tracks and executes the loop.

When engaging in loop engineering, tests should be read-only to prevent LLMs from rewriting them. Tests added to the project should be created by a previous loop step. An essential component is loop.sh, responsible for executing the loop by reading files in their raw form. Scaling the system should consider maintaining test integrity and previous loop-generated tests. Having tests as read-only safeguards against unintended alterations by LLMs, ensuring the project's testing integrity.

We'll talk about that a little bit later. Program.md, which will be just tracking and execution of loop. Yeah, okay. Yeah, and loop which executes a loop. Okay, unfortunately we have one minute, so I'll just slide through spec, plan, agents, progress.md, which just tracks progress. Loop.sh, which executes a loop, by reading all the files in wild raw form. Yeah, important note about tests. When we are doing loop engineering, we should have tests as a read-only because LLMs tend to rewrite tests if they're allowed to do that. Yeah, so that should be read-only. If you're adding tests on the project, they should be created by a previous step of a loop, and when you are scaling the system... Sorry, I just ran out of time and got a little bit nervous.

So tests are read-only. If we need to create tests, we are creating them by a previous loop iteration. Then if we are expanding our project and we're thinking how can we improve our loop, we should be starting by adding a verifier agent which basically takes the previous step, verifies that it runs fine, makes a verdict if it pass or fail, and why if not. And if the verifier fails, there we have a human intervention to actually make a decision on how loops should proceed. Then if we want to add a system which creates new backlog tasks, we can add another agent which will be reading a full goal of a project like spec.md, and within a spec see how which more tasks can we add. And after that, if we still... If, for example, a project is involving complex tasks, not necessarily long-running tasks, we should think about adding a cron or modifying planMd to have more indication to other agents on how they should perform together on the code base.

So, should you use loops? Yes, when the work is repeated, checkable, bounded, and blocked mostly by user attention. And I encourage everyone to start with loops today. We just need to think a work which we are reprompting model constantly and manually. Decisions that follow the same rule and checks where we simply say pass or fail by just glancing on them. And, of course, using loops is only possible when we have a definition of done and good. If we don't know what we're building, if we don't know what's the definition of done for what we're building, then loop is not applicable to that. Thank you.