From Prompting to Orchestrating: Coding Is Now a System

Hi there, excited to be here. My name is Vladimir Novik and I architect AI systems in Scalable Cloud Platforms. Today I want to talk about how we transitioned from prompting to orchestrating and how coding is now a system. And I want to go through different studies and how we approach the systems and how we get better at them.

So we all started in the same place. AI was going to make us type less, ship faster, sort of auto-complete on steroids. And for a while, that's exactly how it felt. But if you try to build something real at scale with more than one agent in the loop, you felt sort of like a floor move. And the bottleneck stopped being how fast you write lines, it became how you coordinate the writing. AI didn't just speed up the coding, it quietly changed what the job is.

I want to think about that as like three errors and three leaps. And each step felt like a leap until you tried to build something real. So we climbed the ladder. First, we started with prompts. You describe, generate, then copilots, leaves in your editor, completing your thoughts, and then agents, obviously. They take a task, run editing files, running commands, opening PRs. Every run on the ladder or step on the ladder felt like a leap. Every step was real. Each one had a cliff right at the edge. And the moment you pushed past that one file, or even multiple files, one task, one session, things collapsed.

So let's talk about how things collapsed. And it started collapsing at memory, unprompting. Stateless, you become the context bus. Prompts don't remember, every session starts from zero, basically a copy-pasting state. Then we went to single agents. A single agent collapses context and self-reviews. So one long thread eventually loses the plot, and there is no one to check it. It creates its own homework, in a sense. So the obvious fix was to add more agents, except naive multi-agent collapses fastest. And it two-ways it once.

Chain steps together and errors don't only cancel, they compound, and it's multiplication. So don't quote me on the number, but if, let's say, we start with 95% reliable, then it can go down to like 36% and stuff. And the shape is measured. If you look at this line, we have a swarm, and p is the success rate, and n is the steps that are chained. And with the swarm of agents, we basically get to 36% of reliability. Starting from 95%. So 95%, 77%, 36%, and then we have the gated version, which I will talk about in a bit. How we do this gated, and that assumes a strong gain. Okay? Now gate also can fail, but this child assumes a strong gain. And these numbers come from a mass study out of Berkeley. It was hand-annotated 200 multi-agent traces, and about 79% of the failures are structural. There is a link to the article for that.

So we're talking about this being not a multi-agent, this is a swarm. So multi-agent isn't the problem, the swarm is. The swarm is n agents chattering peer-to-peer, each editing whatever it wants, nobody owning the truth, and that's what actually collapses. The striking part, different camps agree, for example, on tropic, on pushing more agents, cognition, arguing for fewer. But they agree on the fact that the fix isn't more conversation, it's orchestration. One owner of context, workers that run isolation, they report back. They don't talk to each other, and deterministic gaze deciding what's done. Let's call it orchestra.

Let's talk about numbers a little bit. So quick numbers. On the left side, here, we see the must, it's a UC Berkeley study, multi-agent failure taxonomy. That's the naming of the must. And hundreds over 200 real multi-agent runs and sorted why they failed. Both 79% are structural as I mentioned before, plus what they also call inter-agent misalignment, which is basically context loss or conflicts at handoff. That shows us that the numbers are not good, to say at least. Then on the right side, there is the upside. The cleanest number is this one, dubbed work.

The cleanest number is this one, dubbed work. It's a peer-reviewed and isolated exactly what here is claiming. Same model, just multiple orchestration topology, and you get 12-23% that's on coding and reasoning benchmarks. And that structure is paying off, not the model. So, two-in-three number point, the same way, entropic, plus 90% on research breadth. And then there is a Trinity, it's from Sakana's coordinator model. There is improvement there as well.

So there are different teams, right? Different benchmarks, but same shape, the owner of context, isolated workers, verified gate. So let's reframe it. So if orchestration is the answer, the real question is what does the system need? Not which tool, what property, but what properties, because the tool will change every six months, but the structure underneath them shouldn't. It was never about writing code line by line. It's more about designing a system that produces, validates, and evolves code. And I wrote down as a vendor, neutral spec, called it Orchestrator Coding. It's available on GitHub, you'll get the link in the end.

And the idea is we have three foundations, seven disciplines, and four conformance levels. So you can honestly say where you stand. So let's walk through it. You can check yourself against the system if you're working using this system, or you can just argue with me that it's wrong or something. So let's start with the foundations. Before any orchestration logic, three things have to be true or the whole thing is wrong. First of all, source of truth, the agents obey. It's like a law, it can be architecture, contracts, specs, everything mechanically enforced. Then we have foundation number two, shared context as a protocol. Durable state, everyone reads before, after every unit. And deterministic gates, which is very important. Gates that decide done without hallucinating. And these three things that are basically solving the breaking things, the memory, coordination, and self-grading. So let's talk about foundation one. Source of truth, treated as law. So agents will conform and not reinvent.

We need architecture, contracts, specs, version, explicit, and binding. An agent may never silently divulge. If the reality forces are changed, that change is deliberate and recorded. We need to write down an ADR, architecture decision record, in the same change set. So we can also review that, right? And code and architecture are never left disagreeing. And the most important thing, it's mechanically enforced.

We need links, types, schema conformance. It all defends itself. A doc an agent is told to follow is suggestion, and agents drift from suggestions. Link that fails the build is the law. The architecture defends itself. So 10 agents can't reinvent. If link fails, it fails, right?

Foundation number two, shared context as a protocol. So the answer to prompts don't remember. Each unit context is assembled from prior results, current state, as a source of truth, not smuggled into one giant prompt that dies with a session. So that's kind of glomorous and the single biggest cause of failure in Mast. About third trace of context lost in the handoff, right? So we read things, pass to unit of work, construct the context, and write after. Right? So prior results, current states, that's the source of truth. And we need it to be shared and not just one giant prompt.

Foundation three, the actual gate. So done is decided by checks that can't hallucinate. Mechanically can't. The test types, lints, pass and build. LM can weigh on taste, but nothing done on model say. So it's not like, okay, I marked this task down because the code is there. Right? It has to pass test, has to pass linting, has to pass conform to schema, that kind of stuff. But there is also like a trap, because if you just create lots of tests that are just green and nothing, that didn't actually check. It's worse. And it looks safe. So we need to make sure that our tests actually checking what they supposed to check.

Right? So the seven disciplines are the orchestration itself. And they are not equal. So take them, weigh the spec ranks them. And the first three are minimum that earns the game. Orchestration owns dispatch, isolation, durable handoffs. Then we have the dependency gap loop. We have independent review and merge gate. And then we have multiple harnesses or control loops, which are events and trolls, basically. And after that, it also comes sort of like isolation and multi harnesses sort of enhancement.

So let's talk about this like band one. The core. So D1 orchestrates their own dispatch. Single logical authority decides what runs when. Workers report back. They don't negotiate with each other. It's a consistency requirement, not a single point of error. Just know two things, writing coordination, state of the ones. D2 is the one that matters the most. We talk about isolation here. We put things in a branch, in a work tree, sandbox. Two agents never write the same surface at once. So we don't have these like merge hell and stuff. D3 handoffs as durable artifacts. Not just pass these to agent X, Y, and Z, but having a real deliverable artifact handoff file on disk that other agent can read. That kind of stuff. So if you ever do this tree, you've earned the name orchestrator. Orchestrator, otherwise, no.

Then we have second level is like governed. Now it's safe to land changes on its own. D4 is dependency or a gap loop.

It models dependencies. One dispatch unit whose prerequisites aren't met. So rechecking continuously releasing the moment the blocker clears. So for that, we sort of need the daemon to run and release. I will talk about this a little bit later. And unknown state is treated as blocked. So when in doubt, then fail safe.

Then we have D5 is an independent review plus a merge gate. Reviewer isn't the agent that wrote the code. It's sort of self-review is how single agent failed. It's a different agent that will review the work. So we also need to have human only hold for some scary stuff, right? Like off migrations, breaking changes, that kind of stuff.

And the other three is the resilient. So what survives production? What is absolutely autonomous? So multiple harnesses. Imagine you hit your code limits. Then automatically the system will switch to codex or vice versa. You can compose models. You start with one model and compose to another one. Between multiple harnesses, open code, codex, all of them. Then we also need the control loop. So PR open, review, check failed, recover, blockers, all release. Loop is an event system. Based on the events, you have different agents triggered, and it's constantly working, right? Nothing is dropped. So the difference between automation, you babysit, automation, that runs while you sleep. So the first level, how we get to like from L0 to L3. So L0 is assisted, like a single agent or pilot. No orchestration, human rights, merges everything. Orchestrator is the system that produces work in an organized way. Human still learns every change. Then governed, real jump.

It's the gate, not the person, becomes the merge authority, independent reviews, plus green checks, everything is landed automatically and working. And L3 is resilient. It lands, recovers, pause, asks, resumes, everything is automatic. And the latter is autonomy. So you do it, it produces, you merge. It merges itself as L2, merges and recovers. And then the pause, ask, resume, all of that doing automatically as L3.

And on the cell one, it's an aspirational three, because if you do L1 well, you can totally deliver very good code versus just shaky L3 will probably fail somewhere and will be very scary in production. So you can be the orchestrator. You don't need a demon of sorts. What matters are the disciplines. Who runs the loop, you're a demon, it's a separate thing. So manual, you conduct, you dispatch sub-agents.

You have Git work through per task. You report files as handoff, separate review or sub-agent or CI gate. And you reach an honest L1 with this. Then there is automation that buys you event-driven control loops and dependency and multi-harness and all of that. And so you can use the demon, you can use without the demon. So how would you do it by hand, L1 by hand? Get one skill, verify before done. That's pretty much enough.

So the skill basically saying nothing is done or passing without exact commands that output. A green run you can quote or isn't done. Meaning, if for example, you have a task, you tell Cloud to finalize that and it will mark it as finalized. With verify by done, you need to have it. It will sign that it's verified with this exact command. This is the output, that's the outcome.

And then we have a linear workflow skill. For example, when you pull things from linear, read comments, not just the body, branch, work tree. Use superpowers brainstorm to plan and implement or other, I just use superpowers brainstorm, but you can use others. Do the verification and report the evidence and don't commit. You commit yourself, that's L1. Some custom skills, some public ones you can do it tomorrow.

And then Tomb Raider's L2, without the daemon, you push the gate into a cloud and then basically you deny things. You deny git commands, you deny pushes, you stop post to use who runs. And you have like a separate reviewer in a fresh context. Meaning, either PR review skill or grab tile and call drive it. Usually use both. I use grab tile and I also have a separate PR review skill that will pull the PR and do all the review and do it like separately. A gap loop skill is a good one. What it does, it uses goal and it reruns until it finds all the gaps that are supposed to be there before fixing and then document that. And that basically you can create from that. Other tasks and you iterate on them and so on.

Then you connect linear MCP and then you have a tracker connection. So it's D1, pull issues, read the thread. Also, I build everything into one system, open source system. So basically it's like an L3 level harness. You have a bunch of issues within linear, you can filter them and then enlist. You have a bunch of agents that you can assign things to. You have a dependency graph where you can analyze the dependencies, get the dependency graph, notifications and stuff. But the most important part, we're talking about automation here and get into L3 level. Yeah, we have like timeline and I have like the login and that. But then there is these agent profiles and every profile, for example, these implementer, it has specific gates. What it can do, either comment or post and has its soul, it has its instruction. So every agent is different. For instance, I have a QA agent that can only comment and cannot push the code. And there are a bunch of agents that comes with the install that you can reuse, like automation observer or capability or different debating agents, that kind of stuff. And also you have automation. So for instance, I want to have when I have an issue here, there is this issue that is waiting for input. And it's waiting for me to reply here. So I can say, OK, I want to use this template to be an input responder. And that's my separate input responder agent. It will trigger on specific event. And it basically has this instruction template. You can add instructions and you can combine different filters to trigger this automation. And there are also a bunch of templates for this.

Where is the problem here? The first is obviously the cost. So fleet can burn an order of magnitude of more tokens than a single call. Chance that on big migration, but for three files change, I won't use it. Right? Then there is a parallelism that stops the shared surfaces. So if something is shared, I need to basically decompose that with the engineering effort or actually use a different agent to decompose stories to use them later on. And the gate is still a rubber stamp. About 70% of failures are green checks that didn't check.

Don't validate the gate and you inherit the blind spot. So you still need to be very precise with the gates. And the important thing about the numbers just used. Mass 200 traces and orchestration fixes. That's my production experience, plus the spec design intent. So it's not a controlled study. That's my thoughts from production experience.

So nobody has run these like L3 versus L0 on the same backlog. So it's an open spec and contribute to it, run the study, share the data, argue with me on that. That's the system that works for me. So I didn't invent this. The field keeps landing on the same shape. So it's the same shape again and again. One dispatcher, isolate workers, gate the decides done.

I build Hrevox on top of OpenAI Symphony. It's a spec from OpenAI. Tracker, isolate workspace per task, dispatch, PR on green. Then Sakana, Trinity conductor has the same ideas. And then there are a bunch of independent studies like Adapt-Org and CodeDelegator have the same ideas and everything converged into pretty much the same system that I created a spec from. So I'm not claiming I called it first or something. I wrote down what already works, what works for me. People keep arriving in the same shape from different starting points, which is nice.

That tells me the idea is worth it and load-bearing. And so I'll go with me on whether it's correct or not. So the shift is not from coding to prompting. And it's from coding to designing systems that try to validate and evolve code. And the model rides the lines. And your leverage is not the little code lines. It's the orchestration and the decomposition and the shared context and the gates.

And you can check the spec. You can check later walks, and you can start solo without a daemon. L1, be the conductor, reach the daemon when you need L3. Argue with the spec, add your system's implementation list, run the study, share the data. And there are a bunch of references to look at. Also links to specs and sources I used for different studies that I used. Where I took the numbers from. And thank you. Coding is now a system.

So go ahead and be the orchestrator of the system. Make better code. Hello, hello, welcome back. Hi, Vladimir. Hi there. How's it going? It's going great. Thanks a lot for your talk and your work. It's really incredible. And it's needed nowadays to get a full set of best practices and architecture, high-level views that we need all to follow in the current chaotic world.

I just want to say for our audience, please don't forget to join slido.com if you want to ask our speakers anything. If you want to ask Vladimir about his talk, for example. Just join slido.com. Use the number that we have today for AI Coding Summit, which is 0607. Yeah, and drop your questions there. I'm pretty sure we will be happy to answer that.

In the meantime, we already have a few questions in the backlog. But just to begin with, I have a question regarding, let's say, high-level architecture. And recently we have a lot of talks that are discussing what is the best practices to follow in the current world. And at some point, you see that there are some similarities to traditional architecture practices that we've been familiar with for the last decades. So I wonder, what is your take? What is the key difference and key similarities between modern AI-based architecture and the traditional one? What do you think?

Yeah, that's an amazing question. And actually, it very connects to one part of my talk about the verification gates. And let me elaborate a little bit about this. So when we talk about architectures here, I want to emphasize on AI coding architecture. Basically, our kind of habits of coding correctly. Right. So what unfortunately is happening these days, as we all drift towards using more and more AI tools, being more productive with AI tools, there are a bunch of best practices that were out there for years. Like end-to-end testing, unit testing, all of that. And like CI, proper CI CD, all of that. Proper PR review practice, right? Unfortunately, nowadays, when we run our agents without mechanically checking things, without having these verification gates, when the schema and types conform, when linting passes, unit tests passes, end-to-end tests passes, that's something that mechanically can be enforced. LLM can drift and we see that all the time. Models evolve, but then suddenly the quality drops and LLM can just hallucinate, right? But LLM will never hallucinate if we enforce it to run these verification gates. And I think the key point now is to still think of proper architecture and proper review practices. It can be hard because we're getting huge PRs driven by AI, but my take is, we still need to try to force things mechanically. Hope that answers.

Absolutely, absolutely, yeah. That's a great thought. And to continue this line of thought, we can also predict some tools appear, which summarize the review in some way so they can be done easily and further and further. But just to go on with the questions, and the last question regarding the LLM or particular model. Have you found any specific model that you use as a main orchestrator or any variant combo that works best for you? It's a hard question because it evolves all the time. I would say I use Opus, but then the other week I switched to Codex, then the other week I switched to Fable, then Fable was limited, switched back. It's kind of shifting all the time. But what I do think, as a best practice, you need to constantly evaluate the outputs of your models. And you can evaluate that by creating a local harness that will basically log your sort of lessons learned from you interacting with the agents. And based on that, you can evaluate if one harness works for you versus the other. But I usually do multi-harness. So that's the reason why I build IterVox, for example. So I occasionally have things written with the code, and then I review with Codex or vice versa. And then I let them argue one with each other and find kind of the best solution. And obviously review everything myself as well.

Absolutely, absolutely. I think you are not the only one who's doing it right now. That's a very nice technique indeed. There is another proposal by one of the listeners. Speaking about orchestrators, what do you think if we have layers of orchestrators? So would it benefit us in terms of building a better architecture, you think? Have you tried something like that already? Yeah, definitely. The whole idea is that we need the main one, but then we can obviously branch into multiple ones. For example, I'll give you the general example. If you create multiple agents, right? One of the agents is actually calling others, and this leaf in the tree is the orchestrator, but it repulses the main one. And then in that way, you have sort of a tree that reports back to the first one, to the original one. But the whole point is the whole handoff from orchestrator up to the tree cannot drift, both mechanically and you cannot communicate between each other and work on the same branch and on the same work trees and so on. So the pros, obviously, you separate the concerns, right? If, let's say, you have an agent which is in charge of architecture, you can separate that into backend and frontend, like an obvious separation, right? But then what you have in the end, if you're not aware about how much you put into the context, then you just burn tokens. So the cost is the obvious cons. If you have an infinite amount of money, you just don't care about that. But none of us have an infinite amount of money, so we need to be aware that as you introduce more and more orchestrators and agents and so on, that introduces more tokens cost. And we will see that at the end of the month.

Yeah, no one has it, except maybe a few engineers in Amazon, and that was a couple of months back. Next one would be about concrete tools and software, maybe that you covered a bit. What does a GraphTile and CodeRabbit brings versus AI CodeReview? Have you tried them? So basically, just the convenience, I guess. Like GraphTile and CodeRabbit, you just connect that to the repo, you just use that. As I mentioned in the talk, I also use my own PR review skill, which is because I have very granular control to that. I use it in addition to those tools. What I like specifically about GraphTile and CodeRabbit is that it has this as well, put the summary on your PR and you can basically configure it to conformance level, saying, okay, if it's like five out of five, then only then pass the PR. But then it's not like a magic wand, right? You still need to do your own PR reviews. I usually, in addition, add my own harness on top of that. But GraphTile is very configurable and CodeRabbit as well. It's also configurable.

Nice. Yeah, next one is very interesting. I have some additional questions to that. Have you considered connecting observability tooling into your orchestration pattern? Some specific agents to triage or even request human intervention to anything that has been done during the loop. My additional question to that would be what is actually used under the hood? Which protocol, I mean, for agents to communicate to each other? So yeah, good questions. So here, I want to dive deeper a little bit into EtherVox that I introduced. So the EtherVox under the hood, and to your question as well, under the hood, it's written in Go. It uses an event loop that basically constructs the codex prompts with proper construction prompts from different agents and whatever it pulls into the context. And specifically for observability tooling and agents there, so I showed the automations there and a bunch of agents. So one really interesting one that I have, and actually a production use case that I saw, like some feedback that I got. Let's say if you create a bunch of linear issues and you just create them, not as a product manager, but just saying, oh, fix that stuff. Something very vague. Obviously, it will require additional input from human. So I have specific automations and agents, they come with the template whenever you install the EtherVox. Then actually, there is a cron that runs and pulls linear issues. And when there are new issues, it will basically evaluate with other codex, you have like full configuration to that. It will evaluate if the issue has enough data and also can check the dependencies within the same workspace if it has dependency on the other issues. Based on that, it will ask you questions and kind of have like human in the loop pattern here. And also, observability-wise, there are also additional agents that can triage things, put them into to-do, then to-do picked up by the event loop, it goes into the queue.

There is like this whole system around that. The code base that I share, the EtherVox, it also has a website with a bunch of guides going through very details of how to do that. And I welcome everyone to open issues, also for the spec that I share, share your thoughts about that, argue with me, I'm totally open to discuss it further.

That's awesome. Yeah, the last one we have so far in the queue is, how do you mechanically enforce architecture? Well, let's start from the obvious types, that's like the obvious, right? But that's not just architecture, right? But architecture has multiple layers, every layer you have some sort of enforcement. If we're talking about, let's say, database, then we have like, we can write tests for the database, right? And if we have a certain architecture within the cloud, we can use infrastructure as code, like Terraform or CDK, and then can test that and enforce that as well.

We can take snapshots of the decisions, right? And we can diff them, that's another pattern. Another thing that is actually very useful is to, as you go through your coding process, or talking with agents nowadays, right? You always ask them to output visually within like mermaid syntax, then you have diagrams that you can evaluate easier, and also can actually make diffs between these diagrams and see if something has drifted. You have input on that. There are also a bunch of tools to check, a bunch of like copy-paste things like Knip, for instance, or a bunch of Rust libraries that are more low-level, but they will, instead of asking an agent to find, copy-paste the code and hope for the best, you can actually run these tools.