Building Durable Workflows From Scratch in JavaScript

Hey, I'm Peter, and today I want to talk to you about how you can build durable workflows from scratch in pure JavaScript. So all of you are developers, and you know how hard it is to build truly reliable systems. It's hard to make your applications reliable because when you're building a complex application, just about any step can break anywhere. If you're building something like Money Transfer, for example, you have to worry about your application breaking, like when you start transferring, while you're in the middle of transferring, at the end, when you're trying to send a confirmation. And things can break for just about any reason.

Applications can break and fail because they're process crashes, because they run out of resources or hit a bug. They can fail because of timeouts, because someone takes too long to respond. They can fail because an external API they use breaks down, because it's rate limited, because it transiently fails, because it has an outage. And, of course, failures are an even bigger worry if you're running at large scale, where there are just more things that can fail. Or if you're using AI, which a lot of folks are these days, because AI is kind of inherently brittle, and the AI providers, the model providers that we rely on, frequently have issues and outages that can break AI applications.

If something fails, the easy thing to do is to retry from the beginning, to restart whatever failed. But often you can't do that. If you're restarting a business process, like a Money Transfer from the beginning, you're risking some kind of corruption or duplication, where you transfer money twice, or double book a reservation. If you're trying to restart a really big task, then you're risking wasting compute resources or being incredibly slow. So often, when instead of just restarting something that fails, you have to implement complicated recovery code that figures out exactly what the failure is, and tries to remediate it directly. And that sort of thing is hard to write.

So one new tool that can really help with this are durable workflows. So the idea behind durable workflows is that you regularly checkpoint the state of your program. So that if something fails, you can use those checkpoints to recover your program from its last completed step. And you can think of workflows and their checkpoints as working a lot like save points in a video game, where if you're playing a game, you can save regularly, so that if you die, you can reload from the last save. And when you're running a durable workflow, you're checkpointing every step, so that if your program fails, you can reload, resume, recover from the last completed step.

So these sorts of workflows are really useful for all sorts of applications, for important business processes, for CI pipelines, for data pipelines, and nowadays also for agents. So there are plenty of workflows out systems out there, but most of them are kind of big and complicated. They're not easy to integrate into like a JavaScript application. The classic architecture for durable workflows is what I like to call external orchestration. And the idea here is that you have a central orchestrator surface that orchestrates your workflows. And if you want to run a workflow, now you run the workflow and not on your own JavaScript application, but on the service.

And then the way they work is that your server send a request to the workflow service to start a workflow. The workflow service dispatches the first step in the workflow to a worker. The worker executes the task. The service dispatches the second step and so on until every step has been dispatched and the workflow is done. These are big heavyweight systems and they work, but they require you to take on large dependencies, add new services, re-architect your application. They also can have high latency overhead from step dispatch. You might be adding tens or hundreds of milliseconds to the latency of each of your steps.

You might be adding tens or hundreds of milliseconds to the latency of each of your steps. So effective systems, but heavyweight and maybe not something you want to add to your backend node application. So what I want to tell you is that workflows should be simple. Your workflow should be something you can just put into your node app without needing all this infrastructure. And the workflows you build in your node app should be just as reliable and powerful as the ones that are built and scalable as the ones built using all this infrastructure.

And specifically, I'm going to show you how you can implement workflows as a JavaScript library, as a library that you can install into your application. You can npm install this library and this library will give you your application durable workflows. You'll be able to checkpoint workflows and steps directly from your app servers, just as you would with a big complicated cluster. So this is the architecture. We're going to design this library together. This is what the architecture of this library looks like, where you have the library installed with your app servers and it runs the workflows directly on your servers and checkpoints their workflows and steps to a database.

So here's what the interface of this library that we're going to design looks like. It's actually pretty simple. You write workflows and steps as normal JavaScript functions. Then there are only two methods in our library's interface and we're going to design both methods. The first is register workflow, which registers an ordinary JavaScript function as a workflow. And the other is run step, which runs an ordinary JavaScript function as a step in a workflow. We'll build these two interfaces and from these, we'll get durable workflows. And the way this library is going to work is that it's going to checkpoint workflows and steps in a database. So when a workflow starts, the library will checkpoint the workflows inputs. And then as each step of the workflow completes, the library will checkpoint that step's outcome to a database.

And then finally, when the workflow finishes, the library will checkpoint the workflows output. And because the workflows input and every step is checkpointed, if something ever fails or breaks or crashes, the library can look up the checkpoint to recover each workflow from its last completed step.

So let's start implementing this library together in JavaScript. And it's simple enough that we can fit most of the code, at least a simplified version of the code right here on these slides. So let's start with the first method. Well, we'll look at three methods. And the first one is run workflow. And this method is going to take a JavaScript function, the workflow, and its arguments, and run this function as a durable workflow, checkpointing it and its steps so they can be safely recovered. And I know this block of code might look scary, but we're going to go through it line by line to see exactly what it does.

So the first step in running a workflow is generating a globally unique ID for the workflow that can be used to reference it. Then we're going to checkpoint the workflows, we're going to checkpoint the workflows inputs. So we're going to write our records the database with this workflows ID, its inputs and the pending status. So we now have a record saying that this workflow has started with these inputs and this unique ID. And we can use that record later to recover the workflow if anything breaks. Now that we've, you know, durably persisted a record saying that the workflow started, we can now actually run the workflow function. So the workflow function, just an ordinary JavaScript function, we're going to call it with its original arguments.

And you checkpoint its outputs. Of course, we don't just want to have checkpoints at the beginning and at the end. We want to have some in the middle too. So the next function we have to implement is run step. And this will checkpoint every step in a workflow so that we can checkpoint every step the workflow takes. And if something breaks, you'll recover from the last complicated step. So the first thing you do, this is run step. It takes in a step, which can be any JavaScript function and the steps arguments, and it durability runs and checkpoints the step. So the first thing you do in run step is you retrieve the workflows context. So you retrieve the ID of the workflow and the ID of the step, which is just an index into the number of steps this workflow is running.

So the first thing you do in run step is you retrieve the workflows context. So you retrieve the ID of the workflow and the ID of the step, which is just an index into the number of steps this workflow is running. So you retrieve workflow ID and step ID from context, which might be from a context argument you're passing around or from a local storage. Then after you have your workflow ID and step ID, next, you check if a checkpoint already exists for the step because maybe you're recovering from a failure and the step already ran and we want to retrieve a checkpointed output instead of re-running it. So we go look up in the database, whether a checkpoint exists for this workflow ID and step ID. And if one does, we'll return that checkpoint directly instead of re-executing the workflow. If there's no step, if there's no checkpoint, then we'll just directly execute the step function, calling it on its original arguments and getting its output. And then we'll write this output to the database.

So to sum it all up, when you want to run a step, you're going to first check of a checkpoint that already exists for the step. If one does, return a checkpoint directly, otherwise run the step and check what is the outcome. So now we're actually almost done implementing our little library. The last step, the last thing we need is a way to recover workflows that have previously failed. Using these checkpoints, we just spent so much time building. So here's what our workflow recovery function looks like. And again, pretty simple. This function is going to run every time a program with our library starts up.

The first thing it'll do is it'll get a list of all pending workflows, all workflows that started but didn't finish. These are workflows presumably that failed and we're going to go recover them. We then iterate through each one of these failed workflows and forge these workflows. We're going to retrieve its checkpointed input. Remember how we checkpointed earlier? And we're going to look up the actual workflow function from an in-memory map that we created when each workflow was registered. Now that we have the workflow's unique ID, its arguments and the function itself, we're just going to call run workflow, through which we did earlier on the workflow function, the unique ID and the arguments. And run workflow, we'll just run the workflow again from the beginning.

And run workflow, we'll just run the workflow again from the beginning. But each time the workflow gets to a step, it'll see that the step has already been checkpointed. And instead of re-executing the step, it will return the step's checkpointed output. And by doing this over and over again, the workflow will eventually recover, it'll eventually get to the first step that doesn't have a checkpoint and resume normal execution from that point. So in other words, our workflow will recover from the last completed step by following this procedure. Great.

So now we can see roughly what our library looks like. One last thing I want to show you is a diagrammatic process of what this library might do when applied to a real application. So we're going to look at an example of a workflow. So a checkout service, a simplified checkout service of the sort that might be, that you might run when you click the buy button on a website. So this workflow would be implemented as a JavaScript function. And each of these steps like create order, reserve inventory, process payment, would also be JavaScript functions. So when we run this workflow, we call run workflow.

The first thing we do is checkpoint the inputs. So we'll write a record to the database containing the workflow inputs, the unique ID, and mark the workflow with a pending status. Then once we've written that checkpoint, we can start actually executing the workflow. So we create an order, we reserve inventory, we process payment, and after each step, we checkpoint the steps outcome. Then let's say something breaks. We have just processed payment, and suddenly our server fails. And our process isn't running anymore. Normally, that would be pretty bad because our customers paid for something, and they're expecting to get their order fulfilled, but we don't know that because our process just crashed.

Recovering that might be tricky, but we have durable workflows, so it's much easier. So when our server resumes, it'll look at the database and see that there's this workflow that's in a pending state, so it needs to be recovered. It'll then recover the workflow, looking up each of its steps, and returning the checkpointed outcome. So it'll see that the create order, reserve inventory, and process payment steps were all successful. So it'll fast forward past them, reload from their checkpoints, and now re-execute them. Then finally, we'll get to the first, not yet completed step, fulfill order. We'll run it, we'll fulfill the order, we'll make the customer happy, and then we'll mark the workflows as success, and we'll do that all without re-executing anything, without double preserving inventory, or double charging the customer.