Orchestrating Content Workflows at Netflix Scale

Welcome everyone, I'm Pradeep Shah, a senior software engineer on the Content Understanding Infrastructure team at Netflix. At Netflix, before any title reaches a viewer, it passes through a pipeline of decisions related to content understanding about what it is, what it contains, and how it should be surfaced. Some of these decisions are made by rule engines, some by ML models, and some require humans to make the call.

Today, I want to talk about how we build the infrastructure to coordinate all these three reliably. We will also show you how the same infrastructure extends naturally to AI agents using our prototype example. But before I explain the system details, let's cover the key concepts. One, workflow. It's the process that defines what work needs to happen. And two, task. Tracks the unit of work from start to finish.

Now that we understand the key concepts, let me set the context on why the problem is hard. Netflix has an enormous catalog spanning markets and languages. Every title on the platform carries a large number of metadata attributes, content signals that describe what the title is, what element it contains, etc. This metadata is the engine behind content discoverability, how the right title surfaces to the right viewer. Poor metadata quality degrades discovery, reduces engagement, and erodes consumer trust. The volume and the variety of signals means this cannot be done manually. AI, automation, rule-based engines, etc. are all essential. But here is also the core design challenge.

All decisions are not equal. Rules are fast and cheap, but brittle. They break on edge cases. ML models are scalable, but imperfect. Confidence varies, and we need a mechanism to catch low-confidence predictions before they cause quality problems. Humans are accurate, but this approach does not scale. We want the human's judgment precisely where it adds most value, not as a general fallback. The question is not which one to use. The answer is all three in the right combination for each piece of work. Hence, we need a system which would route each piece of work to the right worker, human, model, AI agent, etc., captures every human decision as structured feedback for model Sharks state across many concurrent workflows. Different teams plug in their own logic without rebuilding the plumbing.

And lastly, scales to new content type and workflows without any code changes. The mantra for our new system is build just once, but reuse across different workflows. So how do we build such an orchestration system? So that it also runs reliably at scale. But before that, let's answer the obvious question. What is the issue with custom workflow implementations? Why do we need one system to do it all? We did build two custom workflow implementations before this, and this was actually critical because we learned the major inflexibilities with our previous implementations.

Capturing multiple workflows meant discrepancies between different implementations, leading to maintenance overhead, longer times to build, and it also had downstream impact. Task management data was captured differently in different systems. That meant data science team, for example, had to normalize it differently for each workflow before they could do any cross-workflow quality analysis. What we observed is that these workflows share a common backbone, triggering, route, execute, verify, and complete. Only the business logic changes. Once we saw that clearly, the right answer was a framework. We would define the backbone once and let teams contribute only the part that's unique to their workflow.

Our new system was built on our learnings from our previous implementations. For example, in the old world, each workflow had a bespoke state machine, depending on whether it required human fulfillment or quality control. One had seven states, the other had nine. Business logic was embedded directly into the state transitions, which meant the state machine and the workflow logic were tightly coupled and impossible to reuse. Adding new phases, such as quality control, once the workflow had shipped meant a lot of work. It also meant operators had to be trained for workflow and data science had to build their pipelines again because now the state transitions itself have changed. The fix was to collapse all state machines through a simple life cycle and decouple the business logic. This helped us achieve consistency across different workflows.

So, how did we do that? By separating state management and workflow execution. We came up with the two-tier architecture. Tier one was the task management tier, which answers questions like, what is the task? Who owns it? Is it waiting for prerequisites, actively being worked upon, or done? This happens to be fast and answers across entities, across different workflows, capturing different metadata.

Tier two is the orchestration layer. Given a task is ready, what are the steps to execute it? Does it depend on angles or AI agents or humans? Where and how does the metadata live, etc? When these two concerns are conflated, as they were in our earlier systems, we ended up with brittle rigid state machines that can't be reused across teams and workflow logic. The workflow logic couldn't be reasoned independently at all. Separating them is what made the system extensible enough to handle rules models and humans as interchangeable worker types.

In order to implement the tier one of our system, we used raw hollow. Raw hollow is Netflix's internal in-memory compressed data store used widely and extensively within Netflix. Instead of making network calls to a database or a cache for every read, every reader instance in raw hollow holds the complete data set in its own JVM heap. A read is just a traversal of in-process memory. Writes in raw hollow propagate to all reader instances within single digit milliseconds. For our use case, this matters enormously as it makes searching across workflows across entities extremely fast. The caveat here is once a task is complete, we need to move it to goal storage like a traditional database in order to keep the memory footprint in check. Netflix has published a research paper on raw hollow, which I will have a link to at the end of the presentation.

To implement the orchestration layer, we used Conductor, which is an open source project built at Netflix. It has built-in retries, timeouts, conditional branching, parallel execution, sub-workflows, etc. Step-within workflows are stateless and language-agnostic. Teams implement individual steps in their own repositories. Steps could be like GraphQL calls, gRPC calls, wait tasks, etc. It also provides visual DAG UI, real-time visibility into each and every step, the inputs and outputs, and retries, histories, etc.

Here is the Sattel Conductor workflow the orchestration layer of our system owns. It allows the team to plug in their sub-workflow while keeping the rest of the execution sane across different use cases. When the orchestration layer is ready to execute our task, our system starts this workflow. It first runs the team's sub-workflow. This is where the team's specific logic lives. Our system is agnostic to the sub-workflow implementation. After the team's sub-workflow completes, it returns a standard response, which contains different flags. Do we need a fulfillment task, or a quality control task, etc.

Based on these flags, our main workflow then decides the rest of the flow of the workflow. The system also provides notifications, reporting, dashboards, etc. This way, a team can go from zero to a fully operational workflow with AI inference, human review, quality control, etc., by writing just the sub-workflow definition and configuration. No other changes are required in the system. So let's answer the obvious question, why do we need two tiers? Conductor is very powerful, so why not just use it alone? Conductor is excellent at step-by-step execution, but it doesn't give us the unified view. It does not answer questions like, how many fulfillment tasks are waiting for human review right now, across workflow types.

RawHolo provides a unified view of the system's state and distributes this data to downstream systems. Meanwhile, Conductor is best suited for the execution of individual tasks. So the two tiers are complementary, not reductive.

Now that we understand the two-tier architecture, let us see what are the various steps which occur when a new workflow is onboarded to our system.

Step 1. We define the events required for triggering. The event router subscribes to event streams. In a configuration, teams declare which event sources to listen to, what conditions to filter, what fields to extract and enrich to create the tasks. There is no custom code. It's completely based on configurations.

Step 2. When the first event arrives, the task is created immediately with state new. New means we know this work needs to happen, but prerequisites are not yet met. This early visibility has a meaningful operational advantage. If ML inference pipeline is slow or packed up, we can see it immediately as a growing pile of tasks stuck in new state, waiting for the model's signal.

Capturing the full lifecycle from new to complete gives us a rich dataset. How long did we wait for the model? How often did tasks require human escalation? How did that rate change after the model updated? All this can be derived by the task history. And because these tasks are in raw HOLO from start, all this is visible in real time for UIs as well as downstream users.

Step 3. When the task becomes ready, the conductor workflow gets kicked in by the framework. Teams configure their sub-workflow name as part of the configuration, which then gets automatically called without any code changes. The business logic that's unique to each workflow lives within this sub-workflow. Here is where we decide when we want to completely rely on rules or AI or escalate to humans.

Step 4. Human in the loop. When the AI model's confidence falls below the threshold, sub-workflows can decide to call for human intervention. This is when our conductor workflow creates a child task in raw HOLO. The parent conductor workflow then enters into a wait state. The child task surfaces on the dashboard. A human reviewer picks it up, sees the model predictions, reviews it, and then makes a correction if needed and submits.

Once submitted, the framework sends a completion signal back to the conductor. This completes the parent conductor workflow, thereby completing the parent task as well. The critical design choice here is the human decision is a structured output. It's not a comment or a note. Every feedback entered during the human-in-the-loop task is seen as denormalized schemas, which work across different workflows. Feedback flows automatically into the model feedback pipeline.

We have purposely used domain-agnostic schemas to allow reusability across model retrainment and data science pipelines across workflows. To reiterate, here is the full workflow. A content event arrives, a new title, or a metadata signal, or an upstream signal, whatever. The event arrives. The event router picks it up, filters it, enriches it, and creates a task in raw holo with state new. Once all prerequisite signals have arrived, the task becomes ready. The framework starts the conductor workflow and moves it to in-progress.

Conductor executes the team sub-workflow, which is where the domain logic is executed. If the model confidence is high, result is applied automatically. If the confidence is below the threshold, conductor creates a human-review-child task. Human picks it up, reviews it, and then corrects it and submits. The workflow resumes with the human's decision. When the task it completes, either automatically or with human review, all decisions are captured. The task state is then written into the raw holo, and the human corrections flow into the model feedback pipeline. We have built notifications, alerts, monitoring, which address our current workflows, as well as automatically scale to future ones as well.

So far we have spoken about humans, but what about AI agents? Let me quickly walk you through an example of how this scales to AI agents as well. An AI agent is just another service that exposes an endpoint. It posts instructions and context. It does the work asynchronously, and once done, it posts back to the parent workflow. The wait mechanism that pauses the workflow when a human reviewer works is identical to the mechanism that pauses while an AI agent is working. The framework does not distinguish between them. It only expects that callback, which then completes the wait task.

We prototype this internally. We use the framework to drive an AI agent that generated code improvement PRs for the framework service itself. The agent received the task context, what changes needed to be made, the files which were in scope, and the constraints to follow. It ran the analysis and created a PR. Once the PR was merged, it called back the parent workflow. The framework treated it like any other completed task. We did not make any changes to the framework to support this new use case. This helped us showcase the power of the framework and how it can integrate with different types of workers seamlessly.

So, what makes this architecture work? Number one, separation of concerns. State management and workflow execution are genuinely different problems. Keeping them separate means we can swap out the orchestration layer or the storage layer and both can evolve independently. Number two, contract-based pluggability. The framework imposes a contract on the sub-workflows. Here is where and how you receive the input. Here is how you must return an output. Within that contract, teams are free to do anything. This is how you scale the platform adoption across many use cases without becoming a bottleneck. Number three, multi-modal workers. This is a very powerful principle. Rural engines, ML models, humans, or AI agents all interact with the framework in the exactly same way. They receive a task, they do the work, and then return a result. Number four, every human correction is a training signal. We capture the human feedback as a structured output, not as a note. We build the feedback pipeline into the framework so every team gets it for free. We must assign humans in the loop from day one and plan how to capture the data so we can build self-improving systems. This is how we build the framework which spans across different workflows here at Netflix using the two-tier architecture, which decouples state management from core business logic. Thank you.