A workflow orchestrator can coordinate thousands of operations and still hit a scalability ceiling—not because individual tasks are too expensive, but because the engine spends too much time deciding what should happen next.
That was the problem Netflix faced with Conductor as its internal adoption grew. The platform now supports roughly 200,000 workflow definitions across about 150 applications and executes around 420 million workflows in a typical month, according to the Netflix Technology Blog. Its workloads span content and studio production, advertising, games, and other business systems. Source: Netflix Technology Blog
Netflix's response was not simply to add more compute. It redesigned the evaluation engine and the data model underneath it.
The distinction between releases matters. Conductor 3.0 modernized the data plane; Conductor 4.0 rebuilt the workflow evaluation engine. Netflix says the 4.0 rewrite increased the supported workflow size from approximately 2,500 tasks to 30,000 and reduced p99 workflow-evaluation latency by roughly 40% in production. Those are Netflix-reported results, not universal performance guarantees.
For engineers building data pipelines, feature-engineering systems, media workflows, or multi-step AI applications, the architectural lessons are highly relevant: keep workflow metadata lightweight, avoid loading state that an evaluation does not need, and move contention-prone decision-making out of a synchronous request path.
Why it matters
A workflow engine is responsible for more than executing tasks. Whenever a task completes, fails, or reports progress, the engine must update state, evaluate dependencies, and determine which tasks can run next.
As the graph grows, that coordination can become the bottleneck.
In Netflix's earlier engine, evaluating a workflow meant loading its entire state into memory. Workflow metadata and task payloads also shared a Cassandra partition, creating large rows and increasing the complexity of concurrent updates. Even if only one task changed, the engine could still need to process far more state than that decision required.
The second problem was synchronization. Workflow evaluation ran inline when updates arrived. Concurrent updates to the same workflow competed for a lock, and unsuccessful attempts could be deferred to a repair process. Under peak load, that caused unpredictable tail latency.
These problems have direct parallels in data and AI infrastructure:
- A feature-engineering workflow with thousands of dependent steps can spend more time coordinating than executing useful work.
- An AI agent that fans out to many tools needs a reliable way to track task status and resume after failures.
- A batch orchestration system can become slow when many workers update a shared execution record.
- Long-running workflows need recovery mechanisms that reconcile missed or delayed updates without corrupting state.
The important performance metric is therefore not just average task duration. It is how efficiently the orchestrator evaluates state, handles concurrent updates, and schedules the next piece of work when the system is under pressure.
What changed: from Conductor 3.0 to 4.0
Netflix describes the evolution as several related changes rather than one isolated rewrite.
| Stage | Main architectural development |
|---|---|
| Conductor 1.0 | Dynomite and DynoQueues supported metadata and queueing, with Elasticsearch for indexing |
| Conductor 2.x | Cassandra became the primary execution datastore, large task inputs and outputs moved to S3, and Timestone replaced DynoQueues |
| Conductor 3.0 | The modernized data plane was consolidated; Dynomite was retired, while Cassandra, Timestone, a Kafka-fed indexer, Elasticsearch, and Iceberg served different roles |
| Conductor 4.0 | The evaluation engine and task data model were redesigned, and evaluation became asynchronous through Timestone |
Conductor 3.0 also introduced automated workflow repair and multi-cluster deployment, allowing workflows to run in isolated contexts. These were important foundations, but the engine still carried an older evaluation design.
The 4.0 work addressed that remaining bottleneck. Netflix's engineering post explains the rewrite in three connected areas: a partitioned task model, safer reconciliation of competing updates, and asynchronous workflow evaluation.
This is an excerpt. The full article continues with:
- 1. Separate workflow metadata from individual task records
- 2. Make concurrent task updates safe without relying on conflicting write paths
- 3. Move workflow evaluation off the synchronous request path
- What the results mean
- 4. Scale task polling as well as workflow evaluation
- 5. Improve concurrency controls and task execution options
- What data and AI engineers should do
- Limitations and open questions
Read the full article on AI Data Legends, with diagrams, code and sources.
Top comments (0)