Your Orchestrator Is an Operating Model, Not a Feature List

Data orchestration became interesting again, although I am not sure it ever stopped being complicated.

In July, Prefect announced that it was acquiring Dagster Labs. The announcement says Dagster will keep its name, support, and open-source license, with no migration required for customers. In September, Kestra released version 2.0 with a rewritten engine, separate control and data planes, remote workers, and flows that agents can call as tools.

Naturally, the familiar question returned: which orchestrator should we use?

The Reddit discussion includes almost every answer you would expect. Stay with Dagster. Use Airflow because it is established. Choose a managed service. Try something smaller for a lean team. A few vendors also arrived to recommend their own products, because some laws of distributed systems are social.

I think the product comparison is starting one step too late.

An orchestrator encodes how a team owns work. The syntax and user interface matter, but the bigger decision is who defines workflows, where code runs, how failures are recovered, and which boundaries the platform enforces.

The DAG is the easy part

Most tools can express this:

Extract
   |
   v
Transform
   |
   v
Validate
   |
   v
Publish

A demo usually ends when those four boxes turn green. Production starts when the second box runs for five hours, the third box discovers bad data after the publish step, and someone needs to backfill Tuesday without rerunning Wednesday.

That is where orchestrators differ in ways a feature matrix struggles to capture.

Does the platform think in tasks, assets, events, or deployments? Does it store enough state to reconstruct what happened? Can one team update a workflow without gaining access to every credential used by the worker? Can workloads run inside the network where the data already lives? How are upgrades tested, rolled back, and supported?

The answers shape team behaviour. They also determine who gets paged.

Start with the ownership boundary

Before comparing Airflow, Dagster, Prefect, Kestra, or a cloud-native scheduler, I would draw the boundary:

Platform team owns
  - control plane
  - identity and secrets
  - worker environments
  - deployment path
  - shared monitoring

Data teams own
  - workflow definitions
  - transformation code
  - data-quality rules
  - schedules and dependencies
  - domain runbooks

This is only one model, but it makes the questions concrete. If every data team needs the platform team to install a Python dependency, self-service is mostly a logo. If every workflow author can run arbitrary code with shared production credentials, self-service has gone too far in the other direction.

The orchestrator should make the desired boundary easier to enforce.

Kestra 2.0 is interesting in this context because its release notes describe workers connecting to the control plane through an outbound gRPC stream. User code runs in the data plane, and workers can sit in another region, cloud, or outbound-only network. That is more than an engine detail. It supports an operating model where a central team manages orchestration while execution stays near the workload.

It also comes with migration work. The 2.0 release removes or changes several constructs, requires an upgrade through 1.3.x, and changes the default behaviour for unmatched worker routing from waiting to failing. Those details belong in the decision because an orchestrator is a long-lived operational dependency, not a library we casually swap on Friday afternoon.

Decide what the system is organizing

The task-versus-asset distinction is another operating choice.

A task-oriented model asks, "What should run next?" It maps naturally to jobs, scripts, APIs, and infrastructure operations. Teams with mixed workloads often find this easy to reason about.

An asset-oriented model asks, "What data should exist, and what does it depend on?" That can make partitions, lineage, freshness, and backfills easier to express for data-heavy platforms.

Neither model is universally better. The useful question is what engineers spend their time debugging.

If incidents usually sound like "this job did not run," task state may be the natural center. If they sound like "the customer dimension is missing yesterday's partition," asset state may be more useful. If the same platform also runs infrastructure automation, ML training, and API workflows, a strongly data-specific abstraction can become awkward.

Pick the model that matches the failure language of the team.

An acquisition is a signal, not a migration plan

Prefect's acquisition of Dagster creates legitimate questions about long-term product direction. It does not, by itself, make a running Dagster deployment unsafe.

The public announcement commits to continued support, the existing name, and the existing open-source license. That is the fact available today. Future convergence is a possibility, not a documented migration requirement.

I would respond with boring engineering work:

• pin the current version and test upgrades in a representative environment;
• document the APIs, metadata, and deployment assumptions that create lock-in;
• keep transformation logic outside orchestration definitions where practical;
• export workflow and run metadata needed for audit or migration;
• define the event that would trigger a reevaluation.

That last item is useful. "We feel nervous" is difficult to act on. "Security fixes fall outside our required window" or "the supported deployment no longer fits our network boundary" is a decision trigger.

The same discipline applies to every orchestrator. Project ownership can change. Cloud pricing can change. A managed feature can move tiers. Even a stable open-source project can become difficult for a small team to operate.

An exit path is part of the architecture.

Run a failure-focused proof of concept

Happy-path evaluations flatter every product. I would test a small workflow and then deliberately make it unpleasant.

Use one hourly ingestion, one daily aggregation, and one data-quality rule. Add a partitioned backfill. Run a task longer than the worker lifetime. Rotate its credential. Remove network access halfway through a run. Deploy an incompatible workflow change and roll it back.

Then score the products on the work the team will actually own:

Decision area	Question to answer
Deployment	Can we upgrade and roll back without improvisation?
Isolation	Can workloads use separate identities and dependencies?
Recovery	Can an operator restart the correct unit without duplicating data?
Backfills	Can historical work run without starving current schedules?
Observability	Can we identify the failed data, code version, and owner quickly?
Self-service	Can a team ship safely without platform tickets?
Cost	What grows with workflow count, run count, retention, and worker size?
Exit	How much domain logic is trapped in the orchestrator?

Two days of deliberate failure will tell you more than two weeks of clicking through feature pages.

My recommendation changes with the starting point

For an organization already running Airflow reliably, I would keep it unless there is a measured problem. A newer orchestration model may look cleaner, but migration creates parallel operations, retraining, rewritten workflows, and a new failure surface. Familiarity is not glamorous. It is still an asset.

For a new data platform, I would compare one task-oriented and one asset-oriented option using a real pipeline. If the team is small, include the cost of operating the control plane, not just the speed of writing the first workflow. A managed service may be cheaper than months of occasional maintenance even when its invoice is higher.

For workloads spread across restricted networks or multiple clouds, I would test the control-plane and worker boundary first. Kestra 2.0 now makes an explicit architectural claim in that area. The PoC should verify identity, connectivity, failure isolation, and upgrade behaviour rather than accepting the diagram.

For a team whose warehouse or data platform already provides adequate scheduling, I would start there. Another orchestrator must earn its database, deployment, alerting, and support burden.

The orchestrator decision is not Airflow versus Dagster versus Prefect versus Kestra. It is centralized versus federated ownership, tasks versus assets, shared execution versus isolated workers, and product convenience versus operational control.

Choose those boundaries first. The product shortlist usually becomes much smaller after that.