TL;DR — Most orchestration debates for ML pipelines fixate on which scheduler to use, but the real failure mode is treating models and datasets as opaque task outputs instead of versioned, lineage-tracked assets. Task-centric DAGs can run perfectly and still produce a model nobody can explain or reproduce. The fix isn't a new tool — it's rethinking what your pipeline-as-code is actually supposed to track.
Every ML platform team eventually has the orchestrator debate. Airflow or Dagster. Managed or self-hosted. Cron-based or event-driven. It feels like an important decision, and vendors are happy to keep it feeling that way. But the debate is mostly noise. The question that actually determines whether your ML pipelines are trustworthy has nothing to do with which scheduler runs your tasks. It's whether your orchestration layer treats data and models as first-class, versioned assets — or just as side effects of tasks that happened to run.
Most teams get this wrong quietly, for years, without noticing. The pipelines run green. The dashboards show success. And nobody can answer the one question that matters when a model misbehaves in production: what data, exactly, trained this thing?
Tasks are not the unit that matters
Classic orchestration — the Airflow model, and honestly most hand-rolled schedulers before it — is built around tasks: discrete units of work with dependencies, retries, and a schedule. A task runs, it succeeds or fails, and the DAG moves on. This model was designed for ETL, where the thing you care about is "did the job complete and did the table get populated." That's a reasonable unit of correctness for a nightly aggregation job.
It is the wrong unit of correctness for ML. In an ML pipeline, the thing you actually care about is not "did the feature engineering task finish." It's "what specific version of the raw data produced this specific version of the feature table, and which version of that feature table trained which version of the model that's currently serving traffic." That's a chain of assets, not a chain of tasks. A task-centric mental model can tell you the pipe ran. It can't tell you what's actually flowing through the pipe.
This gap is where most ML incident postmortems live. A model's performance drifts. Someone checks the DAG — all green, all on schedule. Someone checks the code — unchanged. What actually happened is an upstream table got backfilled with corrected values three weeks after the original training run, silently changing the ground truth the model was trained on, and nothing in the orchestration layer ever represented that as a change worth noticing.
Dagster's asset model is a genuine idea, not just a feature
This is the actual, substantive difference between Airflow and Dagster, and it's more interesting than "one has a nicer UI." Dagster's software-defined assets make the data product — a table, a feature set, a trained model — the primary object in the system, with tasks demoted to the mechanism that produces them. You declare that a model asset depends on a feature-table asset, which depends on a raw-data asset, and the orchestrator tracks that dependency graph as data, with versions and materializations attached to each node.
Airflow can be bent into doing something similar, and people do bend it — dataset-aware scheduling, XCom-based lineage hacks, custom operators that log versions to a metadata store. But it's retrofitted. The DAG is still fundamentally a graph of tasks; the asset semantics are a layer you build on top, and it's only as good as the discipline of whoever built it. Dagster made the asset the default unit, which means the lineage question — what produced this, from what, when — is answerable by construction rather than by convention.
None of this means Dagster is the correct choice for every team. It means the criterion for choosing should be "does this tool make asset lineage a structural property of the pipeline" rather than "which one do I already know."
Pipelines-as-code solved the wrong half of the problem
The pipelines-as-code movement — DAGs defined in Python, checked into version control, reviewed like any other software — was a real improvement over drag-and-drop ETL tools and cron jobs scattered across servers. It gave ML pipelines the same rigor as application code: diffs, tests, CI, rollbacks.
But it solved the reproducibility of the process, not the reproducibility of the result. You can have a pipeline definition that is perfectly version-controlled, perfectly tested, perfectly identical between two runs — and still produce two different models, because the upstream data changed underneath it. Pipelines-as-code makes the code deterministic. It says nothing about the data being deterministic, which for ML is the half that actually matters.
This is why "just use pipelines-as-code" is necessary-but-not-sufficient advice. It fixes bugs caused by pipeline drift — someone manually edited a job in a UI and nobody knows why. It does nothing for the far more common and far more damaging failure: silent data drift flowing through a pipeline that is, by every code-based measure, completely healthy.
What asset-aware orchestration actually buys you
If you take the asset-centric view seriously, a few concrete practices fall out of it, regardless of which tool you use:
Every materialized dataset and model has a version identifier that's queryable, not just implied by a timestamp in a filename.
Lineage is a query you can run, not an investigation you have to conduct. "What trained model X" should be answerable in one lookup, not a week of log archaeology.
Staleness is a first-class signal. An asset that hasn't been rematerialized when its upstream dependency changed should be visibly, structurally stale — not just quietly outdated.
Retraining triggers fire on asset change, not on schedule. A model should retrain because its training data materially changed, not because it's Tuesday.
Backfills propagate through the dependency graph automatically, so a correction to raw data doesn't require someone to remember every downstream table and model that needs to be rerun.
You can implement every one of these on top of Airflow with enough custom tooling. You get most of them by default with an asset-oriented framework. Either way, the point isn't the tool — it's the discipline the tool either enforces or merely permits.
Choose the abstraction, then the tool
The Airflow-versus-Dagster framing puts the tool first and the abstraction second, which is backwards. The right sequence is: decide that your orchestration layer needs to answer lineage questions about data and models as a structural guarantee, not a nice-to-have. Then pick whichever engine makes that guarantee cheapest to enforce and hardest to accidentally violate.
For pure ETL, task-centric orchestration is fine — arguably still the simpler, more battle-tested choice. For ML, where every pipeline run produces an artifact that will eventually make a decision affecting real users, the asset is the thing that needs tracking, and the task is just how you got there. Pick your tool once you've admitted that.
Top comments (0)