TL;DR — Airflow and Dagster both partition work along a single axis: time. ML workflows need backfills along two axes at once — time window and code/config version — and neither tool's partition model was built for that. The result is a mess of duplicated DAGs and broken lineage that teams mistake for an organizational problem instead of an architectural one.
Every mainstream orchestrator — Airflow, Dagster, Prefect, even the newer asset-based frameworks — was built around a single organizing idea: a pipeline run is identified by a point in time. You have a schedule, you have an execution date or a logical date, and when something breaks you backfill by re-running that date range. This model is correct for ETL. It is incomplete for ML.
ML pipelines need to backfill along two axes simultaneously: the time window of the data, and the version of the code or config that processed it. Orchestrators give you a clean, well-supported way to replay the first axis. They give you almost nothing for the second. That gap is not a tooling inconvenience. It's the reason so many teams can't actually answer the question "what would this model look like if we'd used the new feature logic a year ago?" without building a parallel, throwaway system to find out.
The One-Dimensional Assumption
Think about what a partition actually means in Airflow or Dagster. In Airflow, it's the execution date baked into the DAG run. In Dagster, partitions are explicitly modeled, but still almost always keyed on date or some business-time dimension — daily sales, hourly events, monthly cohorts. The entire backfill UX assumes the thing that changes between runs is which slice of data you're processing, not which logic you're processing it with.
That assumption holds fine for a data warehouse load. If yesterday's job failed, you rerun yesterday's job, same code, same config, different date. It falls apart the moment the question becomes: "rerun the last six months of training data through the new feature transform and compare it to the old one." Now you don't have one axis of variation, you have two, and the orchestrator was never designed to let you move along both at once.
Where This Actually Bites
Consider a real pattern: a feature engineering change. Someone fixes a subtle bug in how a rolling aggregate handles late-arriving events. The fix is correct. But now you need to know what the model would have learned if that fix had been in place for the last 180 days of training data, so you can compare against the production baseline fairly. That's not a new DAG. That's the same DAG, run over the same dates, with a different code version — and you need both the old and new outputs to exist side by side, addressable, comparable, and traceable back to exactly which git commit or config hash produced them.
Try to express that in Airflow and you'll end up doing one of three things: cloning the DAG with a suffixed name, parameterizing the DAG with a version tag that lives outside the scheduler's partition model, or spinning up an entirely separate "experiment runner" that bypasses the orchestrator altogether and just calls the underlying tasks directly. All three solve the immediate problem. All three quietly break the thing orchestration was supposed to give you in the first place: a single source of truth for what ran, when, and with what inputs.
Dagster's partition model is genuinely closer to a fix than Airflow's, because it lets you define partition dimensions explicitly rather than relying on an implicit execution date. But in practice almost nobody uses it to encode code version as a first-class partition axis. It's used for backfilling time ranges, maybe combined with a static dimension like region or customer segment — not for "this exact transform logic, run at this exact time window." The primitive exists. The convention doesn't.
Why This Isn't Just a Lineage Problem
It's tempting to wave this away as "well, that's what experiment tracking tools are for." But experiment tracking tools record what happened after the fact. They don't help you orchestrate the actual execution of a two-dimensional backfill matrix — figuring out which (time window, code version) cells you've already computed, which are stale, which need to be recomputed because an upstream dependency changed, and which can be served from cache. That's exactly the kind of dependency resolution problem orchestrators are supposed to be good at. We just haven't given them the right partition key to reason about.
The failure mode compounds in hyperparameter sweeps and model retraining cadences too. A sweep over five configs across twelve monthly windows is sixty cells in a matrix. Today, most teams flatten that matrix into sixty separate DAG runs with ad hoc naming conventions, then reconstruct the grid manually in a notebook or a tracking dashboard. The orchestrator did the easy fifth of the work — running things — and none of the hard four-fifths — knowing what's been run, what's missing, and what's invalidated when either axis changes.
What an Orchestrator Built for This Would Actually Look Like
A composite partition key — (time_window, code_version) — needs to be a native concept, not a workaround bolted onto the date-based scheduler. That means a few concrete things. First, the orchestrator needs to track code or config version as an explicit, queryable dimension with the same first-class status as a date partition, so a backfill can be scoped by either axis independently or both together. Second, invalidation logic needs to understand that a change along the code-version axis invalidates a column of the matrix, not a single cell — the same way a schema change invalidates a row of partitions in a lineage-aware system today. Third, the UI for inspecting runs needs to render as a grid, not a list, because a list hides exactly the comparison you're trying to make.
None of this requires exotic new infrastructure. Dagster's software-defined assets and partition definitions are close enough to the right shape that this could plausibly be built as a convention on top of the existing primitive rather than a rewrite. Airflow would need more work, since its DAG-run identity model is more tightly coupled to a single logical date. But the point isn't which tool wins. The point is that "pipelines-as-code" solved the problem of making ETL dependencies explicit and versioned. It has not yet solved the problem of making the second dimension of ML variation — which code produced which output over which window — equally explicit.
The Real Cost of Ignoring It
Teams that skip this don't notice the gap until someone asks a reasonable question: "are we sure the lift we're seeing came from the new feature, and not from training on three more months of data we didn't have before?" If your orchestrator can't cleanly produce both the old-code-old-window and new-code-old-window runs as comparable, addressable artifacts, you can't answer that question with confidence. You can only answer it with a story. Pipelines-as-code was supposed to replace stories with provenance. For ML workflows, it still only does that along one axis — and the other one is where most of the expensive mistakes happen.
Top comments (0)