Airflow vs Dagster vs Prefect: How to Choose

Keshav — Mon, 26 Oct 2026 19:21:04 +0000

Airflow, Dagster and Prefect are all open-source Python orchestrators. All three schedule pipelines, retry failed steps, keep a run history, show it in a web UI and run on Kubernetes or a cloud container service. "Which one is best?" has no general answer. What differs is the unit you model (a task, a data asset, or a Python function call), how much infrastructure you operate, and how far your existing code and skills already lean one way.

This lesson compares them as a decision framework. It assumes you know Airflow's basics from Dag fundamentals and TaskFlow and scheduling and assets, and does not re-teach them.

Versions compared

As of October 2026: Apache Airflow 3.x (3.3.2 used here), Dagster 1.x (1.13.26) and Prefect 3.x (3.8.8). Older comparisons are often wrong now. Airflow 3 renamed datasets to assets and added event-driven scheduling; in Prefect 3 workers are standard, replacing the agents that were deprecated during Prefect 2. Dagster stays on 1.x and its documentation now scaffolds and runs projects with the create-dagster and dg command-line tools.

The same pipeline in all three

A tiny extract, transform, load job: three order rows go in, revenue per country comes out, and it is written to SQLite with INSERT OR REPLACE, so running it twice gives the same table. In each tool the business logic is the same; only the wrapping changes.

Airflow: a Dag of tasks

import sqlite3
from airflow.sdk import dag, task

DB = "orders.db"


@dag(schedule="@daily", catchup=False)
def orders_pipeline():
    @task
    def extract() -> list[dict]:
        return [
            {"country": "IN", "amount": 120.0},
            {"country": "US", "amount": 80.0},
            {"country": "IN", "amount": 30.0},
        ]

    @task
    def transform(rows: list[dict]) -> dict[str, float]:
        totals: dict[str, float] = {}
        for r in rows:
            totals[r["country"]] = totals.get(r["country"], 0) + r["amount"]
        return totals

    @task
    def load(totals: dict[str, float]) -> None:
        with sqlite3.connect(DB) as con:
            con.execute("CREATE TABLE IF NOT EXISTS revenue (country TEXT PRIMARY KEY, total REAL)")
            con.executemany("INSERT OR REPLACE INTO revenue VALUES (?, ?)", totals.items())
        print(f"loaded {len(totals)} rows: {totals}")

    load(transform(extract()))


pipeline = orders_pipeline()

if __name__ == "__main__":
    pipeline.test()

dag.test() executes a real Dag run, so the file has to sit in the Dags folder and the metadata database must exist (airflow db migrate). Selected lines of the output (the full log is about 50 structured lines):

loaded 2 rows: {'IN': 150.0, 'US': 80.0}
... DagRun Finished: dag_id=orders_pipeline, ... run_duration=7.226455, state=success, run_type=manual, ...

The return values travel between tasks as XComs, which is fine for a small dict and wrong for a DataFrame; see XCom, Variables and Connections.

Dagster: three software-defined assets

import sqlite3
import dagster as dg

DB = "orders.db"


@dg.asset
def raw_orders() -> list[dict]:
    return [
        {"country": "IN", "amount": 120.0},
        {"country": "US", "amount": 80.0},
        {"country": "IN", "amount": 30.0},
    ]


@dg.asset
def country_revenue(raw_orders: list[dict]) -> dict[str, float]:
    totals: dict[str, float] = {}
    for r in raw_orders:
        totals[r["country"]] = totals.get(r["country"], 0) + r["amount"]
    return totals


@dg.asset
def revenue_table(country_revenue: dict[str, float]) -> dg.MaterializeResult:
    with sqlite3.connect(DB) as con:
        con.execute("CREATE TABLE IF NOT EXISTS revenue (country TEXT PRIMARY KEY, total REAL)")
        con.executemany("INSERT OR REPLACE INTO revenue VALUES (?, ?)", country_revenue.items())
    return dg.MaterializeResult(metadata={"rows": len(country_revenue)})


if __name__ == "__main__":
    result = dg.materialize([raw_orders, country_revenue, revenue_table])
    print("success:", result.success)
    print("country_revenue:", result.output_for_node("country_revenue"))

There is no explicit dependency wiring: the parameter name raw_orders is the dependency on the asset of that name. Selected output lines:

... raw_orders - Writing file at: /tmp/tmpd69noogr/storage/raw_orders using PickledObjectFilesystemIOManager...
... raw_orders - ASSET_MATERIALIZATION - Materialized value raw_orders.
... revenue_table - STEP_SUCCESS - Finished execution of step "revenue_table" in 252ms.
... RUN_SUCCESS - Finished execution of run for "__ephemeral_asset_job__".
success: True
country_revenue: {'IN': 150.0, 'US': 80.0}

Note the line about the IO manager: Dagster stored each returned value (pickled, in a temporary directory) and loaded it for the downstream asset. In production you choose an IO manager that writes to your warehouse or object store, or you write the data yourself, as revenue_table does, and return only metadata.

Prefect: a flow calling tasks

import sqlite3
from prefect import flow, task

DB = "orders.db"


@task
def extract() -> list[dict]:
    return [
        {"country": "IN", "amount": 120.0},
        {"country": "US", "amount": 80.0},
        {"country": "IN", "amount": 30.0},
    ]


@task
def transform(rows: list[dict]) -> dict[str, float]:
    totals: dict[str, float] = {}
    for r in rows:
        totals[r["country"]] = totals.get(r["country"], 0) + r["amount"]
    return totals


@task
def load(totals: dict[str, float]) -> int:
    with sqlite3.connect(DB) as con:
        con.execute("CREATE TABLE IF NOT EXISTS revenue (country TEXT PRIMARY KEY, total REAL)")
        con.executemany("INSERT OR REPLACE INTO revenue VALUES (?, ?)", totals.items())
    return len(totals)


@flow(log_prints=True)
def orders_pipeline() -> int:
    rows = load(transform(extract()))
    print(f"loaded {rows} rows")
    return rows


if __name__ == "__main__":
    orders_pipeline()

Full output:

07:22:44.864 | INFO    | prefect - Starting temporary server on http://127.0.0.1:8282
See https://docs.prefect.io/v3/concepts/server#how-to-guides for more information on running a dedicated Prefect server.
07:22:51.455 | INFO    | Flow run 'fierce-kagu' - Beginning flow run 'fierce-kagu' for flow 'orders-pipeline'
07:22:51.533 | INFO    | Task run 'extract-d55' - Finished in state Completed()
07:22:51.538 | INFO    | Task run 'transform-5d2' - Finished in state Completed()
07:22:51.549 | INFO    | Task run 'load-185' - Finished in state Completed()
07:22:51.550 | INFO    | Flow run 'fierce-kagu' - loaded 2 rows
07:22:52.532 | INFO    | Flow run 'fierce-kagu' - Finished in state Completed()
07:22:52.563 | INFO    | prefect - Stopping temporary server on http://127.0.0.1:8282

Calling the flow is just calling a Python function. With no server configured, Prefect started a temporary local API server for the run and stopped it afterwards. Data passed between tasks as ordinary return values in one process.

What the three snippets show: the Airflow and Prefect versions look almost identical, but Airflow parses the file into a Dag the scheduler stores and runs task by task, while Prefect discovers the graph as the function executes. Dagster's version names things that exist (raw_orders, country_revenue, revenue_table) instead of steps that run.

Programming model

Airflow: Dags and tasks. A Dag declares tasks and their dependencies; the Dag processor parses the file and stores a serialized version, and the scheduler creates runs of it. The graph is fixed when the file is parsed, with dynamic task mapping for fan-out at run time. Since Airflow 3, assets are first-class: a task can declare outlets, a Dag can be scheduled on assets, and the @asset decorator is shorthand for a Dag with one task that updates one asset. Assets are still attached to tasks; the task is the primary unit.

Dagster: software-defined assets. The Dagster docs define an asset as an object in persistent storage (a table, file or model) and an asset definition as code describing what should exist and how to produce it. Dependencies are declared on the assets, so the asset graph exists before anything runs, and "run this pipeline" becomes "materialize this selection of assets". Dagster also has ops, graphs and jobs for task-style work, but you do not need them to use assets.

Prefect: flows and tasks. Flows and tasks are decorated Python functions, and dependencies are whatever your code does: function calls, loops, if statements. Calling a task runs it in the caller's thread by default; .submit() and .map() send it to a task runner for concurrency (a thread pool by default, process pools, or Dask and Ray through extras). The graph is recorded at run time rather than declared up front, which makes data-dependent branching natural and makes "what will this run do?" harder to answer before it runs.

Scheduling and event triggers

	Airflow 3	Dagster 1.x	Prefect 3
Time-based	Cron and presets, timetables, data intervals with catchup and backfill	Schedules (cron) on jobs or asset selections; time-partitioned assets	Schedules (cron, interval, RRule) on deployments
Data-aware	Dags scheduled on asset updates, combined with AND and OR conditions	Declarative Automation: conditions on assets such as `on_cron`, `eager`, `on_missing`	Deployment triggers: run a deployment when matching events arrive, for example another flow run completing
External events	`AssetWatcher` on an asset with an event trigger, for example on a message queue (added in 3.0)	Sensors (polling with cursors and run keys); pushing through the GraphQL API	Automations on events; webhooks in Prefect Cloud
Absence of events	Deadline Alerts on Dag runs (experimental since 3.1)	Freshness policies (still under active development) with freshness alerts	Proactive triggers: fire when an expected event does not happen

Two details catch people out. Airflow's event-driven scheduling only accepts triggers that inherit from BaseEventTrigger, not every deferrable trigger. Dagster's Declarative Automation only acts once the default automation condition sensor is switched on in the UI. For Airflow's interval semantics (logical date, data interval, catchup), see scheduling and assets.

Data awareness and lineage

This is where the tools differ most.

Dagster puts the asset graph at the centre of the UI: each asset shows its upstreams, last materialization, metadata and asset checks (data-quality tests attached to an asset). The dbt integration loads each dbt model as an asset, so dbt models and Python assets appear in one graph and can be selected together. Dagster+ adds an asset catalog with column-level lineage.
Airflow tracks assets by URI and shows which Dags produce and consume them; asset events can carry extra metadata (for example a row count). For lineage across systems, the OpenLineage provider emits lineage events to an OpenLineage-compatible backend.
Prefect models assets with the @materialize decorator and a URI key, infers dependencies from how materializations feed each other, and tracks them across workflows. Its documentation marks assets as a Prefect Cloud feature, so check availability before planning around it on a self-hosted server.

If lineage and "is this table fresh and tested?" are the questions your stakeholders ask most, that favours Dagster's model. If they ask "did last night's run succeed?", all three answer it.

Local development and testing

Airflow: dag.test() runs a whole Dag in one process for debugging, but it needs an initialised metadata database and a Dag inside a Dag bundle (the Dags folder by default). Unit tests usually call the task's Python function or load the DagBag to check that files import. The full approach is in testing and CI/CD.
Dagster: assets can be called directly in unit tests with plain inputs and mocked resources, and materialize() runs a selection in process, as in the example. dg dev starts the local UI on a project scaffolded with create-dagster.
Prefect: flows and tasks are called like functions; prefect_test_harness() runs them against a temporary database in tests. The example ran with nothing installed beyond pip install prefect.

Prefect and Dagster have the shorter loop from "edit" to "see it run". Airflow's loop got shorter with dag.test(), but you still carry a metadata database and the parse-then-run model, which matters for tests that build Dags in memory.

Deployment and scaling

	Airflow 3	Dagster OSS	Prefect 3
Long-running services	Scheduler, Dag processor, API server, metadata database; triggerer for deferrable tasks; workers depending on executor	Webserver, daemon (schedules, sensors, run queue), code location servers, database	Prefect server (API, UI) with its database; workers for hybrid work pools
How work is placed	Executor in the scheduler: Local, Celery, Kubernetes, Edge, AWS ECS and others; several executors at once since 2.10	Run launcher starts a run worker (process, Docker, Kubernetes job, ECS task); an executor inside the run runs steps (multiprocess by default, or a pod or task per step)	Work pool holds infrastructure config; a typed worker polls it and starts each flow run (process, Docker, Kubernetes, ECS, Cloud Run and others)
Granularity of isolation	Per task	Per run, or per step with a step executor	Per flow run; tasks run inside it
User code isolation	Task SDK: tasks talk to the API server, not the metadata database	Code locations run in their own processes, separate from the webserver and daemon	Flow code is pulled from an image, Git or storage by the worker

The isolation row matters for cost and reliability. Airflow isolates tasks (with the KubernetesExecutor each task is its own pod); Prefect isolates flow runs, and tasks inside one run share its process and memory; Dagster defaults to a process per step inside a run and can go to a pod per step. In all three, heavy computation should still run in Spark, the warehouse or a container the task launches, not in the orchestrator's workers. Scaling the Airflow side is covered in executors, pools and scaling.

Managed offerings

Airflow: no first-party managed service from the Apache project. The Airflow ecosystem page lists Astronomer's Astro, Google Cloud Composer, Amazon MWAA and Azure's managed Airflow, among others. Each runs its own release schedule and supported versions, so check whether the Airflow 3 version you need is available.
Dagster+: run by the Dagster company in two forms, Serverless (your code runs in Dagster's environment) and Hybrid (Dagster+ runs the control plane, your agent and code run in your infrastructure). It adds Insights (cost and performance trends), alerting, RBAC and audit logs, the asset catalog and branch deployments.
Prefect Cloud: a hosted control plane that, per the self-hosting docs, adds organisation features such as RBAC, audit logs and SSO on top of what a self-hosted server provides. Besides hybrid work pools with your own workers, it offers push work pools (runs submitted straight to serverless cloud infrastructure, no worker) and Prefect Managed pools (runs on Prefect's infrastructure).

All three can be self-hosted. The real question is who upgrades, scales and secures the control plane, and whether your company already buys one of these services.

Ecosystem and providers

Airflow's ecosystem is the largest and oldest: more than 80 community-maintained provider packages (cloud services, databases, warehouses, dbt Cloud, Kafka, Spark and more) that bundle hooks, operators, sensors and asset URI schemes, plus third-party ones. Dagster's integration libraries (dagster-dbt, dagster-k8s, dagster-aws and many more) are built around assets, for example loading dbt models as assets. Prefect's integrations are PyPI packages (prefect-aws, prefect-gcp, prefect-kubernetes, and others) that add blocks, tasks and worker types; because a Prefect task is any function, a plain SDK call is often all you need.

If you depend on a specific operator (a vendor's transfer operator, a mature sensor), check that it exists in the other tool before assuming parity.

Observability

All three UIs show run history, per-step logs and states, and retries. The difference is the default view: Airflow centres on Dags and task instances (grid and graph views per run); Dagster on the asset graph and asset health, with runs underneath; Prefect on flow runs and their states, with automations that turn state changes (or a run stuck in Running) into notifications or actions. For alerting design that applies to any of them, see pipeline observability.

Comparison table

	Airflow 3	Dagster 1.x	Prefect 3
Unit you model	Task in a Dag	Data asset	Python function (flow, task)
Graph defined	At parse time (plus dynamic mapping)	Before run, from asset dependencies	At run time, by your code
Data passing	XCom for small values; data in external storage	IO managers, or write yourself and return metadata	Return values in the flow's process
Data-aware scheduling	Assets, asset expressions, asset watchers	Declarative Automation, asset sensors	Event triggers on deployments
Lineage	Asset graph; OpenLineage provider	Asset graph, asset checks; column lineage in Dagster+	Assets (Prefect Cloud)
Local run	`dag.test()` with a metadata DB	`materialize()`, `dg dev`	Call the flow
Execution placement	Executors (Local, Celery, Kubernetes, Edge, ECS)	Run launchers plus executors	Work pools and workers; push and managed pools in Cloud
Managed options	Astro, Cloud Composer, MWAA and others	Dagster+ Serverless and Hybrid	Prefect Cloud
Ecosystem	Largest provider ecosystem	Strong on dbt and the modern data stack	Python packages; any SDK works in a task

Migration cost

Switching orchestrators is rarely about the decorator syntax; it is everything around it.

Rewriting definitions. Each Dag, operator and sensor needs an equivalent. Task-to-task moves (Airflow to Prefect) map fairly directly; Prefect's migration guide maps Dags to flows, operators to tasks, executors to work pools and workers, and XComs to return values. Moving to Dagster means remodelling tasks as assets, which is more work and also where the benefit is.
Scheduling semantics. Airflow's data intervals, catchup and backfills by logical date have no one-to-one equivalent; Dagster uses partitions, Prefect passes parameters. Any task that relies on data_interval_start must be re-parameterised, and that is where silent bugs come from.
Connections and secrets. Airflow Connections and Variables become Dagster resources or Prefect blocks and settings.
Operations. New deployment, CI/CD, alerting, access control and on-call runbooks.
Running both. Dagster documents an AirflowInstanceComponent that peers with an existing Airflow instance so Airflow Dags appear in Dagster while you migrate incrementally. Running two orchestrators for a while is normal; plan the cut-over per pipeline, not as one big switch.

Staying put and upgrading is a legitimate option too: Airflow 2 to 3 brings assets, event-driven scheduling and the Task SDK without changing tools (see the Airflow 2 to 3 migration question).

When to choose each

If your situation is mostly...	Leaning	Why
Many teams, many systems, lots of time-partitioned batch jobs	Airflow	Mature scheduling semantics, largest provider ecosystem, managed options on every major cloud
An analytics platform built around tables, dbt models and their freshness and quality	Dagster	Assets, asset checks and lineage are the core model; dbt models become assets
Python-heavy, dynamic workflows (ML, API-driven jobs, branching on data), small platform team	Prefect	Plain functions, graph built at run time, low ceremony to go from script to scheduled deployment
A large, working Airflow estate	Airflow (upgrade to 3)	Migration cost usually outweighs feature differences; adopt assets incrementally
A handful of jobs that all live on one platform	That platform's scheduler	Databricks jobs, dbt Cloud or Step Functions may be enough

Common mistakes

Comparing Airflow 2 or Prefect 2 (with agents) against current Dagster. Compare current versions.
Treating Dagster as "Airflow with a different decorator" and then writing one asset per step; you lose the asset graph's value.
Assuming Prefect tasks are isolated like Airflow tasks on Kubernetes; by default they share the flow run's process.
Passing large data between steps (XComs in Airflow, pickled values through Dagster's default IO manager, big in-memory objects in Prefect) instead of writing to storage.
Choosing on the developer loop alone and ignoring who will operate the control plane, upgrades and on-call.
Forgetting that idempotency is your job in all three: retries and reruns are only safe if each step can run twice (idempotency in pipelines).

In an interview

For "Airflow, Dagster or Prefect?", give the framework before the answer: what you model (tasks, assets, functions), how runs are triggered, lineage and data quality needs, how the team develops and tests, who operates it, the ecosystem you depend on, and migration cost. Then commit to a recommendation for the scenario given. For example: "For a dbt-centred analytics platform where freshness and tests matter most, I would lean Dagster for its asset graph and checks; if the company already runs 300 Airflow Dags, I would upgrade to Airflow 3 and use assets instead of migrating." Expect follow-ups on Airflow's data intervals versus partitions, how each isolates work, and what you would never run inside an orchestrator worker. A shorter version of this answer is the choosing an orchestrator interview question.

Key takeaway

Airflow orchestrates tasks, Dagster orchestrates data assets, Prefect orchestrates Python functions. All three are production-ready in their current versions. Choose by the model that matches how your team thinks about its pipelines, the operations you can carry or buy, the integrations you depend on, and what it would cost to move what you already run.

Originally published on DataDank, a free Data Engineering learning site with a 90-day interview plan and 377+ interview questions.

DEV Community: Keshav