"Why use dbt if I have Dagster?" is the number-two Google result for this topic, and it's a Reddit thread. That tells you the question is real. People see two tools that both draw a lineage graph, both run on a schedule, and both run tests, and they wonder if they are paying twice.
They aren't, quite. dbt is a SQL transformation framework. Dagster is an orchestrator. They overlap in the middle, and the overlap is where the confusion lives. My own production pipeline runs on Dagster with no dbt at all, so I'll also say when skipping dbt is the right call.
What each tool actually does
dbt takes a folder of SELECT statements with Jinja templating. It works out their order from ref() calls, compiles them for your warehouse, runs them as tables or views, and runs tests (not_null, unique, custom SQL) against the results. It does not schedule itself. Something has to call dbt build.
Dagster decides when things run, records what each run produced, retries failures, tracks partitions, and runs arbitrary Python. It has its own data-quality checks (asset checks) and its own lineage graph, built from the dependencies you declare.
The overlap: both have a lineage graph and both have tests. The difference: dbt's graph covers SQL models in one warehouse. Dagster's graph covers everything, including the Python that loads data before dbt runs and the models you train after it.
Why use dbt if you have Dagster
You still want dbt when:
-
Most of your transformations are SQL in one warehouse. dbt's templating,
ref(), incremental materializations and the docs site are a decade of work you would otherwise rebuild badly. - Analysts write the models. dbt is the tool analysts already know. Asking them to write Dagster assets in Python is a bigger change than it sounds.
- You want the project to be portable. A dbt project runs under Dagster, Airflow, Prefect, cron or dbt Cloud. Your SQL is not locked to the orchestrator.
You can skip dbt when the transformations are mostly Python, or when one engineer writes everything and is happy to keep the SQL inside Python assets. That is my situation, below.
This is the first part. The full post — including the rest of the working details — is on my site: Dagster and dbt: do you need both?
Top comments (0)