Dagster doesn't have a built-in way to get OpenTelemetry traces out of a run. There's an open issue on the main repo asking for it (dagster-io/dagster#11191), and a more specific one asking for trace/span IDs correlated into log lines (#12353). Neither has shipped.
The workarounds I could find were monkeypatching Dagster internals, or wrapping every @op/@asset in a third-party decorator that effectively takes ownership of the function away from Dagster's own decorators. Neither felt right for something I wanted to actually run.
So I wrote dagster-otel: a small decorator, @traced(), that stacks under @op/@asset instead of replacing them.
from dagster import asset, job, op
from dagster_otel import traced
@op(...)
@traced()
def upstream_op(context) -> int:
...
@op(...)
@traced()
def downstream_op(context, x: int) -> int:
...
@asset(...)
@traced()
def downstream_asset(context) -> None:
...
@job(...)
def my_job():
downstream_op(upstream_op())
No @resource/required_resource_keys to wire up, no manual "which step is the root" bookkeeping. The first @traced() step to run in a given run just becomes the root of that run's trace.
The part that took the most verification: cross-process propagation
Dagster's multiprocess and k8s_job_executor executors usually run each step in its own process, sometimes on its own pod. That means the usual in-memory OTel context propagation doesn't work across steps. dagster-otel propagates trace context through Dagster's own run storage instead.
I didn't want to ship that unverified. What's actually checked against real Dagster runs, not just unit tests with mocks:
-
multiprocessexecutor -
k8s_job_executor(real pods, not a local stand-in) - retry-from-failure runs
- a multi-root, fan-in graph (two independent roots merging into one downstream op) -- the merged op gets a real parent from one root and a
Linkto the other, so both relationships stay visible on the span instead of picking one arbitrarily
dbt: one span per model, not one opaque span for the whole dbt build
If you're running dbt through @dbt_assets, dagster_otel.dbt.traced_dbt() is a drop-in replacement for @traced() that also opens a child span per dbt node (model, seed, test), keyed by the real Dagster asset_key/check_name:
from dagster_otel.dbt import traced_dbt
@dbt_assets(manifest=...)
@traced_dbt()
def my_dbt_assets(context, dbt: DbtCliResource):
yield from dbt.cli(["build"], context=context).stream()
On the included jaffle_shop example, that's 16 spans instead of 1: a step span, 3 asset spans, 12 check spans, each with the actual duration dbt itself measured.
Configuration is just standard OTel env vars
OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT (or _TRACES_ENDPOINT), nothing bespoke. If neither is set, no real OTLP exporter gets created at all -- spans still get created (propagation and log correlation keep working), they just never go anywhere. So trying @traced() with zero setup doesn't make a surprise network call.
Transport defaults to gRPC, but OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf switches to plain HTTP if that's what your collector or network actually accepts.
What's verified and what isn't
Jaeger is the only backend I've actually run this against end to end. configure() builds a plain OTLPSpanExporter() with no backend-specific code, so anything speaking OTLP should work, but "should" isn't the same as checked. Grafana Tempo and SigNoz are on the list (#42, #43) but not done yet.
Pinned and verified against Dagster 1.13.22 and opentelemetry-sdk 1.44.0. pyproject.toml declares a wider floor (dagster >= 1.5) since nothing here relies on version-specific internals beyond an accepted-risk private-API dependency documented in the repo, but that wider range isn't individually spot-checked yet.
pip install dagster-otel
MIT licensed. Repo, design rationale, and the verification write-ups are at github.com/HirofumiTsuda/dagster-otel. Bug reports and feature requests go through GitHub issues there.
Top comments (0)