If you run Dagster pipelines in production, you've probably wanted distributed tracing at some point — seeing exactly how long each op/asset took, how steps nest across a run, and how a run connects to whatever triggered it (a sensor tick, a CI pipeline, another OTel-instrumented service upstream).
A while back I built dagster-otel: a @traced() decorator you stack under @op/@asset, explicit and opt-in, no monkeypatching. It works well, but it means touching every function you want traced.
This post is about the companion package I built on top of it: opentelemetry-instrumentation-dagster — zero-code tracing. No decorators, no imports in your pipeline files, nothing.
The idea
pip install opentelemetry-instrumentation-dagster
OTEL_SERVICE_NAME=my_pipeline \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
opentelemetry-instrument dagster dev -f definitions.py
That's it. Run your existing Dagster command through the standard opentelemetry-instrument launcher (from the OTel Python ecosystem) instead of running it directly, and every @op, @asset, @multi_asset, @asset_check, and @dbt_assets in definitions.py gets a span — automatically. Your definitions.py never imports this package or dagster_otel at all.
from dagster import asset, job, op
@op
def upstream_op(context) -> int:
return 1
@op
def downstream_op(context, x: int) -> int:
return x + 1
@asset
def my_asset(context) -> None:
...
Zero changes to this file. Every function above still gets a span the moment it runs under the launcher.
Why a separate package, not just a flag on dagster-otel
dagster-otel's whole pitch is explicit instrumentation — you decide what gets traced, by writing @traced() yourself, no framework internals touched. Auto-instrumentation is the opposite trade-off: zero code changes, in exchange for monkeypatching and losing per-function visibility. Both are legitimate, but they're different products for different people — the same reasoning the OpenTelemetry Python ecosystem itself follows (opentelemetry-instrumentation-flask, -django, etc. are all separate packages from the manual API/SDK, not toggles on it).
How it actually works
The naive approach would be reaching into an already-built AssetsDefinition and swapping its compute function after the fact — but that means mutating a non-public attribute of an object that was never meant to change post-construction.
Instead, this package patches the decorator factories themselves — dagster.op, dagster.asset, dagster.multi_asset, dagster.asset_check — public, stable API. Each one, when called, first wraps the incoming compute function with dagster_otel.traced() before handing it off to the real decorator. Since the wrapping happens before Dagster ever builds the OpDefinition/AssetsDefinition, no post-hoc mutation is needed at all — the same trick dagster-otel's own manual @traced() already relies on, just applied automatically instead of by hand.
The timing is the interesting part. The patch has to be in place before your Definitions module does from dagster import asset — otherwise you're patching a name nothing refers to anymore. opentelemetry-instrument handles this generically: it's a launcher that discovers every package registered under the opentelemetry_instrumentor entry point and calls .instrument() on each, before your own code ever imports anything.
Where it gets genuinely tricky is cross-process execution. Dagster's multiprocess executor spawns a fresh interpreter per step, which re-imports your Definitions module from scratch — a patch applied only in the original process wouldn't carry over. It turns out opentelemetry-instrument already solves this, just not obviously: it inserts a directory containing a two-line sitecustomize.py at the front of PYTHONPATH, then execl()s into your actual command. Python auto-imports any module named sitecustomize on sys.path at interpreter startup — and since spawned subprocesses inherit PYTHONPATH by default, every child re-triggers the same instrumentation independently. No explicit re-wrapping needed; it rides on Python's own site-import mechanism.
k8s_job_executor breaks this, though — each step becomes a genuinely separate Kubernetes Pod, and PYTHONPATH isn't among the env vars Dagster forwards into it. The fix there is static instead of dynamic: bake sitecustomize.py into the container image itself (copied to site-packages' root at build time), so every pod picks it up on interpreter startup regardless of what got forwarded into it. This is the same shape the OpenTelemetry Operator's own Kubernetes auto-instrumentation uses (a mutating webhook injecting PYTHONPATH into the pod spec) — static injection into the pod spec is just the normal pattern for k8s specifically.
Both paths are verified against real infrastructure, not just reasoned through — a real kind cluster with Postgres-backed run storage and k8s_job_executor, a real Jaeger receiving both spans, correctly parented across genuinely separate pods. dev/kubernetes/ in the repo has the full reproducible setup.
One deliberate exclusion: @graph_asset is not patched. Its decorated function is a composition function called once at definition time to wire up which ops depend on which — it never receives a runtime context at all, so applying traced() to it would be actively wrong, not just unnecessary. (Tracing still works for the ops it composes, for free, since those are plain @ops.)
What's covered today
| Decorator | Status |
|---|---|
@op |
✅ bare and @op(name=...) forms |
@asset |
✅ bare and @asset(name=...) forms |
@multi_asset |
✅ |
@asset_check |
✅ (added in 0.2.0) |
@dbt_assets |
✅ covered for free — it calls multi_asset internally |
@graph_asset |
⬜ deliberately excluded, see above |
What's new: @asset_check support
Until the latest release, asset checks ran with zero tracing, even in an otherwise fully auto-instrumented pipeline — @asset_check was the one decorator that had slipped through. Turns out it's keyword-only (def asset_check(*, asset, ...)), the exact same shape as @multi_asset, so the existing patch mechanism needed no new dispatch logic — just registering a fourth wrapper.
The more interesting wrinkle: AssetCheckExecutionContext (the context an @asset_check function receives at runtime) is a genuinely different shape from OpExecutionContext/AssetExecutionContext — no .job_name, no .selected_asset_keys. dagster_otel.traced() only learned to handle that context type in its own 0.4.0 release, so this package's dependency got bumped accordingly. Verified two ways: a real materialize() run in the test suite, and — closer to how anyone will actually use it — a bare script with zero instrument()/@traced() calls anywhere, run entirely through opentelemetry-instrument, confirming the patch was already live by the time the script's own import dagster executed.
Try it
git clone https://github.com/HirofumiTsuda/opentelemetry-instrumentation-dagster
cd opentelemetry-instrumentation-dagster
docker compose up -d # local Jaeger, no other setup
pip install opentelemetry-instrumentation-dagster
OTEL_SERVICE_NAME=quickstart \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
opentelemetry-instrument dagster asset materialize -f examples/definitions.py --select example_asset
Open http://localhost:16686 (Jaeger), and there's the span — from a file that never imported this package at all.
It's early-stage (MIT-licensed, alpha), but the core mechanism is verified against real Dagster execution, multiprocess, and a real k8s_job_executor cluster. If you're running Dagster and want tracing without touching every function, I'd love feedback:
- Repo: https://github.com/HirofumiTsuda/opentelemetry-instrumentation-dagster
- PyPI: https://pypi.org/project/opentelemetry-instrumentation-dagster/
- Companion project (the explicit
@traced()decorator this is built on): https://github.com/HirofumiTsuda/dagster-otel
Issues and PRs welcome — especially if you hit an execution shape I haven't tested yet.
Top comments (0)