DEV Community: Hirofumi Tsuda

Adding asset staleness metrics to a Dagster Prometheus exporter — and three ways I got it wrong first

Hirofumi Tsuda — Tue, 25 Aug 2026 04:32:14 +0000

A follow-up to the first post about dagster-prometheus-exporter, a small Go binary that polls Dagster's GraphQL API instead of pushing metrics to a Pushgateway. That post ended with "asset materialization status isn't covered yet." It is now — and getting there took three trips back to a real Dagster instance to find out that what I'd assumed wasn't quite true.

The two new metrics

dagster_asset_last_materialization_status{asset_key="customers", status="success"} 1
dagster_asset_stale_status{asset_key="customers", status="fresh"} 1

dagster_asset_last_materialization_status answers "did this asset's most recent run succeed or fail." dagster_asset_stale_status answers "is this asset's data missing, stale, or fresh relative to its upstream."

Two metrics, two GraphQL fields (assetsLatestInfo and assetNodes.staleStatus), that sounds simple. It wasn't, three separate times.

Wrong assumption #1: materializations record failures

The obvious way to get "did this asset succeed" is AssetNode.assetMaterializations — the event log of past materializations. Except it only records successful ones. An asset whose run raised an exception before writing output looks identical, in that field, to an asset that has simply never run. Both show up as nothing.

The actual answer lives one level up, on the run the asset was launched from: assetsLatestInfo.latestRun.status. That field reports SUCCESS/FAILURE/whatever regardless of whether the asset itself produced output — which is also why the metric is named ..._last_materialization_status rather than something implying it's read from assetMaterializations.

Wrong assumption #2: staleness is about time

dagster_asset_stale_status needs a real dependency chain to actually exercise the stale value — two independent assets can be missing or fresh, but stale only happens when an asset's upstream has moved on without it. So I added a small hand-written dbt project to the dev fixtures: raw_customers (seed) → stg_customers (staging view) → customers (mart), via dagster-dbt. Not vendored from dbt Labs' own jaffle_shop tutorial — written from scratch, trimmed to the one chain this needed.

First attempt at demonstrating stale: materialize the whole chain, then re-materialize just stg_customers and leave customers behind. Nothing changed — customers stayed fresh.

Turns out Dagster's staleness for dbt assets isn't timestamp-based at all. It's keyed on a code_version — a checksum of the dbt node's compiled SQL (dagster_dbt.asset_utils.default_code_version_fn, if you want to read it yourself). Re-running a model with unchanged code produces the same code_version, so nothing downstream looks stale, no matter how many times you re-run it or how much wall-clock time passes. You have to actually change the model's SQL for anything to move.

Wrong assumption #3: the running webserver sees that change

Second attempt: edit stg_customers.sql, re-materialize it, check the metric. This time stg_customers itself showed stale too — not just customers, the one I expected.

The long-running dagster dev process doesn't pick up a .sql-only edit on its own. The re-materialization ran with the new file (an out-of-process dbt build re-parses from disk every time), but the webserver serving /metrics's upstream GraphQL was still comparing against the code version it had loaded at startup. Two different views of "current," disagreeing with each other, and the metric reported the disagreement rather than either version cleanly.

The fix is a reloadRepositoryLocation mutation between editing the file and re-materializing:

mutation { reloadRepositoryLocation(repositoryLocationName: "dev-dagster-workspace") { __typename } }

With that, the third attempt finally produced the clean picture: customers=stale, stg_customers/raw_customers=fresh. Dagster's own UI says exactly why, in its own words, if you hover over the stale asset:

1 change since last materialization
1 upstream data version change: stg_customers has a new data version

That's the mechanism, stated plainly — not "time has passed," but "an upstream's data version moved and I haven't caught up."

Bonus wrong assumption, found while wiring up CI

Not part of the original three, but while asserting these metrics in the e2e test: materializing good_asset and bad_asset (two dev fixtures — one always succeeds, one always raises) in a single launchPipelineExecution call made both report status="failure". Since the metric is keyed on the run's status rather than the individual asset's step, and one failing step fails the whole run, the succeeding asset inherits the failing one's status if they're launched together. Splitting them into two separate launches fixed it — obvious in hindsight, surprising in the moment.

Here's both metrics on the actual Grafana dashboard, in the state above (customers stale, bad_asset failed, everything else clean):

What this cost

Three separate live-instance round trips for what looked, from the GraphQL schema alone, like two straightforward field reads. None of these would have shown up from reading docs or reasoning about the schema — each one only became visible by actually materializing something and looking at what came back. The project has a running list of these (issue #56, #98) for anyone building something similar against the same API.

Trying it

git clone https://github.com/HirofumiTsuda/dagster-prometheus-exporter.git
cd dagster-prometheus-exporter
docker compose up --build

brings up Dagster (with the jaffle_shop fixture pre-loaded) + the exporter + Prometheus + a pre-provisioned Grafana dashboard. The README has a full walkthrough for reproducing the stale transition by hand, including the reload step above.

Repo: https://github.com/HirofumiTsuda/dagster-prometheus-exporter — issues and PRs welcome.

A Prometheus exporter for Dagster that polls GraphQL instead of pushing to Pushgateway

Hirofumi Tsuda — Wed, 12 Aug 2026 12:39:22 +0000

Dagster doesn't expose a /metrics endpoint out of the box. The officially documented way to get Dagster metrics into Prometheus is the dagster-prometheus resource, which pushes metrics to a Pushgateway from inside a run.

That works, but it has a structural blind spot: the push only happens if code inside the run gets to call it.

A run that's OOM-killed, or crashes before reaching the push call, never reports anything. The failure is silent from Prometheus's point of view.
A run sitting in QUEUED because of a run-queue concurrency limit hasn't started user code yet, so it can't push either — you can't see queue backlog forming.
Prometheus's own docs are explicit that Pushgateway isn't meant to be a general substitute for pull-based scraping, only for short-lived batch jobs that genuinely can't be scraped.

So I built dagster-prometheus-exporter: a small standalone Go binary that polls Dagster's GraphQL API on an interval and derives metrics from whatever state Dagster itself already has — including runs that never got a chance to push anything.

Architecture

Dagster (GraphQL) <--poll-- exporter --scrape--> Prometheus --> Grafana

The exporter keeps in-memory state, not an external store. Scraping (writing state) and serving /metrics (reading state) are decoupled, so if a GraphQL call is slow or fails, /metrics still serves the last-known state instead of breaking. Completed runs are fetched incrementally (watermark-based, not a full rescan every time), so the cost doesn't grow with total run history.

What it currently covers

Metric	What it answers
`dagster_active_runs`	How many runs are queued/starting/started, per job
`dagster_active_run_duration_seconds`	How long the oldest active run in a job has been stuck there — useful for spotting stalls
`dagster_completed_runs_total` / `dagster_last_run_info` / `dagster_last_run_duration_seconds`	Success/failure counts and timing for completed runs
`dagster_run_queue_concurrency_key_backlog`	Queue backlog per `dagster/concurrency_key` tag — this is the one that needed the most digging, since `instance.concurrencyLimits` in the GraphQL schema looks like the answer but is actually a separate op-pool concurrency mechanism and reports 0 regardless of run-queue backlog. Ended up reading each queued run's own tags instead.
`dagster_code_location_load_error`	Whether a code location is currently failing to load (e.g. broken import) — independent of job-level metrics, since a broken location can't be inferred from run counts
`dagster_schedule_status` / `dagster_schedule_last_tick_status`	Whether a schedule is running, and its last tick outcome
`dagster_sensor_status` / `dagster_sensor_last_tick_status`	Same, for sensors
`dagster_exporter_build_info`	Which version/commit is actually running — handy once you have more than one exporter pod

Full label reference is in the README.

Trying it out

git clone https://github.com/HirofumiTsuda/dagster-prometheus-exporter.git
cd dagster-prometheus-exporter
docker compose up --build

brings up Dagster + the exporter + Prometheus + a pre-provisioned Grafana dashboard together. For an existing Dagster deployment, there's a published image and a Helm chart:

docker run -p 9101:9101 -e DAGSTER_GRAPHQL_ENDPOINT=http://dagster:3000/graphql \
  ghcr.io/hirofumitsuda/dagster-prometheus-exporter:0.2.0

helm install my-dagster-exporter oci://ghcr.io/hirofumitsuda/charts/dagster-prometheus-exporter \
  --version 0.1.3 \
  --set env.DAGSTER_GRAPHQL_ENDPOINT=http://dagster-webserver.dagster.svc.cluster.local/graphql

What's not covered yet

Asset materialization status. I've looked into the GraphQL shape for it (assetNodes + assetsLatestInfo) and the same kind of gotcha as the concurrency backlog shows up: AssetNode.assetMaterializations only records successful materializations, so detecting a failed one means cross-referencing assetsLatestInfo.latestRun.status instead. Tracked in #56 if anyone wants to compare notes.

Repo: https://github.com/HirofumiTsuda/dagster-prometheus-exporter — issues and PRs welcome.