A follow-up to the first post about dagster-prometheus-exporter, a small Go binary that polls Dagster's GraphQL API instead of pushing metrics to a Pushgateway. That post ended with "asset materialization status isn't covered yet." It is now — and getting there took three trips back to a real Dagster instance to find out that what I'd assumed wasn't quite true.
The two new metrics
dagster_asset_last_materialization_status{asset_key="customers", status="success"} 1
dagster_asset_stale_status{asset_key="customers", status="fresh"} 1
dagster_asset_last_materialization_status answers "did this asset's most recent run succeed or fail." dagster_asset_stale_status answers "is this asset's data missing, stale, or fresh relative to its upstream."
Two metrics, two GraphQL fields (assetsLatestInfo and assetNodes.staleStatus), that sounds simple. It wasn't, three separate times.
Wrong assumption #1: materializations record failures
The obvious way to get "did this asset succeed" is AssetNode.assetMaterializations — the event log of past materializations. Except it only records successful ones. An asset whose run raised an exception before writing output looks identical, in that field, to an asset that has simply never run. Both show up as nothing.
The actual answer lives one level up, on the run the asset was launched from: assetsLatestInfo.latestRun.status. That field reports SUCCESS/FAILURE/whatever regardless of whether the asset itself produced output — which is also why the metric is named ..._last_materialization_status rather than something implying it's read from assetMaterializations.
Wrong assumption #2: staleness is about time
dagster_asset_stale_status needs a real dependency chain to actually exercise the stale value — two independent assets can be missing or fresh, but stale only happens when an asset's upstream has moved on without it. So I added a small hand-written dbt project to the dev fixtures: raw_customers (seed) → stg_customers (staging view) → customers (mart), via dagster-dbt. Not vendored from dbt Labs' own jaffle_shop tutorial — written from scratch, trimmed to the one chain this needed.
First attempt at demonstrating stale: materialize the whole chain, then re-materialize just stg_customers and leave customers behind. Nothing changed — customers stayed fresh.
Turns out Dagster's staleness for dbt assets isn't timestamp-based at all. It's keyed on a code_version — a checksum of the dbt node's compiled SQL (dagster_dbt.asset_utils.default_code_version_fn, if you want to read it yourself). Re-running a model with unchanged code produces the same code_version, so nothing downstream looks stale, no matter how many times you re-run it or how much wall-clock time passes. You have to actually change the model's SQL for anything to move.
Wrong assumption #3: the running webserver sees that change
Second attempt: edit stg_customers.sql, re-materialize it, check the metric. This time stg_customers itself showed stale too — not just customers, the one I expected.
The long-running dagster dev process doesn't pick up a .sql-only edit on its own. The re-materialization ran with the new file (an out-of-process dbt build re-parses from disk every time), but the webserver serving /metrics's upstream GraphQL was still comparing against the code version it had loaded at startup. Two different views of "current," disagreeing with each other, and the metric reported the disagreement rather than either version cleanly.
The fix is a reloadRepositoryLocation mutation between editing the file and re-materializing:
mutation { reloadRepositoryLocation(repositoryLocationName: "dev-dagster-workspace") { __typename } }
With that, the third attempt finally produced the clean picture: customers=stale, stg_customers/raw_customers=fresh. Dagster's own UI says exactly why, in its own words, if you hover over the stale asset:
1 change since last materialization
1 upstream data version change:stg_customershas a new data version
That's the mechanism, stated plainly — not "time has passed," but "an upstream's data version moved and I haven't caught up."
Bonus wrong assumption, found while wiring up CI
Not part of the original three, but while asserting these metrics in the e2e test: materializing good_asset and bad_asset (two dev fixtures — one always succeeds, one always raises) in a single launchPipelineExecution call made both report status="failure". Since the metric is keyed on the run's status rather than the individual asset's step, and one failing step fails the whole run, the succeeding asset inherits the failing one's status if they're launched together. Splitting them into two separate launches fixed it — obvious in hindsight, surprising in the moment.
Here's both metrics on the actual Grafana dashboard, in the state above (customers stale, bad_asset failed, everything else clean):
What this cost
Three separate live-instance round trips for what looked, from the GraphQL schema alone, like two straightforward field reads. None of these would have shown up from reading docs or reasoning about the schema — each one only became visible by actually materializing something and looking at what came back. The project has a running list of these (issue #56, #98) for anyone building something similar against the same API.
Trying it
git clone https://github.com/HirofumiTsuda/dagster-prometheus-exporter.git
cd dagster-prometheus-exporter
docker compose up --build
brings up Dagster (with the jaffle_shop fixture pre-loaded) + the exporter + Prometheus + a pre-provisioned Grafana dashboard. The README has a full walkthrough for reproducing the stale transition by hand, including the reload step above.
Repo: https://github.com/HirofumiTsuda/dagster-prometheus-exporter — issues and PRs welcome.
Top comments (0)