A Prometheus exporter for Dagster that polls GraphQL instead of pushing to Pushgateway

Hirofumi Tsuda — Wed, 12 Aug 2026 12:39:22 +0000

Dagster doesn't expose a /metrics endpoint out of the box. The officially documented way to get Dagster metrics into Prometheus is the dagster-prometheus resource, which pushes metrics to a Pushgateway from inside a run.

That works, but it has a structural blind spot: the push only happens if code inside the run gets to call it.

A run that's OOM-killed, or crashes before reaching the push call, never reports anything. The failure is silent from Prometheus's point of view.
A run sitting in QUEUED because of a run-queue concurrency limit hasn't started user code yet, so it can't push either — you can't see queue backlog forming.
Prometheus's own docs are explicit that Pushgateway isn't meant to be a general substitute for pull-based scraping, only for short-lived batch jobs that genuinely can't be scraped.

So I built dagster-prometheus-exporter: a small standalone Go binary that polls Dagster's GraphQL API on an interval and derives metrics from whatever state Dagster itself already has — including runs that never got a chance to push anything.

Architecture

Dagster (GraphQL) <--poll-- exporter --scrape--> Prometheus --> Grafana

The exporter keeps in-memory state, not an external store. Scraping (writing state) and serving /metrics (reading state) are decoupled, so if a GraphQL call is slow or fails, /metrics still serves the last-known state instead of breaking. Completed runs are fetched incrementally (watermark-based, not a full rescan every time), so the cost doesn't grow with total run history.

What it currently covers

Metric	What it answers
`dagster_active_runs`	How many runs are queued/starting/started, per job
`dagster_active_run_duration_seconds`	How long the oldest active run in a job has been stuck there — useful for spotting stalls
`dagster_completed_runs_total` / `dagster_last_run_info` / `dagster_last_run_duration_seconds`	Success/failure counts and timing for completed runs
`dagster_run_queue_concurrency_key_backlog`	Queue backlog per `dagster/concurrency_key` tag — this is the one that needed the most digging, since `instance.concurrencyLimits` in the GraphQL schema looks like the answer but is actually a separate op-pool concurrency mechanism and reports 0 regardless of run-queue backlog. Ended up reading each queued run's own tags instead.
`dagster_code_location_load_error`	Whether a code location is currently failing to load (e.g. broken import) — independent of job-level metrics, since a broken location can't be inferred from run counts
`dagster_schedule_status` / `dagster_schedule_last_tick_status`	Whether a schedule is running, and its last tick outcome
`dagster_sensor_status` / `dagster_sensor_last_tick_status`	Same, for sensors
`dagster_exporter_build_info`	Which version/commit is actually running — handy once you have more than one exporter pod

Full label reference is in the README.

Trying it out

git clone https://github.com/HirofumiTsuda/dagster-prometheus-exporter.git
cd dagster-prometheus-exporter
docker compose up --build

brings up Dagster + the exporter + Prometheus + a pre-provisioned Grafana dashboard together. For an existing Dagster deployment, there's a published image and a Helm chart:

docker run -p 9101:9101 -e DAGSTER_GRAPHQL_ENDPOINT=http://dagster:3000/graphql \
  ghcr.io/hirofumitsuda/dagster-prometheus-exporter:0.2.0

helm install my-dagster-exporter oci://ghcr.io/hirofumitsuda/charts/dagster-prometheus-exporter \
  --version 0.1.3 \
  --set env.DAGSTER_GRAPHQL_ENDPOINT=http://dagster-webserver.dagster.svc.cluster.local/graphql

What's not covered yet

Asset materialization status. I've looked into the GraphQL shape for it (assetNodes + assetsLatestInfo) and the same kind of gotcha as the concurrency backlog shows up: AssetNode.assetMaterializations only records successful materializations, so detecting a failed one means cross-referencing assetsLatestInfo.latestRun.status instead. Tracked in #56 if anyone wants to compare notes.

Repo: https://github.com/HirofumiTsuda/dagster-prometheus-exporter — issues and PRs welcome.

DEV Community: Hirofumi Tsuda

A Prometheus exporter for Dagster that polls GraphQL instead of pushing to Pushgateway

Architecture

What it currently covers

Trying it out

What's not covered yet