Think DolphinScheduler Is Down? You’ll Know with Prometheus + Grafana

Introduction

Ever had this happen? DolphinScheduler quietly goes down, your workflows stop running, and you’re sitting at your desk enjoying your coffee—until your manager suddenly asks, “Why hasn’t the data been updated yet?”

At that moment, your face probably looks just like the production service: completely stuck.

In our previous article, When Workflows Get Stuck: Troubleshooting and Preventing Task Deadlocks in Apache DolphinScheduler, we discussed an uncomfortable reality: if DolphinScheduler goes down because of blocking or a deadlock, it may not be able to alert you about the problem itself. After all, you can’t exactly expect a scheduler that has already “gone offline” to send you a farewell message.

We looked around and found surprisingly few complete, practical guides for integrating DolphinScheduler with Prometheus monitoring.

So this article fills that gap.

We’ll walk through a lightweight DolphinScheduler + Prometheus + Grafana monitoring setup that has been validated in production, covering the entire process from metric collection and alerting rules to dashboard visualization.

By the end, you should be able to find out when DolphinScheduler is in trouble—before someone else does.

1. Architecture Overview

Let’s start with the big picture. The entire monitoring pipeline can be broken down into three simple steps:

Step	Component	Responsibility
1	DolphinScheduler	Exposes the /actuator/prometheus metrics endpoint
2	Prometheus	Periodically scrapes metrics and evaluates alerting rules
3	Grafana	Provides visual dashboards for an at-a-glance view of scheduler health

In simple terms:

DolphinScheduler collects the vital signs, Prometheus monitors them, and Grafana turns the data into something you can actually see.

With the three working together, you can finally enjoy your coffee with a little more peace of mind.

This time, for real.

2. Integrating Prometheus

2.1 Verify That DolphinScheduler Exposes Metrics

DolphinScheduler 2.0.0 and later include a built-in Prometheus metrics endpoint, so no additional plugin is required.

You can verify it directly with curl:

curl http://<dolphin-host>:12345/dolphinscheduler/actuator/prometheus

If the response contains a bunch of metrics beginning with ds_, congratulations—DolphinScheduler is ready to be monitored.

If you get a 404, your version may be too old. Upgrade DolphinScheduler first and then try again.

2.2 Configure Prometheus to Scrape DolphinScheduler

Add the following scrape configuration to prometheus.yml:

scrape_configs:
  # Host monitoring (optional; used to monitor the server running DolphinScheduler)
  - job_name: 'node-exporter'
    static_configs:
      - targets:
          - '<dolphin-host>:9100'
        labels:
          instance: 'dolphin-server'
          nodename: 'dolphin-server'

  # DolphinScheduler metrics
  - job_name: 'dolphinscheduler'
    static_configs:
      - targets: ['<dolphin-host>:12345']
        labels:
          service: dolphinscheduler
    metrics_path: '/dolphinscheduler/actuator/prometheus'
    metric_relabel_configs:
      - source_labels: [application]
        target_label: app
      - regex: 'application'
        action: labeldrop

Tip: Replace <dolphin-host> with the actual address of your DolphinScheduler service. If you’re running DolphinScheduler in a cluster, add all relevant nodes.

2.3 Configure Alerting Rules

This is where things get serious.

What’s the point of monitoring if nobody gets notified when something goes wrong?

Create an alert rule file named dolphinscheduler-rules.yml. The following 10 rules have been validated in production and cover scenarios ranging from “DolphinScheduler is down” to “a workflow has gone off the rails.”

# DolphinScheduler monitoring alert rules
groups:
  - name: dolphinscheduler-alerts
    rules:

      # ============ Core rule: service availability ============

      # Rule 1: No successful tasks in the past hour (likely unavailable)
      - alert: DolphinSchedulerNoSuccessfulTasksForOneHour
        expr: increase(ds_task_instance_count_total{state="success"}[1h]) == 0
        for: 5m
        labels:
          severity: critical
          group: dolphinscheduler
        annotations:
          summary: "No successful DolphinScheduler tasks in the past hour"
          description: "Instance {{ $labels.application }} has produced no successful tasks in the past hour and may be unavailable. Check the service immediately."

      # Rule 2: No successful tasks in the past 30 minutes (early warning)
      - alert: DolphinSchedulerNoSuccessfulTasksFor30Minutes
        expr: increase(ds_task_instance_count_total{state="success"}[30m]) == 0
        for: 5m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "No successful DolphinScheduler tasks in the past 30 minutes"
          description: "Instance {{ $labels.application }} has produced no successful tasks in the past 30 minutes. Check the service."

      # ============ Task health checks ============

      # Rule 3: High task failure rate
      - alert: DolphinSchedulerHighTaskFailureRate
        expr: |
          (
            increase(ds_task_instance_count_total{state="fail"}[10m])
            /
            (increase(ds_task_instance_count_total{state="finish"}[10m]) + 1)
          ) > 0.1
        for: 5m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "High DolphinScheduler task failure rate"
          description: "Instance {{ $labels.application }} has a task failure rate above 10% over the past 10 minutes. Current failure rate: {{ $value | humanizePercentage }}"

      # Rule 4: Task dispatch failures
      - alert: DolphinSchedulerTaskDispatchFailure
        expr: increase(ds_task_dispatch_failure_count_total[10m]) > 0
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "DolphinScheduler task dispatch failure"
          description: "Instance {{ $labels.application }} has experienced {{ $value }} task dispatch failures in the past 10 minutes."

      # ============ Worker health checks ============

      # Rule 5: No active Worker threads (tasks have been submitted, but nothing is executing)
      - alert: DolphinSchedulerWorkerNoActiveThreads
        expr: ds_worker_active_execute_thread == 0 and increase(ds_task_instance_count_total{state="submit"}[5m]) > 0
        for: 10m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "DolphinScheduler Worker has no active execution threads"
          description: "The Worker for instance {{ $labels.application }} has no active execution threads, while tasks have been submitted. Tasks may not be executing."

      # Rule 6: High Worker memory utilization
      - alert: DolphinSchedulerWorkerHighMemoryUsage
        expr: ds_worker_memory_usage > 0.85
        for: 5m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "High DolphinScheduler Worker memory usage"
          description: "The Worker for instance {{ $labels.application }} is using {{ $value | humanizePercentage }} memory, exceeding the 85% threshold."

      # Rule 7: High Worker CPU utilization
      - alert: DolphinSchedulerWorkerHighCPUUsage
        expr: ds_worker_cpu_usage > 0.85
        for: 5m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "High DolphinScheduler Worker CPU usage"
          description: "The Worker for instance {{ $labels.application }} is using {{ $value | humanizePercentage }} CPU, exceeding the 85% threshold."

      # ============ Workflow & Master checks ============

      # Rule 8: Long-running workflow instances (potentially blocked)
      - alert: DolphinSchedulerLongRunningWorkflows
        expr: ds_workflow_instance_running > 10
        for: 30m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "Long-running DolphinScheduler workflows"
          description: "Instance {{ $labels.application }} has {{ $value }} workflow instances running for more than 30 minutes. Possible blocking detected."

      # Rule 9: Master failover check failures
      - alert: DolphinSchedulerMasterFailoverCheckFailure
        expr: increase(ds_master_scheduler_failover_check_count_total{result="fail"}[10m]) > 0
        labels:
          severity: critical
          group: dolphinscheduler
        annotations:
          summary: "DolphinScheduler Master failover check failure"
          description: "Instance {{ $labels.application }} has recorded {{ $value }} Master failover check failures in the past 10 minutes."

      # ============ Alert channel self-check ============

      # Rule 10: Alert delivery failures (monitoring the monitoring system)
      - alert: DolphinSchedulerAlertDeliveryFailure
        expr: increase(ds_alert_send_count_total{status="fail"}[10m]) > 5
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "DolphinScheduler alert delivery failures"
          description: "Instance {{ $labels.application }} has experienced {{ $value }} alert delivery failures in the past 10 minutes."

The 10 rules above can be grouped into four categories:

Group	Rule	Severity	Trigger Condition
Service Availability	No successful tasks for 1h	Critical	0 successful tasks within 1 hour
Service Availability	No successful tasks for 30min	Warning	0 successful tasks within 30 minutes
Task Health	High task failure rate	Warning	Task failure rate > 10% over 10 minutes
Task Health	Task dispatch failure	Warning	Any task dispatch failure within 10 minutes
Worker	No active threads	Warning	Active thread count = 0
Worker	Memory > 85%	Warning	Threshold exceeded for 5 consecutive minutes
Worker	CPU > 85%	Warning	Threshold exceeded for 5 consecutive minutes
Workflow	Long-running workflows	Warning	Running instances > 10 for 30 consecutive minutes
Master	Abnormal failover	Critical	Failover check fails
Self-Monitoring	Alert delivery failure	Warning	More than 5 failures within 10 minutes

2.4 Verify That Alerts Work

When DolphinScheduler stops running or workflows become stalled for some reason, the corresponding alerts will be triggered automatically.

Now, when the service goes down, you can find out immediately instead of waiting until your manager finds out first.

That’s the whole point of monitoring.

3. Building a Grafana Dashboard

Alerts alone are not enough. You also need a dashboard that gives you a quick visual overview of the scheduler's health.

That’s where Grafana comes in.

Prerequisite: Grafana must already be connected to Prometheus as a data source. If it isn’t, go to Grafana → Configuration → Data Sources and add Prometheus.

3.1 Import the Dashboard

Go to:

Grafana → Dashboards → New → Import

Option 1: Import by Dashboard ID

This is the recommended approach.

On the import page, enter the dashboard ID:

24841

Then click Load.

This is the easiest option. One number is all it takes, and you can skip the hassle of manually configuring the dashboard JSON.

Option 2: Import from a JSON Template

If your Grafana instance cannot access the public internet—for example, in a corporate network environment—you can manually import the following JSON template.

The JSON is quite long, so the compressed version is provided below. Copy it directly into Grafana’s JSON import field:

{"__inputs":[{"name":"DS_PROMETHEUS","label":"Prometheus","description":"","type":"datasource","pluginId":"prometheus","pluginName":"Prometheus"}],"__requires":[{"type":"grafana","id":"grafana","name":"Grafana","version":"9.0.0"},{"type":"datasource","id":"prometheus","name":"Prometheus","version":"1.0.0"},{"type":"panel","id":"stat","name":"Stat","version":""},{"type":"panel","id":"gauge","name":"Gauge","version":""},{"type":"panel","id":"timeseries","name":"Time series","version":""},{"type":"panel","id":"piechart","name":"Pie chart","version":""},{"type":"panel","id":"bargauge","name":"Bar gauge","version":""}],"annotations":{"list":[{"builtIn":1,"datasource":{"type":"datasource","uid":"grafana"},"enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"editable":true,"fiscalYearStartMonth":0,"graphTooltip":0,"id":null,"links":[],"liveNow":false,"panels":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"short"},"overrides":[]},"gridPos":{"h":4,"w":4,"x":0,"y":0},"id":1,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"textMode":"auto"},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_task_instance_count_total{state=\"success\"}","refId":"A"}],"title":"Task Success Total","type":"stat"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":1}]},"unit":"short"},"overrides":[]},"gridPos":{"h":4,"w":4,"x":4,"y":0},"id":2,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"textMode":"auto"},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_task_instance_count_total{state=\"fail\"}","refId":"A"}],"title":"Task Failure Total","type":"stat"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"blue","value":null}]},"unit":"short"},"overrides":[]},"gridPos":{"h":4,"w":4,"x":8,"y":0},"id":3,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"textMode":"auto"},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_workflow_instance_running","refId":"A"}],"title":"Running Workflows","type":"stat"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"max":1,"min":0,"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"yellow","value":0.7},{"color":"red","value":0.85}]},"unit":"percentunit"},"overrides":[]},"gridPos":{"h":4,"w":6,"x":12,"y":0},"id":4,"options":{"orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"showThresholdLabels":false,"showThresholdMarkers":true},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_worker_memory_usage","refId":"A"}],"title":"Worker Memory Usage","type":"gauge"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"max":1,"min":0,"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"yellow","value":0.7},{"color":"red","value":0.85}]},"unit":"percentunit"},"overrides":[]},"gridPos":{"h":4,"w":6,"x":18,"y":0},"id":5,"options":{"orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"showThresholdLabels":false,"showThresholdMarkers":true},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_worker_cpu_usage","refId":"A"}],"title":"Worker CPU Usage","type":"gauge"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"short"},"overrides":[{"matcher":{"id":"byName","options":"success"},"properties":[{"id":"color","value":{"fixedColor":"green","mode":"fixed"}}]},{"matcher":{"id":"byName","options":"fail"},"properties":[{"id":"color","value":{"fixedColor":"red","mode":"fixed"}}]}]},"gridPos":{"h":8,"w":12,"x":0,"y":4},"id":6,"options":{"legend":{"calcs":["last","max"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_task_instance_count_total{state=\"success\"}[5m])","legendFormat":"success","refId":"A"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_task_instance_count_total{state=\"fail\"}[5m])","legendFormat":"fail","refId":"B"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_task_instance_count_total{state=\"timeout\"}[5m])","legendFormat":"timeout","refId":"C"}],"title":"Task Execution Trend (5-Minute Increment)","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"hideFrom":{"tooltip":false,"viz":false,"legend":false}},"mappings":[],"unit":"short"},"overrides":[]},"gridPos":{"h":8,"w":6,"x":12,"y":4},"id":7,"options":{"legend":{"displayMode":"table","placement":"right","showLegend":true,"values":["value","percent"]},"pieType":"pie","tooltip":{"mode":"single","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_task_instance_count_total","legendFormat":"{{state}}","refId":"A"}],"title":"Task Status Distribution","type":"piechart"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"max":1,"min":0,"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"yellow","value":0.05},{"color":"red","value":0.1}]},"unit":"percentunit"},"overrides":[]},"gridPos":{"h":8,"w":6,"x":18,"y":4},"id":8,"options":{"orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"showThresholdLabels":false,"showThresholdMarkers":true},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"rate(ds_task_instance_count_total{state=\"fail\"}[10m]) / (rate(ds_task_instance_count_total{state=\"finish\"}[10m]) + 0.001)","refId":"A"}],"title":"Task Failure Rate (10 Minutes)","type":"gauge"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"blue","value":null}]},"unit":"short"},"overrides":[]},"gridPos":{"h":8,"w":12,"x":0,"y":12},"id":9,"options":{"displayMode":"gradient","minVizHeight":10,"minVizWidth":0,"orientation":"horizontal","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"showUnfilled":true},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_task_execution_count_by_type_total > 0","legendFormat":"{{task_type}}","refId":"A"}],"title":"Task Type Distribution","type":"bargauge"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"short"},"overrides":[]},"gridPos":{"h":8,"w":12,"x":12,"y":12},"id":10,"options":{"legend":{"calcs":["last"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_workflow_instance_running","legendFormat":"Running","refId":"A"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_workflow_instance_count_total{state=\"success\"}[5m])","legendFormat":"Success-{{process_definition_code}}","refId":"B"}],"title":"Workflow Instance Trend","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"short"},"overrides":[{"matcher":{"id":"byName","options":"Memory Usage"},"properties":[{"id":"unit","value":"percentunit"},{"id":"custom.axisPlacement","value":"right"}]},{"matcher":{"id":"byName","options":"CPU Usage"},"properties":[{"id":"unit","value":"percentunit"},{"id":"custom.axisPlacement","value":"right"}]}]},"gridPos":{"h":8,"w":12,"x":0,"y":20},"id":11,"options":{"legend":{"calcs":["last","max"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_worker_active_execute_thread","legendFormat":"Active Threads","refId":"A"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_worker_memory_usage","legendFormat":"Memory Usage","refId":"B"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_worker_cpu_usage","legendFormat":"CPU Usage","refId":"C"}],"title":"Worker Resource Monitoring","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"short"},"overrides":[{"matcher":{"id":"byName","options":"Failure"},"properties":[{"id":"color","value":{"fixedColor":"red","mode":"fixed"}}]}]},"gridPos":{"h":8,"w":12,"x":12,"y":20},"id":12,"options":{"legend":{"calcs":["last","max"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_alert_send_count_total{status=\"success\"}[5m])","legendFormat":"Success","refId":"A"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_alert_send_count_total{status=\"fail\"}[5m])","legendFormat":"Failure","refId":"B"}],"title":"Alert Delivery Trend (5-Minute Increment)","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"s"},"overrides":[]},"gridPos":{"h":8,"w":12,"x":0,"y":28},"id":13,"options":{"legend":{"calcs":["mean","max"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"rate(http_server_requests_seconds_sum{uri=~\"/projects.*\"}[5m]) / rate(http_server_requests_seconds_count{uri=~\"/projects.*\"}[5m])","legendFormat":"{{uri}} - {{method}}","refId":"A"}],"title":"Average API Response Time","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"reqps"},"overrides":[]},"gridPos":{"h":8,"w":12,"x":12,"y":28},"id":14,"options":{"legend":{"calcs":["mean","max"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"rate(http_server_requests_seconds_count{uri=~\"/projects.*\"}[5m])","legendFormat":"{{uri}} - {{method}}","refId":"A"}],"title":"API Request Rate","type":"timeseries"}],"refresh":"30s","schemaVersion":38,"style":"dark","tags":["dolphinscheduler","big-data"],"templating":{"list":[{"current":{"selected":false,"text":"Prometheus","value":"Prometheus"},"hide":0,"includeAll":false,"label":"Data Source","multi":false,"name":"datasource","options":[],"query":"prometheus","refresh":1,"regex":"","skipUrlSync":false,"type":"datasource"}]},"time":{"from":"now-6h","to":"now"},"timepicker":{"refresh_intervals":["10s","30s","1m","5m","15m","30m","1h","2h","1d"]},"timezone":"browser","title":"DolphinScheduler Monitoring Dashboard","uid":"dolphinscheduler-overview","version":1,"weekStart":""}

3.2 Dashboard Preview

Once the import is complete, you’ll have a full monitoring dashboard containing the following panels:

Panel	Type	Monitoring Content
Total Success/Failed Tasks	Stat Card	Overview of global task counts
Running Workflows	Stat Card	Number of currently active workflows
Worker Memory/CPU Usage	Gauge	Red/Yellow/Green indicators; turns red above 85%
Task Execution Trend	Time-series Line Chart	Success/Failure/Timeout 5-minute increments
Task Status Distribution	Pie Chart	Proportion of each task status
Task Failure Rate	Gauge	10-minute rolling failure rate
Task Type Distribution	Bar Chart	Task count by type (SQL / Shell / Python)
Workflow Instance Trend	Time-series Line Chart	Workflow execution trends
Worker Resource Monitoring	Time-series Line Chart	Integrated view of Thread Count + Memory + CPU
Alert Notification Trend	Time-series Line Chart	Count of successful/failed alert notifications
API Response Time & Request Rate	Time-series Line Chart	DolphinScheduler API performance

Here’s what the final dashboard looks like:

4. Conclusion

Let’s recap what we built in this article.

1. Metric collection: Prometheus scrapes DolphinScheduler’s built-in /actuator/prometheus endpoint.
2. Alerting: 10 alert rules cover service availability, task health, Worker status, and Master failover.
3. Visualization: Grafana dashboard ID 24841 lets you import the monitoring dashboard with just a few clicks.

If you’re running DolphinScheduler, it’s worth setting up monitoring like this.

After all, being woken up by an alert at 3 a.m. is still better than being woken up by your manager the next morning.

If this guide helps you avoid even one “DolphinScheduler went down and nobody noticed” incident, consider giving it a like or saving it for later.

And if you have questions or improvements, drop them in the comments. Let’s make data engineering a little less painful—one production pitfall at a time.