Maximum DAG Concurrency of 100 Means 100 Workflows or 100 Tasks Running Simultaneously?

Concurrency control in Apache DolphinScheduler spans three distinct tiers: the Master node, individual Process Instances, and Worker nodes. The vast majority of "concurrency bottleneck" issues stem from a simple misunderstanding: mixing up what each parameter actually controls.

1. The Core Dilemma: What Does Concurrency Really Limit?

Almost all scheduling confusion comes down to one classic question: If maximum DAG concurrency is set to 100, does that mean 100 workflows running at once, or 100 parallel tasks inside a single workflow?

The official FAQ breaks this down clearly: each parameter level operates in its own isolated domain, with no overlap.

Tier	Parameter	Scope
Master Node	master.exec-threads	Caps the total number of process instances executed simultaneously on a single Master node
Process Instance	Legacy master.exec.task.number	Caps the maximum number of parallel tasks within a single workflow
Worker Node	worker.physical-task-config.task-executor-thread-size	Caps the total number of task instances executed simultaneously on a single Worker node

Concurrency Control Architecture

Once you map out this hierarchy, troubleshooting the most common failure modes becomes straightforward.

2. Right Configurations, Stuck Tasks: What’s Going On?

Even with high limits across all three tiers, tasks can still get stuck in the "Submitted Successfully" state.

Troubleshooting this requires a different approach than just tweaking concurrency parameters. According to the official FAQ, start by checking whether the WorkerServer is healthy, confirm if the Master is dispatching tasks to the Worker, and make sure the assigned Worker Group actually contains active nodes.

This is usually not a configuration error—it’s a broken execution pipeline. If any node along the path drops offline, no amount of parameter tuning will help.

If the pipeline is healthy and the Worker Group is online, but tasks are still being rejected, you are likely hitting an implicit safeguard introduced in newer releases: Server Load Protection.

3. The Silent Bottleneck: Server Load Protection

Starting in version 3.x, DolphinScheduler introduced server-load-protection. When CPU, memory, or disk usage exceeds specified thresholds, Masters or Workers proactively reject incoming tasks. To the user, the service appears normal, but tasks refuse to move forward.

Because this safety net operates independently of thread pool configurations, the two layer together. It is easy to mistake load shedding for misconfigured thread caps. Recognizing how they interact is key to targeted troubleshooting.

4. Production Tuning: Master vs. Worker Configuration

Master Side (master-server/conf/application.yaml)

Key concurrency controls live here:

• master.exec-threads (Default: 100): The main valve controlling total concurrent workflows.
• master.pre-exec-threads (Default: 10): Limits the number of commands prepared for execution in parallel.
• master.dispatch-task-number (Default: 3): Sets how many tasks are dispatched to Workers per batch. Setting this too low creates dispatch throttling.
• master.server-load-protection.*: CPU, memory, and disk usage thresholds default to 0.7—often an unexpected bottleneck.
• master.server-load-protection.max-concurrent-workflow-instances: Hard cap on total concurrent workflow instances managed by the Master.

Worker Side (worker-server/conf/application.yaml)

On the Worker side, throughput depends on:

• worker.physical-task-config.task-executor-thread-size (Default: 100): Sets the max concurrent tasks a single Worker can execute.
• worker.server-load-protection.*: Defaults to an 0.8 resource utilization threshold, acting as an implicit concurrency cap.

In test environments, these numbers are often dialed down for stress testing (e.g., setting exec-threads to 10). This highlights a key rule: never copy-paste configurations blindly—tailor parameters specifically for Dev, Test, and Production environments.

Kubernetes Deployments

If you deploy via Helm Charts, avoid modifying YAML configuration files directly inside containers. Map parameters to environment variables in values.yaml instead—such as MASTER_EXEC_THREADS, MASTER_EXEC_TASK_NUM, MASTER_DISPATCH_TASK_NUM, and MASTER_SERVER_LOAD_PROTECTION_ENABLED. Check the Helm README for full parameter mappings.

5. Step-by-Step Troubleshooting Checklist

Putting it all together, follow this sequence when diagnosing scheduling bottlenecks:

1. Pinpoint the Bottleneck Tier: Determine if queuing happens at the workflow level (adjust master.exec-threads), the task level within a workflow (adjust pre-exec-threads / task concurrency), or the Worker level (adjust task-executor-thread-size).
2. Check for Load Protection Interventions: Inspect Master and Worker logs for rejection entries triggered by overload protection. If system resources are healthy, increase the max-*-usage-percentage-thresholds values accordingly.
3. Verify Worker Group Status: If settings look fine but tasks keep queuing up, check for offline machines in the designated Worker Group.
4. Tune Incrementally: Start by scaling up dispatch-task-number to monitor throughput changes before touching exec-threads. Avoid sudden spikes in thread counts to protect your database and network from overload.
5. Stick to values.yaml in K8s: Never edit runtime files inside Pods directly; any custom configuration will be wiped out when Pods restart.

Notes

• Legacy releases (1.2.x) relied on master.exec.threads, master.exec.task.number, and worker.exec.threads in master.properties/worker.properties. Version 3.x moved these into application.yaml as master.exec-threads and worker.physical-task-config.task-executor-thread-size. Watch out for these naming changes when upgrading instead of reusing old configs.
• server-load-protection functions as an independent tier on top of thread limits. Because both impact scheduling throughput simultaneously, it is often missed during debugging.
• Due to index boundaries, the complete configuration.md text is omitted here. Pull the complete documentation using a Devin session to verify exact default values before making production changes.