1. What Happened?
1.1 Alert
At around 4 a.m., a series of scheduling failure alerts started coming in:
scheduler failed
projectName: 数仓平台
processName: 【ods】同步MySQL「每小时」
taskName: OPTIMIZE table_records
taskType: SQL
taskState: FAILURE
taskEndTime: 2026-03-13 04:34:01
1.2 What We Saw in the UI
After logging in to DolphinScheduler and checking the task instances, two things immediately stood out:
- Task status: Failed
- Task instance Host: Empty
So, what does an empty Host field actually mean?
No Worker node was willing to take the task.
The task never even got the chance to execute. It was rejected before it could be dispatched to a Worker.
2. Troubleshooting
2.1 Check System Resources First
The first question was straightforward: Was the machine running out of resources?
CPU usage was low, so CPU was not the bottleneck.
Memory usage, however, was a different story.
The immediate priority in production is to stop the bleeding, so the Worker was restarted first:
docker restart dolphinscheduler-worker
After the restart, the task was rerun successfully and the business recovered.
But a restart only restores service. It does not explain why the problem happened in the first place. So the investigation continued.
2.2 Check the Worker Logs
Next, we searched the Worker logs for memory-related warnings and errors:
grep -i "memory\|error\|exception" /data/dolphin/worker/logs/dolphinscheduler-worker.xxx.log | head -20
The logs told us exactly where to look:
[WARN] current cpu load average 0.03 is higher than 1.0
or available memory 0.295 is lower than 0.3
[WARN] current cpu load average 0.03 is higher than 1.0
or available memory 0.294 is lower than 0.3
The Worker heartbeat runs a resource check every 10 seconds. The available memory ratio had repeatedly dropped to 29.5%, below the configured 30% threshold.
2.3 Check the Master Logs
The Master logs showed a similar picture:
grep -i "overload\|memory\|dispatch" /data/dolphin/master/logs/dolphinscheduler-master.xxx.log | head -20
[WARN] Current available memory percentage 0.297 is too low, reserved.memory=0.3
[WARN] The current server is overload, cannot consumes commands.
[WARN] worker 10.0.1.100:1234 current cpu load average 0.0 is too high
or available memory 4.57G is too low
The Master-side behavior was now clear:
- The Master itself did not have enough available memory and stopped consuming scheduling commands.
- Workers were considered overloaded, so the Master stopped dispatching tasks to them.
In other words, the task was not failing because the SQL itself could not run. It was never successfully dispatched to a Worker in the first place.
2.4 Confirm the Memory Situation
We then checked the actual memory usage:
$ free -h
total used free shared buff/cache available
Mem: 15Gi 8.9Gi 482Mi 1.1Gi 5.9Gi 4.9Gi
Available memory was 4.9 GiB out of 15 GiB, or roughly 32%.
That was already dangerously close to the 30% threshold. Once the overnight batch workload caused even a small increase in memory usage, the available memory ratio could easily fall below the protection line.
2.5 Find Out What Was Consuming the Memory
The next step was to identify the biggest memory consumers:
ps -aux --sort=-%mem | head -n 11
The result was straightforward:
Master 4G + Worker 4G + API 1G + Alert 1G + Flink + MySQL + ZooKeeper
Everything was competing for memory on a machine with only 15 GiB of RAM.
3. Root Cause
3.1 What Actually Happened?
DolphinScheduler has a built-in memory protection mechanism:
| Metric | Threshold | Result |
|---|---|---|
| Available Memory Ratio | < reserved.memory (default: 0.3, i.e. 30%) |
Master stops consuming commands; Worker rejects tasks |
| CPU Load | > max.cpu.load.avg (default: 1.0) |
Same |
When the available memory ratio drops below 30%, the protection mechanism kicks in:
Master: "There isn't enough memory, so I won't consume scheduling commands."
Worker: "Memory is too low. Don't send me any more tasks."
Task instance: Host = empty
Result: Task fails
This explains the seemingly strange combination we saw in the UI: a failed task with an empty Host field.
The task was blocked during scheduling and dispatching because both the Master and Worker sides detected insufficient available memory.
3.2 Why Was Memory Running Low?
The machine had only 15 GiB of memory, while the four DolphinScheduler components alone were configured with a combined 10 GiB of JVM heap:
| Component | Heap Configuration | Description |
|---|---|---|
| Master | -Xmx4g |
Responsible for scheduling decisions; 4G is far more than it needs |
| Worker | -Xmx4g |
Tasks are executed in forked processes, so the Worker itself does not need 4G |
| API | -Xmx1g |
Reasonable |
| Alert | -Xmx1g |
Reasonable |
The default 4 GiB heap settings for the Master and Worker were simply too aggressive for this machine.
In practice, their actual RSS usage was only around 1–2 GiB each, meaning a significant amount of memory was reserved without being actively used.
On a resource-constrained host, those JVM heap reservations left too little headroom for the rest of the stack and made the system much more likely to trigger DolphinScheduler's memory protection mechanism.
4. How to Fix It
There are three possible approaches, listed in recommended order.
Option 1: Reduce the Master and Worker JVM Heap Size — Recommended
Add the following settings to the environment section of docker-compose.yml:
dolphinscheduler-master:
environment:
- JAVA_OPTS=-Xms2g -Xmx2g -Xmn1g
dolphinscheduler-worker:
environment:
- JAVA_OPTS=-Xms2g -Xmx2g -Xmn1g
This is the preferred approach because it addresses the underlying resource allocation issue instead of simply weakening the protection mechanism.
Option 2: Lower the Memory Protection Threshold
The default reserved.memory is 0.3, meaning DolphinScheduler reserves 30% of available memory as a safety margin.
It can be lowered to 10%:
dolphinscheduler-master:
environment:
- MASTER_RESERVED_MEMORY=0.1
dolphinscheduler-worker:
environment:
- WORKER_RESERVED_MEMORY=0.1
However, this is more of a workaround than a fundamental fix.
If the machine is genuinely short on memory, lowering the threshold does not create more memory. It simply allows the system to continue running under tighter resource conditions, which increases the risk of an OOM.
Option 3: Set mem_limit for Containers
You can also place explicit memory limits on individual containers to prevent one service from consuming too much memory and starving other components:
dolphinscheduler-master:
mem_limit: 3g
dolphinscheduler-worker:
mem_limit: 3g
This provides stronger resource isolation between containers, but it should be configured carefully according to the actual workload and memory requirements of each component.
5. Verification
In this case, we chose Option 1 and reduced the Master and Worker heap size from 4 GiB to 2 GiB.
After restarting the services, we verified the results.
5.1 Memory Availability Improved
$ free -h
total used free shared buff/cache available
Mem: 15Gi 6.8Gi 2.6Gi 1.1Gi 5.9Gi 7.1Gi
Available memory increased from 4.9 GiB to 7.1 GiB.
The available-memory ratio increased from approximately 32% to 47%, giving the system a much larger safety margin above the 30% protection threshold.
5.2 Check Actual Container Memory Usage
$ docker stats --no-stream dolphinscheduler-master dolphinscheduler-worker
CONTAINER NAME MEM USAGE / LIMIT MEM %
aec302... dolphinscheduler-master 1.098GiB / 15.34GiB 7.16%
cd6df5... dolphinscheduler-worker 1.049GiB / 15.34GiB 6.84%
The numbers looked much healthier:
- Master: 1.1 GiB actual memory usage, with a 2 GiB heap limit
- Worker: 1.05 GiB actual memory usage, with a 2 GiB heap limit
Both components had sufficient headroom for normal workloads.
5.3 Verify the JVM Parameters
Finally, we confirmed that the new JVM parameters were actually applied:
$ docker exec dolphinscheduler-master ps -ef | grep java | head -1
/opt/java/openjdk/bin/java -Xms2g -Xmx2g -Xmn1g ... MasterServer
The parameters were in effect, and the logs were back to normal. The previous memory warnings were no longer appearing.
6. Takeaways
| Item | Before Optimization | After Optimization |
|---|---|---|
| Master/Worker Heap | 4G each | 2G each |
| Available System Memory | 4.9G (32%) | 7.1G (47%) |
| Memory Protection Threshold | Frequently Triggered | Well Above the Threshold |
| Scheduling Status | Nighttime Downtime | Operating Normally |
There are four practical lessons from this incident:
DolphinScheduler's default JVM heap settings may be too large for resource-constrained machines. Always size the heap according to the available hardware and actual workload.
reserved.memory=0.3is a double-edged sword. It protects the system from running completely out of memory, but it can also prevent tasks from being dispatched when available memory falls below the safety threshold.An empty Host field does not necessarily mean a network problem. In this case, it was a sign that the task could not be dispatched because DolphinScheduler's memory protection mechanism rejected the overloaded nodes.
In production, restore service first, then investigate, and finally fix the underlying issue. Restart to stop the immediate impact, inspect the logs to identify the root cause, and adjust the resource configuration to prevent the problem from happening again.
Top comments (0)