Apache DolphinScheduler Zombie Tasks Explained: Safe Recovery Strategies Across Versions

Overview

Have you ever encountered a task in Apache DolphinScheduler that remains stuck in the Running state, even though the underlying process has already terminated?

This "zombie task" scenario creates a mismatch between the UI and the actual execution state, preventing downstream tasks and workflows from progressing normally.

The good news is that the solution depends on your DolphinScheduler version. In this guide, we'll explain why zombie tasks occur and walk through the recommended cleanup methods for both legacy and modern releases.

Why Do Zombie Tasks Occur?

A task that remains in the Running state usually results from one of the following situations:

1. Database latency

Delayed database writes can prevent task status updates from being persisted, causing the UI to display an outdated execution state.

1. Worker node failure

A Worker process unexpectedly stops, but the Master has not yet detected the failure.

1. Network instability

ZooKeeper heartbeat timeouts trigger node removal events even though the task process may still be running.

1. Missing task instance

The logs indicate that the task instance is null, while the task status remains Running.

Solution for Legacy Versions (Earlier than 1.2.1)

Versions prior to Apache DolphinScheduler 1.2.1 require manual cleanup.

Step 1. Clear the ZooKeeper Task Queue

First, remove the stuck task from the ZooKeeper task queue.

# Clear the ZooKeeper task queue
delete /dolphinscheduler/task_queue

This prevents the stale task from continuously blocking the scheduling queue.

Step 2. Update the Task Status in the Database

Modify the task instance status directly in the database.

-- Change the stuck task status to FAILURE (state = 6)
UPDATE t_ds_task_instance
SET state = 6
WHERE state = 1
AND task_instance_id = <STUCK_TASK_ID>;

State definitions:

• 1 — RUNNING_EXECUTION
• 6 — FAILURE

Step 3. Resume the Workflow from the Failed Node

In the DolphinScheduler UI:

1. Locate the corresponding workflow instance.
2. Click Recover from Failure.
3. The workflow resumes execution from the failed task instead of starting over.

Solution for Modern Versions (1.2.1 and Later)

Starting with Apache DolphinScheduler 1.2.1, the platform introduced an automatic fault-tolerance mechanism, eliminating the need for manual cleanup in most scenarios.

Automatic Fault-Tolerance Mechanism

The new fault-tolerance framework is built on ZooKeeper's Watcher mechanism.

Worker Failover Workflow

When a Worker node receives a remove event, the Master performs the following logic:

1. Handle only task instances, not workflow instances.
2. Compare the task instance start time with the Worker service startup time.
3. If the task started after the Worker restarted, skip failover.
4. Otherwise, mark the task as NEED_FAULT_TOLERANCE.
5. The Master Scheduler thread automatically resubmits the task for execution.

This mechanism significantly improves cluster resilience during Worker failures and transient network issues.

Task State Machine

Modern versions also introduce a state machine to manage the complete task lifecycle, including pause and kill operations.

// Task kill event handler
public void onKilledEvent(...) {
    releaseTaskInstanceResourcesIfNeeded(taskExecutionRunnable);
    persistentTaskInstanceKilledEventToDB(taskExecutionRunnable, taskInstanceKillEvent);
    taskExecutionRunnable.getWorkflowExecutionGraph()
        .markTaskExecutionRunnableChainKill(taskExecutionRunnable);
    publishWorkflowInstanceTopologyLogicalTransitionEvent(taskExecutionRunnable);
}

The state machine ensures task transitions remain consistent while automatically releasing resources and updating workflow topology.

Feature Comparison

Feature	Before 1.2.1	1.2.1 and Later
Fault tolerance	Manual recovery	Automatic failover
ZooKeeper queue cleanup	Manual	Automatic
Task state updates	Database modification required	Managed by state machine
Worker failure recovery	Manual intervention	Automatic resubmission
Network jitter handling	Service restart required	Automatic detection and failover

Best Practices for Safe Cleanup

1. Verify the Task Is Actually Dead

Before performing any cleanup, confirm that the task process has indeed terminated.

# Check the Worker process
jps | grep WorkerServer

# Check the task process
ps -ef | grep <task_keyword>

# Review task logs
tail -f /path/to/task.log

2. Use the UI Whenever Possible

For newer versions of DolphinScheduler, always prefer built-in UI operations before modifying the database.

Recommended actions include:

• Click Stop to terminate the task.
• Use Recover from Failure to restart execution.

3. Be Careful with Database Operations

If direct database updates are unavoidable:

• Back up the database first.
• Verify the task instance ID carefully.
• Update only the task status field.
• Never delete task records directly.
• Verify the workflow state after the operation.

4. Exercise Caution When Cleaning ZooKeeper

If ZooKeeper cleanup is required:

• Confirm the target path is correct:

/dolphinscheduler/task_queue

• Use a trusted ZooKeeper client.
• Avoid deleting unrelated coordination nodes.

5. Monitor and Prevent Future Issues

To minimize zombie task occurrences in production:

• Monitor database latency.
• Configure appropriate ZooKeeper session timeout values.
• Regularly check Master and Worker health.
• Deploy monitoring scripts to automatically restart failed services.

Final Thoughts

The correct approach to cleaning up tasks stuck in the Running state depends largely on your DolphinScheduler version.

For versions earlier than 1.2.1, manual cleanup is required by clearing the ZooKeeper task queue, updating the task state in the database, and recovering the workflow from the failed node.

For version 1.2.1 and later, the built-in fault-tolerance mechanism automatically detects and recovers most failures, dramatically reducing operational effort.

If you're still running an older release, upgrading to the latest version is strongly recommended. Modern DolphinScheduler versions provide a more resilient scheduling architecture, automatic failover, and significantly improved cluster reliability.

Notes

This article is based on the official Apache DolphinScheduler FAQ and architecture documentation. The task state transition logic can be found in the AbstractTaskStateAction.java implementation.

For production environments, always validate the recovery procedure in a testing environment before applying it to live workloads.