How Apache DolphinScheduler Manages Task Group Slots Without Timeout-Based Reclamation

Apache DolphinScheduler does not provide a dedicated “force reclaim after timeout” mechanism for task group slots. Instead, what may appear to be an automatic reclamation mechanism is actually a combination of consistency correction, normal slot release, and forced task startup, all handled periodically by the TaskGroupCoordinator background thread.

If you need to address situations where task group resources remain occupied for an extended period, you can leverage the force-start logic in dealWithForceStartTaskGroupQueue, or use the correction mechanism provided by amendTaskGroupUseSize to indirectly achieve the effect of “automatically allowing tasks to proceed after prolonged resource occupancy.”

In-Depth Analysis

1. Core Component: TaskGroupCoordinator

TaskGroupCoordinator is the core component on the Master side responsible for managing task group slots. It implements the ITaskGroupCoordinator interface.

It starts a dedicated daemon thread named TaskGroupCoordinator-Thread, which polls every five seconds and performs four key operations in sequence:

2. The Interface Design Defines the Slot Lifecycle

The class-level documentation of ITaskGroupCoordinator clearly defines the lifecycle of task group slots. When a task instance needs a task group slot, it calls acquireTaskGroupSlot. This operation is non-blocking: the task is simply placed in the waiting queue. Once the task finishes, it calls releaseTaskGroupSlot to release the slot.

3. The First Layer of “Reclamation”: Normal Release and Task Wake-Up

Under normal circumstances, when a task instance finishes, it calls releaseTaskGroupSlot. Internally, this removes the corresponding record from the queue through deleteTaskGroupQueueSlot, thereby releasing the slot.

After a slot becomes available, the background thread runs dealWithWaitingTaskGroupQueue. It scans task groups that are not yet at their capacity limit, selects waiting tasks according to priority, and attempts to atomically update the database by calling acquireTaskGroupSlot. Once the slot is successfully acquired, the waiting task is notified through the RPC method notifyWaitingTaskInstance, allowing the task instance on the corresponding Master to resume execution.

If the notification fails—for example, because the target WorkflowInstance has already failed over, the host is unavailable, or the task is no longer in the SUBMITTED_SUCCESS state—notifyWaitingTaskInstance throws an UnsupportedOperationException. In this case, deleteTaskGroupQueueSlot is called directly to remove the invalid queue record and release the associated slot, preventing a “zombie” task from occupying the resource indefinitely.

4. The Second Layer of “Reclamation”: Force Start

When an operator manually clicks Force Start in the UI, the corresponding TaskGroupQueue record is marked with forceStart = YES.

The background thread then handles these records through dealWithForceStartTaskGroupQueue. It ignores the task group’s resource limit, directly wakes up the corresponding tasks, and removes them from the queue.

This is currently the only manual or semi-automatic entry point for “forced reclamation” provided by the system. Strictly speaking, however, it does not actually reclaim an occupied slot. Instead, it bypasses the task group’s slot limit and allows the task to proceed.

5. Is There an Automatic Timeout Mechanism for Long-Running Slot Occupancy?

At first glance, amendTaskGroupUseSize and amendTaskGroupQueueStatus may appear to be responsible for “reclaiming” resources. In reality, their role is consistency correction: aligning use_size with the actual number of queued or acquired tasks and correcting inconsistent queue states. They are not designed to forcibly evict a task after its slot has been occupied for a certain period of time.

If you need automatic reclamation after a timeout, there are currently two practical approaches:

1. Rely on the task’s own timeout and fault-tolerance mechanisms. When a long-running task is terminated, it can trigger releaseTaskGroupSlot, allowing the slot to be released through the normal lifecycle.

2. Use Force Start through the UI. This requires manual intervention and ultimately relies on the dealWithForceStartTaskGroupQueue process to bypass the task group resource limit.

6. Practical Recommendations

• Check current slot usage: Go to the Task Group Management page and inspect the queue. See the corresponding View Task Group Queue section in the documentation.
• Investigate at the database level: Query the t_ds_task_group_queue table to identify records that have remained in WAIT_QUEUE or ACQUIRE_SUCCESS for an unusually long time, especially when the corresponding task has already terminated unexpectedly. Mapper SQL such as queryTheHighestPriorityTasks for retrieving the highest-priority tasks and queryByTaskId for locating records by task ID can help pinpoint these cases.
• Manually force reclamation: For records confirmed to be associated with “zombie” tasks, use the Force Start operation in the UI to trigger the dealWithForceStartTaskGroupQueue flow. The corresponding queue record is then removed, allowing the resource to become available to subsequent tasks.
• Understand the scope of concurrency control: Task groups only apply to tasks executed by Workers. Master-side nodes such as switch, condition, and sub_workflow are not subject to task group limits. This distinction is important when investigating the source of prolonged resource occupancy.

Notes

The current “automatic reclamation” mechanism is more accurately described as state consistency correction plus fallback deletion when normal release notification fails, rather than true timeout-based forced reclamation. This is fundamentally different from the idle-timeout reclamation mechanisms commonly found in resource pools such as connection pools and thread pools.

The TaskGroupQueueMapper.xml file also contains SQL statements such as updateInQueueLimit1 and updateInQueueCAS, which show that slot state updates use CAS-based operations to ensure concurrency safety. These implementation details are useful for understanding how concurrent slot allocation is controlled, but they are less directly related to the issue of reclaiming resources that have been occupied for too long.