Mastering DolphinScheduler Load Balancing: 3 Core Algorithms + Deep Dive into the Underlying Logic

This article introduces the Worker load balancing mechanism in Apache DolphinScheduler, covering three algorithms — Weighted Random, Smooth Round-Robin, and Linear Load (default) — along with configuration settings, Worker weight and warm-up mechanisms, and a detailed explanation of each algorithm. Refer to the source document at the end for more details.

What Is Load Balancing?

Load balancing refers to the use of routing algorithms (usually in a clustered environment) to reasonably distribute server workloads, thereby maximizing overall server performance.

DolphinScheduler Worker Load Balancing Algorithms

The DolphinScheduler Master assigns tasks to Workers. By default, it provides three algorithms:

• Weighted Random (random)
• Smooth Round-Robin (roundrobin)
• Linear Load (lowerweight)

The default configuration uses Linear Weighted Load Balancing.

Since routing occurs on the client side — that is, in the Master service — you can modify the algorithm in the master.properties file via the following configuration:

Example:
master.host.selector=random (case-insensitive)

Worker Load Balancing Configuration

Configuration file: worker.properties

Weight

All of the above load balancing algorithms are based on weights, which determine how tasks are distributed. You can adjust the value of worker.weight to assign different weights to different machines.

Warm-up

Considering JIT (Just-In-Time) optimization, Workers will operate at reduced power for a short period after startup to gradually reach optimal performance. This process is called warm-up. (If you’re interested, you can read more about JIT-related topics.)

Therefore, after a Worker starts, its weight gradually increases to the maximum value over time (the default warm-up period is ten minutes). This parameter is not configurable by default, but you can modify the code and submit a PR if needed.

Detailed Explanation of Load Balancing Algorithms

Weighted Random

This algorithm is straightforward — it randomly selects one Worker from the available ones, with each Worker’s weight affecting its probability of being chosen.

Smooth Weighted Round-Robin

The traditional weighted round-robin algorithm has a notable drawback: under certain weight combinations, it can generate uneven task sequences, leading to temporary high loads on specific Workers, and even causing system instability or downtime.

To address this issue, DolphinScheduler introduces the Smooth Weighted Round-Robin algorithm.

Each Worker maintains two weight values:

• weight (fixed after warm-up)
• current_weight (dynamically updated)

For each routing iteration:

1. The scheduler traverses all Workers, adding each Worker’s weight to its current_weight.
2. The sum of all weight values is computed as total_weight.
3. The Worker with the highest current_weight is selected to execute the task.
4. The selected Worker’s current_weight is then decreased by total_weight.

Linear Weighted (Default Algorithm)

This algorithm periodically reports each Worker’s load information to the registry.
The load is primarily determined by CPU usage, memory usage, and Worker slot usage.

If any of these metrics fall below a configured threshold, the Worker will be excluded from the load balancing process (i.e., it will not receive new tasks).

📄 Documentation link:
https://dolphinscheduler.apache.org/zh-cn/docs/3.3.1/architecture/load-balance