Chen Debra

Posted on Mar 5

Part 3 | How Does Scheduling Actually “Start Running”?

#apachedolphinscheduler #opensource #bigdata #datascience

In the previous two articles, we broke down what problems a scheduling system is meant to solve, and how a Workflow is abstracted and modeled at the logical level.

But one fundamental question always remains unavoidable:

When time arrives, or when an event is triggered, how does a Workflow actually start “running”, step by step?

In this article, we start from a real scheduling trigger and fully dissect DolphinScheduler’s end-to-end execution path — from Trigger → Scheduling Decision → Task Dispatch → Execution Feedback.

We will focus on explaining its Master / Worker collaboration model, the rationale behind decentralized Workers, and the architectural value of decoupling scheduling from execution.

In a data platform, “the scheduler starts running” is never a trivial statement.

When you click Start in the UI, or when a Cron timestamp quietly arrives, what happens behind the scenes is not simply “executing tasks in sequence”.

Instead, it is a long-running, continuously evaluating, state-driven system behavior.

At its core, DolphinScheduler’s scheduling mechanism is much closer to an operating system kernel for workflows than to a mere timer.

Understanding this is the prerequisite for understanding all of its architectural design choices.

Everything Starts with a Trigger, but the Trigger Itself Is Not Important

In DolphinScheduler, a Trigger is merely a signal source.

Whether it is:

a scheduled trigger,
a manual trigger,
or a dependency-based trigger,

they are all ultimately normalized into one single action:

Create a Workflow Instance and enter the scheduling loop.

This step is critical, because from this moment on, the system no longer focuses on the Workflow Definition, but on an instance with complete runtime state.

Logically, it can be simplified as:

WorkflowInstance instance = workflowInstanceService.create(
    workflowDefinitionId,
    triggerType,
    executionContext
);

The true starting point of “running” is not task execution, but writing state into metadata storage.

The Master Is Not “Running Tasks”, It Is “Making Decisions Continuously”

Many scheduling systems push a large amount of logic into execution nodes.

DolphinScheduler deliberately keeps the Master lightweight.

Once started, the Master enters a continuously running scheduling loop, conceptually similar to:

while (workflowInstance.isRunning()) {
    List<TaskInstance> readyTasks = dag.findRunnableTasks();
    for (TaskInstance task : readyTasks) {
        dispatch(task);
    }
    sleep(scheduleInterval);
}

The key here is not dispatch, but findRunnableTasks().

The essence of scheduling is not dispatching — it is decision-making.

At Runtime, a DAG Is No Longer a Structure, but a State Machine

At definition time, a Workflow is a DAG.

At runtime, it behaves more like a graph whose state is constantly evolving.

Each Task node contains at least the following state dimensions:

Current execution state (SUBMITTED / RUNNING / SUCCESS / FAILURE)
Completion status of upstream tasks
Retry count and failure policies
Conditional branch evaluation results (if any)

In each scheduling loop, what the Master really does is:

Under the current global state snapshot, recompute which nodes are legally runnable at this moment.

The logic can be abstracted as:

boolean canRun(TaskInstance task) {
    return task.state == INIT
        && allUpstreamTasksSuccess(task)
        && conditionSatisfied(task)
        && retryPolicyAllows(task);
}

This is why scheduling is state-driven, not event-driven.

Events only mutate state; scheduling decisions are always based on the current global state.

Master / Worker Collaboration: The Boundary Is Deliberately Sharp

Once the Master determines that a Task Instance is runnable, it does not care how the task is executed.

It does exactly one thing:

Select a suitable Worker and send an execution command.

Worker worker = workerManager.select(task);
workerClient.submit(task);

From this point on, the Master’s direct relationship with the task ends.

This boundary is extremely important. It means:

The Master does not maintain execution threads
The Master does not perceive execution details
The Master does not bear execution risks

The Worker’s Responsibility: Execution Is Dirty Work and Must Be Pushed Down

Workers are where tasks actually run.

After receiving a Task Instance, a Worker will:

Build the execution context (parameters, environment variables, resources)
Launch the corresponding executor (Shell / Spark / Flink / Python)
Continuously monitor process status
Asynchronously report logs, heartbeats, and results

A typical execution flow looks like:

export DS_TASK_ID=12345
export DS_EXECUTION_DATE=2026-02-09

/bin/bash run.sh > task.log 2>&1

The Worker’s world is chaotic, heterogeneous, and unpredictable — which is precisely why it must be thoroughly isolated.

Decentralized Workers Are Not “Nice to Have”, They Are Necessary

In real production environments, tasks are highly heterogeneous:

Spark jobs consume memory
Python scripts are IO-heavy
Shell scripts can do literally anything

If Workers were centralized or tightly bound, the scheduling system would quickly spiral out of control.

DolphinScheduler adopts a fully peer-based Worker model:

Any Worker can execute any task
The Master only perceives Worker status via heartbeats and load metrics
Workers can be added, removed, or replaced at any time

This gives the execution layer native elasticity and fault tolerance.

Decoupling Scheduling from Execution Truly Decouples “Complexity Propagation”

The most dangerous thing for a scheduling system is not task failure, but failure propagating into the system core.

If scheduling threads are blocked by execution, or if the Master needs to understand execution details, then:

One slow task can drag down the entire system
One abnormal execution can contaminate scheduling logic
System complexity grows exponentially

By enforcing strict decoupling, DolphinScheduler locks complexity on the Worker side:

Execution failure → state change
State change → triggers the next scheduling decision
Scheduling logic itself remains pure

This is a highly engineering-driven and mature architectural choice.

From a Global Perspective, What “Runs” Is Not Tasks, but State Flow

At a higher level of abstraction, DolphinScheduler does not run “tasks”.

What actually runs is:

State continuously flowing through the system, with scheduling logic responding to state changes.

Triggers are merely the starting point of state,
Workers are producers of state,
Masters are arbiters of state.

Once you understand this, you will understand why:

A scheduling system must have a metadata center
A DAG must be a computable state model
The execution layer must never erode the scheduling layer

Final Thoughts

Many people use schedulers and only care about one thing: does it run?

Those who maintain scheduling systems long-term care about something else entirely:

Will it lose control under failure?
Can it still hold up as scale grows?
Can it continue to evolve as complexity increases?

DolphinScheduler’s scheduling mechanism is designed precisely for these long-term problems.

In the next article, we will dive deeper into the true soul of a scheduling system: the state machine.

Coming Up Next

Part 4 | Why State Machines Power Reliable Scheduling Systems

DEV Community