Core Architecture Evolves: A Deep Dive into Apache DolphinScheduler 1.3 vs 3.x Architecture

From version 1.3 to the 3.x series, Apache DolphinScheduler has undergone multiple rounds of evolution, introducing significant improvements across architecture, functionality, and extensibility. These changes have attracted considerable attention from the community.

This article systematically compares the key differences between Apache DolphinScheduler 1.3 and 3.x from five perspectives: table structure evolution, task type expansion, registry diversification, architectural optimization, and feature enhancements. It provides an in-depth analysis of the table naming refactoring from t_ds_process_definition to t_ds_workflow_definition, the expansion from 12 task types to more than 30 task types, the evolution from a single ZooKeeper registry to multiple registry implementations, as well as major technical upgrades including the microkernel plugin architecture, decentralized design, fault tolerance mechanisms, and log access architecture.

This deep dive helps users better understand Apache DolphinScheduler’s architectural evolution journey and key upgrade considerations. It is especially valuable for teams planning to upgrade from older versions to the latest 3.x releases.

1. System Architecture Overview

1.1 Core Architecture Design

Apache DolphinScheduler 3.x adopts a microkernel + plugin architecture, where core capabilities such as task types, resource storage, and registry implementations are designed as extension points. The system uses SPI (Service Provider Interface) to improve flexibility and scalability.

1.2 Decentralized Architecture Design

Apache DolphinScheduler adopts a decentralized architecture. The Master and Worker clusters communicate and coordinate through registry implementations such as ZooKeeper, JDBC, and Etcd, enabling a highly available and distributed system without a single central node.

• MasterServer: Primarily responsible for DAG task splitting, task submission monitoring, and monitoring the health status of other MasterServer and WorkerServer instances.
• WorkerServer: Primarily responsible for task execution and providing log services.
• Registry: Supports three implementations: ZooKeeper, JDBC, and Etcd, which are used for cluster management and fault tolerance.

2. Overall Workflow Storage Structure (3.x Version)

2.1 Table Structure Evolution

From the 1.3 series to the 3.x series, Apache DolphinScheduler standardized the naming of core database tables by replacing the term "process" with "workflow". This change better reflects the workflow-oriented concept of the platform.

Table Name in 1.3	Table Name in 3.x	Description
t_ds_process_definition	t_ds_workflow_definition	Workflow definition table
t_ds_process_definition_log	t_ds_workflow_definition_log	Workflow definition version history table
t_ds_process_instance	t_ds_workflow_instance	Workflow instance table
t_ds_process_task_relation	t_ds_workflow_task_relation	Workflow-task relationship table
t_ds_process_task_relation_log	t_ds_workflow_task_relation_log	Workflow-task relationship version history table

2.2 Detailed Analysis of Core Tables

2.2.1 Workflow Definition Table (t_ds_workflow_definition)

Compared with version 1.3, the major changes introduced in 3.x include:

• Uses code as the business unique identifier instead of the original id.
• Adds the execution_type field to support serial execution strategies.
• Adds the warning_group_id field to associate workflows with alert groups.
• Removes the process_definition_json field. Task information is now stored and managed through related tables.

2.2.2 Workflow Data Model Relationship

2.3 Core Service Layer APIs

The ProcessService interface provides core APIs for workflow definition management:

Method	Function
saveWorkflowDefine()	Saves workflow definitions with version management support
saveTaskDefine()	Saves task definitions with version management support
saveTaskRelation()	Saves task relationships and builds DAG structures
genDagGraph()	Generates DAG graphs from workflow definitions
switchVersion()	Switches between workflow versions
findWorkflowDefinition()	Queries workflow definitions

3. Task Type Classification System

3.1 Task Type Architecture

Apache DolphinScheduler 3.x expands the supported task types to more than 30 and categorizes them into five major groups based on functional characteristics:

3.2 Task Type Comparison (1.3 vs 3.x)

Category	1.3 Version	3.x Version	Newly Added Types
General Tasks	SHELL, SQL, PYTHON, SPARK, FLINK, MR, HTTP	+ JAVA, GRPC, DINKY, FLINK_STREAM, HIVECLI, REMOTESHELL	6 types
Cloud-Native Tasks	-	EMR, K8S, DMS, DATA_FACTORY, ALIYUN_SERVERLESS_SPARK	5 types
Logical Tasks	CONDITIONS, SUB_PROCESS, DEPENDENT	+ SWITCH	1 type
Data Integration Tasks	DATAX, SQOOP	+ SEATUNNEL	1 type
Machine Learning Tasks	-	JUPYTER, MLFLOW, OPENMLDB, DVC, SAGEMAKER, PYTORCH, KUBEFLOW	7 types
Other Tasks	-	ZEPPELIN, CHUNJUN, DATASYNC, LINKIS	4 types

3.3 Typical Task Parameter Structure

3.3.1 Common Parameter Structure for All Task Types

All task types support the following common parameters:

3.3.2 Shell Task Parameter Example

{
  "localParams": [],
  "resourceList": [
    {
      "id": 3,
      "name": "run.sh",
      "res": "run.sh"
    }
  ],
  "rawScript": "echo 'Hello from DolphinScheduler 3.x'",
  "conditionResult": {
    "successNode": [""],
    "failedNode": [""]
  }
}

3.3.3 SQL Task Parameter Example

{
  "type": "MYSQL",
  "datasource": 1,
  "sql": "SELECT * FROM users WHERE id = ${id}",
  "udfs": "",
  "sqlType": "0",
  "sendEmail": false,
  "displayRows": 10,
  "preStatements": [],
  "postStatements": []
}

4. Core Architecture Flow

4.1 Workflow Execution Flow

4.2 Registry Architecture

Apache DolphinScheduler 3.x supports multiple registry implementations, including ZooKeeper, JDBC, and Etcd, enabling a truly decentralized architecture.

4.2.1 JDBC Registry

The JDBC registry uses a relational database to implement event listening and distributed locking. It is suitable for environments that already have database infrastructure in place.

Key Features:

• Event Listening: Uses JdbcRegistryDataChangeListenerAdapter to monitor database changes.
• Distributed Locking: Supports both blocking lock acquisition and timeout-based lock acquisition.
• Heartbeat Mechanism: Uses heartbeat detection to monitor client availability and automatically clean up expired locks.

Configuration Example:

registry:
  type: jdbc
  heartbeat-refresh-interval: 3s
  session-timeout: 60s
  hikari-config:
    jdbc-url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
    username: root
    password: root
    maximum-pool-size: 5

4.2.2 Etcd Registry

The Etcd registry is implemented based on the Jetcd client library and is designed for cloud-native environments.

Key Features:

• Watch API: Monitors changes to specified keys or key prefixes.
• Lease Lock: Uses TTL-based lease mechanisms to automatically release locks when clients disconnect.
• Connection Health Monitoring: Tracks connection status through EtcdConnectionStateListener.

5. Fault Tolerance Design

5.1 Failure Recovery Mechanism

Apache DolphinScheduler’s fault tolerance design relies on the Watcher mechanism provided by the registry. It mainly covers two scenarios: Master failover and Worker failover.

5.2 Task Failure Retry

Task failure retry, workflow failure recovery, and workflow failure rerun are three different concepts:

Type	Level	Trigger Method	Execution Scope
Task Failure Retry	Task level	Automatically triggered by the system	Automatically retries until success or the retry limit is reached
Workflow Failure Recovery	Workflow level	Manually triggered	Starts execution from the failed node or current node
Workflow Failure Rerun	Workflow level	Manually triggered	Re-executes from the beginning node

Task Node Categories:

• Business Nodes: Shell, SQL, Spark, Flink, and other executable tasks that support failure retry.
• Logical Nodes: DEPENDENT, SUB_WORKFLOW, CONDITIONS, and other control-flow tasks that do not support failure retry.

6. Task Priority Design

Apache DolphinScheduler adopts a multi-level priority scheduling mechanism to ensure that critical tasks are executed first.

Implementation Mechanism:

• Stores the information of workflow instance priority_workflow instance id_task priority_task id in the registry task queue.
• Uses string comparison to determine the task with the highest priority.
• Both workflow priority and task priority contain five levels: HIGHEST, HIGH, MEDIUM, LOW, and LOWEST.

7. Log Access Mechanism

Apache DolphinScheduler 3.x uses gRPC for remote log access, replacing the earlier Netty-based implementation.

Key Points of Logback Configuration:

• Uses SiftingAppender to separate log files based on task IDs.
• Uses TaskLogFilter to filter task-related logs.
• Uses TaskLogDiscriminator to distinguish different tasks based on taskAppId.
• Uses SensitiveDataConverter to mask sensitive information.

8. System Module Architecture

Apache DolphinScheduler consists of multiple core modules, with each module having clearly defined responsibilities:

9. Core Components Inside MasterServer

MasterServer adopts a distributed, decentralized design. It contains multiple core threads and components that work together to complete workflow scheduling.

Detailed Component Description

Component	Responsibility	Key Operations
DistributedQuartz	Distributed scheduling component	Responsible for starting and stopping scheduled tasks. After a task is triggered, subsequent operations are handled by the thread pool.
MasterSchedulerService	Command scanning thread	Periodically scans the t_ds_command table and executes business operations based on different command types.
WorkflowExecuteRunnable	Workflow execution thread	Responsible for DAG task splitting, task submission monitoring, and processing various event types.
TaskExecuteRunnable	Task execution thread	Handles task processing and persistence, generates task events, and submits them to the event queue.
EventExecuteService	Event execution service	Responsible for polling the workflow instance event queue.
StateWheelExecuteThread	State polling thread	Handles workflow and task timeout detection, task retry management, and dependency polling.
FailoverExecuteThread	Failover execution thread	Handles Master failover and Worker failover logic.

10. Core Components Inside WorkerServer

WorkerServer also adopts a distributed, decentralized design. It is primarily responsible for task execution and log services.

Detailed Component Description

Component	Responsibility	Key Operations
WorkerManagerThread	Task management thread	Continuously retrieves tasks from the task queue and submits them to the thread pool for processing.
TaskExecuteThread	Task execution thread	Executes tasks based on different task types.
RetryReportTaskStatusThread	Retry reporting thread	Periodically reports task status to Master until Master returns a status acknowledgment (ACK).

11. Startup Flow Activity Diagram

The workflow startup process in Apache DolphinScheduler involves collaboration between multiple components.

12. Decentralized vs Centralized Architecture Comparison

Apache DolphinScheduler adopts a dynamic centralized decentralized architecture, which is significantly different from traditional centralized architectures.

13. Core Terminology

To better understand Apache DolphinScheduler, it is important to become familiar with the following core concepts and terminology:

Term	Description
DAG	A Directed Acyclic Graph. Tasks in a workflow are assembled and executed in the form of a DAG.
Workflow Definition	A visual DAG created by dragging task nodes onto the canvas and establishing relationships between them.
Workflow Instance	An instantiated execution of a workflow definition. Each workflow run generates a workflow instance.
Task Instance	An instantiated execution of a task node, representing the execution status of a specific task.
Scheduling Method	Supports both scheduled execution based on cron expressions and manual execution.
Dependency	Supports DAG-based dependencies and task dependency nodes.
Data Backfill	Historical data reprocessing, supporting both interval-based parallel execution and serial execution modes.

14. Summary

Compared with the 1.3 series, Apache DolphinScheduler 3.x has achieved significant improvements in both architecture and functionality.

Architectural Evolution:

• Expanded from a single ZooKeeper registry to support multiple registry implementations, including ZooKeeper, JDBC, and Etcd.
• Evolved from a centralized architecture into a truly decentralized architecture.
• Adopted a microkernel + plugin architecture to improve system extensibility and maintainability.

Table Structure Optimization:

• Standardized table naming conventions by replacing process with workflow.
• Introduced code as the business unique identifier.
• Added the execution_type field to support multiple execution strategies.
• Migrated task information storage from JSON fields to relational tables.

Feature Enhancements:

• Expanded supported task types from 12 to more than 30.
• Added support for cloud-native tasks and machine learning workloads.
• Improved fault tolerance mechanisms and priority scheduling capabilities.
• Upgraded remote log access from Netty to gRPC.

These improvements have transformed Apache DolphinScheduler into a more modern, scalable, and maintainable data orchestration platform, capable of meeting the growing requirements of enterprise data workflows and cloud-native environments.

Notes

• The additional content is based on the architecture design documents:

  • docs/docs/en/architecture/design.md
  • docs/docs/zh/architecture/design.md

• The internal component designs of MasterServer and WorkerServer demonstrate the platform’s modular architecture and clear separation of responsibilities.

• The decentralized architecture leverages registry mechanisms to enable dynamic coordination and fault recovery, reducing the risk of single points of failure.

• For more implementation details, readers are encouraged to refer to the official architecture design documentation.