DEV Community: Chen Debra

🔥 Skip ZooKeeper! Discover two proven registry alternatives for Apache DolphinScheduler and simplify your architecture. 🚀 #ApacheDolphinScheduler #OpenSource #CloudNative #DevOps

Chen Debra — Thu, 02 Jul 2026 07:32:53 +0000

Chen Debra

Jul 2

Running Apache DolphinScheduler Without ZooKeeper: Two Proven Registry Alternatives

#apachedolphinscheduler #zookeeper #datascience #bigdata

6 min read

Running Apache DolphinScheduler Without ZooKeeper: Two Proven Registry Alternatives

Chen Debra — Thu, 02 Jul 2026 07:32:23 +0000

When designing a distributed scheduling platform, the choice of a service registry has a direct impact on both system architecture and operational complexity. Although Apache DolphinScheduler uses ZooKeeper as its default registry center, it also provides multiple alternatives that allow users to choose the solution that best fits their existing infrastructure and operational capabilities.

Instead of forcing every deployment to rely on ZooKeeper, DolphinScheduler offers greater flexibility by supporting both JDBC Registry and Etcd Registry, giving organizations more options for different deployment scenarios.

Why Do Some Teams Prefer Not to Use ZooKeeper?

ZooKeeper has long been the de facto coordination service for distributed systems. It is mature, reliable, and battle-tested. However, many teams still hesitate to introduce it into their environments for several practical reasons.

Higher Operational Complexity

Running ZooKeeper requires deploying and maintaining an independent cluster. In production environments, at least three nodes are typically recommended to achieve high availability. For smaller teams or organizations with limited infrastructure resources, maintaining an additional distributed system increases both operational overhead and maintenance costs.

An Additional Technology Stack

Many organizations already operate around relational databases such as MySQL or PostgreSQL. Introducing ZooKeeper means adopting and maintaining another technology stack, which requires additional expertise and increases the learning curve for operations teams.

Extra Infrastructure Costs

ZooKeeper consumes dedicated computing resources, including CPU, memory, storage, and networking. For organizations aiming to simplify infrastructure or reduce resource consumption, these additional requirements may become an unnecessary burden.

Registry Alternatives in Apache DolphinScheduler

To address different deployment requirements, Apache DolphinScheduler currently provides two production-ready alternatives to ZooKeeper:

JDBC Registry
Etcd Registry

Each solution offers the same core capabilities required by the scheduler while targeting different infrastructure preferences.

Option 1: JDBC Registry

One of the most innovative features in Apache DolphinScheduler is the JDBC Registry, which eliminates the need for an additional registry service by leveraging an existing relational database such as MySQL or PostgreSQL.

How It Works

Instead of relying on an external coordination service, the JDBC Registry reproduces the essential capabilities of ZooKeeper using relational database tables.

Event Notification

The JdbcRegistryDataChangeListenerAdapter converts database changes—including record creation, updates, and deletions—into DolphinScheduler Event notifications that trigger registered SubscribeListener callbacks.

Internally, the registry detects changes through polling or trigger-based mechanisms, effectively simulating ZooKeeper's Watcher functionality.

Distributed Locking

The JDBC Registry provides two methods for acquiring distributed locks:

acquireLock(String key)
acquireLock(String key, long timeout)

These methods support both blocking and timeout-based lock acquisition.

Locks are managed through database records to ensure mutual exclusion across distributed nodes. Registry entries are categorized into two types:

EPHEMERAL — Temporary entries that are automatically cleaned up through heartbeat detection when a client disconnects or fails.
PERSISTENT — Permanent entries that remain available until explicitly removed.

This mechanism enables reliable distributed coordination without requiring ZooKeeper.

Deployment Steps

Step 1: Initialize the Registry Tables

Execute the appropriate initialization script according to your database:

MySQL: src/main/resources/mysql_registry_init.sql
PostgreSQL: src/main/resources/postgresql_registry_init.sql

Step 2: Update the Configuration

Add the following configuration to:

master-server/conf/application.yaml
worker-server/conf/application.yaml
api-server/conf/application.yaml

registry:
  type: jdbc
  heartbeat-refresh-interval: 3s
  session-timeout: 60s
  hikari-config:
    jdbc-url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
    username: root
    password: root
    maximum-pool-size: 5
    connection-timeout: 9000
    idle-timeout: 600000

Step 3: Add the Database Driver

If you are using MySQL, copy the mysql-connector-java.jar driver into the DolphinScheduler classpath. The MySQL JDBC driver is intentionally not bundled with the official distribution package, so it must be provided separately.

Results

With the JDBC registry, the MasterServer and WorkerServer of Apache DolphinScheduler store metadata in a relational database. Database transactions ensure data consistency, while the heartbeat mechanism enables service discovery and failure detection.

This approach is particularly suitable for environments that already have a mature database operations team, allowing organizations to take full advantage of their existing database infrastructure.

Option 2: Etcd Registry

Etcd is a distributed key-value store designed for the cloud-native era and is especially well suited for Kubernetes and other cloud-native environments. The Etcd registry implementation in Apache DolphinScheduler is built on the Jetcd client library and provides functionality similar to ZooKeeper.

How It Works

Event Listening

The EtcdRegistry class uses Etcd's Watch API to monitor changes (create, update, and delete operations) on a specified key or key prefix. It converts the underlying Etcd watch events into DolphinScheduler Event objects and triggers the SubscribeListener callback to provide real-time notifications.

Distributed Locking

EtcdKeepAliveLeaseManager grants leases with a specified TTL and continuously keeps them alive using Etcd's keep-alive mechanism. If a client disconnects, the lease expires automatically, releasing the lock without requiring manual intervention.

Connection Health Monitoring

EtcdConnectionStateListener monitors the connection state between Apache DolphinScheduler and the Etcd cluster. When the connection is lost or restored, it automatically re-establishes distributed locks or re-registers services.

Configuration

1. Update the Configuration Files

Add the following configuration to:

master-server/conf/application.yaml
worker-server/conf/application.yaml
api-server/conf/application.yaml

registry:
  type: etcd
  endpoints: "http://etcd0:2379, http://etcd1:2379, http://etcd2:2379"
  namespace: dolphinscheduler
  connection-timeout: 9s
  retry-delay: 60ms
  retry-max-delay: 300ms
  retry-max-duration: 1500ms
  # Optional SSL configuration
  cert-file: "deploy/kubernetes/dolphinscheduler/etcd-certs/ca.crt"
  key-cert-chain-file: "deploy/kubernetes/dolphinscheduler/etcd-certs/client.crt"
  key-file: "deploy/kubernetes/dolphinscheduler/etcd-certs/client.pem"
  # Optional authentication configuration
  user: ""
  password: ""
  authority: ""

2. SSL Configuration Notes

If SSL is enabled on the Etcd server, make sure your JDK version is newer than Java 8u252 (released in April 2020). JDK 11 is also fully supported. The Docker images currently use JDK 8u362, which works correctly.

This requirement exists because native ALPN support was introduced starting with Java 8u252.

Results

With the Etcd registry, Apache DolphinScheduler can fully leverage Etcd's strong consistency and high availability to deliver low latency, excellent scalability, and simplified deployment in cloud-native environments.

This option is especially suitable for teams that already use Kubernetes and the Etcd technology stack.

Why This Design?

The design philosophy behind providing multiple registry implementations in Apache DolphinScheduler is straightforward: reduce dependencies on external components and allow users to choose the registry that best fits their environment.

This philosophy has been consistently reflected throughout the evolution of Apache DolphinScheduler. The project previously implemented a Redis-based queue, but the Redis implementation was eventually removed to reduce external dependencies.

Today, multiple registry options are provided not to replace ZooKeeper, but to give users greater flexibility when selecting the deployment architecture that best meets their requirements.

For Kubernetes deployments, this flexibility is reflected in the values.yaml configuration of the Helm Chart:

zookeeper:
  enabled: true  # Enabled by default

registryEtcd:
  enabled: false  # Enable manually if needed

registryJdbc:
  enabled: false  # Enable manually if needed

Conclusion

Apache DolphinScheduler provides two production-ready alternatives to ZooKeeper: JDBC Registry and Etcd Registry.

Both implementations support the same core capabilities required by a distributed scheduler, including service registration, service discovery, event notification, heartbeat management, and distributed locking.

The JDBC Registry is particularly attractive for teams with mature relational database infrastructure, while the Etcd Registry offers a seamless experience for organizations embracing Kubernetes and cloud-native technologies.

Instead of forcing every deployment to depend on a single coordination service, DolphinScheduler allows users to make infrastructure decisions based on their own operational requirements.

This flexibility reflects one of the project's core principles: software should adapt to its users—not the other way around.

As distributed systems continue to evolve, reducing unnecessary dependencies while providing deployment choices becomes increasingly valuable. By supporting multiple registry implementations, Apache DolphinScheduler enables organizations to build reliable scheduling platforms using the technologies they already know and trust.

Notes

ZooKeeper remains the default registry implementation in Apache DolphinScheduler and has not been deprecated.
By default, the JDBC Registry uses the same database as DolphinScheduler metadata, although a separate database can also be configured.
The Etcd Registry supports both SSL/TLS encryption and user authentication, allowing deployments to meet different security requirements.
In pseudo-cluster deployments, users can choose among ZooKeeper, MySQL (JDBC Registry), and Etcd as the registry implementation.
All registry configurations support environment variables, making them easy to integrate into containerized and cloud-native deployment workflows.

How to Run Apache Flink Kafka Jobs with Apache DolphinScheduler on Linux

Chen Debra — Thu, 02 Jul 2026 07:18:06 +0000

Apache DolphinScheduler has already been deployed on a virtual machine.
Next, we want to try creating a Flink task node in DolphinScheduler and use Flink to consume data from Kafka.

Apache DolphinScheduler is deployed in standalone mode.
For detailed installation steps, please refer to the official documentation:
DolphinScheduler | Documentation Center
https://dolphinscheduler.apache.org/zh-cn/docs/3.3.2/guide/installation/standalone

Prerequisites

The following components have already been installed:

Java 11
Apache DolphinScheduler 3.3.2
Apache Flink 1.18.1
Apache Kafka 3.6.0
ZooKeeper (using the built-in Kafka version)

It is recommended to download and install binary packages directly on the virtual machine. Installing via system package managers may introduce uncontrollable dependencies.

The downloaded binary packages are shown below:

Configure Flink Environment Variables

1. Edit environment variables

sudo vim ~/.bashrc

Add the Flink installation path:

2. Make the configuration effective

# Reload environment variables
source ~/.bashrc

# Verify Flink environment variable
echo $Flink_HOME

Modify Kafka, Flink, and DolphinScheduler Configuration Files

Since this setup runs inside a virtual machine, configuration changes are required so that services can be accessed from the host machine.

1. Modify Kafka configuration

Navigate to the config directory under the Kafka installation path and edit server.properties.

The listeners and advertised.listeners settings must be modified so that Kafka can be accessed externally. Otherwise, Kafka will default to localhost, which may cause connection failures.

broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
# Replace with your VM IP address
advertised.listeners=PLAINTEXT://192.168.146.132:9092

2. Modify Flink configuration

Go to the conf directory under the Flink installation path and edit flink-conf.yaml.

Replace all localhost addresses with 0.0.0.0 so the Flink Web UI can be accessed externally.
Additionally, adjust JobManager and TaskManager memory settings.

jobmanager.rpc.address: 0.0.0.0
jobmanager.bind-host: 0.0.0.0
jobmanager.cpu.cores: 1
jobmanager.memory.process.size: 1600m

taskmanager.bind-host: 0.0.0.0
taskmanager.host: 0.0.0.0
taskmanager.memory.process.size: 2048m
taskmanager.cpu.cores: 1

3. Modify Apache DolphinScheduler configuration

From the dolphinscheduler-daemon.sh startup script, we can see that DolphinScheduler loads environment variables from:

bin/env/dolphinscheduler_env.sh

View dolphinscheduler-daemon.sh:

Edit dolphinscheduler_env.sh and add Java and Flink paths:

# Replace with your actual Java and Flink paths
export JAVA_HOME=/data/jdk-11.0.29
export Flink_HOME=/data/Flink-1.18.1

Disable Firewall and Start Services

Start all required services, including ZooKeeper, Kafka, Flink, and Apache DolphinScheduler.

# Disable firewall
sudo systemctl stop firewalld

# Start Flink cluster
bin/start-cluster.sh

# Start ZooKeeper
bin/zookeeper-server-start.sh config/zookeeper.properties &

# Start Kafka broker
bin/kafka-server-start.sh config/server.properties &

# Create Kafka topic
bin/kafka-topics.sh --create --topic test \
  --bootstrap-server localhost:9092 \
  --partitions 1 --replication-factor 1

# Produce messages
bin/kafka-console-producer.sh --topic test \
  --bootstrap-server localhost:9092

# Consume messages
bin/kafka-console-consumer.sh --topic test \
  --from-beginning --bootstrap-server localhost:9092

# Start DolphinScheduler Standalone Server
bash ./bin/dolphinscheduler-daemon.sh start standalone-server

Verification

1. Verify Flink Web UI

Access the Flink dashboard at:

http://<VM-IP>:8081

2. Verify DolphinScheduler Web UI

Access DolphinScheduler at:

http://<VM-IP>:12345/dolphinscheduler/ui

Default credentials:

Username: admin
Password: dolphinscheduler123

Sample Implementation

This example demonstrates how Flink consumes data from Kafka, packages the job, uploads it to DolphinScheduler, and executes it as a Flink task.

1. Sample Code

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>Flink-Kafka-demo</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <Flink.version>1.18.1</Flink.version>
        <scala.binary.version>2.12</scala.binary.version>
        <Kafka.version>3.6.0</Kafka.version>
    </properties>

    <dependencies>
        <!-- Flink core dependency -->
        <dependency>
            <groupId>org.apache.Flink</groupId>
            <artifactId>Flink-java</artifactId>
            <version>${Flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.Flink</groupId>
            <artifactId>Flink-streaming-java</artifactId>
            <version>${Flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.Flink</groupId>
            <artifactId>Flink-clients</artifactId>
            <version>${Flink.version}</version>
        </dependency>

        <!-- Connector Base Dependency -->
        <dependency>
            <groupId>org.apache.Flink</groupId>
            <artifactId>Flink-connector-base</artifactId>
            <version>${Flink.version}</version>
        </dependency>

        <!-- Kafka Connector (Key Change) -->
        <dependency>
            <groupId>org.apache.Flink</groupId>
            <artifactId>Flink-connector-Kafka</artifactId>
            <version>3.1.0-1.18</version>
        </dependency>
        <dependency>
            <groupId>org.apache.Kafka</groupId>
            <artifactId>Kafka-clients</artifactId>
            <version>${Kafka.version}</version>
        </dependency>

        <!-- Logging Dependency -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.36</version>
            <scope>runtime</scope>
        </dependency>
    </dependencies>

    <repositories>
        <repository>
            <id>aliyun</id>
            <url>https://maven.aliyun.com/repository/public</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>apache-releases</id>
            <url>https://repository.apache.org/content/repositories/releases/</url>
        </repository>
    </repositories>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <artifactSet>
                                <excludes>
                                    <exclude>org.apache.Flink:force-shading</exclude>
                                    <exclude>com.google.code.findbugs:jsr305</exclude>
                                    <exclude>org.slf4j:*</exclude>
                                </excludes>
                            </artifactSet>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

FlinkKafkaConsumerExample.java

import org.apache.Flink.api.common.functions.FlatMapFunction;
import org.apache.Flink.api.java.tuple.Tuple2;
import org.apache.Flink.api.java.utils.ParameterTool;
import org.apache.Flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.Flink.streaming.api.datastream.DataStream;
import org.apache.Flink.streaming.api.functions.ProcessFunction;
import org.apache.Flink.streaming.api.functions.sink.RichSinkFunction;
import org.apache.Flink.util.Collector;
import org.apache.Flink.streaming.connectors.Kafka.FlinkKafkaConsumer;
import org.apache.Flink.api.common.serialization.SimpleStringSchema;
import org.apache.Kafka.clients.consumer.ConsumerConfig;
import org.apache.Kafka.common.serialization.StringDeserializer;

import java.util.Properties;
import java.util.concurrent.CompletableFuture;


public class FlinkKafkaConsumerExample {
    private static volatile int messageCount = 0;
    private static volatile boolean shouldStop = false;
    public static void main(String[] args) throws Exception {
        // Set the execution environment
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // Kafka configuration
        Properties properties = new Properties();
        properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.146.132:9092"); // Kafka broker 地址
        properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "test-group"); // Consumer group
        properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

        // Create Kafka Consumer
        FlinkKafkaConsumer<String> KafkaConsumer = new FlinkKafkaConsumer<>("test", new SimpleStringSchema(), properties);
        KafkaConsumer.setStartFromEarliest(); // Consume from the earliest messages
        DataStream<String> stream = env.addSource(KafkaConsumer);

        // Process data: tokenization and counting
        DataStream<Tuple2<String, Integer>> counts = stream
                .flatMap(new Tokenizer())
                .keyBy(value -> value.f0)
                .sum(1);


        counts.addSink(new RichSinkFunction<Tuple2<String, Integer>>() {
            @Override
            public void invoke(Tuple2<String, Integer> value, Context context) {
                System.out.println(value);
                messageCount++;

                // Check whether the stop condition is met
                if (messageCount >= 2 && !shouldStop) {
                    System.out.println("Processed 2 messages, stopping job.");
                    shouldStop = true; // Set a flag to indicate that the job should stop
                }
            }
        });

        // Execute the job and obtain JobClient
        CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
            try {
                // Start the job and obtain JobClient
                org.apache.Flink.core.execution.JobClient jobClient = env.executeAsync("Flink Kafka WordCount");
                System.out.println("Job ID: " + jobClient.getJobID());

                // Monitor the condition and cancel the job
                while (!shouldStop) {
                    Thread.sleep(100); // Check every 100 milliseconds
                }

                // Cancel the job when the stop condition is met
                if (shouldStop) {
                    System.out.println("Cancelling the job...");
                    jobClient.cancel().get(); // Cancel the job
                }

            } catch (Exception e) {
                e.printStackTrace();
            }
        });

        // Wait for the job to finish in the main thread
        future.join(); // Wait for the job to finish
            }

    // Tokenizer Class for converting input strings into words
    public static final class Tokenizer implements FlatMapFunction<String, Tuple2<String, Integer>> {
        @Override
        public void flatMap(String value, Collector<Tuple2<String, Integer>> out) {
            String[] tokens = value.toLowerCase().split("\\W+");
            for (String token : tokens) {
                if (token.length() > 0) {
                    out.collect(new Tuple2<>(token, 1));
                }
            }
        }
    }

}

2. Package and Upload to DolphinScheduler

3. Create and Run a Flink Task Node

Start the Kafka producer in the virtual machine and send messages.
Flink successfully consumes and processes the Kafka data.

Demystifying DolphinScheduler’s Plugin Mechanism: Extend Task Types & Data Sources with Ease

Chen Debra — Fri, 26 Jun 2026 09:04:48 +0000

1. High-Level Architecture Overview

DolphinScheduler’s plugin ecosystem is built upon the Java SPI (Service Provider Interface) framework, paired with Google AutoService to auto-generate registration files, enabling non-intrusive plugin-based extension capabilities.

dolphinscheduler-spi                    ← Core Interface Layer (SPI Infrastructure)
dolphinscheduler-datasource-plugin      ← Data Source Plugin Layer
dolphinscheduler-task-plugin            ← Task Plugin Layer
dolphinscheduler-worker                 ← Plugin Consumer Layer (Worker for task execution)

2. SPI Core Infrastructure

PrioritySPI (Root Interface)
  └── getIdentify(): SPIIdentify  ← Unique plugin identifier + priority weight
  └── compareTo(Integer)          ← Priority comparison logic

PrioritySPIFactory<T> acts as the core engine for plugin discovery

// Scan classpath via Java native ServiceLoader
for (T t : ServiceLoader.load(spiClass)) {
    if (map.containsKey(t.getIdentify().getName())) {
        resolveConflict(t);   // Resolve duplicate plugins by priority; throw exception if priorities match
    } else {
        map.put(t.getIdentify().getName(), t);
    }
}

Plugin Registration Mechanism: Each plugin module leverages the @AutoService annotation. During compilation, SPI configuration files are automatically generated under the META-INF/services/ directory, eliminating manual config maintenance.

3. Deep Dive into Data Source Plugin Workflow

3.1 Interface Hierarchy

DataSourceChannelFactory (SPI Entry Point)
  └── getName()          ← Unique plugin identifier, e.g. "MYSQL"
  └── create()           ← Instantiate DataSourceChannel

DataSourceChannel (Connection Channel)
  └── createAdHocDataSourceClient()    ← Create one-off temporary connection client
  └── createPooledDataSourceClient()   ← Create pooled connection client

DataSourceClient (Base Interface)
  └── getConnection(): Connection

PooledDataSourceClient extends DataSourceClient
  └── createDataSourcePool()           ← Initialize HikariCP connection pool

3.2 End-to-End Implementation Example: MySQL Plugin

MySQLDataSourceChannelFactory          ← Registered via @AutoService annotation
  └── create() → MySQLDataSourceChannel
        └── createPooledDataSourceClient() → MySQLPooledDataSourceClient
              └── extends BasePooledDataSourceClient
                    └── createDataSourcePool() → HikariDataSource
                          ├── setDriverClassName()
                          ├── setJdbcUrl()
                          ├── setUsername() / setPassword()
                          ├── setMinimumIdle() / setMaximumPoolSize()
                          └── setConnectionTestQuery()

4. Deep Dive into Task Plugin Workflow

4.1 Interface Hierarchy

TaskChannelFactory (SPI Entry Point) extends UiChannelFactory, PrioritySPI
  └── getName()          ← Unique task type identifier, e.g. "SHELL"
  └── create()           ← Instantiate TaskChannel
  └── getParams()        ← Return UI configuration parameters for frontend rendering

TaskChannel (Task Execution Channel)
  └── createTask(TaskExecutionContext) → AbstractTask
  └── parseParameters(ParametersNode) → AbstractParameters
  └── getResources(parameters)        → ResourceParametersHelper
  └── cancelApplication(boolean)

AbstractTask (Base Task Execution Class)
  └── init()             ← Task initialization logic
  └── handle(callback)   ← Core execution logic (abstract method)
  └── cancel()           ← Task termination logic (abstract method)
  └── getExitStatus()    ← Map exit code to standardized task status

4.2 End-to-End Implementation Example: Shell Task Plugin

ShellTaskChannelFactory                ← Registered via @AutoService annotation
  └── getName() → "SHELL"
  └── getParams() → [nodeName, runFlag, ...]  ← Frontend UI configuration fields
  └── create() → ShellTaskChannel
        └── createTask(ctx) → ShellTask
              └── handle(callback)
                    └── ShellCommandExecutor.run(shellActuatorBuilder)
                          └── Spawn & execute shell script process

5. Side-by-Side Comparison of Two Plugin Types

Feature	Redshift Concurrency Scaling	DolphinScheduler Task Groups
Control Granularity	Cluster-level, auto-scaling	Task/Workflow-level, manual control
Modification Cost	Low; only parameter configuration required	Medium; task grouping planning needed
Cost Impact	Pay-as-you-go, potential extra charges	No additional fees
Applicable Scenarios	Sudden, unpredictable load spikes	Known, stable high-concurrency scheduling scenarios
Recommended Usage	Acts as a "fuse" to handle unexpected traffic surges	Acts as a "throttle valve" for daily concurrency governance

Apache DolphinScheduler + AWS Data Lakehouse: Practical Hybrid Scheduling & Cloud Cost Optimization Guide

Chen Debra — Fri, 26 Jun 2026 08:14:28 +0000

1. Project Overview: Apache DolphinScheduler Meets AWS Data Lakehouse

As data-driven business decision-making becomes mainstream across industries, a high-performance, flexible, cost-efficient data-processing pipeline is the core competitive edge of enterprise data platforms. I’ve collaborated with numerous engineering teams that initially built on-premises Hadoop clusters paired with self-maintained scheduling systems. However, skyrocketing data volumes and growing business complexity have exposed severe bottlenecks in this legacy architecture: lengthy resource scaling cycles, excessive operational overhead, rigid task orchestration logic, and most critically, a lack of balanced performance-cost tradeoffs.

Mature cloud-native data architectures—especially AWS’s full-spectrum suite spanning Amazon S3 data lakes, Amazon Redshift data warehouses, and Amazon EMR big-data processing engines—have delivered a transformative solution to these pain points. Simply migrating standalone components to the cloud is insufficient; seamless interoperability and unified orchestration of these services are the real keys to unlocking the cloud platform's potential. This is where Apache DolphinScheduler, an open-source distributed visual workflow task scheduler, delivers unmatched value. Acting as a unified "data pipeline conductor", it coordinates Amazon EMR for large-scale batch data processing and orchestrates Amazon Redshift to power high-performance analytical queries.

Drawing from a real customer cloud migration and optimization project, this article delivers an in-depth breakdown of deep DolphinScheduler integration within AWS’s Intelligent Data Lakehouse framework. We focus on two core practical scenarios: hybrid scheduling and cost optimization for Amazon EMR (covering traditional EC2-backed clusters and serverless EMR runtime), plus reliable, secure Amazon Redshift task orchestration with built-in mitigations for Redshift’s native concurrency limits. Whether you are a data platform engineer planning cloud migration or an architect aiming to boost efficiency for existing cloud-based data pipelines, the battle-tested patterns, production-ready code snippets, and troubleshooting insights shared in this hands-on guide will deliver actionable, immediate value.

2. Architecture & Tool Selection: Why DolphinScheduler + AWS Is the Optimal Combination

Before diving into hands-on implementation, we first unpack the rationale behind this tech stack pairing: what unique benefits AWS and Apache DolphinScheduler each bring, and which core pain points their combined architecture resolves. Clear alignment on these foundational design decisions streamlines subsequent deployment and customization workstreams.

2.1 Deep Dive into AWS Intelligent Data Lakehouse Architecture

AWS’s Intelligent Data Lakehouse is not a single monolithic product, but a set of industry best practices centered on Amazon S3 as the universal data lake storage layer. Its core design principle eliminates data silos, enabling secure, frictionless data movement across storage, batch processing, real-time analytics, and machine learning workloads.

Within this ecosystem, Amazon S3 acts as infinitely scalable centralized storage for all raw and refined datasets. A suite of fully managed AWS services operates alongside S3 with specialized, decoupled responsibilities:

Data Ingestion: Amazon Kinesis for real-time streaming pipelines, Amazon MSK (Managed Streaming for Apache Kafka) for persistent stream processing, and AWS Glue for serverless ETL and centralized data catalog governance.
Big Data Batch Processing: Amazon EMR serves as the primary compute layer, offering managed deployments of open-source frameworks including Apache Spark, Apache Hive, and Apache Flink for data cleansing, transformation, and complex distributed computing jobs.
Data Warehousing & BI Analytics: Amazon Redshift delivers high-performance petabyte-scale cloud data warehousing, powering complex aggregated queries, business intelligence dashboards, and ad-hoc analytical workloads.
Machine Learning & AI: Amazon SageMaker natively accesses datasets stored in S3 and Redshift to run model training, validation, and real-time inference pipelines.

The "intelligent" differentiator of this architecture lies in native cross-service integration: Amazon EMR and Redshift directly read S3 datasets without costly data duplication or movement, while the AWS Glue Data Catalog functions as a unified metadata store shared across EMR, Redshift, and Amazon Athena. Our core integration challenge lies in building a single orchestration layer capable of reliably sequencing, monitoring, and governing disparate workloads running across all these managed AWS services.

2.2 Core Value & Strategic Positioning of Apache DolphinScheduler

Apache DolphinScheduler is a distributed, visual workflow scheduling platform built for data engineering teams. It outperforms legacy scheduling tools such as cron jobs and custom shell script orchestration via the following key capabilities:

Visual DAG Orchestration: Drag-and-drop canvas construction of complex Directed Acyclic Graph (DAG) workflows, rendering upstream/downstream task dependencies fully transparent and lowering barriers for citizen data developers.
Enterprise-Grade Reliability & HA: Decentralized Master-Worker cluster architecture supporting horizontal elastic scaling; single-node failures do not halt overall platform availability, with native task retry logic, timeout controls, and multi-channel alerting built in.
Multi-Tenant Isolation & Fine-Grained RBAC: Resource segmentation and permission controls organized by project and user team, ideal for collaborative enterprise data teams across departments.
Extensive Native Task Types & Plugin Ecosystem: Out-of-the-box support for Shell, Python, generic SQL, Apache Spark, Apache Flink, and additional task runtimes, paired with a robust plugin extension framework—this extensibility forms the technical foundation for deep native integration with AWS managed services.

Within the AWS data lakehouse stack, DolphinScheduler functions exclusively as the unified orchestration plane. It does not replace EMR’s distributed compute capacity or Redshift’s analytical query engine; instead, it operates at a higher abstraction layer to govern critical execution logic: defining timelines, injecting dynamic runtime parameters, routing Spark and Redshift SQL jobs to designated compute resources (EC2-based EMR clusters, EMR Serverless, Redshift warehouses), and monitoring full task lifecycles end-to-end.

2.3 Hybrid Compute Resource Routing Strategy Design

Following their migration from on-premises Hadoop to Amazon EMR on EC2, the customer initially ran all workloads on persistent EC2-backed EMR clusters. They quickly identified severe cost inefficiencies stemming from uneven resource utilization driven by highly variable job load patterns, split into three distinct workload categories:

Small, Short-Lived Jobs: High volume of lightweight tasks with 20–30 minute average runtime, triggered sporadically around the clock. Provisioning full multi-node EMR clusters for these micro-jobs creates massive idle resource waste, analogous to deploying heavy artillery to eliminate minor targets.
Medium-Dense Daily Batch Jobs: Compute-heavy workloads executing within a fixed 7–8 hour daily processing window, suited for dedicated EMR clusters—yet persistent cluster provisioning incurs idle-hour billing outside the scheduled execution window.
Massive Infrequent Batch Jobs: Multi-day long-running calculations executed only 2–3 times monthly; maintaining dedicated persistent clusters for these rare workloads generates prohibitive recurring cloud costs.

To address this imbalance, we implemented a hybrid intelligent workload routing strategy:

Small sporadic jobs + rare long-running massive jobs → EMR Serverless: AWS EMR Serverless eliminates cluster provisioning and infrastructure management overhead entirely; users only submit jobs and pay per consumed vCPU and memory second. This runtime perfectly aligns with sporadic, bursty workload patterns: micro-jobs avoid persistent cluster idle fees, while infrequent long-running jobs eliminate the cost of permanently reserved compute infrastructure.
Fixed-window medium-dense daily jobs → Time-bound auto-start/auto-terminate EMR on EC2 clusters: For predictable daily batch windows, we retain EC2-backed EMR clusters but automate full cluster lifecycle management via DolphinScheduler: clusters spin up automatically before workflow execution begins and terminate entirely once all daily tasks complete. Supplementing core task nodes with EC2 Spot Instances further cuts compute costs by 60–70%.

The foundational requirement enabling this strategy is DolphinScheduler’s custom intelligent workload routing capability: workflows must automatically route tasks to either EMR Serverless or persistent EMR EC2 clusters based on pre-defined task metadata tags and resource demand profiles. This required targeted custom development to wrap standardized, abstracted job submission interfaces within DolphinScheduler.

3. Integration Practice: Unified API Wrapper for EMR Orchestration

While the hybrid routing strategy delivers clear cost benefits, critical API and runtime behavioral disparities between EMR on EC2 and EMR Serverless present the primary integration roadblock. Our core design objective is full workload transparency for data developers: engineers author tasks in DolphinScheduler without needing to identify the underlying EMR runtime executing their code.

3.1 Core Integration Challenges: Four Key Disparities Between EMR Runtime Modes

Four fundamental technical differences complicate unified orchestration across the two EMR variants:

Job Submission & Synchronization Semantics: EMR on EC2 Step APIs operate in semi-synchronous mode, supporting blocking execution waits with straightforward final status polling logic. EMR Serverless job submission APIs operate fully asynchronously: job IDs return instantly post-submission, requiring separate repeated API polling to retrieve real-time job status—creating compatibility gaps for schedulers reliant on synchronous task execution to trigger downstream DAG branches based on upstream success/failure.
Logging Persistence Paths: EMR on EC2 logs store outputs locally on cluster master nodes or HDFS, with optional CloudWatch Logs forwarding. EMR Serverless enforces mandatory log export to designated Amazon S3 prefixes, with entirely distinct log retrieval endpoints and file structures across the two runtimes.
Divergent AWS SDK Interfaces: Separate boto3 SDK clients govern each service (emr client for EC2 clusters, emr-serverless client for serverless jobs), with mismatched parameter schemas, argument naming conventions, and response payload structures.
Native SQL Support Gaps: EMR on EC2 natively accepts inline SQL strings via Spark SQL or Hive Step definitions. Early EMR Serverless releases lacked direct inline SQL submission, requiring all SQL logic to be wrapped within standalone Spark script files uploaded to S3 prior to job launch.

Forcing developers to maintain dual code paths for each EMR runtime would introduce excessive operational complexity and technical debt. Our resolution is building a centralized abstracted Python SDK named emr_common.

3.2 Implementation: Build a Unified Abstraction Python SDK

We developed the open internal emr_common Python library centered on a standardized top-level Session factory class. This factory initializes specialized child session objects (EMRSession or EMRServerlessSession) based on an input job_type parameter while exposing identical standardized public methods to upstream consumers.

# emr_common/session.py
import boto3
from abc import ABC, abstractmethod

class BaseEMRSession(ABC):
    """Abstract base class for unified EMR runtime sessions"""
    @abstractmethod
    def submit_sql(self, job_name: str, sql: str, **kwargs):
        pass

    @abstractmethod
    def submit_file(self, job_name: str, file_path: str, **kwargs):
        pass

    @abstractmethod
    def get_status(self, job_id: str) -> str:
        pass

    @abstractmethod
    def get_logs(self, job_id: str) -> str:
        pass

class EMRSession(BaseEMRSession):
    """Concrete implementation for EMR on EC2 clusters"""
    def __init__(self, cluster_id: str = None):
        self.client = boto3.client('emr')
        # Auto-locate active waiting clusters if no explicit cluster ID provided
        self.cluster_id = cluster_id or self._find_active_cluster()

    def submit_sql(self, job_name: str, sql: str, **kwargs):
        # Construct standard EMR Step input parameters
        step_args = [
            'spark-sql',
            '-e',
            sql
        ]
        response = self.client.add_job_flow_steps(
            JobFlowId=self.cluster_id,
            Steps=[{
                'Name': job_name,
                'ActionOnFailure': 'CONTINUE',
                'HadoopJarStep': {
                    'Jar': 'command-runner.jar',
                    'Args': step_args
                }
            }]
        )
        return response['StepIds'][0]

    # Supplementary method implementations: submit_file, get_status, get_logs omitted for brevity

class EMRServerlessSession(BaseEMRSession):
    """Concrete implementation for EMR Serverless applications"""
    def __init__(self, application_id: str = None):
        self.client = boto3.client('emr-serverless')
        self.application_id = application_id or self._find_active_application()

    def submit_sql(self, job_name: str, sql: str, **kwargs):
        # Wrap raw SQL inside temporary PySpark script (required limitation for EMR Serverless)
        import tempfile
        with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
            f.write(f"""
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("{job_name}").getOrCreate()
spark.sql(\"\"\"{sql}\"\"\")
""")
            script_path = f.name
        # Internal helper function to upload local script to target S3 bucket
        s3_script_uri = self._upload_to_s3(script_path)
        return self.submit_file(job_name, s3_script_uri)

    def submit_file(self, job_name: str, file_path: str, **kwargs):
        # Construct standardized EMR Serverless job request payload
        response = self.client.start_job_run(
            applicationId=self.application_id,
            executionRoleArn='arn:aws:iam::xxx:role/EMRServerlessRole',
            jobDriver={
                'sparkSubmit': {
                    'entryPoint': file_path,
                    'sparkSubmitParameters': '--conf spark.executor.memory=4g'
                }
            },
            configurationOverrides={...},
            name=job_name
        )
        return response['jobRunId']

    # Supplementary method implementations: get_status, get_logs with async polling & S3 log parsing omitted for brevity

def Session(job_type: int = 0, **kwargs):
    """Factory function returning standardized EMR session instance"""
    if job_type == 0:
        return EMRSession(**kwargs)
    elif job_type == 1:
        return EMRServerlessSession(**kwargs)
    else:
        raise ValueError(f"Unsupported job_type input value: {job_type}")

Key architectural design highlights of the abstraction layer:

Synchronous Wrapper for Asynchronous Serverless APIs: The EMRServerlessSession.get_status() method encapsulates configurable interval polling logic with blocking execution until job completion or terminal failure status, delivering synchronous execution semantics to upstream DolphinScheduler workflows.
Unified Standardized Log Retrieval: The shared get_logs() abstract method internally detects the underlying EMR runtime, fetching logs from S3 prefixes for Serverless workloads or CloudWatch/local cluster storage for EC2-backed EMR jobs, and returns consistently formatted log output for centralized troubleshooting.
Intelligent Default Fallback Logic: When users omit explicit cluster_id or application_id inputs, the SDK automatically scans the target AWS account/region for idle WAITING or STARTED EMR resources, reducing mandatory configuration overhead for end developers.
Transparent Spark Configuration Normalization: A shared standardized configuration dictionary accepts generic tuning parameters including driver_memory and executor_cores, with the SDK internally translating these unified inputs into runtime-specific argument schemas required by each EMR variant. ### 3.3 End-User Workflow Implementation within Apache DolphinScheduler Once the emr_common abstraction SDK is packaged and deployed to all DolphinScheduler Worker nodes, orchestration becomes streamlined via DolphinScheduler’s native Python Operator task type. Python Operator Task Template Supporting Dynamic EMR Runtime Routing:

# Reusable task template dynamically switching EMR runtime based on workflow parameters
from emr_common import Session

def main(**kwargs):
    # Retrieve routing parameter passed via DolphinScheduler custom node/workflow variables
    # emr_job_type: 0 = EMR on EC2, 1 = EMR Serverless
    job_type = int(kwargs.get('emr_job_type', 0))
    task_name = kwargs.get('task_name', 'default_data_processing_job')

    # Initialize standardized abstract EMR session
    session = Session(job_type=job_type)

    # Example: Submit inline Spark SQL transformation job
    sql = """
    SELECT date, count(*) as cnt
    FROM your_core_operational_table
    WHERE date = '${bizdate}'  -- Native DolphinScheduler built-in business date parameter
    GROUP BY date
    """
    job_id = session.submit_sql(job_name=f"{task_name}_daily_agg_sql", sql=sql)

    # Block execution with 30-second polling interval until terminal job state
    status = session.wait_for_completion(job_id, interval=30)
    print(f"Distributed compute job {job_id} completed with final status: {status}")

    # Pull full execution logs for workflow failure troubleshooting
    full_task_logs = session.get_logs(job_id)
    # Append critical log snippets to DolphinScheduler native task log storage
    # ...

    # Trigger workflow success/failure exit codes aligned with job terminal state
    if status == 'SUCCESS':
        return True
    else:
        raise Exception(f"Distributed compute job terminated with failure status: {status}")

if __name__ == "__main__":
    # Parse runtime parameters injected by DolphinScheduler task runtime
    import sys
    import json
    input_params = json.loads(sys.argv[1]) if len(sys.argv) > 1 else {}
    task_execution_success = main(**input_params)
    sys.exit(0 if task_execution_success else 1)

Parameterized Workflow Design Best Practice: Global workflow variables or node-level custom parameters are defined within DolphinScheduler’s workflow editor. For instance, a top-level global variable emr_engine defaults to the value serverless. A lightweight mapping helper translates serverless to job_type=1 and ec2_cluster to job_type=0. This design enables one-click global runtime switching for all tasks within an entire workflow by simply modifying a single global parameter, drastically accelerating cross-runtime testing and iterative optimization.

Unified Metadata Governance Critical Note: To enable seamless cross-runtime data access and consistent table schema visibility across EMR on EC2 and EMR Serverless workloads, organizations must standardize on the AWS Glue Data Catalog as the shared metadata repository. All EMR clusters and Serverless applications must be provisioned with Glue Catalog integration enabled during resource creation. This eliminates the operational burden of maintaining separate Hive metastores for distinct compute runtimes while guaranteeing identical database and table schemas visible to all data processing jobs.

4. Amazon Redshift Task Orchestration & Concurrency Throttling Implementation

Following EMR integration, we shift focus to orchestrating Amazon Redshift, AWS’s enterprise petabyte-scale columnar data warehouse optimized for complex aggregated analytical queries. Starting with DolphinScheduler 3.x releases, the platform’s built-in Datasource Center delivers native Amazon Redshift JDBC connectivity, enabling direct Redshift SQL execution via DolphinScheduler’s standard SQL Operator task type.

4.1 Basic Redshift Orchestration via DolphinScheduler SQL Operator

This represents the most widely adopted, straightforward implementation pattern for Redshift workload scheduling. After completing Redshift data source configuration within DolphinScheduler’s web UI, users create dedicated SQL Operator nodes to execute data warehouse transformation logic.

Step-by-Step Configuration & Critical Operational Notes

Datasource Connection Setup: Standard JDBC connection string format: jdbc:redshift://[Redshift Cluster Endpoint]:[Port]/[Target Database Name]. The primary networking prerequisite is bidirectional connectivity between DolphinScheduler Worker nodes and the Redshift cluster VPC, requiring properly configured VPC peering links and security group inbound/outbound traffic rules.
Dynamic Parameterized SQL Authoring: The SQL editor supports DolphinScheduler’s native ${variable} syntax to reference built-in system parameters and upstream task output payloads, enabling fully dynamic templated warehouse queries.

-- Sample daily partitioned table refresh ETL script for Redshift
CREATE TABLE IF NOT EXISTS dws_user_daily_activity (
    biz_date DATE,
    user_id BIGINT,
    daily_activity_count INT
) DISTKEY(user_id) SORTKEY(biz_date);

DELETE FROM dws_user_daily_activity WHERE biz_date = '${bizdate}';

INSERT INTO dws_user_daily_activity
SELECT
    '${bizdate}'::DATE as biz_date,
    user_id,
    COUNT(*) as daily_activity_count
FROM ods_user_raw_operation_logs
WHERE event_date = '${bizdate}'
GROUP BY user_id;

Redshift-Specific Maintenance Automation: Schedule periodic maintenance operations including ANALYZE (to refresh table query statistics) and VACUUM (to reclaim storage space and re-sort tables) as dedicated low-priority DolphinScheduler tasks, restricted to off-peak business maintenance windows to avoid query performance degradation.

Critical Operational Caveat: Redshift’s transaction semantics diverge sharply from OLTP relational databases; multi-statement BEGIN; ... COMMIT; blocks exhibit inconsistent behavior when wrapping DDL operations or bulk DML inserts. Data engineers are advised to separate DDL (CREATE, ALTER, DROP) and bulk DML (INSERT, UPDATE, DELETE) logic into independent sequential DolphinScheduler tasks with explicit dependency links, mitigating table-level locking risks and unintended transaction rollbacks.

4.2 Resolving Redshift Concurrency Bottlenecks: Two Production-Grade Throttling Strategies

Redshift’s MPP architecture delivers fast analytical query performance but enforces finite concurrency slots (default cluster limits typically cap at 50 concurrent query slots). Unregulated parallel task submission from DolphinScheduler’s large-scale batch workflows frequently exhausts available slots, triggering prolonged query queuing and intermittent task failures. We have validated two complementary mitigation strategies for production environments.

Strategy 1: Enable Amazon Redshift Concurrency Scaling

This managed AWS-native paid feature automatically provisions temporary auxiliary scaling clusters when primary cluster concurrency slots reach saturation, expanding maximum supported parallel query capacity up to 10x baseline limits. The primary advantage is fully transparent horizontal scaling, requiring zero modifications to DolphinScheduler workflows or SQL logic.

Deployment Steps: Modify the target Redshift cluster parameter group via AWS Console or API, setting max_concurrency_scaling_clusters to a positive integer value (e.g., 5), paired with a CPU utilization threshold trigger for auto-scaling activation.
Cost Efficiency Analysis: Auxiliary concurrency scaling clusters bill per-second at higher hourly rates than primary Redshift nodes. Teams must evaluate peak load duration and frequency to calculate total incremental spend. Our field experience confirms this solution delivers strong ROI for predictable daily morning BI report traffic spikes, eliminating the need to over-provision primary cluster capacity permanently for short-duration peak demand. #### Strategy 2: Workload Throttling via DolphinScheduler Native Task Groups This zero-additional-cost strategy delivers granular, fully controllable parallelism limits directly within the orchestration platform.
Task Group Resource Creation: Navigate to DolphinScheduler’s Resource Center to provision a dedicated task group (example label: redshift_query_group).
Enforce Parallelism Caps: Configure a hard maximum concurrent task limit for the group (example value: 15). This threshold should sit comfortably below the Redshift cluster’s safe recommended concurrency ceiling, reserving spare slots for ad-hoc analyst BI tool connections outside scheduled batch pipelines.
Bind Redshift Workloads to the Controlled Group: Assign all DolphinScheduler SQL Operator nodes executing Redshift queries to the defined throttled task group via node configuration panels.
Operational Outcome: Even with hundreds of ready-to-execute Redshift tasks queued within the workflow, DolphinScheduler strictly caps parallel submission to the configured limit, protecting the Redshift warehouse from overwhelming query load and slot exhaustion.

Combined Strategy Recommendation & Comparative Analysis

Our validated production best practice combines both controls simultaneously: baseline traffic is governed via DolphinScheduler task group parallelism limits, while enabling 1–2 concurrency scaling, auxiliary clusters act as a safety buffer to absorb unplanned ad-hoc analyst queries or abnormally long-running scheduled batch jobs consuming excessive concurrency slots.

4.3 Shell Operator Integration with Git CI/CD Pipelines

Beyond the native SQL Operator, DolphinScheduler’s Shell Operator delivers flexible orchestration capabilities for teams maintaining version-controlled SQL script files stored in Git repositories, enabling end-to-end data engineering DevOps workflows.

Standard End-to-End Implementation Pattern

Data developers author versioned Redshift SQL transformation scripts (e.g., transform_sales_daily.sql) locally and commit finalized code to central Git repositories such as GitLab or GitHub.
CI/CD automation pipelines (Jenkins, GitHub Actions) trigger post-code-merge artifact delivery, uploading validated SQL script files to a dedicated versioned Amazon S3 artifact bucket.
A DolphinScheduler Shell Operator task executes the following runtime command to pull and execute remote S3-hosted SQL scripts with dynamic business date parameter injection:

# Assumption: psql client pre-installed on all DolphinScheduler Worker nodes; AWS CLI S3 access configured via IAM roles
export PGPASSWORD=${redshift_password}
psql -h ${redshift_host} -p ${redshift_port} -U ${redshift_user} -d ${redshift_db} \
     -f s3://your-central-script-bucket/etl/transform_sales_daily.sql \
     -v bizdate=\'${bizdate}\'

The -f flag directs the psql client to fetch the SQL script file directly from the defined S3 URI, requiring the Redshift cluster IAM role to include read permissions for the target script bucket.
The -v argument injects runtime variables accessible within the SQL script via :bizdate template syntax.

Advanced Integration with DolphinScheduler Resource Center: A cleaner native alternative leverages DolphinScheduler’s built-in Resource Center, which can mount Amazon S3 buckets as persistent file storage via compatible third-party storage plugins. SQL scripts are uploaded, edited, and version-controlled directly through DolphinScheduler’s web UI, eliminating separate CI/CD artifact upload steps. Shell Operator tasks simply reference the internal Resource Center file path, with DolphinScheduler automatically handling remote file fetching during task initialization.

This unified workflow tightly integrates orchestration (DolphinScheduler), source code version control (Git), continuous delivery automation (CI/CD runners), and object storage (Amazon S3), establishing a fully auditable, reproducible data engineering DevOps pipeline with immutable script version history.

5. Operations, Observability & Continuous Cloud Cost Optimization Playbook

Deep integration between DolphinScheduler and distributed AWS managed services expands operational monitoring scope beyond individual standalone tools to full end-to-end pipeline observability. Additionally, AWS’s pay-as-you-go consumption model requires iterative, ongoing cost governance rather than one-time migration optimization efforts.

5.1 End-to-End Pipeline Monitoring & Multi-Channel Alerting

Task failures can originate from DolphinScheduler platform instability, inter-service networking errors, AWS compute resource exhaustion, or flawed transformation logic. Organizations must implement layered observability covering every architectural tier:

Native DolphinScheduler Platform Alerting: Leverage the built-in alert framework to deliver notifications for task failures, execution timeouts, and workflow suspension via email, DingTalk, Slack, or generic webhook endpoints. Critical monitoring targets include workflow instance state and individual task runtime status.
AWS CloudWatch Managed Service Telemetry:
1. Amazon EMR: Monitor metrics including YARNMemoryAvailablePercentage, ContainerPendingRatio to pre-empt compute resource starvation; track Step execution duration and success/failure ratios.
2. EMR Serverless: Track JobRun success/failure rates, total wall-clock runtime, and consumed vCPU/memory resources (direct cost drivers for serverless billing).
3. Amazon Redshift: Monitor DatabaseConnections, CPUUtilization, average QueryDuration; configure alerts for active concurrency scaling clusters via the ConcurrencyScalingActiveClusters metric to identify peak load frequency.
Custom Application Log Forwarding: The open emr_common SDK extends standardized logging beyond DolphinScheduler’s native task logs, pushing critical lifecycle events (job start, job completion, resource consumption totals) to CloudWatch Log Streams or Amazon SNS topics for centralized analysis and automated Lambda-driven remediation workflows.
Embedded Data Quality Validation Checks: Successful task execution exit codes do not guarantee accurate output datasets. Append dedicated Python validation tasks to the terminal stage of every DolphinScheduler workflow to run data integrity checks against Redshift tables—validating record volume thresholds, null value ratios, and business logic constraints—with immediate alert triggers for anomalous dataset quality deviations.

5.2 Field-Proven Cloud Cost Optimization Tactics

Cloud cost governance for this integrated lakehouse architecture relies on granular workload tuning segmented by compute runtime type:

Cost Optimizations for EMR on EC2 Clusters

Mixed On-Demand + Spot Instance Tiering: Deploy Spot Instances for all Task compute nodes to cut billing costs by 60–70%; reserve On-Demand EC2 instances for Master and Core nodes to guarantee persistent storage and cluster control plane stability.
Dynamic Cluster Auto-Scaling: Configure YARN metric-based horizontal scaling policies responding to pending container backlogs, eliminating persistent over-provisioning of idle cluster capacity.
Scheduled Auto-Lifecycle Orchestration via DolphinScheduler: Embed pre-workflow cluster startup and post-workflow termination Python tasks leveraging boto3 AWS SDK automation. Clusters remain provisioned exclusively during daily batch processing windows, eliminating idle-hour billing outside scheduled execution hours.

Cost Optimizations for EMR Serverless Workloads

Precision Resource Allocation Tuning: Spark job driver and executor memory allocations directly determine serverless billing totals. Analyze historical CloudWatch resource utilization metrics to identify optimal memory sizing thresholds, eliminating over-provisioned reserved memory waste; standardized default tuning parameters are embedded within the shared emr_common SDK.
Provisioned Capacity Warm Pools for Low-Latency Workloads: Enable EMR Serverless pre-provisioned warm capacity pools for latency-sensitive micro-jobs to mitigate cold-start delay overhead. Note this feature incurs persistent background billing, requiring careful latency vs. cost tradeoff evaluation.
File Compaction Pre-Processing: Execute lightweight preprocessing jobs (AWS Glue ETL or S3 DistCp) to merge thousands of tiny source data files into larger optimized partitions before launching heavy EMR Serverless transformation workloads, drastically reducing job startup overhead and cumulative compute runtime spend.

Cost Optimizations for Amazon Redshift Warehouses

Off-Peak Scheduling for Resource-Heavy Maintenance: Schedule resource-intensive warehouse operations, including VACUUM, full table ANALYZE, and bulk INSERT/DELETE ETL loads exclusively during overnight business off-hours, avoiding resource contention with daytime BI analytical queries.
Dedicated WLM Query Queue Segmentation: Configure separate Redshift Workload Management queues for short DolphinScheduler batch ETL jobs with isolated concurrency and memory allocations, preventing long-running complex analytical queries from starving lightweight transformation workloads.
Continuous Slow Query Monitoring & Tuning: Track CloudWatch metrics for abnormal QueryDuration and ScanRowCount values, triggering automated alerts for poorly optimized queries requiring SQL refactoring or table schema adjustments (DISTKEY/SORTKEY reconfiguration).

5.3 Common Production Troubleshooting Checklist

Problem	Possible Causes	Troubleshooting Steps
DolphinScheduler task remains in "Submitting" status	1. Network connectivity failure between Worker nodes and AWS services. 2. The IAM role/access key does not have the corresponding permissions. 3. API calls in the encapsulated SDK time out or are blocked.	1. Test connectivity using `telnet` or `aws cli` on the Worker nodes. 2. Check the policies attached to the IAM user used by the Worker node EC2 or DolphinScheduler. 3. View DolphinScheduler Worker logs to locate the specific error line.
EMR Serverless job submission failed	1. The Application is not in the `STARTED` state. 2. Insufficient permissions for the Execution Role. 3. The specified S3 script path does not exist or access permission is denied.	1. Call the `list-applications` API to check the status. 2. Check the policies of the Execution Role, which must have at least S3 read/write, Glue access, and CloudWatch log write permissions. 3. Manually test whether the S3 script can be read using AWS CLI.
Redshift SQL task connection timeout	1. The connection is blocked by the security group/network ACL. 2. The Redshift cluster is paused. 3. The number of connections is full.	1. Check the inbound rules of the Redshift cluster security group. 2. Check the cluster status in the AWS console. 3. Query the `stv_sessions` system table to check current connections.
Task succeeded but data was not updated	1. SQL logic error (e.g., incorrect WHERE clause). 2. Transaction not committed (some DDLs in Redshift are auto-committed, but script errors may cause subsequent DMLs to not execute). 3. Data latency (e.g., from Kinesis to S3).	1. Manually run the task SQL in the Redshift query editor. 2. Check the complete SQL (after variable replacement) in the DolphinScheduler task logs. 3. Check whether the upstream data source task succeeded.
Abnormal cost surge	1. The EMR cluster was not terminated on time. 2. Excessively high memory configuration for EMR Serverless jobs. 3. Redshift Concurrency Scaling is triggered excessively.	1. Check the EMR cluster runtime duration alarms in CloudWatch. 2. Analyze the `MemoryUtilization` metric of EMR Serverless jobs. 3. View the history of the `ConcurrencyScalingActiveClusters` metric for Redshift.

6. Summary & Future Roadmap Outlook

The core transformative value delivered by this integrated architecture lies in the abstraction layer introduced by Apache DolphinScheduler, unifying heterogeneous AWS compute services (EC2-backed EMR, EMR Serverless, Amazon Redshift) into a logically single, centrally governed data compute platform. Data engineers focus solely on implementing business transformation logic via SQL and PySpark, abstracted entirely from underlying cluster infrastructure management and runtime routing complexity. Platform operations teams gain a single pane of glass for full workflow visibility, standardized observability, and centralized cost governance controls.

This successful deep integration rests on two foundational architectural choices: abstracted unified SDK wrapping to eliminate cross-service API disparities, plus full utilization of DolphinScheduler’s native parameterization, plugin extensibility, and built-in workload throttling controls enabling flexible, customizable orchestration policies.

Looking forward to community-driven enhancements for Apache DolphinScheduler, two high-impact feature developments will further advance intelligent autonomous cloud data pipeline orchestration:

SQL Syntax Tree Parsing for Column-Level Data Lineage: Current lineage tracking mechanisms rely on basic regular expression matching or manual metadata entry, limiting precision. Native SQL AST parsing within DolphinScheduler task execution would generate end-to-end field-level data lineage graphs, drastically elevating enterprise data catalog and governance capabilities.
AI Agent Operator for Autonomous Workflow Optimization: A native AI Agent task runtime would leverage historical job runtime metrics, resource consumption profiles, and input dataset volumes to dynamically auto-tune EMR Serverless executor resource allocations, intelligently route workloads between EMR runtimes, and autonomously parse failure logs to generate diagnostic troubleshooting recommendations or automated retry remediation logic. This autonomous orchestration capability represents a critical milestone toward self-operating data pipeline platforms.

Cloud-native data engineering is an iterative, evolving discipline; deep cross-service integration and intelligent autonomous orchestration remain core levers to boost data team engineering throughput. The standardized architectural patterns, production code samples, and hands-on troubleshooting guidance shared within this guide deliver actionable blueprints for teams building or modernizing cloud-based data scheduling platforms across global markets, including North America, Europe, India, and Southeast Asia.

Create Your First Apache DolphinScheduler Workflow in Five Minutes!

Chen Debra — Thu, 18 Jun 2026 07:58:22 +0000

In this article, we will use DolphinScheduler to create and run a simple workflow step by step. During this journey, you will learn the basic concepts of DolphinScheduler and know the most basic configuration to run the workflow.

Setup Dolphinscheduler

You have to install and start dolphinscheduler first before go ahead. For beginners, we recommend setting up DolphinScheduler with the official Docker image or with the standalone server.

Standalone server
Docker

Build Your First Workflow

You can log in DolphinScheduler through http://localhost:12345/dolphinscheduler/ui and the default username/password is admin/dolphinscheduler123.

Create Tenant

Tenant is an important concept while using DolphinScheduler, so let's briefly introduce the concept of tenant first.

DolphinScheduler maps the admin account you use to log into DolphinScheduler to user. To better control system resources, DolphinScheduler introduces the concept of tenants, which are used to execute tasks.

The brief is as follows:

User: login web UI, do all operations in the web UI, including workflow management and tenant creation.
Tenant: the actual executor of the task, A Linux user for DolphinScheduler worker.

We can create a tenant in DolphinScheduler Security -> Tenant Manage page.

NOTE: The user will bind to a default tenant when it is created, if you use the default tenant, the task will be executed by worker's bootstrap user.

Assign Tenant to User

As we have shown above in Create Tenant section, theuser can not run tasks until we assign it with a tenant.

We can assign a tenant to a specific user in DolphinScheduler Security -> User Manage page.

After we create a tenant and assign it to a user, we can start creating a simple workflow in DolphinScheduler.

Create Project

But in DolphinScheduler, all workflows must belong to a project, so we need to create a project first.

We can create a project in DolphinScheduler Project page by clicking Create Project button.

Create Workflow

Now we can create a workflow for the project tutorial. Click the project we just created, go to Workflow Definition page, click Create Workflow button, and we will redirect to the workflow detail page.

Create Tasks

We can use the mouse to drag the task you want to create from the toolbar in the workflow canvas. In this case, we create a Shell task. Entering the necessary information for the task, we just fill the attribute Node Name with Script to the task for this simple workflow. After that, we can click the Save button to save the task into the workflow. We create another task using the same way.

Set Task Dependency

So we have two different tasks with different names and commands to run in the workflow. The only thing missing from the current workflow is task dependency. We can add dependency using the mouse to drag the arrow from the upstream task to the downstream and then release the mouse. And you can see the link with the arrow between the two tasks is created, from the upstream task to the downstream one. Finally, we can click the Save button from the top right corner to save the workflow, do not forget to enter the name of the workflow.

Run Workflow

After all done, we can run the workflow by clicking the Online and then the Run button from the workflows list. If you want to see the workflow instance, just go to Workflow Instance page, you can see the workflow instance is running and the status is Executing.

View Log

If you want to view the task log, please click the workflow instance from the workflow instance list, then find the task you want to view the log, right-click the mouse and select View Log from the context dialog, and you can see the detailed log of the task.

You can see the task printing Hello DolphinScheduler and Ending... which is the same as we defined when creating the tasks.

You just finished the first tutorial of DolphinScheduler, you can now run some simple workflows in DolphinScheduler, congratulations!

Deploying Apache DolphinScheduler 3.1.9 Cluster with MySQL Instead of PostgreSQL? A Practical Docker Compose Guide

Chen Debra — Thu, 18 Jun 2026 07:37:46 +0000

Background

Today, we'll walk through one of the most requested deployment scenarios in the Apache DolphinScheduler community: deploying an Apache DolphinScheduler 3.1.9 cluster with MySQL as the metadata database using Docker Compose.

As many users know, the official Docker Compose deployment provided by DolphinScheduler uses PostgreSQL as the metadata repository by default. This is mainly because the GPLv2 license of MySQL is not fully compatible with Apache License 2.0, preventing the project from distributing an official MySQL-based package.

However, in real-world production environments, many organizations prefer MySQL due to its mature ecosystem, extensive tooling, operational familiarity, and widespread adoption across enterprises.

In this guide, we'll show how to adapt the official Docker Compose deployment and successfully run a DolphinScheduler cluster backed by MySQL metadata storage.

Pull Required Images

docker pull apache/dolphinscheduler-master:3.1.9

docker pull apache/dolphinscheduler-worker:3.1.9

docker pull apache/dolphinscheduler-tools:3.1.9

docker pull apache/dolphinscheduler-api:3.1.9

docker pull apache/dolphinscheduler-alert-server:3.1.9

docker pull bitnami/zookeeper:3.7.1

Download the MySQL JDBC Driver

wget https://downloads.mysql.com/archives/get/p/3/file/mysql-connector-j-8.0.33.zip

unzip -q mysql-connector-j-8.0.33.zip

cp mysql-connector-j-8.0.33/mysql-connector-j-8.0.33.jar .

Prepare Custom Images

Dockerfile for Master, Worker, API, and Alert Server

# Based on the official DolphinScheduler image

ARG SERVICE=api

FROM apache/dolphinscheduler-${SERVICE}:3.1.9

# Copy the MySQL JDBC driver into the DolphinScheduler library directory

# DolphinScheduler loads JDBC drivers from the lib directory

COPY mysql-connector-j-8.0.33.jar /opt/dolphinscheduler/libs/

Dockerfile for Tools

# Based on the official DolphinScheduler image

ARG SERVICE=tools

FROM apache/dolphinscheduler-${SERVICE}:3.1.9

# Copy the MySQL JDBC driver into the DolphinScheduler tools library directory

# DolphinScheduler loads JDBC drivers from the lib directory

COPY mysql-connector-j-8.0.33.jar /opt/dolphinscheduler/tools/libs/

Build Custom Images

docker build --build-arg SERVICE=master -t apache/dolphinscheduler-master:3.1.9-mysql .

docker build --build-arg SERVICE=worker -t apache/dolphinscheduler-worker:3.1.9-mysql .

docker build --build-arg SERVICE=tools -t apache/dolphinscheduler-tools:3.1.9-mysql .

docker build --build-arg SERVICE=api -t apache/dolphinscheduler-api:3.1.9-mysql .

docker build --build-arg SERVICE=alert-server -t apache/dolphinscheduler-alert-server:3.1.9-mysql .

Update docker-compose.yaml

Disable the PostgreSQL service and add a MySQL service as the metadata database.

# Comment out the PostgreSQL service

# dolphinscheduler-postgresql:
#   image: bitnami/postgresql:15.2.0
#   ...

# MySQL Metadata Database Service

dolphinscheduler-mysql:
  image: mysql:8.0
  container_name: dolphinscheduler-mysql

  profiles:
    - all
    - schema

  environment:
    MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-root}
    MYSQL_DATABASE: ${MYSQL_DATABASE:-dolphinscheduler}

  volumes:
    - dolphinscheduler-mysql:/var/lib/mysql

  ports:
    - "3306:3306"

  # Expose MySQL to allow workers on other servers to connect

  healthcheck:
    test:
      [
        "CMD",
        "mysqladmin",
        "ping",
        "-h",
        "localhost",
        "-u",
        "${MYSQL_USERNAME:-root}",
        "-p${MYSQL_PASSWORD:-root}"
      ]

    interval: 5s
    timeout: 60s
    retries: 120

  networks:
    - dolphinscheduler

The remaining services (ZooKeeper, Schema Initializer, API, Alert Server, Master, Worker, Network, and Volume configurations) remain the same as the official Docker Compose deployment, with the following key modifications:

Schema Initializer Dependency

depends_on:
  dolphinscheduler-mysql:
    condition: service_healthy

Master JVM Configuration

environment:
  JAVA_OPTS: >
    -server
    -Duser.timezone=${SPRING_JACKSON_TIME_ZONE}
    -Xms8g
    -Xmx8g
    -Xmn4g
    -XX:+PrintGCDetails
    -Xloggc:gc.log
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:HeapDumpPath=dump.hprof

Worker JVM Configuration

environment:
  JAVA_OPTS: >
    -server
    -Duser.timezone=${SPRING_JACKSON_TIME_ZONE}
    -Xms8g
    -Xmx8g
    -Xmn4g
    -XX:+PrintGCDetails
    -Xloggc:gc.log
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:HeapDumpPath=dump.hprof

Volume Configuration

volumes:
  # Comment out PostgreSQL volume

  # dolphinscheduler-postgresql:

  dolphinscheduler-mysql:
  dolphinscheduler-zookeeper:
  dolphinscheduler-worker-data:
  dolphinscheduler-logs:
  dolphinscheduler-shared-local:

Update the .env File

# Docker Hub Repository and Image Tag

HUB=apache
TAG=3.1.9

# MySQL Configuration

MYSQL_ROOT_PASSWORD=root
MYSQL_DATABASE=dolphinscheduler
MYSQL_USERNAME=root
MYSQL_PASSWORD=root

# DolphinScheduler Database Configuration

TZ=Asia/Shanghai

# Use MySQL as the metadata database

DATABASE=mysql

SPRING_JACKSON_TIME_ZONE=GMT+8

SPRING_DATASOURCE_URL=jdbc:mysql://dolphinscheduler-mysql:3306/${MYSQL_DATABASE}?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai&allowPublicKeyRetrieval=true

SPRING_DATASOURCE_USERNAME=${MYSQL_USERNAME}

SPRING_DATASOURCE_PASSWORD=${MYSQL_PASSWORD}

REGISTRY_ZOOKEEPER_CONNECT_STRING=dolphinscheduler-zookeeper:2181

MASTER_FETCH_COMMAND_NUM=10

Initialize the Database

docker compose --profile schema up -d

Start the Entire Cluster

docker compose --profile all up -d

Start the Worker Service

docker compose up -d dolphinscheduler-worker

Start the Master Service

docker compose up -d dolphinscheduler-master

Start the Alert Service

docker compose up -d dolphinscheduler-alert

Start the API Service

docker compose up -d dolphinscheduler-api

Restart All Services

docker compose --profile all restart

Conclusion

Although Apache DolphinScheduler officially ships with a PostgreSQL-based Docker Compose deployment, many enterprises continue to standardize on MySQL for operational consistency and ecosystem compatibility.

By adding the MySQL JDBC driver, rebuilding the DolphinScheduler images, and adjusting the Docker Compose and environment configurations, you can quickly deploy a fully functional Apache DolphinScheduler 3.1.9 cluster powered by MySQL metadata storage.

This approach enables teams already invested in the MySQL ecosystem to integrate DolphinScheduler into their infrastructure with minimal friction while preserving the benefits of containerized deployment and cluster-based scheduling.

If your organization relies on MySQL as a strategic database platform, this solution provides a practical and production-friendly path to running Apache DolphinScheduler at scale.

Hands-On Demo: Incremental MySQL-to-Doris Synchronization with Apache DolphinScheduler and Apache SeaTunnel

Chen Debra — Thu, 11 Jun 2026 03:48:39 +0000

Data synchronization is one of the most common requirements in enterprise data platform development. As business volume continues to grow, full data synchronization can place increasing pressure on source databases while consuming substantial computing and storage resources. As a result, incremental synchronization has become the preferred approach in most production environments.

In this demo, we will combine Apache DolphinScheduler and Apache SeaTunnel to implement a typical offline incremental synchronization scenario. DolphinScheduler retrieves the synchronization checkpoint from the target system and passes it to SeaTunnel as a runtime parameter, enabling incremental data synchronization from MySQL to Apache Doris.

This article is based on an actual demonstration and provides a complete walkthrough of the environment setup, SeaTunnel configuration, and DolphinScheduler workflow configuration process.

For the full demo, please refer to:

1. Environment Setup

This demonstration uses the following components:

Component	Version
Apache SeaTunnel	2.3.9
Apache DolphinScheduler	3.x
MySQL	8.4
Apache Doris	2.x

In this architecture:

MySQL serves as the source database.
Doris serves as the target database.
SeaTunnel is responsible for data synchronization.
DolphinScheduler handles workflow orchestration and scheduling.

2. Preparing Test Data

Before configuring the synchronization task, we first prepare sample business data.

In this demonstration, a database named shopping is used as the sample database, and an orders table is created.

The orders table contains an auto-incrementing primary key column:

order_id

This field will later be used as the incremental synchronization checkpoint.

To verify synchronization results, a batch of sample records is inserted into the table. Approximately 300 order records are generated using a script.

The following information is then inspected:

Current total number of orders
Current maximum order ID

These values will serve as references when configuring incremental synchronization logic later.

It is worth noting that order_id is used only for demonstration purposes. In real-world production scenarios, timestamp fields such as update_time or create_time are often used as incremental synchronization conditions.

3. Incremental Synchronization Design

Before configuring SeaTunnel, let's first understand the overall synchronization strategy.

The core idea is to use data that has already been synchronized into Doris to determine the current synchronization progress.

The workflow operates as follows:

Query the current maximum order ID in Doris.
Use this value as the synchronization checkpoint.
SeaTunnel reads records from MySQL whose order IDs are greater than this checkpoint.
Newly added records are written into Doris.
During the next execution, synchronization resumes from the latest checkpoint.

For example, if the current maximum order ID in Doris is:

SeaTunnel will execute the following condition:

WHERE order_id > 300

This ensures that only newly inserted records are processed during each run, preventing duplicate synchronization of existing data.

As emphasized during the demonstration, the incremental field does not necessarily have to be a primary key. Any field that can accurately identify newly added or modified data can be used.

4. Configuring the SeaTunnel Job

After defining the synchronization strategy, we can start configuring the SeaTunnel job.

Configure the JDBC Source

Since the source data resides in MySQL, JDBC Source is used to read the data.

The core query is:

SELECT *
FROM orders
WHERE order_id > ${order_id}

The most important part is:

${order_id}

This value is not hardcoded. Instead, it will be dynamically supplied by DolphinScheduler.

When the workflow runs, SeaTunnel automatically replaces this variable with the actual synchronization checkpoint, enabling incremental extraction.

Configure Parallelism

The demonstration also configures task parallelism:

parallelism = 4

Increasing parallelism can significantly improve synchronization performance.

In production environments, the appropriate value should be determined based on available server resources and database workload.

Configure Partitioned Reads

To improve performance when reading large tables, partitioned reading is also introduced.

The partition column is:

order_id

Configuration:

partition_column = "order_id"

Combined with the partition_num parameter, the dataset is divided into multiple partitions that can be processed in parallel.

This approach can greatly improve synchronization efficiency for large-scale datasets.

Configure Fetch Size

Within the JDBC Connector, fetch_size can be used to control the number of records retrieved from the database per fetch operation.

Proper configuration of this parameter can reduce database round trips and improve overall read performance.

5. Configuring the Doris Sink

After completing the Source configuration, the next step is configuring the Doris Sink.

Automatic Table Creation

The demonstration first introduces:

create_schema

This parameter enables automatic creation of target tables.

By leveraging automatic table creation, users can significantly reduce the effort required to manually maintain Doris table schemas.

Configure Write Mode

Since this example uses incremental synchronization, append mode is selected:

save_mode = APPEND_DATA

APPEND_DATA is used because each synchronization run only processes newly added records and does not need to overwrite historical data.

Enable Two-Phase Commit

To ensure data consistency, the demonstration also introduces:

enable_2pc = true

Enabling this option activates the two-phase commit mechanism, providing more reliable data writes.

It also helps guarantee Exactly-Once semantics during data synchronization.

Performance Optimization Parameters

Several performance-related parameters are also discussed, including:

batch_size

and

buffer_size

These parameters primarily control batch write behavior and can significantly improve Doris ingestion performance.

6. Configuring the DolphinScheduler Runtime Environment

After completing the SeaTunnel configuration, the next step is setting up DolphinScheduler.

Create a Tenant

First, navigate to the Security Center.

Open the Tenant Management page and create a new tenant.

The demonstration specifically emphasizes that all tasks in DolphinScheduler are ultimately executed under a tenant identity. Therefore, tenant configuration is an essential step in preparing the runtime environment.

Create a User and Associate It with the Tenant

Next, navigate to the User Management page.

Create a user and associate it with the tenant created in the previous step.

Once configured, the user will have permission to execute tasks under the corresponding tenant.

Create an Environment

Next, open the Environment Management page.

Create a runtime environment for SeaTunnel.

Configure the following environment variable:

SEATUNNEL_HOME=/soft/seatunnel

This configuration tells DolphinScheduler where SeaTunnel is installed.

When a workflow executes a SeaTunnel task, DolphinScheduler uses this path to locate the corresponding execution scripts.

The demonstration highlights that this configuration is mandatory and should not be skipped.

7. Creating the Project and Workflow

After completing the environment setup, create a new project.

In this demonstration, a project named:

shopping

is created.

After entering the project, create a new workflow.

The workflow contains two core nodes:

SQL Task
SeaTunnel Task

The SQL Task is responsible for retrieving the synchronization checkpoint, while the SeaTunnel Task performs the actual data synchronization.

8. Configuring the SQL Task to Retrieve the Synchronization Checkpoint

This is the most critical step in the entire solution.

First, create an SQL Task and select the Doris datasource.

The purpose of this task is to determine how far synchronization has progressed by querying the latest synchronized order ID.

The SQL statement is:

SELECT IFNULL(MAX(order_id),0) AS order_id
FROM orders;

The demonstration specifically explains why the following statement should not be used directly:

SELECT MAX(order_id)
FROM orders;

The reason is that during the initial synchronization, the Doris table may still be empty.

In this case:

MAX(order_id)

returns:

NULL

If NULL is passed directly to the downstream SeaTunnel task, it may generate an invalid query condition.

Therefore:

IFNULL(MAX(order_id),0)

is used to convert NULL values into 0.

This ensures that the initial synchronization starts correctly from the very first record.

Configure an OUT Parameter

The query result must be passed to downstream tasks.

To accomplish this, create a custom parameter within the SQL Task.

Select the parameter type:

OUT

Set the parameter name to:

order_id

The SQL query result will then be stored as a workflow variable.

The SeaTunnel task can subsequently reference this variable directly.

9. Incremental Synchronization Workflow Logic

Once the SQL Task is configured, the entire incremental synchronization pipeline is established.

When the workflow runs:

The SQL Task queries the current maximum order_id in Doris.
The result is stored as a workflow variable.
SeaTunnel uses ${order_id} as the query condition.
Newly added records are extracted from MySQL.

Through this approach, offline incremental synchronization based on a business primary key can be implemented efficiently and reliably.

10. Conclusion

This example demonstrates how to implement offline incremental synchronization by combining Apache DolphinScheduler and Apache SeaTunnel.

SeaTunnel handles data extraction and loading, while DolphinScheduler manages synchronization checkpoint retrieval, parameter passing, workflow orchestration, and scheduling.

The key idea behind this solution is querying the maximum order_id from the target Doris table through an SQL Task and passing the result to SeaTunnel through an OUT parameter. SeaTunnel then uses the checkpoint to perform incremental extraction from MySQL.

For data warehouse construction, ODS synchronization, and recurring offline synchronization scenarios, this solution offers several advantages:

Simple implementation
Easy maintenance
Strong extensibility
Production-ready incremental synchronization capabilities

As a result, it provides a practical and highly valuable reference architecture for enterprise data platform development.

Apache DolphinScheduler 3.4.2 Released! Introducing Amazon EMR Serverless Support, Enhanced Monitoring, and Improved Backfill Capabilities

Chen Debra — Thu, 11 Jun 2026 03:45:35 +0000

The Apache DolphinScheduler community has officially announced the release of Apache DolphinScheduler 3.4.2.

As an important maintenance release in the 3.4.x series, version 3.4.2 continues the community’s tradition of delivering high-quality iterations. This release introduces several user-facing enhancements, including the new Amazon EMR Serverless Task Plugin and improvements to the Monitoring Center. It also delivers extensive optimizations and refactoring across complement data processing, task plugin capabilities, security governance, permission control, and the underlying architecture.

From a release perspective, DolphinScheduler 3.4.2 includes more than one hundred merged pull requests covering feature enhancements, stability fixes, code governance, documentation improvements, and CI/CD optimizations. Contributors from across the global community participated in this release, further advancing DolphinScheduler as an enterprise-grade cloud-native workflow orchestration platform.

Amazon EMR Serverless Officially Joins the DolphinScheduler Ecosystem

As more organizations deploy data platforms on AWS, serverless architectures are becoming an increasingly important direction for big data computing.

To address this demand, the community introduced the Amazon EMR Serverless Task Plugin in this release (PR #18069, contributed by @norrishuang).

With this plugin, users can directly submit and manage Amazon EMR Serverless jobs from within DolphinScheduler, enabling unified orchestration and scheduling of Spark and other big data workloads without maintaining the underlying compute clusters. For data lakes, offline data warehouses, and AI-driven data processing scenarios, this translates into lower operational costs and higher resource utilization.

The addition of this capability further expands DolphinScheduler’s coverage within the AWS cloud-native ecosystem.

Enhanced Monitoring Center Delivers Greater Operational Visibility

In production environments, one of the most common questions for operations teams is understanding which workflows and tasks are currently running.

Previously, troubleshooting often required logging into servers and manually inspecting logs or process information. To improve this experience, the community enhanced the Monitoring Center through PR #18138 (contributed by @ruanwenjun).

After the upgrade, users can directly view workflows currently being executed by Active Masters and tasks currently running on Active Workers from the Monitoring UI. This provides a much clearer view of cluster activity and significantly improves troubleshooting efficiency.

For large-scale deployments, this enhancement further strengthens DolphinScheduler’s observability capabilities.

Complement Data Capabilities Further Improved

Complement Data (backfill processing) remains a core capability for offline data warehouses and modern data platforms.

In previous releases, dependency handling in certain backfill scenarios still had limitations. To address this issue, the community completed the implementation related to DSIP-95.

PR #18003, contributed by @det101, improves dependency task handling in complement data scenarios. After the upgrade, the system can correctly identify and calculate dependencies during historical data backfill operations, ensuring both accuracy and completeness throughout the backfill process.

This enhancement is particularly valuable for T+1 data warehouses, historical data recomputation, and offline metric correction scenarios.

Continued Enhancements to SQL, Flink, and Other Task Plugins

Task plugins continue to be one of the most actively evolving areas of DolphinScheduler.

For SQL Tasks, PR #18020 (contributed by @macdoor) introduces support for loading SQL statements directly from resource files while also supporting parameter placeholder substitution.

This enhancement allows complex SQL logic to be maintained in Resource Center files rather than embedded directly within task definitions. As a result, organizations can manage SQL assets more systematically while simplifying version control and collaborative development.

Within the Flink ecosystem, PR #17987 (also contributed by @macdoor) adds parameter substitution support to both Flink Task and FlinkStream Task plugins.

With this capability, users can dynamically generate task parameters at runtime, enabling workflows to adapt more easily to different environments and business scenarios. For organizations that manage multi-tenant configurations, dynamic date calculations, or environment switching, this feature significantly improves orchestration flexibility and operational efficiency.

In addition, the community further improved configuration flexibility for Flink tasks. PR #17909 (contributed by @leocook) exposes the Flink -sae parameter in the Web UI and sets its default value to disabled.

Through graphical configuration, users can control relevant runtime parameters without modifying underlying configuration files, further lowering the barrier to adopting Flink tasks within DolphinScheduler.

Continuous Improvements in Security and Permission Governance

Enterprise-grade scheduling platforms require robust security governance.

In this release, the community completed implementation work related to DSIP-37. PR #18119, contributed by @SbloodyS, introduces the ability to disable Jetty HTTP TRACE requests through configuration.

TRACE requests are rarely used in business scenarios but can potentially be exploited for information discovery. Disabling this capability helps reduce potential security risks and further strengthens system security.

Regarding datasource access control, PR #18073 (contributed by @njnu-seafish) introduces permission validation mechanisms for interfaces including connectionTest, getDatabases, getTables, and getTableColumns.

After the upgrade, unauthorized users can no longer retrieve related metadata information, further reinforcing security boundaries around data access.

Continuous Optimization and Refactoring

While new features are often the most visible improvements for users, consistent code standards, architectural refinements, and logic optimization are equally critical to the long-term sustainability of an open-source project.

In this release, the community continued optimizing and refactoring several core modules, including API, DAO, Datasource, Alert, Worker, and Storage. These improvements standardized implementation patterns, removed redundant logic, and further enhanced the maintainability and extensibility of the system.

Related pull requests include:

PR18103, PR18104, PR18105, PR18107, PR18111, PR18112, PR18113, PR18114, PR18115, PR18116, PR18117, and PR18118.

Meanwhile, @ruanwenjun continued advancing the Repository DAO refactoring initiative and the optimization of return object types.

Related pull requests include:

PR18226, PR18227, PR18228, PR18229, PR18230, PR18232, PR18233, PR18234, PR18236, PR18245, as well as multiple Repository refactoring pull requests from PR18250 through PR18263.

These improvements replace a large number of Map-based return structures with strongly typed objects, establishing a cleaner architectural foundation for future development.

Improved Stability Through Numerous Critical Fixes

In addition to feature enhancements, DolphinScheduler 3.4.2 resolves multiple critical issues encountered in production environments.

PR #18033 (contributed by @ruanwenjun) improves JDBC connection creation by switching to a Driver-based approach, resulting in more reliable database connectivity.

PR #18042 and PR #18044 (contributed by @njnu-seafish) fix parameter passing issues affecting Kubernetes Task and Zeppelin Task plugins.

PR #18155 (contributed by @SbloodyS) resolves an issue where date parameters were incorrectly propagated to sub-workflows during complement data execution.

PR #18146 (contributed by @SbloodyS) fixes a problem that could cause workflows using the CONTINUE strategy to remain in the RUNNING state indefinitely.

PR #18183 (contributed by @ruanwenjun) addresses a permission vulnerability that allowed users to delete task definitions from unauthorized projects.

PR #18212 and PR #18300 (contributed by @ruanwenjun) further strengthen permission validation for Workflow Trigger and Access Token-related operations.

In addition, the community resolved several other stability-related issues, including:

Docker Compose deployment exceptions
Helm Chart configuration problems
JDBC Registry REMOVE event anomalies
SQL License Header parsing issues
Multiple defects affecting deployment reliability and runtime stability

Together, these fixes significantly improve the robustness of DolphinScheduler in enterprise production environments.

Continued Improvements to Documentation and Engineering Infrastructure

Beyond code-level enhancements, the community continues investing heavily in documentation and engineering excellence.

This release updates the FAQ documentation, improves parameter passing explanations, introduces AGENT.md and CLAUDE.md files, and fixes multiple documentation formatting and broken-link issues.

On the CI/CD front, the community restored Python end-to-end testing, improved unit test execution efficiency, optimized the Docker release process, and enhanced GitHub Code Owner management mechanisms.

Although these improvements may not appear directly in product feature lists, they serve as essential infrastructure that supports the long-term health and sustainability of the project.

Thanks to All Contributors

The successful release of Apache DolphinScheduler 3.4.2 would not have been possible without the collective efforts of contributors from around the world.

Special thanks go to Release Manager @ruanwenjun for his outstanding leadership and coordination throughout the release cycle.

Under his guidance, a total of 19 contributors successfully delivered this release:

@ruanwenjun

@SbloodyS

@det101

@njnu-seafish

@macdoor

@norrishuang

@leocook

@wcmolin

@asadjan4611

@HEEKDragonOne

@CloudExtreme

@Mrhs121

@hiSandog

@pjfanning

@sanjana2505006

@shaolei7788

@llphxd

@includetts

@shrihari7396

We would also like to thank everyone who contributed code, submitted issues, improved documentation, or provided valuable community feedback.

It is the continued dedication of every contributor that enables Apache DolphinScheduler to keep evolving and moving forward.

Download and Upgrade

Apache DolphinScheduler 3.4.2 is now officially available.

This release delivers substantial improvements across cloud-native capabilities, monitoring and observability, security governance, task plugins, and overall system stability. It provides a more reliable, efficient, and secure workflow orchestration experience for enterprise production environments.

We encourage all users to upgrade and explore the latest enhancements.

Download:
https://dolphinscheduler.apache.org/zh-cn/download/3.4.2

Release Notes:
https://github.com/apache/dolphinscheduler/releases/tag/3.4.2

Inside DolphinScheduler’s May 2026 Release: Better Failover, Stronger Security, and More Reliable Plugins

Chen Debra — Fri, 05 Jun 2026 10:07:17 +0000

The May 2026 DolphinScheduler community update can be summarized with two keywords: stability and precision.

On one hand, major stability risks such as Master failover issues—which can have a significant impact on production environments when failures occur—have been addressed. On the other hand, long-standing usability problems, including API authorization gaps, plugin dependency conflicts, and RemoteShell null pointer exceptions, have been systematically fixed.

This monthly report highlights the key changes merged into the dev branch during May, including their impact on users, whether upgrades should be considered, and how to validate them.

Monthly Statistics

Merged PRs: 50
Contributors: 7
Code Changes: +10,036 / -8,542
Major Modules Involved: API, DAO, Master, Task Plugin, CI/Testing

You will notice that testing-related changes account for a large proportion of the updates. This reflects the community's effort to build a stronger foundation for future iterations. Stable and efficient CI/UT pipelines enable faster feature delivery and more reliable bug fixes.

Who Should Read This?

End users and business teams: Want to know which common issues were fixed and whether production environments will become more stable.
Operations and platform engineers: Care about failover, permissions, logging, and plugin stability.
Developers: Want a quick overview of recent engineering governance efforts, including CI, unit testing, and quality assurance improvements.

The 6 Improvements Users Will Notice Most

1. More Reliable Master Failover

Typical scenario: after a Master node crashes, cluster recovery is slow or failover becomes stuck.

One of May's major fixes addresses failover lock leaks, reducing the likelihood that the scheduler remains unavailable for an extended period after failures.

2. More Rigorous Authorization for Critical APIs

Project-level authorization checks have been added to APIs such as view-gantt, view-variables, and trigger workflow.

This makes the permission model more intuitive: users without proper authorization should not be able to access these resources.

3. Fewer Null Pointer Exceptions in RemoteShell Tasks

Null pointer exceptions in remote tasks are notoriously difficult to troubleshoot due to distributed logs and complex execution contexts.

This month introduces fixes for RemoteShell-related NPEs, making task failures easier to understand and resolve.

4. Improved Dependency Conflict Management for Task Plugins

Plugins such as AliyunServerlessSpark previously suffered from dependency conflicts that could lead to ClassNotFound or compatibility issues.

Enhancements to dependency management and exception handling improve overall plugin reliability.

5. Faster and More Reliable CI and Unit Testing

This is not a user-facing feature, but it matters greatly.

More stable CI pipelines catch problems before code is merged, and stronger testing reduces the likelihood of production incidents.

Related PRs:

6. More Flexible Region and Endpoint Support for AWS S3 Remote Logs

Users relying on S3-compatible storage services or private endpoints now have greater flexibility when configuring regions and endpoints.

This reduces troubleshooting time for connectivity issues caused by storage configuration differences.

Upgrade and Validation Recommendations

This report is based on PRs merged into the dev branch during May 2026, making it valuable for tracking development trends and performing early validation.

If you are running DolphinScheduler in production, prioritize upgrades based on risk:

Recommended for Immediate Attention

Master failover improvements
Authorization and security-related fixes
Task plugin stability enhancements

Can Be Adopted as Needed

CI and testing optimizations
Documentation and formatting updates
Return-type migration and engineering quality improvements

Since all changes were merged into the dev branch, validation in testing or integration environments is recommended:

git fetch origin dev
git checkout dev
git pull --rebase

Focus regression testing on the following scenarios:

Master restart and failover recovery
Critical API authorization validation
Common task plugins such as RemoteShell and ServerlessSpark

Contributor Acknowledgements

Thanks to all contributors who submitted and merged PRs to Apache DolphinScheduler during May 2026.

Your contributions continue to improve the platform's stability, usability, and ecosystem capabilities.

GitHub Username	Main Contribution	Merged PRs	+Lines	-Lines	Score
@ruanwenjun	Test Cases	40	7367	6506	349.69
@SbloodyS	Test Cases	4	2503	1988	45.83
@hiSandog	Documentation	2	34	7	15.12
@leocook	Debug & Fix	1	34	29	9.15
@includetts	Debug & Fix	1	16	6	9.06
@llphxd	Documentation	1	4	4	9.02
@wcmolin	Test Cases	1	78	2	8.26

In-Depth Analysis of Key Technical Changes

A total of 50 PRs were merged this month.

The primary focus areas include:

Stability
Security and Authorization
Plugin Reliability
CI and Testing Efficiency

To help readers quickly understand the most important developments, the following section analyzes five representative changes in detail.

1. [Fix-18197][Master] Fix master failover lock leak (#18207)

Link: https://github.com/apache/dolphinscheduler/pull/18207
Author: @ruanwenjun
Base/Head: dev ← dev_wenjun_fix18197
Diff Stats: +171 / -10

Background and Challenges

Master failover relies on distributed locks to ensure that failover for a given address is not executed concurrently.

If lock release logic is incorrect, lock nodes may leak, preventing future failover operations and leaving the cluster unable to resume scheduling after failures.

Design and Implementation

The lock acquisition interface was redesigned to return an AutoCloseable handle.

Using try-with-resources guarantees symmetric acquire/release behavior.

Additionally, callers now retain the exact lock path, preventing subtle mistakes such as releasing parent paths.

Suggested Metrics

Simulate failover storms in a three-Master cluster by repeatedly issuing kill -9 and automatic restarts.

Compare:

Failover success rate
Mean Time To Recovery (MTTR)
Failover thread blocking duration

Registry lock node count should also be monitored, as lock leaks accumulate over time.

Compatibility and Rollback

Interface signature changes may affect callers.

Rollback is straightforward but requires cleanup of leaked lock nodes to prevent continued service disruption.

2. [Fix][API] Add missing project authorization on view-gantt/view-variables and trigger workflow APIs (#18212)

Link: https://github.com/apache/dolphinscheduler/pull/18212
Author: @ruanwenjun
Base/Head: dev ← dev_wenjun_fixCvePermissionCheck
Diff Stats: +321 / -16

Background and Challenges

Workflow APIs without project-level authorization checks can create privilege escalation risks.

In multi-tenant enterprise environments, this becomes a serious security concern.

Design and Implementation

Authorization validation was added to:

view-gantt
view-variables
trigger workflow

Permission checks are enforced consistently through Controller and Service layers.

Suggested Validation

Benchmark authorization overhead before and after implementation.

Security regression tests should include cross-project access attempts.

Best Practices

Enterprise users should enable stricter tenant isolation policies and audit sensitive API operations.

3. [Fix-18201][TaskPlugin] Fix RemoteShell task NullPointerException and… (#18210)

Link: https://github.com/apache/dolphinscheduler/pull/18210
Author: @leocook
Base/Head: dev ← fix-18201-remoteshell-npe
Diff Stats: +34 / -29

Background and Challenges

RemoteShell tasks are commonly used for operations and integration workloads.

Network interruptions, command output handling differences, and SSH channel inconsistencies can easily lead to NPEs and incomplete logs.

Design and Implementation

Input/output stream handling for SSH channels was improved to eliminate null pointer scenarios.

Exception handling paths were also enhanced to preserve root-cause information.

Suggested Validation

Inject failures such as:

Remote disconnections
Empty output streams
Immediate command termination

Execute 1,000 test runs and compare:

NPE occurrence rates
Log completeness

Risks and Rollback

Changes are isolated to the plugin layer and are relatively easy to revert.

Regression tests should continue covering:

Empty output
Large output
Non-zero exit codes

4. [Fix-18177][Task Plugin] Fix AliyunServerlessSpark plugin dependency conflicts and improve exception handling (#18180)

Link: https://github.com/apache/dolphinscheduler/pull/18180
Author: @includetts
Base/Head: dev ← fix/aliyun-serverless-spark-deps-v2
Diff Stats: +16 / -6

Background and Challenges

Dependency conflicts are classic runtime problems that often manifest as:

NoSuchMethodError
NoSuchFieldError

They are difficult to reproduce because they only occur under specific dependency combinations.

Design and Implementation

Critical dependency versions were corrected and exception wrapping improved.

Users can now directly identify conflicting classes and methods from logs.

Suggested Validation

Execute smoke tests under multiple Hadoop and Spark dependency trees.

Measure:

Startup success rate
Exception readability
Time-to-diagnosis

Best Practices

Production environments should consider dependency isolation techniques such as:

Shading
Relocation
Dedicated ClassLoaders

5. [Chore] Unit-Test performance optimize (#18213)

Link: https://github.com/apache/dolphinscheduler/pull/18213
Author: @SbloodyS
Base/Head: dev ← ut_performance_optimize
Diff Stats: +22 / -6

Background and Challenges

Slow, flaky, or frequently skipped tests delay problem detection until production deployment.

Testing infrastructure directly impacts community development speed and software quality.

Design and Implementation

Unit test execution and CI configurations were optimized.

Temporary safeguards were also introduced to maintain CI stability during environmental issues.

Suggested Validation

Compare:

Total CI duration
Number of executed unit tests
Percentage of skipped tests
Flaky test rerun counts

Risks and Rollback

Temporary test disablement should always include a documented recovery plan.

Conditions for re-enabling tests should be tracked through issues and PRs.

Appendix

PR #18204: https://github.com/apache/dolphinscheduler/pull/18204
PR #18208: https://github.com/apache/dolphinscheduler/pull/18208
PR #18206: https://github.com/apache/dolphinscheduler/pull/18206
PR #18207: https://github.com/apache/dolphinscheduler/pull/18207
PR #18205: https://github.com/apache/dolphinscheduler/pull/18205
PR #18213: https://github.com/apache/dolphinscheduler/pull/18213
PR #18209: https://github.com/apache/dolphinscheduler/pull/18209
PR #18180: https://github.com/apache/dolphinscheduler/pull/18180
PR #18212: https://github.com/apache/dolphinscheduler/pull/18212
PR #18210: https://github.com/apache/dolphinscheduler/pull/18210
PR #18214: https://github.com/apache/dolphinscheduler/pull/18214
PR #18221: https://github.com/apache/dolphinscheduler/pull/18221
PR #18218: https://github.com/apache/dolphinscheduler/pull/18218
PR #18225: https://github.com/apache/dolphinscheduler/pull/18225
PR #18227: https://github.com/apache/dolphinscheduler/pull/18227
PR #18241: https://github.com/apache/dolphinscheduler/pull/18241
PR #18240: https://github.com/apache/dolphinscheduler/pull/18240
PR #18226: https://github.com/apache/dolphinscheduler/pull/18226
PR #18228: https://github.com/apache/dolphinscheduler/pull/18228
PR #18229: https://github.com/apache/dolphinscheduler/pull/18229
PR #18232: https://github.com/apache/dolphinscheduler/pull/18232
PR #18223: https://github.com/apache/dolphinscheduler/pull/18223
PR #18230: https://github.com/apache/dolphinscheduler/pull/18230
PR #18234: https://github.com/apache/dolphinscheduler/pull/18234
PR #18242: https://github.com/apache/dolphinscheduler/pull/18242
PR #18236: https://github.com/apache/dolphinscheduler/pull/18236
PR #18233: https://github.com/apache/dolphinscheduler/pull/18233
PR #18245: https://github.com/apache/dolphinscheduler/pull/18245
PR #18250: https://github.com/apache/dolphinscheduler/pull/18250
PR #18251: https://github.com/apache/dolphinscheduler/pull/18251
PR #18252: https://github.com/apache/dolphinscheduler/pull/18252
PR #18257: https://github.com/apache/dolphinscheduler/pull/18257
PR #18270: https://github.com/apache/dolphinscheduler/pull/18270
PR #18271: https://github.com/apache/dolphinscheduler/pull/18271
PR #18258: https://github.com/apache/dolphinscheduler/pull/18258
PR #18253: https://github.com/apache/dolphinscheduler/pull/18253
PR #18260: https://github.com/apache/dolphinscheduler/pull/18260
PR #18259: https://github.com/apache/dolphinscheduler/pull/18259
PR #18256: https://github.com/apache/dolphinscheduler/pull/18256
PR #18263: https://github.com/apache/dolphinscheduler/pull/18263
PR #18262: https://github.com/apache/dolphinscheduler/pull/18262
PR #18261: https://github.com/apache/dolphinscheduler/pull/18261
PR #18254: https://github.com/apache/dolphinscheduler/pull/18254
PR #18279: https://github.com/apache/dolphinscheduler/pull/18279
PR #18284: https://github.com/apache/dolphinscheduler/pull/18284
PR #18288: https://github.com/apache/dolphinscheduler/pull/18288
PR #18268: https://github.com/apache/dolphinscheduler/pull/18268
PR #18296: https://github.com/apache/dolphinscheduler/pull/18296
PR #18300: https://github.com/apache/dolphinscheduler/pull/18300
PR #18301: https://github.com/apache/dolphinscheduler/pull/18301

Upgrading DolphinScheduler Across Major Versions: From 3.1.3 to 3.4.1 via API Automation

Chen Debra — Thu, 21 May 2026 07:22:53 +0000

1. Background: Why Perform a Major Version Upgrade?

Existing Environment

Current DolphinScheduler Version: 3.1.3
Current SeaTunnel Version: 2.1.3

Deployment Scale:
1 Master + 2 Workers, with over 3,700 workflow definitions and more than 20,000 scheduled tasks executed daily.

Years in Production:
The system has been running stably for over three years.

Drivers Behind the Upgrade

Functional Requirements:
With growing business demands, the limitations of the current version became increasingly apparent, including architectural design constraints, metadata database processing bottlenecks, and insufficient server resources.
Community Support:
The official community recommends adopting the latest stable release to obtain better technical support.
Performance Optimization:
Version 3.4.1 delivers significant improvements in scheduling performance and system stability.

Why We Did Not Use the Official Upgrade Path

Large Version Gap:
The upgrade path would require multiple intermediate upgrades:
3.1.3 → 3.2.0 → 3.3.0 → 3.4.1.
Production Environment Constraints:
Multiple maintenance windows were unacceptable due to strict business continuity requirements.
Architectural Change Risks:
Risks included resource center refactoring, metadata schema changes, and compatibility issues.
Workload Considerations:
With thousands of workflows, manually rebuilding tasks would require enormous effort, making automation essential.

2. Overall Migration Strategy: Bypassing the Official Upgrade Path with a “Rebuild + API” Approach

2.1 Core Idea

Instead of pursuing an “in-place incremental upgrade,” we adopted a “new environment + data migration” strategy:

The old 3.1.3 cluster continued running without impacting production workloads.
A brand-new 3.4.1 cluster was deployed to ensure a clean architecture.
Custom scripts were developed to retrieve workflow definitions and task configurations from the old-version APIs.
New-version APIs were used to batch-create workflows in the new cluster.
Business scheduling traffic was gradually switched to the new environment.

2.2 Comparison of Advantages and Risks

Dimension	Official Upgrade Approach	Rebuild + API Approach
Downtime	Multiple upgrades with cumulative downtime potentially lasting hours or even days	Nearly seamless cutover by stopping old schedules and enabling new ones
Rollback Difficulty	Difficult, as the database schema has already changed	Easy, since the old environment remains intact
Data Consistency	Requires validation of all schema migrations	Only core business data (workflow definitions) is migrated; historical execution records are excluded
Version Compatibility	Must handle compatibility issues across all intermediate versions	Directly adapts to 3.4.1 with only necessary parameter transformations
Workload	Requires repeated validation cycles	Effort mainly concentrated on script development
Suitable Scenarios	Minor version upgrades	Major version jumps and large-scale task migration

3. Detailed Implementation Steps

3.1 Environment Preparation

3.1.1 Deploying the New Environment

Deploy a brand-new DolphinScheduler 3.4.1 cluster.
Configure dependencies such as the database and Registry. ZooKeeper was deprecated and replaced with JDBC Registry.
Configure required components such as DataX and SeaTunnel.
Understand key changes in the new version, including:
- SeaTunnel 2.1.3 integration startup mode changed from Spark engine execution to start-seatunnel-spark.sh;
- Default configurations such as tenants, worker groups, and environments are now managed through project preferences;
- Parameter passing behavior changed: downstream tasks must explicitly define IN parameters to receive upstream variable values.
Verify the basic functionality of the new environment and manually validate representative workflows.

3.1.2 API Access Configuration

Configure API access permissions in the new environment by creating new tokens in token management.
Obtain administrator tokens for API calls.
Verify API connectivity.

3.2 Metadata Database Initialization

Replicating Base Tables: Rebuild foundational metadata tables in the new metadata database according to the old-version configuration, including preserving IDs wherever possible. This significantly reduced modification complexity during script-based workflow restoration.

Verification Item	Table
Tenant Table	`t_ds_tenant`
Project Table	`t_ds_project`
User Table	`t_ds_user`
Environment Tables	`t_ds_environment`, `t_ds_environment_worker_group_relation`
Worker Group Table	`t_ds_worker_group`
Data Source Table	`t_ds_datasource`

3.3 Migration Script Development

3.3.1 Preliminary Preparation and Testing

Categorize workflows into template-based and non-template-based tasks.
Select representative workflows and execute them in the new environment to verify successful execution and data synchronization accuracy.

3.3.2 Code Development — Reading Original Workflow Definitions

...
 // Retrieve all workflows and process them iteratively
        String processDefinitionUrl = OLD_URL + "/dolphinscheduler/projects/" + oldProjectCode +
                "/process-definition/query-process-definition-list";
        Map<String, String> map = new HashMap<>();
        map.put("projectCode", oldProjectCode);
        String pdRes = httpClientUtilOld.doGetRequest(processDefinitionUrl, map);
        ArrayList<JSONObject> dataList = parseResDataToList(pdRes);
        for (JSONObject job : dataList) {
            String oldWFCode = job.get("code").toString();
            Map<String, String> mapPara = new HashMap<>();
            String oldurl = OLD_URL + "/dolphinscheduler/projects/" + oldProjectCode
                    + "/process-definition/" + oldWFCode;
            mapPara.put("code", oldWFCode);
            mapPara.put("projectCode", oldProjectCode);

            String res = httpClientUtilOld.doGetRequest(oldurl, mapPara);
            JSONObject jsonObject = JSON.parseObject(res);
            JSONObject data = (JSONObject) jsonObject.get("data");
            JSONObject processDefinition = data.getJSONObject("processDefinition");
            JSONArray processTaskRelationList = data.getJSONArray("processTaskRelationList");
            JSONArray taskDefinitionList = data.getJSONArray("taskDefinitionList");

            // TODO: generate new task codes and replace old ones
            // Populate workflow information and create workflows
            createWF(processDefinition, processTaskRelationList, taskDefinitionList, NEW_IP, newProjectCode);

3.3.3 Creating Workflows via API

 // Generate task codes based on task count
        int taskCnt = taskDefinitionList.size();
        List<String> taskCodeList = taskDefinitionList.stream()
                .map(obj -> (JSONObject) obj)
                .map(obj -> obj.getString("code"))
                .collect(Collectors.toList());

        try {
// TODO: generate task codes
            String taskCodeUrl = NEW_URL + "/dolphinscheduler/projects/" + newProjectCode + "/task-definition/gen-task-codes";
            HashMap<String, String> taskCodeMap = new HashMap<>();
            // Generate n task codes
            taskCodeMap.put("genNum", String.valueOf(taskCnt));
            String codeData = httpClientUtilNew.doGetRequest(taskCodeUrl, taskCodeMap);
            Object codes = JSON.parseObject(codeData).get("data");
            JSONArray taskCodeArr = JSON.parseArray(codes.toString());

// Add downstream input parameters based on actual task requirements
            for (int i = 0; i < taskDefinitionList.size(); i++) {
                JSONObject logTask = (JSONObject) taskDefinitionList.get(i);
                if (“Condition Logic”)) {
                    JSONObject taskParams = logTask.getJSONObject("taskParams");
                    JSONArray localParams = taskParams.getJSONArray("localParams");

                    JSONObject hiveParam = new JSONObject();
                    hiveParam.put("prop", "hiveAmount");
                    hiveParam.put("direct", "IN");
                    hiveParam.put("type", "VARCHAR");
                    hiveParam.put("value", "");
                    localParams.add(hiveParam);

                    logTask.put("taskParamList", localParams);

                    JSONObject paramMap = new JSONObject();
                    for (Object obj : localParams) {
                        JSONObject param = (JSONObject) obj;
                        paramMap.put(param.getString("prop"), param.getString("value"));
                    }

                    logTask.put("taskParamMap", paramMap);
                    ....


// Replace required parameters such as task code and SeaTunnel execution engine
 for (int i = 0; i < taskCodeList.size(); i++) {
                String oldCode = taskCodeList.get(i);
                String newCode = taskCodeArr.getString(i);

                // Replace task code
                // Replace SeaTunnel engine: "SPARK" -> "start-seatunnel-spark.sh"
                taskDefinitionListJsonStr = taskDefinitionListJsonStr
                        .replace("\"code\":" + oldCode + ",", "\"code\":" + newCode + ",")
                        .replace("\"engine\":\"SPARK\",", "\"startupScript\":\"start-seatunnel-spark.sh\",");

                taskRelationListJsonStr = taskRelationListJsonStr
                        .replace("TaskCode\":" + oldCode + ",", "TaskCode\":" + newCode + ",");

                locationsJsonStr = locationsJsonStr
                        .replace(oldCode, newCode);

...
  }
            Map<String, String> map = new HashMap<>();
            map.put("taskDefinitionJson", taskDefinitionListJsonStr);
            map.put("taskRelationJson", taskRelationListJsonStr);
            map.put("locations", locationsJsonStr);

            map.put("name", processDefinition.getString("name"));
            map.put("tenantCode", "omm");
            map.put("executionType", processDefinition.getString("executionType"));
            map.put("description",
                    processDefinition.getString("description") == null ? "" : processDefinition.getString("description"));
            map.put("globalParams", processDefinition.getString("globalParams"));
            map.put("timeout", processDefinition.getString("timeout"));

            String processDefinitionUrl = NEW_URL + "/dolphinscheduler/projects/" + newProjectCode + "/workflow-definition";
            String processDefinitionRes = httpClientUtilNew.doPostRequest(processDefinitionUrl, map);

3.4 Migration Execution

3.4.1 Migration Procedure

Back Up Old Scheduling Tables:
Example: t_ds_schedules_20260416_10
Select a Pilot Project:
Choose a project with moderate workload and limited business impact.
Migrate Workflow Definitions:
Migrate approximately 200 workflow definitions, including scheduling configurations.
Deploy Workflows Without Enabling Schedules:
Deploy workflows first without activating schedules.
Manual Validation:
Execute workflows manually in batches and verify conflicts with the original cluster. Since most workflows run daily, hourly, or every 15 minutes, conflicts were minimal.
Investigate Failed Tasks:
Analyze root causes, fix issues, and rerun failed workflows.
Enable Scheduling Configurations:
Enable schedules after all workflows pass validation.
Disable Old Cluster Scheduling:
After confirming stable operation in the new environment, disable corresponding schedules in the old cluster.
Migrate Project by Project, Batch by Batch

3.4.2 Migration Execution Results

Following this process, we first selected a representative project containing 199 workflows. After migration, it was tested in production for one week without issues.

Subsequently, we completed migration for 50 projects, totaling approximately 3,700 workflows, within about 10 days.

Tracking Table

No.	Project Name	Project Code	Workflow Count	Progress	Remarks
1	Project 1	13*******	199	Completed on 04/16	One-week stability test completed
2	Project 2	...	...	...	...
...	...	...	...	...	...
50	...	...	...	...	...

3.4.3 Runtime Status

The new cluster has now been running for nearly one month.

Previously, scheduling delays ranged from 10 seconds to over one minute in severe cases. After the upgrade, scheduling latency has been virtually eliminated.

Issues related to missing scheduling instances have also not reoccurred.

So far, the system has been running smoothly without any identified problems, and continuous monitoring remains in place.

4. Risk Control and Contingency Planning

4.1 Major Risks

Risk of Data Loss:
Some configurations could potentially be missed during migration.

Compatibility Issues:
Certain configurations supported in the old version may not be supported in the new version.

Business Interruption Risks:
Scheduling delays could occur during the switchover process.

4.2 Contingency Plans

4.2.1 Rollback Strategy

Immediately stop scheduling in the new environment.
Restore scheduling services in the old environment.
Analyze root causes and retry after issue resolution.

4.2.2 Data Backup

Perform a complete backup of the old environment database.
Back up the initial configuration of the new environment.

5. Conclusion

5.1 Project Outcomes

Successfully completed a cross-version upgrade with zero business interruption.
Automated migration scripts significantly improved efficiency and reduced manual errors.
The new version delivered major performance gains and substantial stability improvements.

5.2 Lessons Learned

Major version upgrades require comprehensive evaluation of architectural changes.
API-based migration is highly suitable for configuration migration, though parameter compatibility must be handled carefully.
Thorough testing and validation are critical to success.
A robust monitoring system is essential for operational stability.
Comprehensive documentation is invaluable for long-term maintenance.

5.3 Future Plans

The current upgrade primarily addressed DolphinScheduler scheduling bottlenecks. To align with upcoming Spark cluster upgrades, the next step will be upgrading SeaTunnel from version 2.1.3 to 2.3.12, most likely using the same migration methodology.
Explore automated testing solutions.
Share migration experience with other teams.

DolphinScheduler Agent Is Now Open-Source! Bringing Self-Healing Automation to DataOps

Chen Debra — Mon, 18 May 2026 03:17:26 +0000

At the 2026 Apache DolphinScheduler Meetup technical session, the DolphinScheduler Agent solution presented by Liu Xiaodong immediately became one of the hottest topics in the community. This end-to-end system, connecting “group alert → intelligent diagnosis → automatic recovery → reporting loop,” effectively solves the fragmentation, high manual overhead, and constant context switching of traditional operations workflows, bringing big data incident handling from the era of “manual firefighting” into the age of “intelligent autonomous operations.”

The project’s core supporting tool, dolphinscheduler-cli (dsctl), has now officially been open-sourced on GitHub and is freely available for all developers!

Watch the Replay: https://youtu.be/mnGC-XOf8xU

The Pain of Traditional Operations: Slow Recovery Isn’t About Commands — It’s About Fragmented Context

When using Apache DolphinScheduler in daily production, handling failed tasks has always been a major burden for operations teams.

The workflow is all too familiar:

A Feishu alert pops up → open the DS UI to check instance status → inspect logs to locate the failure → compare with the Runbook → manually decide what to do → return to the group chat and reply with the result...

What truly slows down efficiency is not executing a command itself, but the constant loss of context across multiple systems. Facts, evidence, and risks are scattered across different tools, forcing operators to spend enormous time “searching for information, stitching logic together, and rebuilding context.” Collaboration breaks frequently, troubleshooting costs soar, and incident recovery cycles become unnecessarily long.

With DolphinScheduler Agent, all of this changes.

A Major Upgrade: From Fragmented Human Coordination to an Intelligent End-to-End Closed Loop

To solve these operational gaps, the goal of the DolphinScheduler Agent solution is crystal clear:

Transform every failure alert into a continuous, traceable, and reusable handling workflow.

The old model treated alerts, UI pages, logs, group chats, and postmortems as isolated systems heavily dependent on human coordination.

The new model starts from a Feishu alert and flows through Channel conversations, intelligent orchestration, execution control, verification, and automated reporting, forming a seamless end-to-end process from trigger to resolution — without requiring engineers to jump repeatedly between systems.

Simply put:

Once an alert is triggered, the Agent automatically takes over.
Once handling is complete, it automatically replies in the group and generates a full incident report.

Operations engineers only need to review the conclusion instead of “running around everywhere.”

Five-Layer Core Architecture: Not Just Scripts, but a Safe and Controllable Intelligent Control Chain

Many people mistakenly think automated operations simply mean “bots + scripts.”

However, DolphinScheduler Agent takes a much more robust and engineering-oriented approach: a five-layer decoupled control chain. Each layer has clear responsibilities, ensuring both execution capability and strict safety boundaries.

1. L1 Event & Collaboration

Alerts directly enter Feishu threads, allowing human intervention and questioning at any time. The workflowInstanceId serves as the unique incident anchor, ensuring information is never lost or fragmented.

2. L2 Session Integration

Feishu events synchronize into local sessions, maintaining full conversational context and eliminating interruptions caused by switching systems.

3. L3 Intelligent Orchestration

Claude Code handles information organization and invocation orchestration, while Skills encapsulate DolphinScheduler domain expertise for more accurate decision-making.

4. L4 Execution Control

dsctl centrally handles the core actions of evidence collection, fault recovery, and result verification, providing standardized, reusable, and stable command execution.

5. L5 Governance & Reporting

The system automatically generates Feishu replies, incident reports, and audit logs, balancing real-time collaboration with long-term governance and postmortem analysis.

This architecture directly addresses real operational requirements:
Only through decoupled architecture can capabilities scale reliably; only through clear boundaries can automation safely enter production environments.

Four Core Modules: Making Self-Healing Truly Production-Ready

Built on top of the five-layer architecture, four tightly integrated modules make the system practical, scalable, and trustworthy.

📌 Channel: Native Feishu Entry Point for Unified Collaboration

Feishu groups become the alert entrance, collaboration interface, and result feedback page all in one.

Agents, humans, and on-call workflows collaborate within the same thread. Group chats display concise conclusions, while detailed evidence is preserved in reports for future reference.

📌 Runtime: Intelligent Orchestration Engine with Decoupled Rules and Execution

Claude Code manages conversation orchestration logic, while Skills encapsulate operational expertise such as fault response, workflow design, and data quality governance.

By separating orchestration, rules, and execution into independent layers, the system becomes highly extensible and continuously evolvable.

📌 Control Plane: dsctl as the Unified Execution Foundation

dsctl is the core execution engine powering the entire Agent system.

It provides standardized CLI capabilities that can be safely invoked by automation systems:

Evidence collection: doctor / digest / log
Fault repair: recover-failed / edit --dry-run
Result verification: watch / digest
Unified outputs: fully observable, traceable, and auditable

With dsctl, manual commands become stable automation primitives.

A Seven-Step Standard Closed Loop: Dual-Path Protection for Production Safety

From alert triggering to incident reporting, the Agent strictly follows a seven-step state machine:

Alert Parsing → Diagnosis → Decision → Execution → Verification → Response → Reporting

Two execution paths guarantee safety:

Happy Path
For low-risk scenarios with sufficient evidence:
collect evidence → generate execution plan → recover failed tasks → verify → reply in the group → generate report
Escalation Path
For insufficient evidence, high-risk situations, or failed verification:
escalate to human operators while preserving complete context — never falsely reporting success.

Everything is traceable, auditable, and reviewable, enabling safe and stable production deployment.

📌 Safety: Four-Level Risk Governance — Safety Comes First

In production automation, safety always matters more than speed.

The system classifies operations into four risk levels:

Automatically Allowed: read-only queries and log viewing
Automatic + Protection: low-risk recovery operations like recover-failed
Human Approval Required: high-risk modifications
Forbidden: dangerous operations such as force-success are directly blocked

This defines the system’s core philosophy:

The true strength of an Agent is not “daring to execute,” but knowing “when not to execute.”

A Pragmatic Roadmap: Gradual Delegation Toward Autonomous Operations

To ensure safe production adoption, the Agent follows a gradual empowerment strategy:

MVP Stage: read-only diagnosis + automated short replies
V1 Stage: enable low-risk automatic recovery via recover-failed
V2 Stage: integrate approval mechanisms for broader controllable operations
V3 Stage: accumulate Runbooks and Skills for community collaboration

The true value of this solution is not a single prompt, but an entire engineering framework built around:

Channel + Skill + CLI + Report + Safety

A reusable and portable operational architecture.

Demo

To help the audience better understand DolphinScheduler Agent’s capabilities, Liu Xiaodong also demonstrated a live demo during the session.

Please refer to the video starting from 57:10 for the full demonstration.

🎉 Official Open Source Release: dsctl Is Now Available on GitHub

The great news is that the core project powering DolphinScheduler Agent — dolphinscheduler-cli (dsctl) — has officially been open sourced!

GitHub Repository:
dolphinscheduler-cli GitHub Repository

The project provides a complete CLI toolkit supporting:

DolphinScheduler configuration and environment management
Workflow authoring, lint checking, and DryRun simulation
Runtime monitoring, instance inspection, and log retrieval
Failure recovery, rerun handling, and batch operations
Standardized outputs fully compatible with automation and Agent integration

The project is released under the Apache-2.0 license, supports one-line installation via pip, and is compatible with mainstream DolphinScheduler versions including 3.3.2, 3.4.0, and 3.4.1.

Final Thoughts

DolphinScheduler Agent is redefining the operational paradigm for big data systems:

Free people from repetitive tasks, fragmented workflows, and endless context switching — let systems handle incidents, while humans focus on decision-making and governance.

From alert triggering to automatic recovery, automated replies, and report generation, the entire process becomes a seamless one-click closed loop.

If everything runs smoothly, operations teams really can “lie back and let the system do the work.”

Developers, operators, and big data engineers are all welcome to explore dsctl on GitHub and join the community in building a simpler, smarter, and more efficient future for operations.