DEV Community: Chen Debra

Part 7 | Where Scheduling Systems Really Break and the Hidden Bottlenecks Beyond CPU and Scale

Chen Debra — Fri, 10 Apr 2026 09:58:17 +0000

In production environments, performance issues in a scheduling platform are never caused by a single bottleneck. Instead, they arise from the combined effects of scheduling decisions, task execution, metadata storage, and coordination mechanisms. Taking Apache DolphinScheduler as an example, focusing on just one component, such as the Master or Worker, often leads to misidentifying the root cause.

This article is based on real-world production experience. It systematically breaks down performance bottlenecks in a scheduling platform and provides practical, actionable optimization strategies.

1. From the overall architecture, where exactly are the bottlenecks?

The core workflow of DolphinScheduler can be abstracted as:

Scheduling → Execution → Storage → Coordination

Any layer can become a bottleneck, but the most common issues are concentrated in four areas:

Insufficient scheduling throughput on the Master
Mismatch between Worker execution capacity and workload
Excessive pressure on the database (MySQL/PostgreSQL)
Latency or instability in ZooKeeper (coordination layer)

2. The Master bottleneck is not CPU, but the “scheduling model”

Many assume the Master’s CPU is the issue. In practice, the real bottleneck is the combination of the scheduling model and database I/O.

2.1 Scheduling mechanism

The Master’s core loop looks like this:

// MasterSchedulerService.java
while (true) {
    List<ProcessInstance> instances = processService.findNeedScheduleProcessInstances();

    for (ProcessInstance instance : instances) {
        submitProcessInstance(instance);
    }
}

This is a polling + database-driven model. The key limitation is that scheduling capacity is directly tied to database throughput.

2.2 Typical symptoms

High scheduling latency:

Tasks are ready but delayed by tens of seconds before execution, while Master CPU usage remains low and database QPS is high.

Low throughput:

The system may only schedule a few hundred tasks per minute, and adding more Masters yields limited improvement.

2.3 Optimization strategies

Reduce database scanning pressure

Typical SQL:

SELECT * FROM t_ds_process_instance
WHERE state = 'READY'
LIMIT 100;

Optimization:

CREATE INDEX idx_state_priority_time 
ON t_ds_process_instance(state, priority, create_time);

Additional measures include limiting scan batch sizes and tuning scheduling intervals to avoid excessive polling.

Increase scheduling concurrency

Key configuration:

master:
  exec-threads: 100
  dispatch-task-number: 50

Practical guidelines:

exec-threads should be approximately 2 to 4 times the number of CPU cores.
dispatch-task-number should not be too large to avoid overwhelming Workers.

Scale out Masters

DolphinScheduler supports multiple Masters, but scaling is not linear due to shared database bottlenecks and ZooKeeper coordination overhead.

3. More Workers is not always better

Adding more Workers blindly can overload the database and worsen queuing.

3.1 Worker configuration

worker:
  exec-threads: 50

Workers act as both execution units and resource isolation boundaries.

3.2 Estimation formula

Worker count ≈ Total concurrent tasks / Per-Worker concurrency
Per-Worker concurrency ≈ CPU cores × (2 to 4)

3.3 Example

For 1,000 concurrent tasks and 16-core Workers:

Per Worker ≈ 32 to 64 concurrent tasks
Required Workers ≈ 1000 / 50 ≈ 20

3.4 Task type matters more

Short tasks (<5 seconds):

Scheduling overhead exceeds execution time, making the Master the bottleneck.

Long tasks (>10 minutes):

Workers become resource bottlenecks due to long occupation time.

4. Different strategies for short and long tasks

4.1 Short tasks optimization

Typical scenarios include SQL queries and API calls.

Batching example:

-- Before: multiple small queries
SELECT * FROM table WHERE id = 1;
SELECT * FROM table WHERE id = 2;

-- After: batch query
SELECT * FROM table WHERE id IN (1,2,3,...);

Other strategies include reducing DAG granularity and moving loops into scripts.

4.2 Long tasks optimization

Typical scenarios include Spark or Flink jobs.

The bottleneck lies in resource systems rather than the scheduler.

Strategies:

Bind workloads to YARN queues or Kubernetes namespaces and enforce concurrency limits.

5. The database bottleneck is the most underestimated

Around 80% of production performance issues ultimately relate to the database.

5.1 Common problems

Slow queries
Row-level lock contention
Connection pool exhaustion

5.2 Typical SQL

UPDATE t_ds_task_instance
SET state = 'RUNNING'
WHERE id = ?;

Frequent updates to the same rows lead to lock contention and reduced throughput.

5.3 Optimization strategies

Read-write separation

Masters handle writes, while APIs and queries use read replicas.

Reduce update frequency

Inefficient pattern:

RUNNING → RUNNING → RUNNING

Optimization:

Reduce heartbeat frequency.

Batch updates

// Batch update task states
updateBatch(taskInstances);

6. ZooKeeper as a hidden bottleneck

ZooKeeper is responsible for coordination, including Master election, Worker registration, and heartbeat management.

6.1 Common symptoms

Scheduling jitter under high load
Workers falsely marked as dead
Frequent Master failovers

6.2 Root causes

Improper session timeout settings
Too many nodes and connections
Network instability

6.3 Optimization

Example configuration:

tickTime=2000
initLimit=10
syncLimit=5

Recommendations:

Increase session timeout to at least 20 seconds to tolerate transient failures.
Deploy ZooKeeper independently to avoid resource contention.

7. A real-world optimization case

Background

Daily tasks: 200,000
DAGs: 30,000
Masters: 2
Workers: 30

Issues

Scheduling latency exceeded 1 minute during peak hours
Database CPU usage reached 90 percent

Optimization process

Step 1: Database indexing
Result: latency reduced by 40 percent

Step 2: Reduce short tasks
Result: DAG count reduced by 30 percent

Step 3: Adjust Master parameters

exec-threads: 50 → 120

Result: throughput doubled

Final results

Scheduling latency reduced from 60 seconds to 8 seconds
Database CPU usage reduced from 90 percent to 50 percent
Overall throughput improved by 2 to 3 times

8. Summary: the essence of scheduling performance optimization

The core insight is that performance is a balance of:

Scheduling capacity × Execution capacity × Storage capacity × Coordination capability

Optimization must be holistic:

The Master controls the scheduling rhythm
Workers provide execution capacity
The database defines system limits
ZooKeeper ensures coordination stability

Ultimately:

The limit of a scheduling system is not how many tasks it can dispatch, but how long the database can sustain the load.

Part 1 | Scheduling Systems Are More Than Just “Timers”
Part 2 | The Core Abstraction Model of Apache DolphinScheduler
Part 3 | How Scheduling Actually Runs
Part 4 | The State Machine: The Real Soul of Scheduling Systems
Part 5 | What Happens When Tasks Fail? A Complete Guide to Retry and Backfill in Apache DolphinScheduler
Part 6 | Enterprise Multi-Tenancy and Resource Isolation Techniques in DolphinScheduler You Might Not Know
Next: The boundaries between DolphinScheduler and Flink, Spark, and SeaTunnel

Can Your Scheduler Fix Itself at 2 AM? Inside the DolphinScheduler Agent Meetup

Chen Debra — Thu, 02 Apr 2026 10:18:14 +0000

If you’ve ever worked with scheduling systems, you’ve probably had moments like this:

At 2 AM, your phone suddenly lights up.
Not a message—an alert. A job has failed.

You stare at the screen, with only one thought in your head:
“Can it just fix itself?”

It sounds a bit idealistic.
But this time, we actually want to take it seriously.

Soon, the Apache DolphinScheduler community will host a new online Meetup.

This time, we won’t dive into grand architectures or complex theories.
Instead, we start with a very “engineer-like” question:

👉 Can a scheduling system require less human effort?

📅 Event Info

Time: April 21, 2026, 14:00–15:00
Format: Online livestream
Register your seat now: https://meeting.tencent.com/dm/sdXKjKfLewVe

🎤 Who’s Speaking?

This session features Liu Xiaodong,
an algorithm engineer from Shanghai FamilyMart Co., Ltd.

His self-introduction is quite fun:

Not limited to one direction—he tinkers with everything.
Writes code, builds systems, explores new ideas.
And occasionally “wanders around Hyrule to discover new landscapes.”

Sounds like this won’t be a conventional talk.

💡 What’s the Topic?

The topic is simple yet vivid:
“DolphinScheduler Agent: I Just Want to Lie Down and Still Get Work Done”

It starts from a very real idea:

The dream state of a “lazy engineer” is:
When something breaks, the system detects and fixes it automatically.
Humans just take a glance and say a word—everything else is handled.

Sounds exaggerated?

This talk will explore:
👉 How far can we actually go in this direction

🧠 What Will You Learn?

This is not a purely conceptual talk, but an ongoing exploration:

The design of DolphinScheduler Agent
How to make scheduling systems more “self-healing”
Real-world attempts and lessons learned
A working demo

Rather than giving standard answers, it’s more like:
a journey recap + new ways of thinking

🎁 Bonus

There will also be a lucky draw during the livestream 🎉

You might even win a custom Apache DolphinScheduler keychain—
a must-have for community members!

👀 Who Should Join?

This Meetup is for you if:

You’re using or exploring DolphinScheduler
You’re interested in automation, agents, or intelligent operations
You want to see real demos, not just slides
Or you simply want to “work less” in a smarter way

📢 Final Thought

We’re used to fixing problems when they occur.
But rarely do we ask:
Can systems prevent problems—or even solve them on their own?

Maybe that’s the next step for scheduling systems.

📅 April 21
Let’s talk about building systems that are a little less exhausting.

Apache DolphinScheduler Local Setup Made Simple: A Beginner-Friendly Guide

Chen Debra — Thu, 02 Apr 2026 10:08:09 +0000

This article is intended for developers who want to read and debug the core source code of Apache DolphinScheduler locally. The example environment is based on Windows + IntelliJ IDEA + Docker Desktop + PostgreSQL + ZooKeeper.

If you only want to quickly تجربه features rather than debug the full chain of master / worker / api, it is recommended to use StandaloneServer first. If you want to debug the distributed scheduling workflow, follow this guide to start services separately.

Use Cases

Start MasterServer, WorkerServer, and ApiApplicationServer individually in IntelliJ IDEA
Use Docker Desktop to host PostgreSQL and ZooKeeper
Debug Java services locally on the host machine
Run the frontend locally and connect it to backend APIs

Environment Requirements

Docker Desktop
JDK 8 or 11
Maven 3.8+ (or use the built-in mvnw.cmd)
Node.js 16+
pnpm 8+
IntelliJ IDEA

The java.version in the root pom.xml is 1.8. It is recommended to use JDK 8 or 11 for local debugging.

1. Start PostgreSQL and ZooKeeper

First, navigate to the deploy/docker directory:

cd <your-path>\dolphinscheduler\deploy\docker

If you are using the docker-compose-windows.yml provided in the appendix, ensure that dolphinscheduler-zookeeper exposes port 2181.

master, worker, and api all connect to localhost:2181 by default. If ZooKeeper runs only inside the container without port mapping, Java processes started in IDEA will fail to connect.

Ensure the following configuration exists:

dolphinscheduler-zookeeper:
  image: zookeeper:3.8
  ports:
    - "2181:2181"

Start services:

docker-compose -f docker-compose-windows.yml up -d dolphinscheduler-postgresql dolphinscheduler-zookeeper

Optional verification:

docker ps
Test-NetConnection 127.0.0.1 -Port 5432
Test-NetConnection localhost -Port 2181

Expected results:

Port 5432 is reachable
Port 2181 is reachable

If you are using local or remote installations instead of Docker, skip this step but ensure configurations match your environment.

2. Build the Project

cd <your-path>\dolphinscheduler
.\mvnw.cmd spotless:apply
.\mvnw.cmd clean install -DskipTests

Notes:

spotless:apply formats code to avoid check failures
The first build may take a while

3. Initialize PostgreSQL Metadata Database

Before starting master and api, initialize metadata tables.

SQL script location:

dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_postgresql.sql

Using Docker PostgreSQL:

Get-Content -Path .\dolphinscheduler-dao\src\main\resources\sql\dolphinscheduler_postgresql.sql -Raw |
docker exec -i -e PGPASSWORD=root docker-dolphinscheduler-postgresql-1 psql -U root -d dolphinscheduler

Alternatively, use DataGrip, DBeaver, or psql.

Note: This script contains DROP TABLE IF EXISTS. Do NOT run it on production databases.

Verification:

select version from t_ds_version;

Expected: one record returned (e.g., 3.4.0)

4. Verify Local Configuration

Default configs (usually no changes needed):

PostgreSQL: 127.0.0.1:5432
DB: dolphinscheduler
Username: root
Password: root
ZooKeeper: localhost:2181

Config files:

dolphinscheduler-master/.../application.yaml
dolphinscheduler-api/.../application.yaml
dolphinscheduler-worker/.../application.yaml

If needed, modify:

spring.datasource.url
spring.datasource.username
spring.datasource.password
registry.zookeeper.connect-string

Do NOT use:

-Dspring.profiles.active=mysql

Use instead:

-Dspring.profiles.active=postgresql

5. Configure IntelliJ IDEA Run Configurations

Common settings:

JDK: 8 or 11
Use the classpath of the module
Enable: Add dependencies with "provided" scope to classpath
Working directory: project root

This option is critical to avoid missing dependency issues.

Create these configurations:

MasterServer

Main class: org.apache.dolphinscheduler.server.master.MasterServer

Ports:

RPC: 5678
Spring Boot: 5679

WorkerServer

Main class: org.apache.dolphinscheduler.server.worker.WorkerServer

Ports:

RPC: 1234
Spring Boot: 1235

ApiApplicationServer

Main class: org.apache.dolphinscheduler.api.ApiApplicationServer

Ports:

HTTP: 12345
Gateway: 25333

Startup order:

MasterServer
WorkerServer
ApiApplicationServer

6. Start Frontend

cd <your-path>\dolphinscheduler\dolphinscheduler-ui
pnpm install
pnpm run dev

Access:

http://localhost:5173

Default credentials:

Username: admin
Password: dolphinscheduler123

7. Verification

API

/actuator/health → should return UP
/swagger-ui → should load successfully

Frontend

Access UI and log in successfully

Logs

Check for fatal errors in the IDEA console

8. Common Issues

ZooKeeper connection failed

ZooKeeper is not running
Port 2181 not exposed

Missing `t_ds_version` table

DB not initialized
Wrong database

Missing dependencies in IDEA

Check the “provided scope” option

Port 12345 occupied

Stop conflicting processes

Built by the Community: Apache DolphinScheduler March 2026 Highlights

Chen Debra — Thu, 02 Apr 2026 09:59:10 +0000

Hey there! The March 2026 monthly report is here! The Apache DolphinScheduler community has been on fire 🔥

A total of 13 contributors actively submitted code. Version 3.4.1 was released, bringing enhanced scheduling, upgraded task plugins, improved API & UI, and fixing 15+ bugs.

Meanwhile, infrastructure has also been upgraded. Both enterprise and individual users are encouraged to upgrade and explore the latest features. Let’s grow with the community 🚀

Reporting period: March 1, 2026 – March 30, 2026

1. Release

Version	Release Date	Notes
3.4.1	2026-03-01	Latest stable release

📎 Download: https://dolphinscheduler.apache.org/download

2. Key Feature Updates

2.1 Scheduling Enhancements

Feature	Description	PR
Configurable Max Runtime	Set maximum runtime limits for workflows/tasks	#17932
Worker Group Optimization	Allow creation of Worker Groups without Workers	#17927
Scheduling Timeout Detection	Handle cases with missing or unavailable Workers	#17796

2.2 Task Plugin Improvements

Task Type	Improvement	PR
Java Task	Support built-in & custom variables	#17860
Zeppelin Task	Support parameter parsing	#17862
Procedure Task	Support cancellation & output parameters	#17696, #17973
HTTP Task	Fix nested JSON sending issue	#17911

2.3 API & UI Improvements

Module	Improvement	PR
API	Remove import/export (DSIP-104)	#17941
UI	Improve Spark parameter validation	#17958
UI	Fix Keycloak icon 404 issue	#18007
UI	Fix lock not released on request failure	#17989

3. Bug Fixes

Module	Issue	PR
Master	Fix timeout alert failure	#17818
Master	Fix workflow failure strategy issue	#17851
Master	Fix task not marked failed on init error	#17821
Dependent	Fix PostgreSQL dependency SQL error	#17837
API	Fix token deletion issue for non-admin users	#17997
API	Add tenant validation	#17970
DAO	Fix type mismatch in workflow_definition_code	#17988
Alert	Fix timeout unit inconsistency	#17920
SeaTunnel	Fix broken documentation link	#17905
Params	Fix Procedure Task param passing issue	#17968

4. Community Updates

Top Contributors

In March, 31 PRs were merged. Thanks to all 9 contributors 🙌

Full list: https://github.com/apache/dolphinscheduler/graphs/contributors

Infrastructure Updates

Upgrade ZooKeeper to 3.8.3
Upgrade Testcontainers to 1.21.4
Update license year
Add AI usage confirmation to PR template

5. Enterprise Recommendations

🔧 Upgrade Advice

Production environments are recommended to upgrade to 3.4.1
Includes multiple bug fixes and stability improvements

📋 Key Features to Watch

Runtime limits for workflows/tasks
Flexible Worker Group management
Enhanced Procedure Task capabilities

⚠️ Notes

No major API changes this month
Follow official docs for latest configurations

6. Statistics

Metric	March Data
Releases	1 (3.4.1)
Improvements	10+
Bug Fixes	15+
Contributors	13+

Meet ASF’s New Member Xiang Zihao: How He Impacts the Community with Code and the Apache Way

Chen Debra — Fri, 27 Mar 2026 03:24:08 +0000

Congratulations to @xiang Zihao on being recently invited to become an ASF Member! As a PMC Member of Apache DolphinScheduler, the community is truly delighted by this well-deserved recognition.

Over the years, his continuous contributions to the community have been evident to all—from documentation improvements to code enhancements, from active discussions to helping newcomers. His presence can be seen everywhere. Beyond Apache DolphinScheduler, he is also deeply involved in multiple ASF open source projects, consistently practicing the Apache Way year after year. All his persistent efforts have finally led him to this milestone.

On this occasion, the community conducted another in-depth interview with him. This time, through five chapters—Personal Background, Open Source Contributions & Growth, Becoming an ASF Member, DolphinScheduler Community Development, and Open Source Culture—we take a closer look at his journey, his growth story in open source, and the passion and persistence he has accumulated within the community.

Part 1: Personal Background

Q1: Could you briefly introduce yourself, including how you entered the big data and open source fields?

A: I’m Xiang Zihao / SbloodyS 👋
My hobbies include coding during the day, gaming at night, taking my kid out on weekends, backpacking during holidays, and enjoying tea chats when I need a break.
My life philosophy is: explore the world through code, and heal through life.

Q2: When did you start contributing to Apache DolphinScheduler? What was the trigger?

A: I first encountered Apache DolphinScheduler in 2021. It was actually quite accidental—an opportunity at work introduced me to this scheduling system. Unexpectedly, this “chance encounter” gradually drew me in, and I began contributing to the community.

Q3: What key work or features have you contributed to DolphinScheduler?

A: I have mainly worked on documentation optimization, performance improvements, bug fixes, code reviews, and CI/CD optimization.

Part 2: Open Source Contributions & Growth

Q4: In open source collaboration, what do you think is the most important ability? Technical skills, communication, or something else?

A: I believe the most important ability in open source collaboration is not a single dimension, but a combination of technical skills, communication ability, and an open mindset.
Technical skills are the foundation, communication determines efficiency and quality, and an open mindset is the key to long-term growth.
If I had to prioritize, I’d say openness is the most fundamental—it determines whether you are willing to learn, ask, and evolve.

Q5: What advice would you give to newcomers in open source?

A: Start by “using” rather than “building.”
Become a real user first, identify problems during usage, submit issues, then gradually move to documentation fixes, bug fixes, and eventually core feature development.
Don’t aim to contribute “big features” right away—every small PR is the beginning of building trust with the community.

Part 3: Becoming an ASF Member

Q6: Congratulations on becoming an ASF Member! What was your first reaction?

A: Thank you! Honestly, my first reaction was a mix of surprise and gratitude.

Surprise—because becoming an ASF Member was never my initial goal. In 2021, I simply started contributing to solve problems and give back to the community, and I never imagined this journey would lead here.

Gratitude—because this honor represents the trust and support of the entire community. Without patient reviewers and fellow contributors, I wouldn’t be here today.

For me, becoming an ASF Member is not an endpoint, but a new beginning. It means greater responsibility and a commitment to give back even more.

Q7: How closely related is this achievement to DolphinScheduler? What other factors contributed?

A: DolphinScheduler was an important foundation, but not the only reason.

On one hand, it’s the first Apache project I deeply engaged in, where I built experience and credibility through contributions.

On the other hand, ASF evaluates broader impact:

Cross-project contributions
Community-building efforts
Practicing the Apache Way

In short, DolphinScheduler was my starting point, but sustained and sincere contributions to the broader Apache ecosystem made this possible.

Q8: What does becoming an ASF Member mean to you and the community?

A: For me, it’s recognition from the global open source community—not for one achievement, but for long-term commitment. It’s also a responsibility to keep improving.

For the community, ASF Members are core contributors responsible for project incubation, governance, and cultural inheritance.

For China’s open source ecosystem, more ASF Members represent growing global recognition and diversity.

Q9: How important is the Apache Way to project success?

A: It can be summed up in one phrase: “Community Over Code.”
Code can be replaced, but a healthy, collaborative community cannot.
The Apache Way ensures openness, transparency, and consensus-driven development—proven principles behind many successful projects.

Part 4: DolphinScheduler Community Development

Q10: What are the key milestones in DolphinScheduler’s growth?

A: Three major turning points:

Donation to Apache
Graduation from incubation
Globalization

These milestones transformed it into a globally recognized project.

Q11: How do you see its positioning and future?

A: DolphinScheduler is evolving into a next-generation cloud-native workflow orchestration platform, connecting the full data lifecycle.
Its future lies in integrating with modern data stacks and becoming essential for data engineers worldwide.

Q12: What are your future plans as an ASF Member?

A: Three directions: Deepening, Expanding, and Passing On.

Deepening: continue contributing to core tech and governance
Expanding: engage in more Apache projects
Passing On: help more developers enter open source

Open source has given me a lot—I want to pass it forward.

Part 5: Open Source Culture & Personal Growth

Q13: How has open source changed you?

A: It reshaped my definition of growth.
Before, growth meant improving skills. Now, it means expanding impact—helping others grow.
I’ve transformed from a solo problem-solver into a global collaborator.

Q14: How would you summarize the spirit of open source in one sentence?

A: Open source is a belief that sharing is more powerful than owning.

That concludes our interview! If you found this inspiring, feel free to like, share, and spread the word so more people can discover valuable insights from the open source world 🏅

Part 6 | Enterprise Multi-Tenancy and Resource Isolation Techniques in DolphinScheduler You Might Not Know

Chen Debra — Fri, 27 Mar 2026 03:22:57 +0000

In Apache DolphinScheduler, multi-tenancy is not just an “auxiliary permission feature,” but the core execution model of the scheduling system. What it truly solves is not “who can use the system,” but:

Under what identity tasks run, what resources they consume, and how to prevent interference between them

Only by understanding this can we grasp the essence of DolphinScheduler’s multi-tenant design.

What Are Single-Tenant and Multi-Tenant?

First, let’s clarify what single-tenant and multi-tenant mean.

In enterprise scheduling platforms, how different teams or business units share platform resources is a fundamental design concern. Single-tenancy and multi-tenancy are two common models, with clear differences in resource isolation, stability, and scalability. Understanding these differences helps organizations choose the right architecture for efficient and controllable scheduling.

A single-tenant system serves only one team or business unit. All tasks share the same execution environment, resource pool, and permission system.

A multi-tenant system, on the other hand, allows multiple teams to share one platform. Each team is logically isolated as an independent Tenant and mapped to underlying execution identities (Linux users), resource queues (YARN queues), or cloud-native namespaces (Kubernetes namespaces), enabling independent management of tasks and resources.

Compared with single-tenancy, multi-tenancy provides significant advantages in resource isolation, stability, and scalability. While single-tenancy is simple to deploy and manage, resource contention and task interference become inevitable as the number of users grows. Multi-tenancy avoids this by clearly isolating Tenants and assigning dedicated resource pools per team or environment.

Core Mechanism: Tenant-Centric Execution Model

To overcome the limitations of single-tenancy, Apache DolphinScheduler adopts a multi-tenant design.

At the heart of this design is a single concept: Tenant.

However, a Tenant is not just a logical label—it is an execution context container. When a task is scheduled, the system determines three key aspects based on the Tenant:

1. Execution Identity

Tasks do not run abstractly on Worker nodes; they must run as a specific OS user. A Tenant is bound to a Linux user, and tasks execute under that identity, inheriting file permissions and system-level isolation.

Example: Executing tasks as a Linux user

# Switch to the Linux user corresponding to the Tenant
sudo su - team_alpha_user

# Execute workflow task
spark-submit --class com.example.Job /opt/jobs/job.jar

Description: Tenant is bound to an OS user, and tasks run under this identity on Worker nodes, achieving file permission and environment isolation.
Tip: Ensure each Tenant has an independent home directory to avoid unauthorized access. ### 2. Resource Ownership

When tasks are submitted to engines like Spark or Flink, they must enter a resource pool. The Tenant determines the target resource queue or namespace, ensuring controlled resource usage.

Example: Create a Tenant and bind a YARN Queue

curl -X POST http://dolphinscheduler-api:12345/tenants \
  -H "Content-Type: application/json" \
  -d '{
        "name": "team_alpha",
        "queue": "team_alpha_queue",
        "description": "Team Alpha Tenant"
      }'

Description: Each Tenant corresponds to a YARN Queue or K8s Namespace, ensuring exclusive resource usage.
Tip: After creating a Tenant, remember to configure the queue or namespace in the resource scheduling system. ### 3. Isolation Boundary

Tenant defines a clear boundary for data access, task execution, and resource usage, forming logical isolation between teams.

Together, these three aspects form the foundation of DolphinScheduler’s multi-tenant mechanism.

How Resource Isolation Is Achieved

Multi-tenancy alone at the scheduling layer is not enough. The key design of DolphinScheduler is mapping Tenants to real underlying resource systems.

YARN-Based Isolation

In traditional big data architectures, Tenants are mapped to YARN queues. Each Tenant corresponds to a queue with defined capacity and limits. Tasks are submitted with queue information and scheduled accordingly, preventing resource contention.

YARN Mapping Example:

Queue configuration

<queue name="team_alpha_queue">
  <capacity>30</capacity>
  <maximum-capacity>50</maximum-capacity>
  <user-limit-factor>1.0</user-limit-factor>
</queue>

Description: Tasks automatically enter the queue when submitted, avoiding resource conflicts between Tenants.
Tip: Capacity and maximum capacity can be dynamically adjusted based on team workload.

Even if one team submits a large number of tasks, it only consumes resources within its own queue.

Kubernetes-Based Isolation

In cloud-native environments, Tenants are mapped to Kubernetes namespaces. Tasks run as Pods, and:

ResourceQuota limits total resource usage
LimitRange restricts per-task resource consumption

apiVersion: v1
kind: Namespace
metadata:
  name: team-alpha
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-quota
  namespace: team-alpha
spec:
  hard:
    cpu: "20"
    memory: "64Gi"
    pods: "50"

Description: Limits total resources and number of Pods to achieve cloud-native isolation.
Tip: Combine with LimitRange to control per-task resource limits and prevent a single task from monopolizing resources.

This approach isolates not only resources but also runtime environments and networking.

OS-Level Isolation

At the execution layer, Linux users provide the final isolation boundary. Even on the same machine, tasks from different Tenants cannot access each other’s files or scripts.

End-to-End Execution Flow

Putting everything together, the execution flow looks like this:

A workflow is triggered in DolphinScheduler
The system determines the Tenant
The Master assigns tasks to Workers
Workers switch to the corresponding Linux user
Tasks are submitted with resource metadata (YARN queue / K8s namespace)
Tasks run within the assigned resource pool under defined limits

This creates full isolation from scheduling logic to resource execution.

Technical Architecture

The architecture can be understood in three layers:

Top Layer: DolphinScheduler (Tenant / Workflow)
Middle Layer: Mapping (Linux User / YARN Queue / K8s Namespace)
Bottom Layer: Resource systems (Compute nodes / Big data clusters / Kubernetes clusters)

The key idea is:

The scheduling layer does not directly manage resources—it controls them through Tenant mapping

Why This Design Works in Enterprises

This design becomes especially powerful in enterprise environments.

When multiple teams share a platform, resource contention is inevitable. Without Tenant-to-resource mapping, a high-load workload could impact the entire system. With proper isolation, each team operates within its own boundaries.

It also simplifies troubleshooting. Issues can be traced to a specific Tenant and then to its corresponding resource pool, without affecting the entire system.

Most importantly, the design is highly scalable. Adding new teams or integrating new compute engines only requires extending Tenant mappings, without redesigning the scheduling system.

Summary

DolphinScheduler’s multi-tenant design is essentially a way to embed the scheduling system into the resource ecosystem. Instead of relying on complex logic, it leverages operating systems, resource schedulers, and container platforms to build a stable, clear, and controllable execution model.

For engineers, the real focus is not:

“How to create a Tenant”

but rather:

“How to map Tenants to resources effectively to achieve true isolation and stability”

That is the core value of multi-tenant design.

Previous articles: Part 1 | Scheduling Systems Are More Than Just “Timers”

Part 2 | The Core Abstraction Model of Apache DolphinScheduler

Part 3 | How Scheduling Actually Runs

Part 4 | The State Machine: The Real Soul of Scheduling Systems

Part 5 | What Happens When Tasks Fail? A Complete Guide to Retry and Backfill in Apache DolphinScheduler

Next article preview: Part 7 | Where Are the Performance Bottlenecks in Scheduling Platforms?

Apache SeaTunnel 2.3.13 Major Release! Top 10 Features You Should Know

Chen Debra — Fri, 20 Mar 2026 09:35:27 +0000

Apache SeaTunnel community officially released version 2.3.13! This release is a milestone for Apache SeaTunnel, bringing important features such as Checkpoint API, Flink engine upgrade, large file parallel processing, multi-table sync, AI Embedding Transform, and richer connector extensions. Whether for batch processing or real-time CDC syncing to Lakehouse, SeaTunnel can now support your data integration tasks more efficiently, stably, and intelligently.

Thanks to 50+ community contributors, this release includes 100+ PRs of new features, optimizations, and bug fixes. If you are building data warehouses, real-time sync platforms, or AI data pipelines, this release is worth your attention.

No time to read the full Release Notes? No worries, here are the Top 10 features of this release with PR references for your reference.

Full Release Note: https://github.com/apache/seatunnel/releases/tag/2.3.13

01 New Checkpoint API Enhances Task Fault Tolerance

In data sync tasks, checkpoints are one of the core mechanisms to ensure task reliability. SeaTunnel 2.3.13 introduces Checkpoint API (#10065), making task state management more flexible and providing a solid foundation for future scheduling and operation capabilities. The Zeta engine supports min-pause configuration (#9804) to avoid system pressure caused by frequent checkpoints.

Monitoring has also been enhanced, such as adding Sink commit metrics and calculating commit rate (#10233), returning PendingJobs information in the task overview interface (#9902), and providing REST API to view the Pending queue (#10078).

These capabilities help users better understand task execution status and optimize checkpoint strategies.

02 Flink 1.20.1 Support and Enhanced CDC

On the engine side, this version improves Apache Flink support. SeaTunnel now supports Flink 1.20.1 (#9576), and CDC sync capabilities have been enhanced. CDC Source now supports Schema Evolution (#9867), automatically adapting sync tasks to source table structure changes.

Additionally, NO_CDC Source also supports checkpoints (#10094), improving task recovery. These changes make SeaTunnel more stable in scenarios with frequent database schema changes.

03 Large File Parallel Reading Significantly Improved

In real data platforms, large amounts of data often exist as files, such as HDFS, object storage, or local file systems.

This release significantly optimizes file processing performance. HDFS File Connector supports true large file parallel splitting (#10332), LocalFile Connector supports CSV, Text, JSON large file parallel reading (#10142), and Parquet files now support Logical Split (#10239).

HDFS File also supports multi-table reading (#9816). These improvements significantly increase throughput for TB-scale file processing.

04 File Connector Adds Update Sync Mode

Previously, file sync tasks only supported append or overwrite. In this version, multiple file connectors add sync_mode=update, including FTP, SFTP, and LocalFile Source (#10437), and HdfsFile Source (#10268). This allows file sync tasks to support update semantics, better fitting incremental data processing scenarios.

05 Connector Ecosystem Expansion

SeaTunnel 2.3.13 continues to expand and enhance the connector ecosystem. For analytical databases, it adds DuckDB Source and Sink support (#10285), suitable for local analysis and data exploration.

New or enhanced connectors include Apache HugeGraph Sink (#10002), AWS DSQL Sink (#9739), Lance Dataset Sink (#9894), IoTDB 2.x Source and Sink (#9872).

Existing connectors have also been improved: PostgreSQL supports TIMESTAMP_TZ (#10048), Hive Sink supports SchemaSaveMode and DataSaveMode (#9743), MongoDB Sink supports multi-table writing and adds SaveMode (#9958 / #9883).

These updates significantly improve SeaTunnel’s adaptability in database and Lakehouse scenarios and the efficiency of building data pipelines.

Category	Connector	Type	Feature Highlights	PR
Analytical DB	DuckDB	Source/Sink	Read and write data from DuckDB, suitable for local analysis and exploration	#10285
Graph DB	Apache HugeGraph	Sink	Write data into HugeGraph	#10002
SQL Lakehouse	AWS DSQL	Sink	Write data into AWS DSQL	#9739
File/Dataset	Lance Dataset	Sink	Write data into Lance Dataset	#9894
Time Series DB	IoTDB 2.x	Source/Sink	Add IoTDB 2.x source and sink support	#9872
Relational DB	PostgreSQL	Source	Support TIMESTAMP_TZ type	#10048
Data Warehouse	Hive	Sink	Support SchemaSaveMode and DataSaveMode	#9743
Document DB	MongoDB	Sink	Support multi-table write and new SaveMode	#9958 / #9883

06 Kafka Supports Protobuf Schema Registry

In real-time scenarios, Kafka often uses Schema Registry. This release adds Protobuf Schema Registry Wire Format support (#10183) to Kafka Connector, allowing SeaTunnel to directly parse Protobuf data managed via Schema Registry, making real-time pipeline construction easier.

07 New AI Embedding Transform

With AI and data engineering integration, more companies need vector data pipelines.

SeaTunnel adds Multimodal Embedding Transform (#9673) in the Transform component, generating vector data directly in pipelines for vector databases, RAG systems, and AI retrieval applications. RegexExtract Transform (#9829) further enhances data cleaning.

08 Markdown Parser Supports RAG Scenarios

Markdown documents are common in AI data preparation. This release adds Markdown Parser (#9760) and related documentation (#9834) for parsing and structuring Markdown, facilitating RAG pipeline construction.

09 Stability and Performance Improvements

This release includes numerous stability and performance optimizations, such as ClickHouse Connector parallel read strategy (#9801), MySQL Connector shard calculation (#9975), JSON parsing for nested structures (#10000), Zeta engine task metrics (#9833), and more.

It also fixes production issues like Zeta engine memory leak on task cancellation (#10315), ClickHouse ThreadLocal memory leak (#10264), MongoDB multi-task submit (#10116), HBase Source scan exception (#10287), Hive Sink init failure (#10331), etc.

10 Bug Fixes and Documentation Updates

Fixes include CDC Snapshot Split null pointer (#10404), ClickHouse memory leak (#10264), MongoDB multi-task submit (#10064, #10116), HBase scan exceptions (#10336, #10287), JDBC schema merge overflow (#10387, #9942, #10093), Hive Sink overwrite semantics (#10279, #9823, #9743), Elasticsearch Sink task exit issue (#10038), and other Connector, Transform, Engine, UI, CI fixes (#10422, #10013, etc.).

Documentation improvements include SeaTunnel MCP & x2SeaTunnel docs (#10108), connector config examples (#10283, #10250, #10241, #10202), multi-table sync examples (#10241), upgrade incompatibility notes (#10068), and doc structure optimizations (#10262, #10395, #10351, #10420, #10438, #10424, #10109, #10382, #10385), helping new users get started and developers better understand architecture and features.

Thanks to Contributors ❤️

Special thanks to release manager @xiaochen-zhou for strong support in planning and execution. Thanks to all volunteers; your efforts keep the SeaTunnel community growing!

Adam Wang, AzkabanWarden.Gf, Bo Schuster, cloud456, CloverDew, corgy-w, CosmosNi, Cyanty, David Zollo, dotfive-star, dy102, dyp12, Frui Guo, Jarvis, Jast, Jeremy, JeremyXin, Jia Fan, Joonseo Lee, krutoileshii, 老王, Leon Yoah, Li Dongxu, LiJie20190102, limin, LimJiaWenBrenda, liucongjy, loupipalien, mengxpgogogo-eng, misi, 巧克力黑, shfshihuafeng, silenceland, Sim Chou, Steven Zhao, wanmingshi, wtybxqm, yzeng1618, zhan7236, zhangdonghao, zhuxt2015, zy

Download & Try

Download: https://seatunnel.apache.org/download
Upgrade Guide: https://seatunnel.apache.org/docs/upgrade-guide

Upgrade Note: If you are on SeaTunnel 2.3.x, upgrading to 2.3.13 is generally safe as it focuses on feature enhancement and stability. Back up config files and test in staging. For tasks using checkpoints, stop tasks and confirm state consistency to avoid checkpoint conflicts. Check connector config changes (Hive, MongoDB, Kafka). If using Flink engine, consider upgrading to Flink 1.20.x for better compatibility and CDC support.

Airflow Is Overkill for Most Teams-Here’s a Better Option

Chen Debra — Fri, 20 Mar 2026 07:32:35 +0000

Last year, when our team was selecting a data platform, my boss directly said:“Airflow is too heavy. The operational cost is too high. Find a lighter alternative.”

To be honest, I was a bit overwhelmed at the time. Airflow is indeed heavy. There are a lot of Python dependencies, and the Celery Executor also requires Redis or RabbitMQ. Once the scale grows a bit, you basically need to use Kubernetes.

But our data team only has a few people. Asking them to maintain crontab scripts? That would be going backwards.

Later, after browsing GitHub, I found DolphinScheduler in the Apache Incubator. It has 14.1K stars, is under the Apache 2.0 license, and was open-sourced by a Chinese company (Analysys). Now it has graduated and become a top-level Apache project.

After trying it out, I found that this thing really has something special.

Low-Code Drag-and-Drop, You Can Get Things Done Without Writing YAML

Everyone understands Airflow’s DAG configuration: workflows are written in Python code. It’s flexible, but data analysts can’t understand it.

DolphinScheduler directly provides you with a visual drag-and-drop interface. You can configure task dependencies just by clicking and dragging with your mouse.

It supports more than 30 task types: Shell, SQL, Spark, Flink, HTTP, DataX, Python… basically covering all common tasks in big data scenarios.

Want to run a Hive SQL? Drag a SQL node, configure the data source and script, connect upstream dependencies, done. No need to write a single line of Python, and no need to deal with BashOperator or SparkSubmitOperator.

This is much more friendly to non-developer roles. Data analysts can configure workflows themselves, without coming to you every day asking you to write DAGs.

Decentralized High Availability, No Dependence on ZooKeeper

Everyone knows Airflow’s architecture. The Scheduler is a single point. Although it later supports multi-Scheduler HA, it still relies on database locks to ensure tasks are not scheduled repeatedly.

DolphinScheduler was designed with decentralization from the very beginning. The architecture is very clear, with five core components:

API Server: the entry point for frontend interaction, including workflow configuration and user permission management
Master Server: DAG parsing and task distribution; multiple Masters can be deployed, and each can work independently
Worker Server: task execution nodes that receive tasks from Master and return results
Alert Server: alert notifications, supporting email, DingTalk, WeCom, Feishu, and more
Registry: registry center responsible for service discovery and distributed locks, supporting three options: JDBC, ZooKeeper, and Etcd

Let’s focus on the Master’s decentralized design.

There is no master-slave relationship between multiple Masters. After starting, each Master registers itself to the Registry, and then competes for tasks using a slot partitioning algorithm.

How is the partitioning done? It uses modulo on ID:

Command ID % total number of Masters = the slot of the current Master

For example, if you have 3 Masters, and the Command ID is 1001, then it will be assigned to slot 2 (1001 % 3 = 2, slots start from 0).

If one Master goes down, its slot will be taken over by other Masters, and tasks will not be lost.

This design is much simpler than Airflow’s Scheduler HA. It does not require complex leader election logic, and Masters can scale horizontally at any time.

Use JDBC as Registry, Say Goodbye to ZooKeeper Dependency

In the past, when building distributed scheduling systems, you couldn’t avoid ZooKeeper. Early versions of Airflow also relied on ZK. Later it switched to database locks, but there are still performance bottlenecks.

DolphinScheduler supports three types of registries: JDBC, ZooKeeper, and Etcd.

The official recommendation is to use JDBC. You can directly reuse your business database (MySQL or PostgreSQL), without deploying additional ZK or Etcd clusters.

For small and medium-sized teams, maintaining one less component means reducing cost and improving efficiency.

Of course, if you already have a ZK cluster, or have extremely high performance requirements (tens of thousands of concurrent scheduling tasks), you can still choose ZK or Etcd.

Task Dispatch Mechanism: Active Push Instead of Pull

Airflow’s Celery Executor is a typical task queue model. The Scheduler puts tasks into a Redis queue, and Workers pull them themselves.

This approach is flexible, but when the queue gets backlogged, it becomes troublesome.

DolphinScheduler uses active push. After the Master parses the DAG, it directly pushes tasks to Workers via Netty RPC.

Workers do not need to poll. The Master tells them exactly what to do.

During task allocation, load balancing is performed. By default, it uses dynamic weighted round-robin, considering CPU, memory, and thread pool usage of Workers, and assigning tasks to nodes with lower load.

If a Worker is about to be overloaded, the Master will automatically schedule tasks to other nodes.

The advantage of this push mechanism is low scheduling latency. The Master can grasp Worker status in real time, and tasks will not sit in the queue for dozens of seconds waiting to be consumed.

Plugin-Based Architecture, Replace Anything You Want

DolphinScheduler’s plugin system is quite thorough:

Task plugins: more than 30 built-in task types, and you can write your own plugins
Alert plugins: email, DingTalk, WeCom, Feishu, Telegram; if not enough, implement the Alert Plugin interface yourself
Data source plugins: MySQL, PostgreSQL, Hive, Spark SQL, ClickHouse… supporting hundreds of data sources
Storage plugins: task logs and resource files can be stored locally, on HDFS, S3, or OSS

Want to switch an alert channel? Write a plugin, package it into a JAR, drop it in, restart the service—done.

No need to modify source code, and maintenance cost is low.

Flexible Deployment, One-Click Experience with Docker

The official provides four deployment methods:

Standalone: single-machine mode, for development and testing, can run with one command
Cluster: cluster mode, standard for production, manually deploy each component
Docker: start a complete environment with one click, suitable for quick experience
Kubernetes: deploy with Helm Chart, preferred for cloud-native teams

If you want to try quickly, just use Docker Compose:

docker-compose -f docker/docker-compose.yaml up -d

After the containers start, open your browser at:
http://localhost:12345/dolphinscheduler

Default account: admin / dolphinscheduler123

Drag a Shell task and try it—you can run a workflow in a few minutes.

For production deployment, it is recommended to have at least 3 Masters plus several Workers. Use MySQL master-slave or PostgreSQL for the database, and choose JDBC as the registry.

Highlights of Version 3.4.0

The 3.4.0 version released at the end of last year mainly optimized several points:

Task priority queue: high-priority tasks can jump the queue instead of waiting
Dynamic resource allocation: Workers can dynamically adjust thread pool size based on task type
Workflow version management: DAG changes automatically save history versions, supporting one-click rollback
Enhanced lineage analysis: visualization of upstream and downstream dependencies of data tables

The most practical one is the task priority queue. Previously, when inserting urgent tasks, you had to manually pause other tasks to free resources. Now you just assign a high priority label, and the scheduler will handle it automatically.

What Kind of Teams Is It Suitable For?

That said, after talking about so many advantages, it’s only fair to discuss where it actually fits.

Suitable teams for DolphinScheduler:

Data teams with fewer than 10 people and limited operational resources
Tasks mainly based on offline batch processing, such as ETL, data synchronization, reporting scheduling
Need for a low-code platform so that analysts and business users can configure workflows
Already using MySQL/PostgreSQL and do not want to deploy ZooKeeper

Not very suitable scenarios:

Mainly real-time streaming tasks (although Flink is supported, scheduling granularity is still batch-oriented)
Heavy reliance on Python ecosystem with highly customized workflow logic (Airflow is more flexible)
Extremely large task volume with tens of thousands of concurrent scheduling tasks

Final Thoughts

Overall, DolphinScheduler’s positioning is a user-friendly, stable, and lightweight data scheduling platform.

It doesn’t have as many fancy features as Airflow, but all the core capabilities are there, and the maintenance cost is much lower.

After our team migrated from Airflow to DolphinScheduler, the cluster size was reduced from 5 nodes to 3 nodes, and operational manpower was cut by half.

Now data analysts can configure workflows themselves, and no longer need to urge me every day to write DAGs.

There is no absolute good or bad scheduling tool. The one that fits your team is the best.

If you are also looking for an alternative to Airflow, you might want to try DolphinScheduler—it might be exactly what you need.

See You in Beijing This August! CFP for Community Over Code Asia 2026 Is Now Open

Chen Debra — Fri, 20 Mar 2026 07:09:11 +0000

Community Over Code Asia 2026 will take place from August 7–9, 2026 in Beijing, and the Call for Proposals (CFP) is now officially open.

Developers, Apache Committers, open-source contributors, technology leaders, and practitioners from around the world will gather in Beijing to explore the latest practices in AI, cloud-native technologies, big data, open-source community governance, and the broader Apache ecosystem.

If you are contributing to an open-source project or using the Apache technology stack in production, this is the perfect opportunity to share your experience and step onto a global stage.

Conference Info

Date: August 7 – August 9, 2026
Location: Zhongguancun National Innovation Demonstration Zone Conference Center, Beijing

19 Tracks Covering Key Areas of the Apache Ecosystem

This year’s conference will run for three days and feature 19 technical tracks, showcasing the latest technical breakthroughs in Apache projects and innovative practices from the Apache Incubator.

The conference invites developers, technical experts, and open-source contributors worldwide to submit proposals and share insights into Apache projects, cutting-edge technologies, and open-source collaboration.

Submit to the DataOps Track!

If you have hands-on experience using Apache DolphinScheduler, optimization practices, or deep insights into new features, you are welcome to submit a talk to the DataOps Track and share your experience with the global community.

Conference website: https://asia.communityovercode.org/
Submit your proposal now:
Submission deadline: April 21, 2026, 23:59 (Beijing Time, UTC+8)
Submission language: Please submit proposals in English. Presentations can be delivered in either Chinese or English.

What Makes Community Over Code Asia 2026 Special?

Curated by top Apache experts
Each track is led by experienced contributors from the Apache Software Foundation who carefully curate high-quality sessions focusing on real technical innovation and open collaboration.

A one-stop event for Apache ecosystem trends
From Agentic Coding and AI Infrastructure to Data + AI and Streaming, the conference covers the most important topics in modern open-source development.

Connect with global open-source leaders
Meet Apache Committers, foundation members, and open-source contributors face-to-face. Exchange ideas, grow your network, and explore the spirit of “The Apache Way”.

Open source is more than code — it’s a way of collaboration and a culture of innovation. Whether you are an experienced Apache Committer or someone who just submitted your first pull request, Community Over Code Asia 2026 welcomes your voice.

See you in Beijing this August.

Part 5 | What Happens When Tasks Fail? A Complete Guide to Retry and Backfill in Apache DolphinScheduler

Chen Debra — Fri, 13 Mar 2026 08:26:12 +0000

This article is the fifth installment of the series “Understanding Apache DolphinScheduler: From Scheduling Principles to DataOps Practices.” Using Apache DolphinScheduler as an example, it explains failure retry, manual rerun, and backfill mechanisms in scheduling systems, clarifies the meaning of Exactly Once semantics in scheduling, and summarizes common misuse scenarios and best practices to help build a stable and reliable data scheduling system.

In the daily operation of data platforms, task failures are almost inevitable. Network fluctuations, insufficient resources, downstream dependency failures, and code bugs can all cause scheduled tasks to fail. When failures occur, many teams rely on automatic retries, manual reruns, or backfill operations to recover the data pipeline.
However, an often overlooked fact is:

Failure retry, manual rerun, and backfill in scheduling systems actually have completely different semantics.
If these differences are not clearly understood, it can easily lead to duplicate data, data misalignment, or even data corruption. This article analyzes the design mechanisms of Apache DolphinScheduler to explain three of the most common but frequently misunderstood capabilities in scheduling systems: failure retry, manual rerun, and backfill, and further explores the real meaning of “Exactly Once” in scheduling systems.

1 Failure Retry vs Manual Rerun: Two Completely Different Recovery Mechanisms

In scheduling systems, failed tasks are usually recovered in two ways:

Automatic Retry
Manual Rerun Many people assume the only difference between them is how they are triggered. In reality, they are fundamentally different in terms of execution semantics.

1 Automatic Retry: Re-execution within the Same Instance

In Apache DolphinScheduler, every schedule generates a Workflow Instance, which contains multiple Task Instances.
When a task fails and Retry Times is configured, the system automatically retries the task within the same task instance.
Its characteristics include:

Belongs to the same workflow instance
Keeps the same Schedule Time
Dependency relationships remain unchanged
Only the failed task is re-executed Execution flow illustration:

The design goal of automatic retry is to handle:

Transient failures
For example:

Network fluctuations
Temporary resource shortages
Short-term unavailability of external systems In such cases, automatic retry can usually restore the task quickly.

2 Manual Rerun: Creating a New Instance

Unlike automatic retry, a manual rerun creates a new instance.
In Apache DolphinScheduler, users can choose to:

Rerun failed nodes
Rerun from the current node
Rerun the entire workflow from the beginning In these scenarios, the system generates a new Workflow Instance.

This means two instances may process data for the same logical time, and downstream tasks may write data repeatedly.

If tasks are not idempotent, this may lead to duplicate data issues.

2 Backfill and Data Recovery: Reconstructing Time in Scheduling Systems

In data warehouse scenarios, backfill is a very common operation. For example:

Backfilling historical data after creating a new task
Rerunning tasks for days when execution failed
Filling missing data due to upstream delays In Apache DolphinScheduler, backfill is typically performed using Backfill Run.

1 The Nature of Backfill: Creating Multiple Historical Instances

Assume a task runs daily.
Backfill range:

2025-03-01 → 2025-03-05

The system will create multiple instances:

Instance (2025-03-01)
Instance (2025-03-02)
Instance (2025-03-03)
Instance (2025-03-04)
Instance (2025-03-05)

Each instance has:

Independent execution status
Independent dependency relationships
Independent parameters The schedule time will be set to the corresponding historical time.

2 The Key to Backfill: Schedule Time vs Execution Time

In scheduling systems, two concepts are extremely important.
Schedule Time

Logical data time

Execution Time

Actual task runtime

Example:

Schedule Time : 2025-03-01
Execution Time: 2025-03-10

If the SQL uses:

WHERE dt = ${schedule_time}

Backfill is safe.
But if the SQL uses:

WHERE dt = today()

Backfill will produce incorrect data.
This is also the root cause of many data quality issues.

3 Exactly Once in Scheduling Systems: What Does It Really Mean?

In stream processing systems such as Apache Flink, Exactly Once usually means:

Each record is processed only once.
However, in scheduling systems, Exactly Once has a completely different meaning.
A scheduling system cannot guarantee that tasks will not run multiple times, nor can it guarantee that data will not be written repeatedly. This is because automatic retries may re-execute tasks, manual reruns may re-execute tasks, and backfill may rerun historical logic.
Therefore, in scheduling systems, Exactly Once is closer to the idea that:
Only one logical instance is generated for the same schedule time.
But the task itself may still run multiple times.
Thus, true Exactly Once semantics must be guaranteed by idempotent task logic.
Common implementations include:

1 Overwrite Write

INSERT OVERWRITE TABLE

2 Partition-based Writing

partition dt='${schedule_time}'

3 Deduplicated Writing

MERGE INTO

4 Common Misuse Scenarios

Many data incidents actually stem from misunderstandings of scheduling semantics.

1 Using Current Time as Data Date

Incorrect example:

dt = today()

Correct approach:

dt = ${schedule_time}

2 Non-idempotent Writes

Example:

INSERT INTO table

If the task is rerun:

duplicate data will occur

3 Manually Rerunning the Entire Workflow

Many users habitually do:

Failure → rerun from the beginning

But a safer approach is:

rerun only the failed nodes

5 Best Practice Recommendations

Based on experience using Apache DolphinScheduler, several important practices can be summarized.

1 Tasks Must Be Designed to Be Idempotent

All tasks should allow:

repeated execution

without affecting data correctness.

2 Data Logic Must Be Based on Schedule Time

Avoid using:

now()
today()

Always use:

${schedule_time}

3 Use Retry Strategies Appropriately

Recommended configuration:

Retry Times: 1~3
Retry Interval: 1~5 min

Avoid infinite retries.

4 Control Concurrency During Backfill

If the backfill range is too large:

a large number of instances may be generated at once

which may cause:

scheduling queue congestion
cluster resource exhaustion Recommendation:

perform backfill in batches

Conclusion

In data platforms, scheduling systems are often regarded as simple “task triggers.” In reality, they are responsible for time management, dependency control, and failure recovery.
By understanding the true semantics of failure retry, manual rerun, and backfill, we can build stable and reliable data production systems.
Modern scheduling systems, such as Apache DolphinScheduler, already provide powerful mechanisms. However, the ultimate factor determining data quality is still:

Correct understanding of scheduling semantics + idempotent data task design.
Only in this way can data platforms remain recoverable, traceable, and reconstructable even when failures occur.

Previous articles:
Part 1 | Scheduling Systems Are More Than Just “Timers”
Part 2 | The Core Abstraction Model of Apache DolphinScheduler
Part 3 | How Scheduling Actually Runs
Part 4 | The State Machine: The Real Soul of Scheduling Systems
Next article preview:
Part 6 | Multi-Tenant and Resource Isolation Design in Apache DolphinScheduler

Apache DolphinScheduler 3.4.1 Released with Task Dispatch Timeout Detection

Chen Debra — Fri, 13 Mar 2026 08:06:35 +0000

The 3.4.1 version of Apache DolphinScheduler has been officially released by the community. As a maintenance release in the 3.4.x series, this update focuses on improving scheduling stability, enhancing task execution control, and fixing system issues.

The new version introduces a task dispatch timeout detection mechanism and maximum runtime control for tasks, while also resolving multiple issues in scheduling logic, plugin functionality, and API behavior. In addition, system documentation, development processes, and project structure have been further optimized.

For more details, see the Release Note: https://github.com/apache/dolphinscheduler/releases/tag/3.4.1
Source code download: https://dolphinscheduler.apache.org/zh-cn/download/3.4.1

Key Highlights

Task Dispatch Timeout Detection Mechanism

A task dispatch timeout checking logic has been added to the Master scheduling module. When a task is dispatched to a Worker for execution, if the Worker Group does not exist or no Worker nodes are available, the scheduler can detect the dispatch exception within a certain period and handle it accordingly.

This mechanism prevents tasks from remaining in a waiting state for an extended time and improves the system’s fault tolerance in scenarios involving resource anomalies (#17795, #17796).

Support for Configuring Maximum Runtime for Workflow and Task Instances

The new version allows users to configure a maximum runtime for both Workflow Instances and Task Instances.

Users can define the maximum execution duration for tasks or workflows. If the runtime exceeds the configured threshold, the system can trigger timeout handling mechanisms, preventing tasks from hanging or occupying resources indefinitely and improving overall operational controllability (#17931, #17932).

Key Fixes and Improvements

Scheduling System Stability Fixes

Fixed an issue where task timeout alerts were not triggered (#17820, #17818)
Fixed the issue where the workflow failure strategy did not take effect (#17834, #17851)
Automatically mark a task as failed when task execution context initialization fails (#17758, #17821)
Fixed incorrect parallelism calculation in backfill tasks under parallel execution mode (#17831, #17853)

Database and Compatibility Fixes

Fixed SQL execution errors for dependent tasks in PostgreSQL environments (#17690, #17837)
Fixed mismatched INT/BIGINT column types in database tables (#17979, #17988)

API and Permission Fixes

Removed the WAIT_TO_RUN state and added a FAILOVER state when querying workflow instances (#17838, #17839)
Added tenant validation for the Workflow API (#17969, #17970)
Fixed an issue where non-admin users could not delete their own Access Tokens (#17995, #17997)

Plugin and Task Execution Fixes

Fixed incorrect JVM parameter position in Java Task (#17848, #17850)
Fixed an issue where Procedure Task parameters could not be passed correctly (#17967, #17968)
Fixed the issue where ProcedureTask could not return parameters or execute query stored procedures (#17971, #17973)
Fixed an issue where the HTTP plugin could not send nested JSON structures (#17912, #17911)
Fixed inconsistent timeout units in the HTTP alert plugin (#17915, #17920)

UI and Documentation Fixes

Removed the STOP state from task instances in the UI (#17864, #17865)
Fixed an issue where locks were not released when workflow definition list loading failed (#17984, #17989)
Fixed the Keycloak login icon 404 issue (#18006, #18007)
Corrected errors in the installation documentation (#17901, #17903)
Fixed a SeaTunnel documentation link 404 issue (#17904, #17905)

In-Depth Feature Analysis

In modern data platform architectures, scheduling systems often serve as key infrastructure connecting various computing engines. Tasks from systems such as Apache Spark, Apache Flink, and Apache Hive are commonly orchestrated through a unified scheduler.

However, in production environments, scheduling systems often face challenges such as:

Worker resource anomalies preventing tasks from being scheduled
Uncontrollable task execution time
Unstable plugin execution behavior

The newly introduced task dispatch timeout detection mechanism enables the scheduler to quickly identify anomalies when Workers do not exist or resources are unavailable, preventing tasks from waiting indefinitely (#17795, #17796).

At the same time, the maximum runtime control capability provides a more flexible management approach for task execution. By setting a maximum runtime for workflows or tasks, the system can take action when tasks hang or run abnormally long, preventing resources from being occupied for extended periods (#17931, #17932).

These improvements further enhance DolphinScheduler’s stability and controllability in production-grade data platform environments.

Acknowledgements

The release of Apache DolphinScheduler 3.4.1 would not have been possible without the contributions of community developers. Special thanks to the release manager @ruanwenjun and the following contributors for their work on this version:

SbloodyS
njnu-seafish
Mrhs121
ylq5126
qiong-zhou
XpengCen
iampratap7997-dot
yzeng1618
Alexander1902
maomao199691
asadjan4611
dill21yu

Final Thoughts

Apache DolphinScheduler 3.4.1 is a maintenance release focused on improving scheduling stability and enhancing task runtime control.

With the introduction of scheduling fault-tolerance mechanisms, maximum task runtime control, and numerous bug fixes, this version further strengthens the system’s reliability in production environments.

As the community continues to grow, Apache DolphinScheduler is steadily improving its capabilities in the data workflow orchestration space, providing enterprises with a more stable and efficient infrastructure for building modern data platforms. We welcome more contributors to join the community and help drive the development of the project forward.

Apache DolphinScheduler February 2026: What the Community Shipped

Chen Debra — Thu, 05 Mar 2026 09:41:59 +0000

In February 2026, the Apache DolphinScheduler community maintained an active development pace. This month’s work mainly focused on improving system stability, enhancing existing features, and optimizing code quality. Community members made significant contributions in bug fixing, user experience improvements, documentation updates, and advancing important architectural decisions.

Key Highlights

1. Support for Configurable Maximum Runtime of Workflow/Task Instances

One of the most important features introduced this month is support for configurable maximum runtime for workflow and task instances (Feature-17931). Users can now set a maximum runtime for a workflow or an individual task. When the instance exceeds this time limit, the system will automatically handle it (for example, marking it as failed or canceling it). This provides stronger guarantees for resource management and preventing runaway tasks.

2. Master Node Dispatch Timeout Check Logic

To improve system robustness, timeout checking logic for task dispatch has been added to the Master node (Improvement-17795). When a Worker group does not exist or no available Workers are present, this feature can handle dispatch timeout scenarios, preventing tasks from remaining in a waiting state for a long time and improving scheduling reliability.

3. Proposal to Remove the Import/Export Feature

The community is discussing an important improvement proposal (DSIP-104) that suggests removing the import and export functionality from the project (DSIP-104). This usually indicates that the community is considering adopting more modern and reliable ways to manage and migrate workflows, such as through GitOps or other version-control-friendly approaches. This is an architectural evolution worth watching.

Fixes and Improvements

UI/UX

Fixed the issue where the KeyCloak icon returned a 404 error (Fix-18006).
Improved the validation logic for Spark parameters, enhancing the experience when configuring Spark tasks (Improvement-17957).
Fixed an issue where the workflow definition list loading lock was not released when a request failed (Fix-17984).

API and Backend

Stored Procedure Task Enhancements: This month the community focused on improving and fixing the Procedure task type. Issues with parameter passing not working (Fix-17967) and local parameters not being passed correctly (Fix-17971) were resolved, improving the stability of this task type.
Fixed a permission issue where non-admin users could not delete their own access tokens (Fix-17995).
Fixed missing tenant validation in workflows, strengthening multi-tenant security (Fix-17969).
Fixed inconsistent timeout unit settings in the HTTP alert plugin (Fix-17915).

Database

Fixed the mismatch between the INT and BIGINT types of the workflow_definition_code field in the t_ds_serial_command table (Fix-17979), ensuring database stability and data consistency.

Other Improvements

Added support for creating Worker groups without Workers, providing more flexible resource configuration (Improvement-17926).
Hardened the startup scripts and parameter handling for SeaTunnel tasks (Improvement-17994).

Community and Ecosystem

Documentation

Community members fixed several spelling and wording issues in multiple README files (Doc).
Added a section on frontend code checks in the development documentation to help new contributors better follow project standards (Doc-17913).

Code Quality and Refactoring

Upgraded the Zookeeper dependency version to 3.8.3 (Chore).
Upgraded the testcontainer dependency version to 1.21.4 to resolve Docker environment issues in CI (Chore).
Refactored the datasource plugin manager and processor manager to improve code structure (Chore).
Refactored Kubernetes task code by moving the generateK8sTaskExecutionContext method into the more specific K8sTaskParameters class, making code responsibilities clearer.

Community Governance and Continuous Integration (CI)

Added AI usage confirmation to the PR template, reflecting the community’s attention to code contribution quality and originality (Chore).
Updated the CI configuration so that when new commits are pushed to a PR, previous review comments automatically become outdated. This helps ensure that code reviews are always based on the latest changes, improving collaboration efficiency.

Acknowledgments to Contributors

Thanks to all community members who contributed to Apache DolphinScheduler in February (in no particular order):

Wenjun Ruan
xiangzihao
yzeng1618
Divyansh Pratap Singh
dill
Muhammad Asad
huangsheng
XpengCen
njnu-seafish
maomao_zero

Special thanks to Wenjun Ruan, who was very active in February and contributed numerous fixes, improvements, and code refactoring to the community.

Outlook

From the February updates, it is clear that the Apache DolphinScheduler community continues to move steadily toward becoming more stable, more user-friendly, and more powerful. In the coming months, the community is expected to continue:

Continuously improving stability: Bug fixes and system improvements will remain a top priority.
Advancing architectural optimization: As seen in discussions about the import/export feature, the community will keep exploring and implementing better architectural solutions.
Focusing on user experience: Ongoing UI/UX improvements will provide users with a better operational experience.

Thanks to all developers who contributed to the DolphinScheduler community.

Note: The references in parentheses (for example, Fix-18006) correspond to Issue or Pull Request numbers on GitHub for DolphinScheduler, allowing readers to find more detailed information.

DEV Community: Chen Debra

Part 7 | Where Scheduling Systems Really Break and the Hidden Bottlenecks Beyond CPU and Scale

1. From the overall architecture, where exactly are the bottlenecks?

2. The Master bottleneck is not CPU, but the “scheduling model”

2.1 Scheduling mechanism

2.2 Typical symptoms

2.3 Optimization strategies

Reduce database scanning pressure

Increase scheduling concurrency

Scale out Masters

3. More Workers is not always better

3.1 Worker configuration

3.2 Estimation formula

3.3 Example

3.4 Task type matters more

4. Different strategies for short and long tasks

4.1 Short tasks optimization

4.2 Long tasks optimization

5. The database bottleneck is the most underestimated

5.1 Common problems

5.2 Typical SQL

5.3 Optimization strategies

Read-write separation

Reduce update frequency

Batch updates

6. ZooKeeper as a hidden bottleneck

6.1 Common symptoms

6.2 Root causes

6.3 Optimization

7. A real-world optimization case

Background

Issues

Optimization process

Final results

8. Summary: the essence of scheduling performance optimization

Previous articles:

Can Your Scheduler Fix Itself at 2 AM? Inside the DolphinScheduler Agent Meetup

📅 Event Info

🎤 Who’s Speaking?

💡 What’s the Topic?

🧠 What Will You Learn?

🎁 Bonus

👀 Who Should Join?

📢 Final Thought

Apache DolphinScheduler Local Setup Made Simple: A Beginner-Friendly Guide

Use Cases

Environment Requirements

1. Start PostgreSQL and ZooKeeper

2. Build the Project

3. Initialize PostgreSQL Metadata Database

4. Verify Local Configuration

5. Configure IntelliJ IDEA Run Configurations

MasterServer

WorkerServer

ApiApplicationServer

6. Start Frontend

7. Verification

API

Frontend

Logs

8. Common Issues

ZooKeeper connection failed

Missing t_ds_version table

Missing dependencies in IDEA

Port 12345 occupied

Built by the Community: Apache DolphinScheduler March 2026 Highlights

1. Release

2. Key Feature Updates

2.1 Scheduling Enhancements

2.2 Task Plugin Improvements

2.3 API & UI Improvements

3. Bug Fixes

4. Community Updates

Top Contributors

Infrastructure Updates

5. Enterprise Recommendations

🔧 Upgrade Advice

📋 Key Features to Watch

⚠️ Notes

6. Statistics

Missing `t_ds_version` table