DEV Community: Chen Debra

💡 Open source isn't just for developers. Learn how anyone can contribute with docs, design, translation, and AI—no coding required!

Chen Debra — Thu, 23 Jul 2026 02:06:46 +0000

Chen Debra

Jul 23

You Don't Need to Code to Contribute: A Beginner's Guide to Open Source

#nocode #opensource #beginners #tutorial

11 min read

You Don't Need to Code to Contribute: A Beginner's Guide to Open Source

Chen Debra — Thu, 23 Jul 2026 02:06:14 +0000

Many people assume that open source is a world reserved for elite programmers and seasoned developers. If you can't read code or write software, it can feel like there's no place for you.

The reality is very different.

Countless contributors from non-technical backgrounds have faced the same challenges—feeling overwhelmed by GitHub, intimidated by unfamiliar terminology, or convinced they simply weren't "technical enough." Yet many of them have gone on to become trusted community members and invaluable contributors.

If you've ever wanted to get involved in open source but didn't know where to begin, this guide is for you. Based on real-world experience and lessons learned along the way, it will help you understand how open source works, overcome the common hurdles, and discover meaningful ways to contribute—even if you never write a single line of code.

Open Source Isn't as Intimidating as You Think

Before contributing to an open source project, it's worth setting aside some of the myths that often surround open source.

Open source software (OSS) is software whose source code is made publicly available under an open source license. These licenses allow anyone to view, study, modify, and redistribute the source code, provided they comply with the terms of the license.

At its core, open source is about collaboration. It's not just a way to develop software—it's a model for building technology in the open, where people from around the world can improve projects together.

A few key ideas help explain what open source really means.

Freedom to Use, Learn, and Build

Open source licenses don't mean that software has "no owner." Copyright remains with the original author or organization. What the license does is grant others the right to use, modify, and distribute the software under clearly defined conditions.

Depending on the license, you may be free to:

Use the software for personal or commercial purposes
Study how it works
Modify the source code
Share your own versions with others

As long as you follow the license requirements, these freedoms are protected.

Open Collaboration at Scale

Unlike traditional proprietary software, open source projects are developed in public.

Developers, technical writers, designers, translators, community managers, and users from around the world all contribute through discussions, documentation, code reviews, bug reports, feature requests, and countless other forms of collaboration.

In many ways, open source resembles community-driven knowledge platforms: everyone has the opportunity to make the project better.

Where the Term "Open Source" Came From

The term "open source software" emerged in the late 1990s as a way to make the principles of free software more accessible to businesses and the broader technology industry.

While the philosophy has evolved over time, the goal remains the same: encouraging innovation through openness, collaboration, and shared knowledge.

Why Do Open Source Foundations Matter?

If an open source project were a startup, an open source foundation would be the organization responsible for ensuring that the project can thrive over the long term.

Beyond providing visibility and credibility, foundations play a critical role in project governance. They act as independent legal entities that manage trademarks, intellectual property, and other shared assets, helping ensure that no single company can take unilateral control of the project. This vendor-neutral governance model is one of the defining characteristics of many successful open source communities.

Today, several foundations have become pillars of the global open source ecosystem.

Apache Software Foundation (ASF)

The Apache Software Foundation (ASF) is home to hundreds of influential open source projects, particularly in the big data ecosystem. Beyond hosting projects, ASF is widely recognized for its mature community governance model and its emphasis on meritocracy, openness, and collaborative decision-making.

Linux Foundation

The Linux Foundation stewards the Linux kernel while supporting some of the world's largest collaborative open source initiatives. It also plays a major role in advancing industry standards and fostering innovation across cloud computing, networking, security, AI, and many other technology domains.

Cloud Native Computing Foundation (CNCF)

The Cloud Native Computing Foundation (CNCF) focuses on cloud-native technologies such as Kubernetes, Prometheus, Envoy, and many other projects that power modern infrastructure. It has become one of the most influential organizations in the cloud-native ecosystem.

OpenAtom Foundation

The OpenAtom Foundation is one of China's leading open source foundations. It supports the development of domestic open source projects while promoting collaboration between Chinese open source communities and the global ecosystem.

But foundations do much more than simply host projects.

They provide governance frameworks, establish community standards, help projects adopt sustainable operating models, and create opportunities for collaboration across communities. For emerging projects, joining a foundation often means gaining access to proven governance practices, experienced mentors, and a broader developer ecosystem.

Open Source Licenses: The Legal Foundation of Every Project

For many newcomers—especially those without a technical background—open source licenses can feel like an afterthought. In reality, they're one of the most important parts of the open source ecosystem.

An open source license defines what you can do with the code, what obligations you have when using it, and how intellectual property is protected. Whether you're building a commercial product, contributing to an existing project, or simply experimenting with an open source library, understanding the license is essential.

A simple way to think about it is this: an open source license is similar to a content license for creative work. The author decides how others may use their work, and anyone who benefits from it must respect those terms.

While dozens of open source licenses exist today, most fall into three broad categories.

MIT and BSD Licenses: Maximum Flexibility

The MIT License and BSD Licenses are among the most permissive open source licenses available.

As long as you retain the original copyright notice, you're generally free to use, modify, distribute, and even incorporate the code into proprietary commercial software.

Their simplicity and flexibility have made them popular choices for libraries, frameworks, and developer tools.

Apache License 2.0: Enterprise-Friendly by Design

The Apache License 2.0 strikes a balance between openness and legal clarity.

In addition to allowing commercial use, modification, and redistribution, it explicitly addresses patent grants, contributor protections, and trademark usage—areas that are especially important for organizations building products on top of open source software.

That's one of the reasons why many widely adopted projects in big data, cloud computing, and distributed systems are released under Apache License 2.0.

GNU GPL: Share-Alike by Design

The GNU General Public License (GPL) follows a different philosophy.

Often described as a strong copyleft license, the GPL requires that if you distribute software derived from GPL-licensed code, the derivative work must also be released under the GPL.

It's worth noting, however, that this obligation is generally triggered by distribution. If GPL-licensed software is modified solely for internal use and never distributed outside your organization, those source code disclosure requirements typically do not apply.

Each license reflects a different approach to collaboration, software freedom, and intellectual property. Understanding those differences helps you choose the right projects to use—and the right license if you ever publish software of your own.

You Don't Need to Write Code to Contribute

One of the biggest misconceptions about open source is that every meaningful contribution must involve code.

It doesn't.

In fact, healthy open source communities depend on contributors with a wide range of skills. Documentation, design, education, community building, localization, and user support are all critical to a project's long-term success.

That said, it's also fair to acknowledge that most open source communities are still designed primarily with developers in mind. For first-time contributors without a technical background, the learning curve can feel steep.

Learning the Workflow Can Be the Hardest Part

For many newcomers, the biggest challenge isn't understanding the project itself—it's understanding the contribution workflow.

Concepts like forking a repository, creating a branch, opening a pull request, or syncing with the upstream repository are second nature to experienced developers, but they can be overwhelming if you've never used GitHub before.

It's not unusual for someone's very first contribution to be delayed simply because they're trying to figure out how Git works.

And that's perfectly normal.

Non-Code Contributions Matter More Than You Think

Every successful open source project relies on much more than software engineering.

Communities also need people who can:

Write and improve documentation
Design diagrams, illustrations, and presentation materials
Translate documentation into multiple languages
Organize meetups, webinars, and community events
Produce tutorial videos and educational content
Share real-world use cases and success stories
Moderate discussion forums and community channels
Improve onboarding guides for new contributors

These contributions may not appear in the project's source code, but they often have an equally significant impact on community growth and user adoption.

Many long-time community leaders built their reputation not through code contributions alone, but by consistently helping others, improving documentation, and strengthening the community over time.

Trust Is Built Through Consistency

Unlike a merged pull request, the impact of non-code contributions isn't always visible immediately.

Community building is a long-term investment.

Whether you're answering questions, improving documentation, translating tutorials, or organizing events, every contribution helps make the project more welcoming and accessible.

Over time, those consistent efforts build trust—and trust is one of the most valuable forms of contribution in any open source community.

Three Challenges Almost Every New Contributor Faces

Whether you're joining Apache DolphinScheduler, Kubernetes, or virtually any other open source community, you'll likely encounter a few common obstacles when you're getting started.

The good news? They're all part of the learning process.

1. Learning the Tooling

Compared with the business applications most people use every day—such as office suites, knowledge bases, or CRM platforms—developer tools often feel much more intimidating.

Applications like GitHub Desktop, Git clients, IDEs, and terminal environments introduce new concepts and workflows that can take time to learn.

For many newcomers, simply becoming comfortable with these tools is the first major milestone.

2. Understanding Merge Conflicts

Imagine this scenario: you've finally submitted your first pull request, only to discover that GitHub reports a merge conflict.

If you've never encountered one before, the term alone can be discouraging.

Fortunately, the concept is much simpler than it sounds.

Think of it like two people editing the same paragraph in a shared document at the same time. Since both versions change the same content differently, Git can't automatically determine which version should be kept. Human intervention is needed to resolve the conflict.

Once you understand the analogy, merge conflicts become far less intimidating.

3. Debugging Error Messages

Error messages are another common source of frustration for new contributors.

You know something has gone wrong—but you may have no idea where to start.

A failed build, a Git error, or a confusing terminal message can feel like reading a foreign language. Without prior experience, identifying the root cause is often the most difficult part of solving the problem.

The important thing to remember is that every experienced contributor has been in the same position.

Learning to troubleshoot is a skill that develops over time—not something anyone is expected to master on day one.

How AI Is Lowering the Barrier to Open Source

The rise of generative AI is transforming how people learn, contribute to, and collaborate within open source communities.

For contributors without a software engineering background, AI isn't a replacement for learning—it's an accelerator. It helps newcomers overcome technical hurdles faster, understand unfamiliar concepts, and create high-quality contributions with greater confidence.

Whether you're improving documentation, designing visual assets, translating content, or troubleshooting an issue, AI can dramatically reduce the time it takes to make meaningful contributions.

Here are a few practical ways AI is making open source more accessible.

Turn Complex Technical Concepts into Visuals

One of the biggest challenges for newcomers is understanding architecture diagrams, system workflows, and technical documentation.

AI-powered design tools can transform dense technical descriptions into clear, easy-to-understand diagrams, making complex systems far more approachable.

For example, instead of asking new users to interpret a lengthy explanation of a distributed workflow engine, you can generate a simplified architecture diagram or request flow illustration that highlights the key components and interactions. Visual learning materials like these can significantly shorten the onboarding process for new community members.

Create Real-World Architecture and Solution Diagrams

Open source projects often need more than technical documentation—they also need compelling ways to demonstrate real-world business value.

With the right prompts, AI can generate polished architecture diagrams, workflow illustrations, and solution overviews tailored to specific industries or use cases.

Imagine showcasing how an open source workflow orchestration platform supports a manufacturing production line, or illustrating how data flows through a modern data platform. These visuals make it much easier for prospective users, customers, and decision-makers to understand the practical value of a project.

For community advocates and technical marketers, this can be just as valuable as writing code.

Use AI to Explain Error Messages

Debugging can be one of the most intimidating parts of contributing to open source.

Instead of struggling through pages of unfamiliar error logs, try asking an AI assistant to explain them in plain language.

Rather than simply identifying the error, modern AI models can often:

Explain what the message actually means
Point out the likely root cause
Recommend a step-by-step troubleshooting approach
Suggest possible fixes
Clarify why a merge conflict or build failure occurred

For newcomers, having an "always-available mentor" dramatically reduces the frustration of solving technical problems independently.

Translate and Localize Documentation

Most successful open source communities are global communities.

That means documentation, tutorials, release notes, and user guides often need to be available in multiple languages.

AI can dramatically speed up translation while also helping contributors adapt content for local audiences. Instead of relying on literal machine translation, you can use AI to rewrite documentation in a way that feels natural to readers in different regions.

Just as importantly, AI can help transform highly technical documentation into beginner-friendly tutorials—making projects more welcoming to first-time users around the world.

Build Better Community Knowledge

Every active open source community eventually encounters the same challenge: the same questions get asked over and over again.

Issues, Discussions, Slack channels, Discord servers, and mailing lists all accumulate valuable knowledge—but much of it remains scattered across different platforms.

AI can help identify recurring questions, summarize community discussions, and organize them into well-structured FAQs, troubleshooting guides, or onboarding documentation.

This not only reduces the support burden on maintainers but also makes it much easier for newcomers to find answers on their own.

In many communities, improving knowledge sharing is one of the highest-impact contributions anyone can make.

Your First Contribution Doesn't Have to Be Code

Open source has never been exclusively about software development.

At its heart, it's about openness, collaboration, and the collective effort to build something that benefits everyone. Great communities thrive because they bring together people with different backgrounds, experiences, and skill sets—not because everyone writes code.

If you're just getting started, you don't need to begin by submitting a complex pull request.

A much simpler path is often the best one.

Start by choosing a project you genuinely use or care about. Fix a typo in the documentation, improve an installation guide, or suggest a clearer explanation for new users. These small contributions are often the first step taken by long-time community members.

Next, become part of the community itself. Join discussion forums, attend online meetups, participate in community events, or help organize local gatherings. Open source is built on relationships just as much as it is on repositories.

Finally, look for ways to contribute using the skills you already have. Whether you're a designer, translator, technical writer, marketer, educator, product manager, or simply someone who enjoys helping others, your expertise can make a meaningful difference.

The first few weeks may feel unfamiliar. Learning GitHub, understanding community workflows, and navigating technical terminology all take time.

But with today's AI tools acting as learning companions—and with communities becoming more welcoming than ever—the barriers to entry are lower than they've ever been.

Every thriving open source project needs developers.

But it also needs writers.

Designers.

Translators.

Educators.

Community builders.

Advocates.

And people who are simply willing to help.

If you've been waiting for the "right time" to join an open source community, this is it. Your first contribution doesn't have to be code—it just has to be yours.

🚀 Run Apache DolphinScheduler with Spark on Kubernetes! Learn cloud-native deployment, shared storage, and production-ready best practices. ☸️⚡ #ApacheDolphinScheduler #Kubernetes #ApacheSpark #CloudNative

Chen Debra — Thu, 23 Jul 2026 01:50:02 +0000

Chen Debra

Jul 23

Running Apache DolphinScheduler with Spark on Kubernetes: A Practical Deployment Guide

#apachedolphinscheduler #spark #kubernetes #opensource

9 min read

Running Apache DolphinScheduler with Spark on Kubernetes: A Practical Deployment Guide

Chen Debra — Thu, 23 Jul 2026 01:46:59 +0000

As enterprise data volumes continue to grow, deploying Apache DolphinScheduler on traditional physical servers or virtual machines is becoming increasingly challenging. Complex environment setup, low resource utilization, and limited scalability often make operations difficult. Running Spark and other big data workloads further amplifies these issues, requiring teams to maintain multiple runtime environments, resolve dependency conflicts, and handle unpredictable traffic spikes.

Deploying DolphinScheduler on Kubernetes provides a cloud-native approach to solving these challenges. Containerization ensures consistent runtime environments, shared storage simplifies the distribution of Spark binaries and other dependencies, and Kubernetes enables elastic resource scaling to maximize infrastructure efficiency.

In this guide, you'll learn how to deploy Apache DolphinScheduler on Kubernetes and integrate Spark workloads step by step. By the end, you'll have a highly available, scalable, and cloud-native workflow orchestration platform ready for production.

Environment Preparation

Prerequisites

Before getting started, ensure your environment meets the following requirements:

Kubernetes cluster (v1.19 or later)
kubectl command-line tool
Helm 3.x
A storage class that supports the ReadWriteMany (RWX) access mode (required for shared storage)

Deployment Architecture

Basic Deployment

Step 1. Add the Helm Repository

helm repo add dolphinscheduler https://dolphinscheduler.apache.org/helm
helm repo update

Step 2. Create a Namespace

kubectl create namespace dolphinscheduler

Step 3. Configure `values.yaml`

Create a custom values.yaml file with the following key configurations.

# Shared storage configuration
# Required for storing Spark binaries and other shared resources
common:
  sharedStoragePersistence:
    enabled: true
    mountPath: "/opt/soft"
    accessModes:
      - "ReadWriteMany"
    storageClassName: "your-storage-class"   # Replace with your storage class
    storage: "20Gi"

  configmap:
    # Spark environment variables
    SPARK_HOME: "/opt/soft/spark"
    HADOOP_HOME: "/opt/soft/hadoop"
    HADOOP_CONF_DIR: "/opt/soft/hadoop/etc/hadoop"
    JAVA_HOME: "/opt/java/openjdk"

# External database configuration
externalDatabase:
  enabled: true
  host: "your-mysql-host"
  port: "3306"
  database: "dolphinscheduler"
  username: "root"
  password: "your-password"
  type: "mysql"

# Worker configuration
worker:
  replicas: 2
  env:
    WORKER_EXEC_THREADS: "10"

Key configuration notes

Shared Storage must be enabled to support Spark integration.
The shared directory is mounted at /opt/soft, where Spark and Hadoop binaries will be stored.
Ensure your Kubernetes storage class supports the ReadWriteMany (RWX) access mode so that all Worker Pods can access the same files.
Using an external MySQL database is recommended for production deployments.

Step 4. Deploy Apache DolphinScheduler

Install DolphinScheduler using Helm.

helm install dolphinscheduler dolphinscheduler/dolphinscheduler \
  --namespace dolphinscheduler \
  --values values.yaml \
  --version 3.2.0

It is recommended to specify an explicit chart version to ensure deployment consistency.

Step 5. Verify the Deployment

After installation, verify that all components are running correctly.

Check the Pod status:

kubectl get pods -n dolphinscheduler

Check the services:

kubectl get svc -n dolphinscheduler

Check the Persistent Volume Claims:

kubectl get pvc -n dolphinscheduler

If all Pods are in the Running state and the PVC has been successfully bound, your Apache DolphinScheduler deployment is ready for Spark integration.

Spark Integration

Once Apache DolphinScheduler has been successfully deployed, the next step is to integrate Apache Spark so that Spark applications can be scheduled and executed directly from DolphinScheduler.

Since the shared storage has already been mounted at /opt/soft, Spark only needs to be installed once. All Worker Pods can then access the same binaries through the shared volume.

Step 1. Download the Spark Binary Package

In this example, we'll use Apache Spark 3.2.1.

wget https://archive.apache.org/dist/spark/spark-3.2.1/spark-3.2.1-bin-hadoop2.7.tgz

Step 2. Copy Spark to a Worker Pod

Because the shared storage is mounted under /opt/soft, you only need to copy the Spark package into one Worker Pod. The extracted files will automatically become available to all Workers through the shared volume.

Copy the Spark package:

kubectl cp spark-3.2.1-bin-hadoop2.7.tgz dolphinscheduler-worker-0:/opt/soft -n dolphinscheduler

kubectl exec -it dolphinscheduler-worker-0 -n dolphinscheduler bash

cd /opt/soft

tar zxf spark-3.2.1-bin-hadoop2.7.tgz

rm -f spark-3.2.1-bin-hadoop2.7.tgz

ln -s spark-3.2.1-bin-hadoop2.7 spark

Verify that Spark has been installed successfully:

$SPARK_HOME/bin/spark-submit --version

If the Spark version information is displayed correctly, the installation has completed successfully.

Step 3. Configure Hadoop (Optional for Spark on YARN)

If you plan to run Spark jobs in YARN mode, Hadoop also needs to be installed and configured.

Download the Hadoop binary package:

wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz

Copy it into the Worker Pod:

kubectl cp hadoop-3.3.1.tar.gz dolphinscheduler-worker-0:/opt/soft -n dolphinscheduler

Extract and configure Hadoop:

kubectl exec -it dolphinscheduler-worker-0 -n dolphinscheduler bash

cd /opt/soft

tar zxf hadoop-3.3.1.tar.gz

rm -f hadoop-3.3.1.tar.gz

ln -s hadoop-3.3.1 hadoop

Set the Hadoop environment variables (these have already been configured in values.yaml):

export HADOOP_HOME=/opt/soft/hadoop

export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop

After the symbolic link is created and the environment variables are configured, DolphinScheduler Workers can locate the Hadoop installation automatically when submitting Spark jobs to YARN.

Verify Spark Jobs

Once Spark has been installed, it's recommended to verify the environment before running production workloads.

Method 1. Verify with a Shell Task

The quickest way to validate your Spark environment is by creating a Shell Task in the DolphinScheduler Web UI.

Run the following command:

$SPARK_HOME/bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  $SPARK_HOME/examples/jars/spark-examples_2.12-3.2.1.jar

After the task finishes, review the execution logs.

If you see output similar to:

Pi is roughly 3.14...

your Spark runtime has been configured successfully.

Method 2. Verify with a Spark Task

Upload the Example JAR

Before creating a Spark task, upload the example JAR through the DolphinScheduler Resource Center.

Upload:

spark-examples_2.12-3.2.1.jar

Create a Spark Task

Configure the Spark task with the following settings.

Parameter	Value
Program Type	Java
Main Class	`org.apache.spark.examples.SparkPi`
Main Program Package	`spark-examples_2.12-3.2.1.jar`
Deploy Mode	`local`
Master	`local[*]`

Execute the workflow and confirm that the task completes successfully.

Method 3. Verify with a Spark SQL Task

You can also validate the Spark SQL engine by creating a Spark SQL task.

Use the following SQL statements:

-- Create a test table
CREATE TABLE IF NOT EXISTS test_table (
  id INT,
  name STRING
) USING parquet;

-- Insert sample data
INSERT INTO test_table VALUES
(1, 'test'),
(2, 'dolphinscheduler');

-- Query the data
SELECT * FROM test_table;

Configure the task using these parameters:

Parameter	Value
Program Type	SQL
Deploy Mode	`local`
Master	`local[*]`

If the SQL statements execute successfully and return the expected results, the Spark SQL environment has been configured correctly.

Supported Spark Deployment Modes

According to the current Apache DolphinScheduler implementation, Kubernetes deployments support different Spark execution modes to varying degrees.

Deployment Mode	Support Status	Notes
Spark Local (client)	Supported (via shared storage)	Requires Spark binaries to be installed in the shared volume
Spark on YARN (cluster)	Supported (with additional configuration)	Requires a properly configured Hadoop environment
Spark Standalone (cluster)	Not Supported	Currently unavailable
Spark on Kubernetes (cluster)	Not Supported	Native Kubernetes deployment mode has not yet been implemented

Although native Spark on Kubernetes is not currently available, the shared-storage approach provides an effective solution for running Spark Local and Spark on YARN workloads within a Kubernetes-based DolphinScheduler deployment.

Production-Ready Configuration

While manually copying Spark binaries into the shared volume is sufficient for evaluation and testing, production deployments typically package Spark directly into the Worker image. This approach simplifies operations, shortens deployment time, and guarantees a consistent runtime environment across every Worker Pod.

Build a Custom Worker Image

Create a custom Worker image with Spark pre-installed.

FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.0

# Install Apache Spark
RUN wget https://archive.apache.org/dist/spark/spark-3.2.1/spark-3.2.1-bin-hadoop2.7.tgz && \
    tar zxf spark-3.2.1-bin-hadoop2.7.tgz -C /opt/soft && \
    rm -f spark-3.2.1-bin-hadoop2.7.tgz && \
    ln -s /opt/soft/spark-3.2.1-bin-hadoop2.7 /opt/soft/spark

# Configure environment variables
ENV SPARK_HOME=/opt/soft/spark

Build and push the image to your container registry.

docker build -t your-registry/dolphinscheduler-worker:spark .
docker push your-registry/dolphinscheduler-worker:spark

Then update your values.yaml file to use the custom Worker image.

worker:
  image:
    repository: your-registry/dolphinscheduler-worker
    tag: spark

With this approach, every newly created Worker Pod comes with Spark pre-installed, eliminating manual setup after deployment.

Configure Resource Requests and Limits

Proper resource allocation is essential for maintaining stable task execution, especially when multiple Spark jobs are running concurrently.

Configure CPU and memory limits for Worker Pods as follows:

worker:
  resources:
    limits:
      memory: "8Gi"
      cpu: "4"
    requests:
      memory: "4Gi"
      cpu: "2"

In production environments, it's recommended to configure resource requests and limits based on your workload characteristics. This helps Kubernetes schedule Pods more efficiently while preventing individual tasks from consuming excessive cluster resources.

Enable Worker Auto Scaling with KEDA

Workload volumes often fluctuate throughout the day. Instead of provisioning a fixed number of Workers, Apache DolphinScheduler supports KEDA (Kubernetes Event-driven Autoscaling), allowing Worker Pods to scale automatically based on workload demand.

Install KEDA

kubectl create namespace keda

helm install keda kedacore/keda \
  --namespace keda \
  --version v2.0.0

Enable Auto Scaling

After KEDA is installed, enable auto scaling when upgrading your DolphinScheduler deployment.

helm upgrade dolphinscheduler dolphinscheduler/dolphinscheduler \
  --namespace dolphinscheduler \
  --set worker.keda.enabled=true \
  --set worker.keda.minReplicaCount=1 \
  --set worker.keda.maxReplicaCount=10

In this example:

The Worker deployment scales down to 1 replica during idle periods.
It can automatically scale up to 10 replicas when task volume increases.

This elasticity helps reduce infrastructure costs while ensuring sufficient computing capacity during peak workloads.

Troubleshooting Common Issues

If a Spark task fails to execute, the following checks can help identify the root cause.

Check Worker Logs

Worker logs are usually the first place to investigate execution failures.

View the live logs:

kubectl logs dolphinscheduler-worker-0 \
  -n dolphinscheduler \
  -f

Search for logs related to a specific task:

kubectl logs dolphinscheduler-worker-0 \
  -n dolphinscheduler | grep "task"

Verify the Shared Storage

Ensure the shared volume has been mounted correctly and is accessible from the Worker Pods.

Check the Persistent Volume Claim:

kubectl get pvc -n dolphinscheduler

Verify the mounted directory inside the container:

kubectl exec -it dolphinscheduler-worker-0 \
  -n dolphinscheduler bash

df -h | grep opt/soft

ls -la /opt/soft

Confirm that the Spark and Hadoop directories are present and accessible.

Verify Environment Variables

Check whether the required runtime environment variables have been configured correctly.

kubectl exec -it dolphinscheduler-worker-0 \
  -n dolphinscheduler bash

echo $SPARK_HOME

echo $HADOOP_HOME

$SPARK_HOME/bin/spark-submit --version

If the environment variables return the expected paths and spark-submit reports the installed Spark version, the runtime environment has been configured successfully.

Common Issues

Spark Commands Cannot Be Found

Symptoms

The Worker reports errors such as spark-submit: command not found.

Possible Causes

SPARK_HOME is not configured correctly.
Spark was not extracted into the shared storage directory.
The symbolic link to the Spark installation is missing or incorrect.

Resolution

Verify that SPARK_HOME points to the correct installation path.
Confirm that Spark has been extracted under /opt/soft.
Recreate the symbolic link if necessary.

Shared Storage Fails to Mount

Symptoms

Worker Pods cannot access files stored in /opt/soft.

Possible Causes

The selected StorageClass does not support the ReadWriteMany (RWX) access mode.
The PersistentVolumeClaim is not bound successfully.
The storage configuration in values.yaml is incorrect.

Resolution

Verify that your storage backend supports RWX.
Check the status of the PVC and PersistentVolume.
Review the storage configuration in your Helm values file.

Spark on YARN Jobs Fail

Symptoms

Spark applications fail during submission or cannot connect to the YARN cluster.

Possible Causes

Hadoop is not installed correctly.
HADOOP_CONF_DIR points to an incorrect location.
Network connectivity to the YARN cluster is unavailable.

Resolution

Verify the Hadoop installation.
Confirm that HADOOP_CONF_DIR contains the correct configuration files.
Check connectivity between the Worker Pods and the YARN cluster.

Best Practices for Production Deployments

The following recommendations can help improve reliability, scalability, and operational efficiency in production environments.

Use High-Performance Shared Storage

Since Spark binaries and shared resources are accessed by all Worker Pods, choose a high-performance storage backend—such as SSD-backed network storage—to minimize startup latency and improve overall job performance.

Define Appropriate Resource Quotas

Configure CPU and memory requests and limits according to your workload profile. Proper resource management improves cluster utilization and reduces resource contention between concurrent jobs.

Implement Monitoring and Alerting

Monitor both the Kubernetes infrastructure and DolphinScheduler workloads. Collect metrics for Pod health, resource consumption, task execution status, and workflow success rates, and configure alerts for abnormal conditions.

Deploy for High Availability

Run multiple replicas for critical components to eliminate single points of failure. In production, it's generally recommended to deploy at least two Master replicas and two Worker replicas to improve service availability.

Establish a Backup Strategy

Regularly back up the DolphinScheduler metadata database as well as workflow resources and configuration files. A well-defined backup and recovery strategy helps minimize downtime during unexpected failures.

Notes

This guide is based on Apache DolphinScheduler 3.2.0. Configuration options and supported features may vary across releases.

At the time of writing, native Spark on Kubernetes (cluster mode) is not yet supported by DolphinScheduler. If your workload depends on this capability, keep an eye on future project releases for updates.

Finally, remember to tailor the deployment to your own infrastructure. In particular, review environment-specific settings such as the storage class, external database configuration, networking, and cluster resource allocation before moving into production.

Apache DolphinScheduler Zombie Tasks Explained: Safe Recovery Strategies Across Versions

Chen Debra — Fri, 17 Jul 2026 06:37:35 +0000

Overview

Have you ever encountered a task in Apache DolphinScheduler that remains stuck in the Running state, even though the underlying process has already terminated?

This "zombie task" scenario creates a mismatch between the UI and the actual execution state, preventing downstream tasks and workflows from progressing normally.

The good news is that the solution depends on your DolphinScheduler version. In this guide, we'll explain why zombie tasks occur and walk through the recommended cleanup methods for both legacy and modern releases.

Why Do Zombie Tasks Occur?

A task that remains in the Running state usually results from one of the following situations:

Database latency

Delayed database writes can prevent task status updates from being persisted, causing the UI to display an outdated execution state.

Worker node failure

A Worker process unexpectedly stops, but the Master has not yet detected the failure.

Network instability

ZooKeeper heartbeat timeouts trigger node removal events even though the task process may still be running.

Missing task instance

The logs indicate that the task instance is null, while the task status remains Running.

Solution for Legacy Versions (Earlier than 1.2.1)

Versions prior to Apache DolphinScheduler 1.2.1 require manual cleanup.

Step 1. Clear the ZooKeeper Task Queue

First, remove the stuck task from the ZooKeeper task queue.

# Clear the ZooKeeper task queue
delete /dolphinscheduler/task_queue

This prevents the stale task from continuously blocking the scheduling queue.

Step 2. Update the Task Status in the Database

Modify the task instance status directly in the database.

-- Change the stuck task status to FAILURE (state = 6)
UPDATE t_ds_task_instance
SET state = 6
WHERE state = 1
AND task_instance_id = <STUCK_TASK_ID>;

State definitions:

1 — RUNNING_EXECUTION
6 — FAILURE

Step 3. Resume the Workflow from the Failed Node

In the DolphinScheduler UI:

Locate the corresponding workflow instance.
Click Recover from Failure.
The workflow resumes execution from the failed task instead of starting over.

Solution for Modern Versions (1.2.1 and Later)

Starting with Apache DolphinScheduler 1.2.1, the platform introduced an automatic fault-tolerance mechanism, eliminating the need for manual cleanup in most scenarios.

Automatic Fault-Tolerance Mechanism

The new fault-tolerance framework is built on ZooKeeper's Watcher mechanism.

Worker Failover Workflow

When a Worker node receives a remove event, the Master performs the following logic:

Handle only task instances, not workflow instances.
Compare the task instance start time with the Worker service startup time.
If the task started after the Worker restarted, skip failover.
Otherwise, mark the task as NEED_FAULT_TOLERANCE.
The Master Scheduler thread automatically resubmits the task for execution.

This mechanism significantly improves cluster resilience during Worker failures and transient network issues.

Task State Machine

Modern versions also introduce a state machine to manage the complete task lifecycle, including pause and kill operations.

// Task kill event handler
public void onKilledEvent(...) {
    releaseTaskInstanceResourcesIfNeeded(taskExecutionRunnable);
    persistentTaskInstanceKilledEventToDB(taskExecutionRunnable, taskInstanceKillEvent);
    taskExecutionRunnable.getWorkflowExecutionGraph()
        .markTaskExecutionRunnableChainKill(taskExecutionRunnable);
    publishWorkflowInstanceTopologyLogicalTransitionEvent(taskExecutionRunnable);
}

The state machine ensures task transitions remain consistent while automatically releasing resources and updating workflow topology.

Feature Comparison

Feature	Before 1.2.1	1.2.1 and Later
Fault tolerance	Manual recovery	Automatic failover
ZooKeeper queue cleanup	Manual	Automatic
Task state updates	Database modification required	Managed by state machine
Worker failure recovery	Manual intervention	Automatic resubmission
Network jitter handling	Service restart required	Automatic detection and failover

Best Practices for Safe Cleanup

1. Verify the Task Is Actually Dead

Before performing any cleanup, confirm that the task process has indeed terminated.

# Check the Worker process
jps | grep WorkerServer

# Check the task process
ps -ef | grep <task_keyword>

# Review task logs
tail -f /path/to/task.log

2. Use the UI Whenever Possible

For newer versions of DolphinScheduler, always prefer built-in UI operations before modifying the database.

Recommended actions include:

Click Stop to terminate the task.
Use Recover from Failure to restart execution.

3. Be Careful with Database Operations

If direct database updates are unavoidable:

Back up the database first.
Verify the task instance ID carefully.
Update only the task status field.
Never delete task records directly.
Verify the workflow state after the operation.

4. Exercise Caution When Cleaning ZooKeeper

If ZooKeeper cleanup is required:

Confirm the target path is correct:

/dolphinscheduler/task_queue

Use a trusted ZooKeeper client.
Avoid deleting unrelated coordination nodes.

5. Monitor and Prevent Future Issues

To minimize zombie task occurrences in production:

Monitor database latency.
Configure appropriate ZooKeeper session timeout values.
Regularly check Master and Worker health.
Deploy monitoring scripts to automatically restart failed services.

Final Thoughts

The correct approach to cleaning up tasks stuck in the Running state depends largely on your DolphinScheduler version.

For versions earlier than 1.2.1, manual cleanup is required by clearing the ZooKeeper task queue, updating the task state in the database, and recovering the workflow from the failed node.

For version 1.2.1 and later, the built-in fault-tolerance mechanism automatically detects and recovers most failures, dramatically reducing operational effort.

If you're still running an older release, upgrading to the latest version is strongly recommended. Modern DolphinScheduler versions provide a more resilient scheduling architecture, automatic failover, and significantly improved cluster reliability.

Notes

This article is based on the official Apache DolphinScheduler FAQ and architecture documentation. The task state transition logic can be found in the AbstractTaskStateAction.java implementation.

For production environments, always validate the recovery procedure in a testing environment before applying it to live workloads.

🚀 Stop running incremental syncs manually! Automate daily data pipelines with Apache SeaTunnel + Apache DolphinScheduler for reliable scheduling and effortless data integration. ⚡

Chen Debra — Fri, 17 Jul 2026 06:32:10 +0000

Chen Debra

Jul 17

Stop Running Data Syncs Manually: Automate Incremental Pipelines with Apache SeaTunnel & DolphinScheduler

#database #datascience #apachedolphinscheduler #opensource

4 min read

Stop Running Data Syncs Manually: Automate Incremental Pipelines with Apache SeaTunnel & DolphinScheduler

Chen Debra — Fri, 17 Jul 2026 06:28:13 +0000

Still spending time manually synchronizing new data every day? Combine Apache SeaTunnel with Apache DolphinScheduler to build a fully automated incremental data pipeline that runs on schedule—so you can focus on building applications instead of maintaining scripts.

Is this a challenge your team faces?

Your production database grows every day, and yesterday's newly generated data needs to be synchronized to another database or data warehouse for reporting, analytics, or downstream business applications.

Running the synchronization manually every day? That's tedious—and it's easy to forget.

Writing cron jobs? They work, but they quickly become difficult to maintain as your workloads grow.

Fortunately, there's a much better approach.

In this tutorial, we'll show you how to use Apache SeaTunnel as your data integration engine and Apache DolphinScheduler as your workflow orchestration platform. Together, they create a reliable, fully automated solution for daily incremental data synchronization.

Why Combine SeaTunnel and DolphinScheduler?

Simply put, this open-source combination automates recurring data synchronization jobs.

Imagine a common enterprise scenario.

Your business database is continuously receiving new records. Every day, those new records need to be synchronized to another database or data warehouse for dashboards, analytics, or downstream services.

There are two ways to solve this problem.

Using SeaTunnel alone

SeaTunnel can easily perform both full and incremental data synchronization. However, you'll still need to execute the job manually every day or rely on external scheduling tools such as crontab.

Adding DolphinScheduler

Everything becomes much simpler.

Configure the workflow once, and DolphinScheduler automatically triggers your SeaTunnel job on schedule every day. It also provides centralized monitoring, execution history, retry mechanisms, and alert notifications whenever a task fails.

The biggest value of this architecture is straightforward:

It transforms repetitive data synchronization into an automated, visualized, and manageable workflow, freeing engineers from repetitive operational work.

In this article, we'll walk through a practical example that synchronizes yesterday's data from a MySQL table into another table automatically every day.

Prerequisites

Before building the automation pipeline, let's prepare the environment.

Environment

The following software versions are used in this example:

Apache DolphinScheduler 3.4.0 (Workflow Scheduler)
Apache SeaTunnel 2.3.12 (Data Integration Engine)
MySQL 5.7 (Source and Target Database)

Prepare Test Data

First, create a simple table in MySQL and insert a few records to simulate daily incremental data.

CREATE TABLE `paramtest` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `rq` date DEFAULT NULL,
  PRIMARY KEY (`id`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

INSERT INTO `paramtest` VALUES (1, '张三', '2026-02-06');
INSERT INTO `paramtest` VALUES (2, '李四', '2026-02-05');

Our objective is simple:

Synchronize all records whose rq equals yesterday's date automatically every day.

Configuring Apache DolphinScheduler

Step 1: Install the SeaTunnel Plugin

First, DolphinScheduler needs to recognize SeaTunnel as one of its task types.

Open the following configuration file under your DolphinScheduler installation directory:

vi conf/plugins_configdolphinscheduler-task-seatunnel

Then install the plugin:

bash ./bin/install-plugins.sh 3.4.0

Step 2: Configure the SeaTunnel Environment Variable

Next, tell DolphinScheduler where SeaTunnel is installed.

If you're using the Standalone Server deployment mode, configuring the environment variable is required so that DolphinScheduler can locate the SeaTunnel installation correctly.

# Required environment variable

# Add the following to your system environment
# (for example, /etc/profile)

export SEATUNNEL_HOME=/your/path/to/apache-seatunnel-2.3.12

# Reload the environment
source /etc/profile

# The following two files were also configured in this example.
# Depending on your deployment, they may or may not be required.

# ${SEATUNNEL_HOME}/bin/env/dolphinscheduler_env.sh
export SEATUNNEL_HOME={SEATUNNEL_HOME:/your/path/to/apache-seatunnel-2.3.12}

# ${SEATUNNEL_HOME}/standalone-server/conf/dolphinscheduler_env.sh
export SEATUNNEL_HOME={SEATUNNEL_HOME:/your/path/to/apache-seatunnel-2.3.12}

After completing the configuration, restart the DolphinScheduler services.

Step 3: Create the Workflow

Now it's time to create the scheduled workflow.

In the DolphinScheduler Web UI:

Navigate to Data Integration → Create SeaTunnel Task.
Configure the following parameters.

Startup Script

seatunnel.sh

Command Options

This is where the dynamic date parameter comes into play.

-i date=$(date -d "1 day ago" +"%Y-%m-%d")

Every time the workflow executes, this Bash command automatically calculates yesterday's date and passes it into the workflow as the variable date.

Deployment Mode

Select:

local

Script

Paste the following SeaTunnel configuration.

The Core Magic: SeaTunnel Configuration

The configuration below defines both the data source and the destination.

The key is the ${date} variable, which is injected dynamically by DolphinScheduler at runtime so that only yesterday's records are queried.

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Jdbc {
    url = "jdbc:mysql://10.0.12.100:3306/cdc?serverTimezone=UTC"
    driver = "com.mysql.cj.jdbc.Driver"
    connection_check_timeout_sec = 100
    user = "root"
    password = "root"
    query = "select * from paramtest where rq = '${date}'"
  }
}

sink {
  Jdbc {
    url = "jdbc:mysql://10.0.12.100:3306/cdc?serverTimezone=UTC"
    driver = "com.mysql.cj.jdbc.Driver"
    user = "root"
    password = "root"
    generate_sink_sql = true
    database = cdc
    table = "paramtest2"
    primary_keys = ["id"]
  }
}

After saving the task:

Publish the workflow.
Configure a schedule (for example, every day at 1:00 AM).
Enable the schedule.

Once completed, the entire synchronization process becomes fully automated.

Monitor and Manage Your Workflows

After the workflow is online, DolphinScheduler takes care of everything automatically.

Open the Workflow Instance page to view the execution history of every scheduled run.

You can easily monitor:

Execution Status — Running, Successful, or Failed at a glance.
Execution Logs — If a task fails, simply open the task instance, right-click it, and select View Log. You'll quickly identify whether the problem is related to networking, SQL, or configuration, making troubleshooting significantly easier.

Final Thoughts

Data synchronization is part of every data engineer's daily work—but it doesn't have to be repetitive.

By combining Apache SeaTunnel with Apache DolphinScheduler, you can transform manual, error-prone synchronization tasks into reliable, fully automated workflows with built-in scheduling, monitoring, retries, and centralized management.

The result is fewer operational errors, higher productivity, and more time to focus on what really matters—building better data platforms and delivering business value.

If your team is still running incremental synchronization jobs manually, now is the perfect time to modernize your workflow with this powerful open-source combination.

🚀 Apache DolphinScheduler June Update is here! Stronger failover, safer logs, S3 improvements, Helm fixes, and enhanced production stability. 💪🔒☁️⚡ Thanks to every contributor! ❤️ #ApacheDolphinScheduler #OpenSource #DataEngineering #DevOps #CloudNative

Chen Debra — Thu, 09 Jul 2026 09:16:16 +0000

Chen Debra

Jul 9

Apache DolphinScheduler June Community Update: Strengthening Stability Across the Entire Workflow Operations Lifecycle

#apachedolphinscheduler #opensource #programming #datascience

7 min read

Apache DolphinScheduler June Community Update: Strengthening Stability Across the Entire Workflow Operations Lifecycle

Chen Debra — Thu, 09 Jul 2026 09:14:37 +0000

Hi, Community!

Our June community report is here! Instead of focusing on major feature releases, the community spent the month strengthening the platform's foundation and production readiness.

Throughout June, contributors concentrated on improving operational stability by finalizing the 3.4.2 maintenance release, enhancing task failover capabilities, fixing high-concurrency edge cases, introducing sensitive data masking for logs, and addressing numerous production issues across deployment, storage, CI, and engineering workflows.

In this report, we'll walk you through the most important production-ready improvements delivered in June, provide upgrade recommendations for production environments, and recognize our top community contributors. Let's dive in! ✨

Project Highlights

Reporting Period: June 1, 2026 – July 1, 2026

Merged Pull Requests: 22
Contributors: 9
Code Changes: +630 / -473
Major Components: Master, API, Storage, Helm, UI, Documentation & Project Governance

June was primarily a maintenance and platform hardening release cycle. Alongside the Apache DolphinScheduler 3.4.2 release and follow-up fixes, the community delivered meaningful improvements in concurrency safety, log security, Helm deployment, S3 storage handling, and documentation quality.

This update is especially valuable for:

End users and platform owners who want to quickly understand whether June introduced important stability or usability improvements.
Platform engineers and DevOps teams who should pay close attention to failover behavior, Helm deployment, S3 storage improvements, and logging security.
Contributors and developers interested in seeing how the community continues investing in long-term platform reliability and technical governance.

Six Improvements That Matter Most to Users

1. Apache DolphinScheduler 3.4.2 Is Fully Released and Maintained

June included both the official 3.4.2 release and follow-up documentation corrections, representing a typical maintenance release cycle focused on polishing production quality.

Related PRs: #18317, #18319

2. Safer Worker Logs with Password Masking

Worker logs now prevent plaintext passwords from being written to log files.

This improvement is particularly important in enterprise and multi-tenant environments, where credential exposure through logs can become a far greater security risk than ordinary software bugs.

Related PR: #18333

3. Improved Helm Deployment Experience

Several deployment-related improvements landed in June, including fixes for duplicate ConfigMap labels and an updated MySQL Helm chart version.

These changes directly improve deployment reliability and long-term maintainability for Kubernetes users.

Related PRs: #18341, #18336

4. Better Concurrency Safety

A key stability improvement replaces HashMap with ConcurrentHashMap, eliminating potential ConcurrentModificationException issues under high-concurrency workloads.

Although these failures were relatively rare, they could have significant production impact when they occurred.

Related PR: #18331

5. Smarter Task Failover

A new failover enhancement allows Apache DolphinScheduler to automatically terminate external applications when task failover occurs.

This is particularly valuable for Yarn and Kubernetes deployments, preventing orphaned applications from continuing to consume cluster resources after failover.

Related PR: #18353

6. S3 Resource Listings No Longer Stop at 1,000 Objects

Users relying on Amazon S3-compatible storage will no longer encounter incomplete resource listings once object counts exceed 1,000.

This fixes a long-standing usability issue that directly affected storage visibility and operational correctness.

Related PR: #18381

Upgrade Recommendations

If you're running Apache DolphinScheduler in production, we recommend validating this release from two perspectives: maintenance updates and platform stability improvements.

Priority validation areas include:

Log masking to ensure sensitive credentials are never persisted.
High-concurrency workload stability.
S3 resource visibility.
Helm deployment and upgrade workflows.

We also recommend verifying release documentation and upgrade guides, especially if your organization maintains internal deployment documentation or embedded platform references.

Before incorporating June's updates, we recommend keeping your development environment synchronized using the following commands:

git fetch origin dev
git checkout dev
git pull --rebase

June Contributors

Based on Git author names, nine community members contributed code during June:

eye-gu
Jarek Potiuk
luxiaolong
njnu-seafish
suyc
Victor Laborie
Wenjun Ruan
xiangzihao
Yanjun Qiu

Technical Deep Dive

1. Concurrency Safety: Fixing the Root Cause Instead of the Symptoms

Representative PR: #18331

The UserGroupInformationFactory cache has been upgraded from HashMap to ConcurrentHashMap, accompanied by dedicated concurrency test cases.

The real challenge with concurrency issues isn't that they happen—it's that they occur only under specific high-load conditions and are extremely difficult to reproduce.

When Hive authentication and Kerberos renewal threads accessed the same cache simultaneously, a standard HashMap could trigger ConcurrentModificationException during concurrent iteration and modification.

Instead of applying a temporary synchronization workaround, the community addressed the underlying design by replacing the shared data structure with a thread-safe implementation and introducing dedicated tests that simulate concurrent forEach and logout/login operations.

This fundamentally eliminates the issue rather than masking its symptoms.

For production environments using Hive, Kerberos authentication, or long-lived sessions, this significantly reduces intermittent failures that disappear after retries but remain difficult to diagnose.

Recommended validation focuses on Hive workloads, Kerberos renewal scenarios, and high-concurrency datasource access.

2. Smarter Failover That Cleans Up External Applications

Representative PR: #18353

June introduces a new configuration option, kill-application-when-task-failover, integrated into MasterConfig, task execution logic, and multiple deployment configurations.

Historically, failover primarily ensured scheduling continuity.

However, for external execution engines such as Yarn and Kubernetes, the larger operational risk often comes from orphaned applications that continue consuming resources even after the scheduler has recovered.

This enhancement expands failover beyond scheduler metadata management to include cleanup of execution environments.

The implementation spans core Master logic as well as application.yaml, Docker Compose configurations, Kubernetes Helm values, and documentation, demonstrating that this is a production-ready capability rather than an experimental feature.

For organizations running Yarn or Kubernetes clusters, this provides a unified mechanism for preventing resource leakage after failover events.

Recommended validation includes Master failover testing, interrupted task execution, and verification that legitimate applications remain untouched while orphaned applications are correctly terminated.

3. S3 Pagination Support That Scales Beyond Development Environments

Representative PR: #18381

S3StorageOperator.listStorageEntity() has been enhanced to iterate through paginated results using ContinuationToken instead of relying on a single ListObjectsV2 request.

This addresses a classic production issue.

Development environments rarely exceed 1,000 objects, making the limitation difficult to detect. Once production storage grows beyond that threshold, however, users encounter missing resources even though the underlying files still exist.

Rather than implementing a workaround at the application layer, the community fixed pagination directly within the storage plugin.

Listing now continues until isTruncated() returns false, ensuring every object is returned.

As a result, resource centers, file management interfaces, and directory browsers all benefit automatically.

For organizations using large S3 buckets or treating DolphinScheduler as a centralized resource management platform, this restores correctness—not just convenience.

Recommended validation includes directories containing more than 1,000 objects, mixed folder structures, and confirmation that UI and API results match the actual contents of the storage bucket.

4. Small Change, Major Security Improvement: Log Desensitization

Representative PR: #18333

From a code perspective, this pull request modifies only a small portion of PasswordUtils.

Its impact, however, is far greater than the size of the code change suggests.

Security improvements should never be measured by the number of modified lines.

Plaintext credentials appearing in logs don't usually cause immediate system failures. Instead, they silently propagate through centralized logging platforms, troubleshooting systems, backups, and collaborative workflows, dramatically increasing the attack surface.

This update reflects the community's continued investment in secure-by-default behavior.

Many real-world security incidents originate not from broken authentication systems, but from sensitive information accidentally exposed through ordinary application logs.

For organizations already aggregating Worker logs into ELK, Alibaba Cloud SLS, Amazon CloudWatch, or similar platforms, this enhancement provides even greater value because centralized logging significantly amplifies the potential impact of leaked credentials.

When upgrading, verify that sensitive information is properly masked while ensuring logs still retain sufficient detail for effective troubleshooting.

Development Focus

Looking across all 22 merged pull requests, June represents a classic platform hardening cycle focused on resolving numerous high-frequency production issues that affect reliability over time.

Key investment areas included:

Release Management & Project Governance: #18310, #18317, #18319, #18363, #18374, #18375
Runtime Stability: #18331, #18347, #18351, #18352, #18353
Cloud-Native Deployment: #18336, #18341, #18355, #18360
Security & Compliance: #18310, #18333, #18363
Storage & Ecosystem: #18381
Documentation & User Experience: #18321, #18323, #18327, #18308

If we had to summarize Apache DolphinScheduler's June development in a single sentence, it would be this:

June wasn't about shipping major new features—it was about making Apache DolphinScheduler more reliable, more secure, and better prepared for production by strengthening the operational details that matter most.

Appendix A: Merged Pull Requests in June 2026

2026-06-02 #18310 Add AGENTS.md + SECURITY.md to make the security model discoverable
2026-06-04 #18317 Release 3.4.2
2026-06-04 #18319 Hotfix 3.4.2 doc error
2026-06-05 #18321 Update the parameter priority explanation in the docs
2026-06-05 #18323 Add a note to the parameter priority documentation specifying and fix some issues
2026-06-07 #18316 Remove dead code after t_ds_relation_user_alertgroup table dropped
2026-06-10 #18331 Replace HashMap with ConcurrentHashMap in UserGroupInformationFactory
2026-06-12 #18341 Fix duplicate app.kubernetes.io/name label on ConfigMap
2026-06-12 #18333 Remove plaintext passwords from the logs
2026-06-12 #18336 Update mysql helm chart version
2026-06-15 #18347 countTaskInstanceStateByProjectCodes uses submit_time filtering
2026-06-16 #18327 Fix and enrich the documentation for parameter priority
2026-06-16 #18355 Change the description field to optional
2026-06-17 #18353 Add kill-application-when-task-failover logic
2026-06-17 #18352 Fix rerun workflow instance should follow the specified workerGroup
2026-06-18 #18360 Change the description field to optional in k8s config
2026-06-22 #18351 Fix forceTaskSuccess cannot reset the last unsuccessful workflow instance to success
2026-06-22 #18363 Fix sonar token leak
2026-06-23 #18374 Remove sonar check
2026-06-23 #18375 Fix auto labeler error
2026-06-23 #18308 Frontend correctly uses the preferred values of the associated project for task creating and workflow scheduling
2026-06-26 #18381 Fix list resources returned only 1000 records in s3 storage type

Appendix B: References

PR #18308: https://github.com/apache/dolphinscheduler/pull/18308
PR #18310: https://github.com/apache/dolphinscheduler/pull/18310
PR #18316: https://github.com/apache/dolphinscheduler/pull/18316
PR #18317: https://github.com/apache/dolphinscheduler/pull/18317
PR #18319: https://github.com/apache/dolphinscheduler/pull/18319
PR #18321: https://github.com/apache/dolphinscheduler/pull/18321
PR #18323: https://github.com/apache/dolphinscheduler/pull/18323
PR #18327: https://github.com/apache/dolphinscheduler/pull/18327
PR #18331: https://github.com/apache/dolphinscheduler/pull/18331
PR #18333: https://github.com/apache/dolphinscheduler/pull/18333
PR #18336: https://github.com/apache/dolphinscheduler/pull/18336
PR #18341: https://github.com/apache/dolphinscheduler/pull/18341
PR #18347: https://github.com/apache/dolphinscheduler/pull/18347
PR #18351: https://github.com/apache/dolphinscheduler/pull/18351
PR #18352: https://github.com/apache/dolphinscheduler/pull/18352
PR #18353: https://github.com/apache/dolphinscheduler/pull/18353
PR #18355: https://github.com/apache/dolphinscheduler/pull/18355
PR #18360: https://github.com/apache/dolphinscheduler/pull/18360
PR #18363: https://github.com/apache/dolphinscheduler/pull/18363
PR #18374: https://github.com/apache/dolphinscheduler/pull/18374
PR #18375: https://github.com/apache/dolphinscheduler/pull/18375
PR #18381: https://github.com/apache/dolphinscheduler/pull/18381

Talk to your workflows! 💬⚡ We integrated Apache DolphinScheduler CLI into Tencent Music's SuperSonic via Java SPI for AI-powered workflow orchestration. 🚀🤖 #ApacheDolphinScheduler #AI #DataEngineering #OpenSource

Chen Debra — Thu, 09 Jul 2026 07:55:20 +0000

Chen Debra

Jul 9

From Headless BI to Workflow Operations: Integrating Apache DolphinScheduler CLI with Tencent Music's SuperSonic

#bigdata #apachedolphinscheduler #cli #tencent

18 min read

From Headless BI to Workflow Operations: Integrating Apache DolphinScheduler CLI with Tencent Music's SuperSonic

Chen Debra — Thu, 09 Jul 2026 07:54:45 +0000

Project Overview

SuperSonic

SuperSonic is Tencent Music's open-source next-generation AI-powered Business Intelligence (AI + BI) platform, bringing together two complementary paradigms in a single architecture:

Chat BI, powered by Large Language Models (LLMs)
Headless BI, powered by a semantic layer

By combining conversational AI with semantic modeling, SuperSonic enables both business users and analytics engineers to interact with data more efficiently.

Key Features

Chat BI Interface
Business users can ask questions in natural language and receive instant visualized insights through charts and dashboards.
Headless BI Interface
Analytics engineers can build semantic data models by defining metrics, dimensions, tags, and their relationships, allowing the platform to understand business context rather than raw tables.
Extensible Architecture
Built on Java SPI (Service Provider Interface), SuperSonic allows developers to extend the platform through custom parsers, executors, and plugins.
Semantic Enhancement
Business terminology, schema metadata, and column values are injected into LLM prompts, significantly reducing hallucinations and improving answer accuracy.
LLM Workload Reduction
Complex SQL logic—including joins, aggregations, and business formulas—is handled by the semantic layer instead of the language model, resulting in more reliable SQL generation.

Core Architecture

Component	Responsibility
Knowledge Base	Extracts metadata from semantic models and builds dictionaries and indexes
Schema Mapper	Identifies schema entities referenced in user queries
Semantic Parser	Converts natural language into semantic queries using rules and LLMs
Semantic Corrector	Validates and refines generated semantic queries
Semantic Translator	Translates semantic queries into executable SQL
Chat Plugin	Extends capabilities through third-party integrations
Chat Memory	Stores conversation history for few-shot prompting

GitHub: https://github.com/tencentmusic/supersonic (⭐ 4.9K)

Apache DolphinScheduler

Apache DolphinScheduler is a modern open-source workflow orchestration platform under the Apache Software Foundation, purpose-built for orchestrating complex data pipelines and task dependencies.

Key Features

Easy Deployment
Supports Standalone, Cluster, Docker, and Kubernetes deployment modes.
Developer Friendly
Create and manage workflows through the Web UI, Python SDK, or Open API.
Highly Reliable
Decentralized architecture with multiple Masters and Workers, providing native horizontal scalability.
High Performance
Processes tens of millions of tasks per day while delivering significantly higher throughput than traditional workflow schedulers.
Cloud Native
Orchestrates workflows across multiple cloud environments and data centers.
Version Control
Supports version management for both workflows and task definitions.
Flexible Workflow Lifecycle Management
Pause, stop, resume, or rerun workflows at any time.
Multi-Tenant Architecture
Designed for enterprise environments with built-in tenant isolation.

GitHub: https://github.com/apache/dolphinscheduler (⭐ 14.3K)

dsctl (DolphinScheduler CLI)

dsctl is the official command-line interface for managing Apache DolphinScheduler resources.

Major Capabilities

Project Management (project)
Workflow Management (workflow)
Task Management (task)
Data Source Management (datasource)
Environment Management (environment)
User Management (user)
Monitoring & Auditing (monitor / audit)

Example Commands

dsctl project list                    # List all projects
dsctl workflow run daily-etl          # Run a workflow
dsctl workflow-instance watch 123     # Monitor a workflow instance
dsctl task-instance log 456 --raw     # View raw task logs

Why Integrate SuperSonic with DolphinScheduler?

Although DolphinScheduler already provides a powerful Web UI, many operational tasks still require multiple manual steps. By integrating dsctl into SuperSonic through its SPI extension mechanism, users can manage workflows directly through natural language conversations.

Before vs. After Integration

Scenario	Traditional Workflow	With SuperSonic + dsctl
Run a workflow	Open the DolphinScheduler Web UI → Locate the workflow → Click Run	"Run the daily-etl workflow."
Monitor an instance	Open the Web UI → Navigate to workflow instances → Find the target instance	"Monitor workflow instance 123."
View logs	Locate the task instance in the UI → Open logs	"Show the logs for task 456."
Batch operations	Repeat multiple UI operations	Execute multiple actions through a single conversation

Business Value

Improved Productivity

Reduce repetitive UI interactions and execute complex workflow operations with a single natural-language command.

Lower Learning Curve

Even non-technical users can manage workflows without understanding the DolphinScheduler interface or CLI syntax.

Unified User Experience

Users can perform both data analytics and workflow orchestration within the same conversational interface, eliminating context switching between multiple systems.

Architecture Design

The integration leverages SuperSonic's SPI (Service Provider Interface) extension mechanism, enabling seamless integration with DolphinScheduler CLI without modifying the framework's core code.

User Natural Language Input
            │
            ▼
WorkflowParser (ChatQueryParser SPI)
    ├── accept(): Check whether the Agent is configured with the DolphinScheduler CLI tool
    └── parse(): Invoke the LLM to translate natural language into a dsctl subcommand
                 Store the parsed result in SemanticParseInfo.properties
            │
            ▼
WorkflowExecutor (ChatQueryExecutor SPI)
    ├── accept(): Check whether queryMode == "WORKFLOW_CTL"
    └── execute(): Execute the dsctl command using ProcessBuilder
                   Inject DS_API_URL and DS_API_TOKEN as environment variables
                   Return QueryResult.textResult in Markdown format
            │
            ▼
Frontend ChatItem
    └── Extend the rendering whitelist to support the WORKFLOW_CTL query mode

Request Flow

The user submits a natural-language request through the SuperSonic Chat interface.
WorkflowParser receives the request and invokes the LLM to generate the corresponding dsctl command.
WorkflowExecutor executes the generated command by launching a dsctl process.
dsctl communicates with the Apache DolphinScheduler REST API.
The execution result is formatted as Markdown and rendered in the chat interface.

Backend Implementation

File 1: `AgentToolType.java` (Modified)

Location

chat/server/src/main/java/com/tencent/supersonic/chat/server/agent/AgentToolType.java

package com.tencent.supersonic.chat.server.agent;

import java.util.HashMap;
import java.util.Map;

public enum AgentToolType {
    DATASET("Text2SQL Dataset"),
    PLUGIN("Third-party Plugin"),
    WORK_FLOW_CTL("Workflow CLI");

    private final String title;

    AgentToolType(String title) {
        this.title = title;
    }

    public static Map<AgentToolType, String> getToolTypes() {
        Map<AgentToolType, String> map = new HashMap<>();
        map.put(DATASET, DATASET.title);
        map.put(PLUGIN, PLUGIN.title);
        map.put(WORK_FLOW_CTL, WORK_FLOW_CTL.title);
        return map;
    }
}

After registering the new tool type, the frontend automatically retrieves it through the getToolTypes() API. As a result, DolphinScheduler CLI becomes available in the Agent configuration page without requiring any frontend changes.

File 2: `WorkflowTool.java` (New)

Location

chat/server/src/main/java/com/tencent/supersonic/chat/server/agent/WorkflowTool.java

package com.tencent.supersonic.chat.server.agent;

import java.util.List;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@NoArgsConstructor
@AllArgsConstructor
public class WorkflowTool extends AgentTool {

    /** Path to the dsctl executable (default: "dsctl") */
    private String dsctlPath = "dsctl";

    /** Default DolphinScheduler project */
    private String defaultProject;

    /** DolphinScheduler API endpoint */
    private String dsApiUrl;

    /** DolphinScheduler API token */
    private String dsToken;

    /** Example questions used for semantic retrieval */
    private List<String> exampleQuestions;
}

File 3: `WorkflowParser.java` (New)

Path：

chat/server/src/main/java/com/tencent/supersonic/chat/server/parser/WorkflowParser.java

package com.tencent.supersonic.chat.server.parser;

import com.alibaba.fastjson2.JSON;
import com.alibaba.fastjson2.JSONArray;
import com.alibaba.fastjson2.JSONObject;
import com.tencent.supersonic.chat.server.agent.AgentToolType;
import com.tencent.supersonic.chat.server.agent.WorkflowTool;
import com.tencent.supersonic.chat.server.pojo.ParseContext;
import com.tencent.supersonic.common.pojo.ChatApp;
import com.tencent.supersonic.common.pojo.enums.AppModule;
import com.tencent.supersonic.common.util.ChatAppManager;
import com.tencent.supersonic.headless.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.headless.api.pojo.response.ParseResp;
import dev.langchain4j.model.chat.ChatModel;
import dev.langchain4j.model.input.Prompt;
import dev.langchain4j.model.input.PromptTemplate;
import dev.langchain4j.model.output.structured.Description;
import dev.langchain4j.provider.ModelProvider;
import dev.langchain4j.service.AiServices;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.collections4.CollectionUtils;

/**
 * Agent Management Page
 * └── Add Tool:
 *     type = WORK_FLOW_CTL
 *     dsctlPath = /usr/bin/dsctl
 *     defaultProject = etl-prod
 *         ↓ Stored in
 * Agent.toolConfig (JSON)
 *
 * WorkflowParser.accept()
 * └── Activate when
 *     agent.getTools(AgentToolType.WORK_FLOW_CTL) is not empty
 *
 * WorkflowParser.parse()
 * └── Read the WorkflowTool configuration
 * └── Use the LLM to generate a dsctl command
 * └── Write the result into parseInfo.properties
 *
 * WorkflowExecutor.execute()
 * └── Read dsctl_path + dsctl_cmd
 * └── Execute the command through Runtime.exec()
 * └── Return the execution result
 */
@Slf4j
public class WorkflowParser implements ChatQueryParser {

    public static final String QUERY_MODE = "WORKFLOW_CTL";
    public static final String APP_KEY = "WORKFLOW_PARSER";

    /** Cache the dsctl schema to avoid repeated execution. */
    private static volatile String cachedSchema;
    private static volatile long schemaCacheTime;
    private static final long SCHEMA_CACHE_TTL_MS = 5 * 60 * 1000; // 5 minutes

    private static final String INSTRUCTION =
            """
            # Role: You are a DolphinScheduler CLI expert using dsctl.

            # Task: Convert user's natural language into dsctl commands.

            # dsctl Usage:
            {{dsctl_help}}

            # Available Commands:
            {{dsctl_commands}}

            # Command Generation Rules:
            1. Match user intent to the appropriate command
            2. For resource selectors, use discovery commands listed below
            3. Include required arguments and options
            4. Output ONLY the dsctl command

            # Intent Patterns:
            - "list/view/show" + resource → dsctl <resource> list
            - "get/detail" + resource → dsctl <resource> get <selector>
            - "create/new" + resource → dsctl <resource> create --name <name>
            - "delete/remove" + resource → dsctl <resource> delete <selector> --force
            - "run/execute" + workflow → dsctl workflow run <workflow>
            - "stop" + instance → dsctl workflow-instance stop <id>
            - "watch/monitor" + instance → dsctl workflow-instance watch <id>
            - "log" + task → dsctl task-instance log <id> --raw

            # Resource Selectors:
            - project: name or code (discover: dsctl project list)
            - workflow: name or code (discover: dsctl workflow list)
            - task: name or code (discover: dsctl task list)
            - workflow-instance: numeric id (discover: dsctl workflow-instance list)
            - task-instance: numeric id (discover: dsctl task-instance list)

            # Output Rules:
            1. Output ONLY the dsctl command
            2. If intent is unclear, output: unknown
            3. Do NOT include explanations

            # Question: {{question}}
            # Command:
            """;

    public WorkflowParser() {
        ChatAppManager.register(
                APP_KEY,
                ChatApp.builder()
                        .prompt(INSTRUCTION)
                        .name("Workflow CLI")
                        .appModule(AppModule.CHAT)
                        .description("Convert natural language into DolphinScheduler workflow commands")
                        .enable(true)
                        .build());
    }

    /** Structured output model. @Description provides field semantics to AiServices. */
    @Data
    static class DsctlCommand {

        @Description("the dsctl subcommand without 'dsctl' prefix, e.g. 'workflow run daily-etl'")
        private String command;

        @Description("brief explanation of what this command does")
        private String thought;
    }

    /** Extraction interface. AiServices automatically generates the implementation. */
    interface DsctlCommandExtractor {
        DsctlCommand extractCommand(String text);
    }

    @Override
    public boolean accept(ParseContext parseContext) {
        // Activate only when the Agent has a Workflow CLI tool configured.
        List<String> tools = parseContext.getAgent().getTools(AgentToolType.WORK_FLOW_CTL);
        return !CollectionUtils.isEmpty(tools);
    }

    @Override
    public void parse(ParseContext parseContext) {

        // Load the first WorkflowTool configured for the Agent.
        List<String> toolJsonList =
                parseContext.getAgent().getTools(AgentToolType.WORK_FLOW_CTL);
        WorkflowTool workflowTool =
                JSONObject.parseObject(toolJsonList.getFirst(), WorkflowTool.class);

        // Load the ChatApp configuration, including the LLM binding and prompt template.
        ChatApp chatApp =
                parseContext.getAgent().getChatAppConfig().get(APP_KEY);

        if (Objects.isNull(chatApp) || !chatApp.isEnable()) {
            log.warn("WorkflowParser ChatApp [{}] is not enabled, skip.", APP_KEY);
            return;
        }

        // Build the prompt by replacing template variables.
        Map<String, Object> variables = new HashMap<>();

        // 1. Execute "dsctl --help" to retrieve CLI usage information.
        String dsctlHelp =
                executeDsctlCommand(workflowTool.getDsctlPath(), workflowTool, "--help");

        // 2. Execute "dsctl schema" to retrieve and format the command list (cached).
        String dsctlSchema = getCachedOrExecuteSchema(workflowTool);
        String formattedCommands = formatCommandsFromSchema(dsctlSchema);

        variables.put("dsctl_help", dsctlHelp);
        variables.put("dsctl_commands", formattedCommands);
        variables.put("question", parseContext.getRequest().getQueryText());

        Prompt prompt =
                PromptTemplate.from(chatApp.getPrompt()).apply(variables);

        // Invoke the LLM using AiServices for structured output.
        ChatModel model =
                ModelProvider.getChatModel(chatApp.getChatModelConfig());

        DsctlCommandExtractor extractor =
                AiServices.create(DsctlCommandExtractor.class, model);

        DsctlCommand cmd =
                extractor.extractCommand(prompt.toUserMessage().singleText());

        log.info(
                "WorkflowParser input=[{}] -> command=[{}] thought=[{}]",
                parseContext.getRequest().getQueryText(),
                cmd == null ? "null" : cmd.getCommand(),
                cmd == null ? "null" : cmd.getThought());

        if (cmd == null || "unknown".equalsIgnoreCase(cmd.getCommand())) {
            // Intent not recognized. Skip this parser and continue with the remaining parsers.
            return;
        }

        // Store the parsing result in parseInfo for WorkflowExecutor.
        SemanticParseInfo parseInfo = new SemanticParseInfo();
        parseInfo.setQueryMode(QUERY_MODE);
        parseInfo.setId(1);

        Map<String, Object> props = new HashMap<>();

        // e.g. "workflow run daily-etl"
        props.put("workflow_ctl_cmd", cmd.getCommand());

        // LLM-generated explanation
        props.put("workflow_ctl_thought", cmd.getThought());

        // Path to the dsctl executable
        props.put("workflow_ctl_path", workflowTool.getDsctlPath());

        if (workflowTool.getDefaultProject() != null) {
            props.put("default_project", workflowTool.getDefaultProject());
        }

        if (workflowTool.getDsApiUrl() != null) {
            props.put("ds_api_url", workflowTool.getDsApiUrl());
        }

        if (workflowTool.getDsToken() != null) {
            props.put("ds_token", workflowTool.getDsToken());
        }

        parseInfo.setProperties(props);

        parseContext.getResponse().getSelectedParses().add(parseInfo);
        parseContext.getResponse().setState(ParseResp.ParseState.COMPLETED);
    }

    /**
     * Execute a dsctl command and return its output.
     *
     * @param dsctlPath Path to the dsctl executable
     * @param workflowTool Workflow tool configuration (including environment variables)
     * @param args Command arguments
     * @return Command output
     */
    private String executeDsctlCommand(
            String dsctlPath,
            WorkflowTool workflowTool,
            String... args) {

        try {
            String[] cmdArray = new String[args.length + 1];
            cmdArray[0] = dsctlPath;
            System.arraycopy(args, 0, cmdArray, 1, args.length);

            ProcessBuilder pb = new ProcessBuilder(cmdArray);
            pb.redirectErrorStream(true);

            // Inject environment variables.
            Map<String, String> env = pb.environment();

            if (workflowTool.getDsApiUrl() != null) {
                env.put("DS_API_URL", workflowTool.getDsApiUrl());
            }

            if (workflowTool.getDsToken() != null) {
                env.put("DS_API_TOKEN", workflowTool.getDsToken());
            }

            Process process = pb.start();

            String output;

            try (BufferedReader reader =
                    new BufferedReader(
                            new InputStreamReader(process.getInputStream()))) {

                output = reader.lines().collect(Collectors.joining("\n"));
            }

            boolean finished = process.waitFor(30, TimeUnit.SECONDS);

            if (!finished) {
                process.destroyForcibly();
                log.warn("dsctl command timed out: {}", String.join(" ", cmdArray));
                return "";
            }

            if (process.exitValue() != 0) {
                log.warn(
                        "dsctl command failed with exit code {}: {}",
                        process.exitValue(),
                        output);
            }

            return output;

        } catch (Exception e) {
            log.error("Failed to execute dsctl command: {}", e.getMessage(), e);
            return "";
        }
    }

    /**
     * Retrieve the cached schema or execute a new schema command.
     *
     * @param workflowTool Workflow tool configuration
     * @return dsctl schema JSON
     */
    private String getCachedOrExecuteSchema(WorkflowTool workflowTool) {

        long now = System.currentTimeMillis();

        if (cachedSchema != null
                && (now - schemaCacheTime) < SCHEMA_CACHE_TTL_MS) {

            log.debug("Using cached dsctl schema");
            return cachedSchema;
        }

        synchronized (WorkflowParser.class) {

            // Double-check locking.
            if (cachedSchema != null
                    && (now - schemaCacheTime) < SCHEMA_CACHE_TTL_MS) {

                return cachedSchema;
            }

            String schema =
                    executeDsctlCommand(
                            workflowTool.getDsctlPath(),
                            workflowTool,
                            "schema");

            if (schema != null && !schema.isBlank()) {
                cachedSchema = schema;
                schemaCacheTime = now;
            }

            return schema;
        }
    }

    /**
     * Format the command list from the dsctl schema JSON.
     *
     * @param schemaJson JSON output from "dsctl schema"
     * @return Formatted command list
     */
    private String formatCommandsFromSchema(String schemaJson) {

        if (schemaJson == null || schemaJson.isBlank()) {
            return getDefaultCommands();
        }

        try {

            JSONObject schema = JSON.parseObject(schemaJson);
            JSONObject data = schema.getJSONObject("data");

            if (data == null) {
                return getDefaultCommands();
            }

            JSONArray commands = data.getJSONArray("commands");

            if (commands == null) {
                return getDefaultCommands();
            }

            StringBuilder sb = new StringBuilder();

            for (int i = 0; i < commands.size(); i++) {

                JSONObject group = commands.getJSONObject(i);

                if (!"group".equals(group.getString("kind"))) {
                    continue;
                }

                String groupName = group.getString("name");
                String groupSummary = group.getString("summary");

                sb.append("\n## ").append(groupName).append("\n");
                sb.append("# ").append(groupSummary).append("\n");

                JSONArray cmdList = group.getJSONArray("commands");

                if (cmdList != null) {

                    for (int j = 0; j < cmdList.size(); j++) {

                        JSONObject cmd = cmdList.getJSONObject(j);

                        String cmdName = cmd.getString("name");
                        String summary = cmd.getString("summary");

                        // Build the command syntax.
                        StringBuilder syntax = new StringBuilder();

                        syntax.append("dsctl ")
                                .append(groupName)
                                .append(" ")
                                .append(cmdName);

                        // Append required arguments.
                        JSONArray args = cmd.getJSONArray("arguments");

                        if (args != null) {

                            for (int k = 0; k < args.size(); k++) {

                                JSONObject arg = args.getJSONObject(k);

                                if (arg.getBooleanValue("required")) {

                                    syntax.append(" <")
                                            .append(arg.getString("name"))
                                            .append(">");
                                }
                            }
                        }

                        sb.append("- ")
                                .append(syntax)
                                .append(": ")
                                .append(summary)
                                .append("\n");

                        // Append the discovery command if available.
                        if (args != null) {

                            for (int k = 0; k < args.size(); k++) {

                                JSONObject arg = args.getJSONObject(k);

                                String discovery =
                                        arg.getString("discovery_command");

                                if (discovery != null && !discovery.isEmpty()) {

                                    sb.append("  Discovery: ")
                                            .append(discovery)
                                            .append("\n");

                                    break;
                                }
                            }
                        }
                    }
                }
            }

            return sb.toString();

        } catch (Exception e) {

            log.error("Failed to parse dsctl schema: {}", e.getMessage(), e);

            return getDefaultCommands();
        }
    }
    /**
     * Returns the default command list.
     * This is used as a fallback when parsing the schema fails.
     *
     * @return Default command list
     */
    private String getDefaultCommands() {

        return """
            - dsctl project list: List projects
            - dsctl project get <project>: Get one project
            - dsctl project create --name <name>: Create a project
            - dsctl project update <project>: Update a project
            - dsctl project delete <project>: Delete a project
            - dsctl workflow list: List workflows
            - dsctl workflow get <workflow>: Get workflow details
            - dsctl workflow run <workflow>: Run a workflow
            - dsctl workflow delete <workflow>: Delete a workflow
            - dsctl workflow-instance list: List workflow instances
            - dsctl workflow-instance get <id>: Get workflow instance details
            - dsctl workflow-instance watch <id>: Monitor a workflow instance
            - dsctl workflow-instance stop <id>: Stop a workflow instance
            - dsctl workflow-instance digest <id>: View workflow instance summary
            - dsctl task list --workflow <workflow>: List tasks in a workflow
            - dsctl task get <task>: Get task details
            - dsctl task-instance list --workflow-instance <id>: List task instances
            - dsctl task-instance log <id>: View task logs
            - dsctl cluster list: List clusters
            - dsctl datasource list: List data sources
            - dsctl environment list: List environments
            - dsctl user list: List users
            - dsctl tenant list: List tenants
            - dsctl schedule list: List schedules
            - dsctl doctor: Run diagnostics
            """;
    }
}

Key Design Highlights

ChatAppManager.register() registers the prompt template during initialization, making it configurable from the Agent management UI and allowing different LLMs to be bound without changing code.
AiServices.create() automatically injects a JSON schema into the prompt and deserializes the LLM response into a strongly typed DsctlCommand object, eliminating manual output parsing.
accept() activates the parser only when the Agent is configured with a WORK_FLOW_CTL tool, enabling fine-grained, Agent-level feature control.

File 4: `WorkflowExecutor.java` (New)

Location

chat/server/src/main/java/com/tencent/supersonic/chat/server/executor/WorkflowExecutor.java

WorkflowExecutor is responsible for executing the dsctl command generated by WorkflowParser. It serves as the execution layer of the entire integration, receiving parsed commands, launching the DolphinScheduler CLI, and returning execution results to the SuperSonic chat interface.

Unlike SQL execution, which queries databases directly, WorkflowExecutor delegates workflow operations to the official dsctl CLI. This approach ensures that all existing DolphinScheduler authentication, permission management, and API interactions remain unchanged while allowing users to control workflows through natural language.

Execution Flow

SemanticParseInfo
        │
        ▼
Read workflow_ctl_cmd
        │
        ▼
WorkflowExecutor.accept()
        │
        ▼
WorkflowExecutor.execute()
        │
        ├── Build ProcessBuilder
        ├── Inject environment variables
        ├── Execute dsctl
        ├── Capture stdout/stderr
        └── Return Markdown result

The executor first retrieves the following properties generated by WorkflowParser:

workflow_ctl_path
workflow_ctl_cmd
workflow_ctl_thought
ds_api_url
ds_token

It then launches a child process using Java's ProcessBuilder to execute the corresponding dsctl command.

For example, if the user asks:

Run the daily-etl workflow.

the parser generates:

workflow run daily-etl

and the executor actually runs:

dsctl workflow run daily-etl

Environment Variable Injection

One important implementation detail is that Java child processes do not always inherit shell environment variables, especially in containerized or service environments.

To avoid configuration issues, the executor explicitly injects the required DolphinScheduler connection settings before launching the CLI:

Map<String, String> env = pb.environment();

env.put("DS_API_URL", dsApiUrl);
env.put("DS_API_TOKEN", dsToken);

This guarantees that every dsctl invocation communicates with the correct DolphinScheduler cluster regardless of how SuperSonic itself was started.

Unified Output Handling

Both standard output (stdout) and error output (stderr) are merged into a single stream:

pb.redirectErrorStream(true);

Instead of handling two separate streams, the executor captures all command output through one reader, simplifying both logging and error reporting.

The complete output is then returned to the chat interface.

Timeout Protection

Workflow-related operations such as monitoring instances may take longer than ordinary commands.

To prevent hanging processes from occupying server resources indefinitely, every CLI invocation has a timeout.

process.waitFor(60, TimeUnit.SECONDS);

If execution exceeds 60 seconds, the child process is terminated automatically.

process.destroyForcibly();

This prevents zombie processes from accumulating and protects the stability of the SuperSonic service.

Markdown Rendering

Instead of returning raw terminal output, the executor formats the response as Markdown.

For example:

**Execute the specified workflow**

```bash
$ dsctl workflow run daily-etl
Workflow submitted successfully.

Instance ID: 12345
Status: RUNNING
```

Because the frontend already supports Markdown rendering, users receive CLI output in a clean, readable format directly within the chat window.

Key Design Highlights

Uses ProcessBuilder instead of Runtime.exec() for greater flexibility and better process control.
Explicitly injects DS_API_URL and DS_API_TOKEN, avoiding dependency on shell environments.
Merges standard output and error output into a unified stream for simplified processing.
Enforces a 60-second timeout to prevent long-running or stalled processes.
Formats CLI output as Markdown for an improved conversational experience.

File 5: `spring.factories` (Modified)

Location

launchers/standalone/src/main/resources/META-INF/spring.factories

To enable automatic discovery through Java SPI, both the parser and executor must be registered in spring.factories.

Append the following entries to the existing configuration:

com.tencent.supersonic.chat.server.parser.ChatQueryParser=\
    com.tencent.supersonic.chat.server.parser.NL2PluginParser,\
    com.tencent.supersonic.chat.server.parser.NL2SQLParser,\
    com.tencent.supersonic.chat.server.parser.WorkflowParser,\
    com.tencent.supersonic.chat.server.parser.PlainTextParser

com.tencent.supersonic.chat.server.executor.ChatQueryExecutor=\
    com.tencent.supersonic.chat.server.executor.PluginExecutor,\
    com.tencent.supersonic.chat.server.executor.WorkflowExecutor,\
    com.tencent.supersonic.chat.server.executor.SqlExecutor,\
    com.tencent.supersonic.chat.server.executor.PlainTextExecutor

Registration Order Matters

The registration order is critical.

WorkflowExecutor must appear before SqlExecutor.

This is because SqlExecutor.accept() always returns true. If it is registered first, it will intercept every request before WorkflowExecutor has an opportunity to process workflow-related commands.

The correct execution pipeline is therefore:

PluginExecutor
        │
        ▼
WorkflowExecutor
        │
        ▼
SqlExecutor
        │
        ▼
PlainTextExecutor

This ordering guarantees that workflow commands are correctly routed to the CLI executor while SQL queries continue to be handled by the existing semantic query engine.

Why SPI?

One of the biggest advantages of this integration is that no modifications are required to the SuperSonic framework itself.

By implementing the ChatQueryParser and ChatQueryExecutor interfaces and registering them through Java SPI, the Workflow CLI becomes a first-class capability that integrates seamlessly with the existing architecture.

This plug-in design makes future extensions straightforward. Additional enterprise tools—such as Kubernetes CLI, Spark CLI, Flink CLI, Airflow CLI, or even custom internal command-line utilities—can be integrated using exactly the same extension pattern without changing the SuperSonic core.

In the next section, we'll cover the Frontend Implementation, including type.ts, ToolModal.tsx, ChatItem, and ExecuteItem, demonstrating how the new Workflow CLI tool is exposed in the Agent UI and how Markdown command output is rendered in the chat interface.

Frontend Implementation

The frontend changes are intentionally lightweight. Since SuperSonic already supports a flexible plugin architecture, only a few components need to be extended to expose the new Workflow CLI capability.

The implementation consists of four small modifications:

Add a new tool type.
Extend the Agent configuration dialog.
Allow the new query mode to pass through the message pipeline.
Render command output as Markdown.

No core frontend architecture needs to be modified.

File 6: `type.ts` (Modified)

Location

webapp/packages/supersonic-fe/src/pages/Agent/type.ts

First, introduce a new tool type named WORK_FLOW_CTL.

export enum AgentToolTypeEnum {
  NL2SQL_RULE = 'NL2SQL_RULE',
  NL2SQL_LLM = 'NL2SQL_LLM',
  PLUGIN = 'PLUGIN',
  DATASET = 'DATASET',
  WORK_FLOW_CTL = 'WORK_FLOW_CTL',
}

Then extend the tool definition with the configuration required by DolphinScheduler CLI.

export type AgentToolType = {
  id?: string;
  type: AgentToolTypeEnum;
  name: string;
  queryModes?: QueryModeEnum[];
  plugins?: number[];
  metricOptions?: MetricOptionType[];
  exampleQuestions?: string[];
  modelIds?: number[];

  dsctlPath?: string;
  dsApiUrl?: string;
  dsToken?: string;
  defaultProject?: string;
};

These additional fields allow administrators to configure the CLI executable, API endpoint, authentication token, and default project directly from the Agent management interface.

File 7: `ToolModal.tsx` (Modified)

Location

webapp/packages/supersonic-fe/src/pages/Agent/ToolModal.tsx

After the existing PLUGIN configuration form, add a new form section dedicated to WORK_FLOW_CTL.

{toolType === AgentToolTypeEnum.WORK_FLOW_CTL && (
  <>
    <FormItem
      name="dsApiUrl"
      label="DS API URL"
      rules={[
        {
          required: true,
          message: 'Please enter the DolphinScheduler API endpoint.',
        },
      ]}
    >
      <Input
        placeholder="Example: http://ds-host:12345"
        allowClear
      />
    </FormItem>

    <FormItem
      name="dsToken"
      label="DS API Token"
    >
      <Input.Password
        placeholder="DolphinScheduler API Token"
        allowClear
      />
    </FormItem>

    <FormItem
      name="dsctlPath"
      label="dsctl Path"
    >
      <Input
        placeholder="Path to the dsctl executable (default: dsctl)"
        allowClear
      />
    </FormItem>

    <FormItem
      name="defaultProject"
      label="Default Project"
    >
      <Input
        placeholder="Default DolphinScheduler project"
        allowClear
      />
    </FormItem>
  </>
)}

One advantage of this design is that the tool type dropdown requires no additional frontend changes.

The available tool types are loaded dynamically through the backend getToolTypes() API. Once WORK_FLOW_CTL is added to the backend enum, DolphinScheduler CLI automatically appears in the dropdown menu.

This keeps the frontend completely data-driven.

File 8: `index.tsx` (Modified)

Location

webapp/packages/chat-sdk/src/components/ChatItem/index.tsx

Next, update the message processing logic to recognize the new query mode.

Inside the updateData() function, extend the response whitelist by adding WORKFLOW_CTL.

if (
    (queryColumns && queryColumns.length > 0 && queryResults) ||
    queryMode === 'WEB_PAGE' ||
    queryMode === 'WEB_SERVICE' ||
    queryMode === 'PLAIN_TEXT' ||
    queryMode === 'WORKFLOW_CTL'
) {
    data = res.data;
    tip = '';
}

At the same time, make the component compatible with responses that return code: 1.

else if (res.code !== 200 && res.code !== 1) {
    tip = SEARCH_EXCEPTION_TIP;
}

Without these two changes, Workflow CLI responses would be treated as invalid and never reach the rendering layer.

File 9: `ExecuteItem.tsx` (Modified)

Location

webapp/packages/chat-sdk/src/components/ChatItem/ExecuteItem.tsx

Two small changes are required in this component.

Update the Message Title

Near the top of the file, modify the title prefix logic.

const titlePrefix =
    queryMode === 'PLAIN_TEXT' ||
    queryMode === 'WEB_SERVICE' ||
    queryMode === 'WORKFLOW_CTL'
        ? 'Q&A'
        : 'Data';

Workflow operations are conversational interactions rather than analytical queries, so they should be categorized as Q&A instead of Data.

Render Markdown Output

Next, add a rendering branch for the new query mode.

data?.queryMode === 'WORKFLOW_CTL' ? (
    <ReactMarkdown>
        {data?.textResult ?? ''}
    </ReactMarkdown>
) : (
    <ChatMsg
        ...
    />
)

The backend already formats command output as Markdown, making ReactMarkdown the ideal renderer.

For example, when a user asks:

Run the daily-etl workflow.

The chat interface displays:

**Execute the specified workflow**

```bash
$ dsctl workflow run daily-etl

Workflow submitted successfully.

Instance ID: 12345
Status: RUNNING
```

Compared with rendering plain text, Markdown significantly improves readability by preserving code formatting and command output structure.

Since ReactMarkdown has already been imported into the component, no additional dependencies are required.

Frontend Extension Summary

The frontend integration is intentionally minimal. Only four files require changes:

File	Purpose
`type.ts`	Register the new `WORK_FLOW_CTL` tool type and define its configuration fields.
`ToolModal.tsx`	Add a configuration form for DolphinScheduler CLI.
`index.tsx`	Allow the new query mode to pass through the message processing pipeline.
`ExecuteItem.tsx`	Render Workflow CLI responses as Markdown in the chat interface.

With these lightweight modifications, SuperSonic gains a fully conversational workflow orchestration capability while preserving its existing frontend architecture.

Usage Configuration

Before using the integration, make sure both dsctl and Apache DolphinScheduler are properly configured.

Prerequisites

1. Install dsctl

pip install -e .

2. Configure Environment Variables

export DS_API_URL=http://your-dolphinscheduler-host:12345/dolphinscheduler
export DS_API_TOKEN=your-api-token

These environment variables enable dsctl to communicate with the target DolphinScheduler cluster.

3. Verify the Installation

Run the following command to ensure everything is configured correctly:

dsctl doctor

If the diagnostic completes successfully, the CLI is ready to be integrated with SuperSonic.

Configuring an Agent

Once the backend and frontend have been deployed, the final step is to configure an Agent that can invoke the DolphinScheduler CLI.

Step 1: Open the Agent Management Page

Navigate to the Agent management page and either create a new Agent or edit an existing one.

(Screenshots omitted.)

Step 2: Add a New Tool

Open the Tools tab and click Add Tool.

(Screenshots omitted.)

Step 3: Select Workflow CLI

From the Tool Type dropdown, select Workflow CLI.

(Screenshots omitted.)

This option becomes available automatically after the backend registers the WORK_FLOW_CTL tool type.

Step 4: Configure the Connection

Fill in the required connection settings.

(Screenshots omitted.)

Field	Description	Example
DS API URL	DolphinScheduler REST API endpoint	`http://ds-host:12345/dolphinscheduler`
DS API Token	API authentication token	`your_token_here`
dsctl Path	Path to the `dsctl` executable	`/usr/local/bin/dsctl`
Default Project	Default DolphinScheduler project	`etl-prod`

Once the configuration is saved, the Agent is ready to execute workflow operations through natural language.

Step 5: Start Chatting

Users can now interact with DolphinScheduler directly from the SuperSonic chat interface.

(Screenshots omitted.)

Instead of navigating through multiple pages in the DolphinScheduler Web UI, users simply describe what they want to accomplish in plain language.

Supported Natural Language Commands

The following examples illustrate how natural-language requests are translated into dsctl commands.

Natural Language Request	Generated dsctl Command
Run a health check.	`dsctl doctor`
List all projects.	`dsctl project list`
Switch to the etl-prod project.	`dsctl use project etl-prod`
List all workflows.	`dsctl workflow list`
Run the daily-etl workflow.	`dsctl workflow run daily-etl`
Monitor workflow instance 123.	`dsctl workflow-instance watch 123`
Show the summary of workflow instance 123.	`dsctl workflow-instance digest 123`
List tasks for workflow instance 123.	`dsctl task-instance list --workflow-instance 123`
Show the raw log of task 456.	`dsctl task-instance log 456 --raw`

These examples demonstrate that users no longer need to memorize CLI syntax. Instead, they can interact with DolphinScheduler naturally while the LLM automatically translates their intent into executable commands.

Extension Summary

The entire integration is built around SuperSonic's SPI extension mechanism, making it highly modular and easy to maintain.

Extension Point	Interface	Responsibility
Intent Parsing	`ChatQueryParser`	Converts natural-language requests into `dsctl` commands
Command Execution	`ChatQueryExecutor`	Executes the generated `dsctl` command and returns the result
Tool Registration	`AgentToolType`	Registers the new Workflow CLI tool in the Agent management page
Tool Configuration	`AgentTool` subclass	Stores CLI configuration, API endpoint, authentication token, and project settings
LLM Integration	`AiServices` + `ChatAppManager`	Generates structured commands while allowing prompts and models to be configured through the UI

One of the biggest advantages of this approach is that no modifications to the SuperSonic framework itself are required.

By implementing the standard SPI interfaces and registering them in spring.factories, developers can seamlessly integrate DolphinScheduler into SuperSonic while preserving the framework's pluggable architecture.

More importantly, this pattern is not limited to DolphinScheduler. Any CLI-based system—including Kubernetes, Spark, Flink, Airflow, or internal enterprise tools—can be integrated using the same extension model, making SuperSonic a powerful conversational gateway for enterprise operations.

References

SuperSonic GitHub
Apache DolphinScheduler
dsctl GitHub
SuperSonic Documentation
Apache DolphinScheduler Documentation

How to Upgrade Apache DolphinScheduler from 1.3.6 to 3.2.2 Without Downtime

Chen Debra — Thu, 09 Jul 2026 06:52:15 +0000

Migrating a production Apache DolphinScheduler cluster across multiple major releases can be challenging, especially when upgrading from legacy versions such as v1.3.6 to the latest v3.2.2.

Because the database schema, task definition model, configuration files, and upgrade utilities have evolved significantly between releases, skipping intermediate versions is not recommended. This guide walks you through a verified upgrade path, highlights common pitfalls, and documents the fixes required during the migration process.

1. Upgrade Overview

Follow the upgrade sequence below:

v1.3.6
   ↓
v2.0.0
   ↓
v2.0.9
   ↓
v3.0.0
   ↓
v3.1.0
   ↓
v3.1.8
   ↓
v3.2.2

Download the Binary Packages

Download the binary distribution for each target version from the Apache archive.

Example:

https://archive.apache.org/dist/dolphinscheduler/3.1.8/apache-dolphinscheduler-3.1.8-bin.tar.gz

Prepare Independent Databases

Create a dedicated MySQL database for each intermediate version.

Example databases:

dolphinscheduler_2_0_0
dolphinscheduler_2_0_9
dolphinscheduler_3_0_0
dolphinscheduler_3_1_0
dolphinscheduler_3_1_8
dolphinscheduler_3_2_2

Example SQL statement:

CREATE DATABASE dolphinscheduler_3_2_2
DEFAULT CHARACTER SET utf8
DEFAULT COLLATE utf8_general_ci;

Using separate databases for every upgrade stage allows you to verify each migration independently and provides an easy rollback path if necessary.

Extract Each Release

Extract every binary package into the same working directory.

./upgrade_dolphin

After extraction:

Update the database connection configuration.
Run the corresponding upgrade script.
Verify that the upgrade completes successfully before proceeding to the next version.

2. Upgrade from v1.3.6 to v2.0.0

This is the most complex migration in the entire upgrade path because Apache DolphinScheduler introduced significant changes to its internal metadata model.

Step 1. Back Up the Existing Database

Export the original v1.3.6 database.

mysqldump -uroot -pmysql dolphinscheduler > .mysql_bak/dolphinscheduler.sql

Step 2. Restore the Backup

Import the exported data into the new database:

dolphinscheduler_2_0_0

Step 3. Extract the Binary Package

tar -zxvf apache-dolphinscheduler-2.0.0-bin.tar.gz \
-C ./upgrade_dolphin/

Step 4. Patch the Source Code

During the migration from v1.3.6 to v2.0.0, the task definition storage model changed significantly.

Previously, every workflow stored all task definitions inside the

t_ds_process_definition.process_definition_json

field.

Starting with v2.0.0, every task is migrated into the newly introduced

t_ds_task_definition

table.

During this conversion process, several edge cases may trigger NullPointerExceptions (NPEs).

For example:

To resolve these issues, four locations in the upgrade source code were patched by adding null checks to safely handle task and dependency parsing during schema migration.

Depending on the workflow definitions in your production environment, additional exceptions may occur. If so, update the corresponding migration logic, rebuild the project, and rerun the upgrade.

Examples:

Step 5. Replace the Patched DAO Library

The modified source file is:

UpgradeDao.java

After rebuilding the project, replace the original DAO library in the lib directory with the newly compiled JAR.

The following files are included:

Apache DolphinScheduler v2.0.0 source code
Rebuilt dolphinscheduler-dao-2.0.0.jar

Step 6. Run the Upgrade

Update the database configuration.

Navigate to:

apache-dolphinscheduler-2.0.0-bin

Edit:

conf/datasource.properties

Configure it to connect to:

dolphinscheduler_2_0_0

Copy mysql-connector-java-8.0.16.jar into the lib directory.
Run the upgrade script:

sh script/upgrade-dolphinscheduler.sh

Monitor the output carefully and resolve any errors that occur during execution.

After the script completes successfully, verify the database version by checking the t_ds_version table.

The version should now be:

2.0.0

At this point, the upgrade from v1.3.6 to v2.0.0 is complete.

3. Upgrade from v2.0.0 to v2.0.9

Compared with the previous migration, upgrading from v2.0.0 to v2.0.9 is much more straightforward. No source code modifications are required, and the upgrade can be completed using the official upgrade scripts.

Step 1. Back Up the Database

Export the upgraded v2.0.0 database.

mysqldump -uroot -pmysql dolphinscheduler_2_0_0 > .mysql_bak/dolphinscheduler_2_0_0.sql

Step 2. Restore the Backup

Import the backup into the target database:

dolphinscheduler_2_0_9

Step 3. Extract the Binary Package

tar -zxvf apache-dolphinscheduler-2.0.9-bin.tar.gz \
-C ./upgrade_dolphin/

Step 4. Run the Upgrade

Update the database configuration.

Navigate to:

apache-dolphinscheduler-2.0.9-bin

Edit the following configuration file:

conf/config/install_config.conf

Configure it to connect to:

dolphinscheduler_2_0_9

Copy mysql-connector-java-8.0.16.jar into the lib directory.
Run the upgrade script:

sh script/upgrade-dolphinscheduler.sh

Monitor the upgrade logs and resolve any errors if they occur.

Once the upgrade finishes successfully, verify the schema version in the t_ds_version table.

The version should now be:

2.0.9

The upgrade from v2.0.0 to v2.0.9 is now complete.

4. Upgrade from v2.0.9 to v3.0.0

Apache DolphinScheduler v3.0.0 introduces a new database upgrade mechanism. Unlike previous releases, schema migrations are executed using the upgrade-schema.sh utility located under the tools directory.

Step 1. Back Up the Database

Export the v2.0.9 database.

mysqldump -uroot -pmysql dolphinscheduler_2_0_9 > .mysql_bak/dolphinscheduler_2_0_9.sql

Step 2. Restore the Backup

Import the backup into the new database:

dolphinscheduler_3_0_0

Step 3. Extract the Binary Package

tar -zxvf apache-dolphinscheduler-3.0.0-bin.tar.gz \
-C ./upgrade_dolphin/

Step 4. Run the Upgrade

Update the database configuration.

Navigate to:

apache-dolphinscheduler-3.0.0-bin

Edit:

bin/env/dolphinscheduler_env.sh

Configure the database connection to:

dolphinscheduler_3_0_0

Copy mysql-connector-java-8.0.16.jar into the lib directory.
Modify the database upgrade script by removing the SQL statement that adds the alert_type column.

File location:

tools/sql/sql/upgrade/3.0.0_schema/mysql/dolphinscheduler_ddl.sql

After updating the SQL file, execute the schema upgrade:

sh tools/bin/upgrade-schema.sh

Review the output carefully and fix any reported errors before rerunning the script.

After a successful upgrade, verify that the t_ds_version table reports:

3.0.0

The migration to v3.0.0 is now complete.

5. Upgrade from v3.0.0 to v3.1.0

The upgrade procedure for v3.1.0 follows the same workflow as previous releases, with one additional adjustment to the database migration script.

Step 1. Back Up the Database

Export the upgraded v3.0.0 database.

mysqldump -uroot -pmysql dolphinscheduler_3_0_0 > .mysql_bak/dolphinscheduler_3_0_0.sql

Step 2. Restore the Backup

Import the backup into:

dolphinscheduler_3_1_0

Step 3. Extract the Binary Package

tar -zxvf apache-dolphinscheduler-3.1.0-bin.tar.gz \
-C ./upgrade_dolphin/

Step 4. Run the Upgrade

Navigate to:

apache-dolphinscheduler-3.1.0-bin

Update the database connection in:

bin/env/dolphinscheduler_env.sh

Configure it to use:

dolphinscheduler_3_1_0

Then:

Copy mysql-connector-java-8.0.16.jar into the lib directory.
Edit the schema upgrade script and remove the SQL statement that adds the other_params_json column.

File location:

tools/sql/sql/upgrade/3.1.0_schema/mysql/dolphinscheduler_ddl.sql

Run the upgrade:

sh tools/bin/upgrade-schema.sh

Monitor the execution logs for any errors.

After the migration completes successfully, verify the schema version in t_ds_version.

Expected version:

3.1.0

The upgrade from v3.0.0 to v3.1.0 is now complete.

6. Upgrade from v3.1.0 to v3.1.8

The upgrade from v3.1.0 to v3.1.8 follows the standard Apache DolphinScheduler schema migration workflow.

Before upgrading, always create a complete backup of the current database to ensure rollback capability in case of unexpected issues.

Step 1. Back Up the Database

Export the upgraded v3.1.0 database:

mysqldump -uroot -pmysql dolphinscheduler_3_1_0 > .mysql_bak/dolphinscheduler_3_1_0.sql

Step 2. Restore the Backup

Import the backup into the target database:

dolphinscheduler_3_1_8

Step 3. Extract the Binary Package

Extract the Apache DolphinScheduler v3.1.8 distribution:

tar -zxvf apache-dolphinscheduler-3.1.8-bin.tar.gz \
-C ./upgrade_dolphin/

Step 4. Run the Upgrade

Navigate to:

apache-dolphinscheduler-3.1.8-bin

Update the database connection configuration in:

bin/env/dolphinscheduler_env.sh

Configure the database as:

dolphinscheduler_3_1_8

Copy mysql-connector-java-8.0.16.jar into the lib directory.
Execute the schema upgrade script:

sh tools/bin/upgrade-schema.sh

Carefully monitor the upgrade process and check the logs for any errors.

After successful execution, verify the schema version:

SELECT * FROM t_ds_version;

The expected version should be:

3.1.8

The upgrade from v3.1.0 to v3.1.8 is now complete.

7. Upgrade from v3.1.8 to v3.2.2

Apache DolphinScheduler v3.2.2 is the final target version in this upgrade path.

The migration process is similar to the previous upgrades.

Step 1. Back Up the Database

Export the upgraded v3.1.8 database:

mysqldump -uroot -pmysql dolphinscheduler_3_1_8 > .mysql_bak/dolphinscheduler_3_1_8.sql

Step 2. Restore the Backup

Import the backup into:

dolphinscheduler_3_2_2

Step 3. Extract the Binary Package

tar -zxvf apache-dolphinscheduler-3.2.2-bin.tar.gz \
-C ./upgrade_dolphin/

Step 4. Run the Upgrade

Navigate to:

apache-dolphinscheduler-3.2.2-bin

Update the database configuration file:

bin/env/dolphinscheduler_env.sh

Configure it to connect to:

dolphinscheduler_3_2_2

Copy mysql-connector-java-8.0.16.jar into the lib directory.
Execute the schema migration script:

sh tools/bin/upgrade-schema.sh

Monitor the output carefully and resolve any migration errors if necessary.

After successful completion, verify the version:

3.2.2

by checking the t_ds_version table.

The upgrade from v3.1.8 to v3.2.2 is now complete.

8. Post-Upgrade Configuration and Troubleshooting

After completing the upgrade, additional configuration adjustments may be required to ensure all existing workflows and data sources continue working correctly.

8.1 Update Script Dependencies

After upgrading to v3.2.2, update the script dependency configuration.

Edit:

bin/env/dolphinscheduler_env.sh

Update paths for dependencies such as:

DataX
Python
Other external execution environments

Example:

Make sure all environment variables point to the correct installation paths after the upgrade.

8.2 Fix MySQL Data Source Configuration

After upgrading, clicking Edit Data Source for an existing MySQL data source may result in errors.

Root Cause

The issue is caused by an incompatible format in the:

t_ds_datasource.connection_params

field.

The connection parameter structure changed between versions.

Existing records created in older versions may not match the format expected by Apache DolphinScheduler v3.2.2.

Solution

Manually update the connection_params field in the t_ds_datasource table according to the new v3.2.2 format.

Example:

{
  "user": "root",
  "password": "doris",
  "address": "jdbc:mysql://127.0.0.1:9030",
  "database": "PROD_ODS",
  "jdbcUrl": "jdbc:mysql://127.0.0.1:9030/PROD_ODS",
  "driverClassName": "com.mysql.cj.jdbc.Driver",
  "validationQuery": "select 1"
}

After updating the connection parameters, the MySQL data source can be edited normally from the DolphinScheduler web interface.

Upgrade Completed Successfully 🎉

Following this guide, you can safely migrate Apache DolphinScheduler from:

v1.3.6
      ↓
v2.0.0
      ↓
v2.0.9
      ↓
v3.0.0
      ↓
v3.1.0
      ↓
v3.1.8
      ↓
v3.2.2

The key lessons from this upgrade journey:

Always perform database backups before every version migration.
Use intermediate upgrade versions instead of skipping major releases.
Review schema migration scripts carefully when moving across major versions.
Be prepared to patch migration logic for customized workflows or legacy metadata.
Validate data source configurations after completing the upgrade.

A carefully planned migration process can significantly reduce risks when upgrading production Apache DolphinScheduler environments across multiple major releases.

DEV Community: Chen Debra

💡 Open source isn't just for developers. Learn how anyone can contribute with docs, design, translation, and AI—no coding required!

You Don't Need to Code to Contribute: A Beginner's Guide to Open Source

You Don't Need to Code to Contribute: A Beginner's Guide to Open Source

Open Source Isn't as Intimidating as You Think

Freedom to Use, Learn, and Build

Open Collaboration at Scale

Where the Term "Open Source" Came From

Why Do Open Source Foundations Matter?

Apache Software Foundation (ASF)

Linux Foundation

Cloud Native Computing Foundation (CNCF)

OpenAtom Foundation

Open Source Licenses: The Legal Foundation of Every Project

MIT and BSD Licenses: Maximum Flexibility

Apache License 2.0: Enterprise-Friendly by Design

GNU GPL: Share-Alike by Design

You Don't Need to Write Code to Contribute

Learning the Workflow Can Be the Hardest Part

Non-Code Contributions Matter More Than You Think

Trust Is Built Through Consistency

Three Challenges Almost Every New Contributor Faces

1. Learning the Tooling

2. Understanding Merge Conflicts

3. Debugging Error Messages

How AI Is Lowering the Barrier to Open Source

Turn Complex Technical Concepts into Visuals

Create Real-World Architecture and Solution Diagrams

Use AI to Explain Error Messages

Translate and Localize Documentation

Build Better Community Knowledge

Your First Contribution Doesn't Have to Be Code

🚀 Run Apache DolphinScheduler with Spark on Kubernetes! Learn cloud-native deployment, shared storage, and production-ready best practices. ☸️⚡ #ApacheDolphinScheduler #Kubernetes #ApacheSpark #CloudNative

Running Apache DolphinScheduler with Spark on Kubernetes: A Practical Deployment Guide

Running Apache DolphinScheduler with Spark on Kubernetes: A Practical Deployment Guide

Environment Preparation

Prerequisites

Deployment Architecture

Basic Deployment

Step 1. Add the Helm Repository

Step 2. Create a Namespace

Step 3. Configure values.yaml

Step 4. Deploy Apache DolphinScheduler

Step 5. Verify the Deployment

Spark Integration

Step 1. Download the Spark Binary Package

Step 2. Copy Spark to a Worker Pod

Step 3. Configure Hadoop (Optional for Spark on YARN)

Verify Spark Jobs

Method 1. Verify with a Shell Task

Method 2. Verify with a Spark Task

Upload the Example JAR

Create a Spark Task

Method 3. Verify with a Spark SQL Task

Supported Spark Deployment Modes

Production-Ready Configuration

Build a Custom Worker Image

Configure Resource Requests and Limits

Enable Worker Auto Scaling with KEDA

Install KEDA

Enable Auto Scaling

Troubleshooting Common Issues

Check Worker Logs

Verify the Shared Storage

Verify Environment Variables

Common Issues

Spark Commands Cannot Be Found

Shared Storage Fails to Mount

Spark on YARN Jobs Fail

Best Practices for Production Deployments

Use High-Performance Shared Storage

Define Appropriate Resource Quotas

Implement Monitoring and Alerting

Deploy for High Availability

Establish a Backup Strategy

Notes

Apache DolphinScheduler Zombie Tasks Explained: Safe Recovery Strategies Across Versions

Overview

Why Do Zombie Tasks Occur?

Solution for Legacy Versions (Earlier than 1.2.1)

Step 3. Configure `values.yaml`

File 1: `AgentToolType.java` (Modified)