DEV Community

Chen Debra
Chen Debra

Posted on

How to Upgrade Apache DolphinScheduler from 1.3.6 to 3.2.2 Without Downtime

#opensource #dataengineering #apachedolphinscheduler #devops

Migrating a production Apache DolphinScheduler cluster across multiple major releases can be challenging, especially when upgrading from legacy versions such as v1.3.6 to the latest v3.2.2.

Because the database schema, task definition model, configuration files, and upgrade utilities have evolved significantly between releases, skipping intermediate versions is not recommended. This guide walks you through a verified upgrade path, highlights common pitfalls, and documents the fixes required during the migration process.

1. Upgrade Overview

Follow the upgrade sequence below:

v1.3.6
   ↓
v2.0.0
   ↓
v2.0.9
   ↓
v3.0.0
   ↓
v3.1.0
   ↓
v3.1.8
   ↓
v3.2.2
Enter fullscreen mode Exit fullscreen mode

Download the Binary Packages

Download the binary distribution for each target version from the Apache archive.

Example:

https://archive.apache.org/dist/dolphinscheduler/3.1.8/apache-dolphinscheduler-3.1.8-bin.tar.gz
Enter fullscreen mode Exit fullscreen mode

Prepare Independent Databases

Create a dedicated MySQL database for each intermediate version.

Example databases:

dolphinscheduler_2_0_0
dolphinscheduler_2_0_9
dolphinscheduler_3_0_0
dolphinscheduler_3_1_0
dolphinscheduler_3_1_8
dolphinscheduler_3_2_2
Enter fullscreen mode Exit fullscreen mode

Example SQL statement:

CREATE DATABASE dolphinscheduler_3_2_2
DEFAULT CHARACTER SET utf8
DEFAULT COLLATE utf8_general_ci;
Enter fullscreen mode Exit fullscreen mode

Using separate databases for every upgrade stage allows you to verify each migration independently and provides an easy rollback path if necessary.

Extract Each Release

Extract every binary package into the same working directory.

./upgrade_dolphin
Enter fullscreen mode Exit fullscreen mode

After extraction:

  • Update the database connection configuration.
  • Run the corresponding upgrade script.
  • Verify that the upgrade completes successfully before proceeding to the next version.

2. Upgrade from v1.3.6 to v2.0.0

This is the most complex migration in the entire upgrade path because Apache DolphinScheduler introduced significant changes to its internal metadata model.

Step 1. Back Up the Existing Database

Export the original v1.3.6 database.

mysqldump -uroot -pmysql dolphinscheduler > .mysql_bak/dolphinscheduler.sql
Enter fullscreen mode Exit fullscreen mode

Step 2. Restore the Backup

Import the exported data into the new database:

dolphinscheduler_2_0_0
Enter fullscreen mode Exit fullscreen mode

Step 3. Extract the Binary Package 

tar -zxvf apache-dolphinscheduler-2.0.0-bin.tar.gz \
-C ./upgrade_dolphin/
Enter fullscreen mode Exit fullscreen mode

Step 4. Patch the Source Code

During the migration from v1.3.6 to v2.0.0, the task definition storage model changed significantly.

Previously, every workflow stored all task definitions inside the

t_ds_process_definition.process_definition_json
Enter fullscreen mode Exit fullscreen mode

field.

Starting with v2.0.0, every task is migrated into the newly introduced

t_ds_task_definition
Enter fullscreen mode Exit fullscreen mode

table.

During this conversion process, several edge cases may trigger NullPointerExceptions (NPEs).

For example:

To resolve these issues, four locations in the upgrade source code were patched by adding null checks to safely handle task and dependency parsing during schema migration.

Depending on the workflow definitions in your production environment, additional exceptions may occur. If so, update the corresponding migration logic, rebuild the project, and rerun the upgrade.

Examples:

Step 5. Replace the Patched DAO Library

The modified source file is:

UpgradeDao.java
Enter fullscreen mode Exit fullscreen mode

After rebuilding the project, replace the original DAO library in the lib directory with the newly compiled JAR.

The following files are included:

  • Apache DolphinScheduler v2.0.0 source code
  • Rebuilt dolphinscheduler-dao-2.0.0.jar

Step 6. Run the Upgrade

Update the database configuration.

Navigate to:

apache-dolphinscheduler-2.0.0-bin
Enter fullscreen mode Exit fullscreen mode

Edit:

conf/datasource.properties
Enter fullscreen mode Exit fullscreen mode

Configure it to connect to:

dolphinscheduler_2_0_0
Enter fullscreen mode Exit fullscreen mode

Next:

  • Copy mysql-connector-java-8.0.16.jar into the lib directory.
  • Run the upgrade script: 
sh script/upgrade-dolphinscheduler.sh
Enter fullscreen mode Exit fullscreen mode

Monitor the output carefully and resolve any errors that occur during execution.

After the script completes successfully, verify the database version by checking the t_ds_version table.

The version should now be:

2.0.0
Enter fullscreen mode Exit fullscreen mode

At this point, the upgrade from v1.3.6 to v2.0.0 is complete.

3. Upgrade from v2.0.0 to v2.0.9

Compared with the previous migration, upgrading from v2.0.0 to v2.0.9 is much more straightforward. No source code modifications are required, and the upgrade can be completed using the official upgrade scripts.

Step 1. Back Up the Database

Export the upgraded v2.0.0 database.

mysqldump -uroot -pmysql dolphinscheduler_2_0_0 > .mysql_bak/dolphinscheduler_2_0_0.sql
Enter fullscreen mode Exit fullscreen mode

Step 2. Restore the Backup

Import the backup into the target database:

dolphinscheduler_2_0_9
Enter fullscreen mode Exit fullscreen mode

Step 3. Extract the Binary Package 

tar -zxvf apache-dolphinscheduler-2.0.9-bin.tar.gz \
-C ./upgrade_dolphin/
Enter fullscreen mode Exit fullscreen mode

Step 4. Run the Upgrade

Update the database configuration.

Navigate to:

apache-dolphinscheduler-2.0.9-bin
Enter fullscreen mode Exit fullscreen mode

Edit the following configuration file:

conf/config/install_config.conf
Enter fullscreen mode Exit fullscreen mode

Configure it to connect to:

dolphinscheduler_2_0_9
Enter fullscreen mode Exit fullscreen mode

Next:

  • Copy mysql-connector-java-8.0.16.jar into the lib directory.
  • Run the upgrade script: 
sh script/upgrade-dolphinscheduler.sh
Enter fullscreen mode Exit fullscreen mode

Monitor the upgrade logs and resolve any errors if they occur.

Once the upgrade finishes successfully, verify the schema version in the t_ds_version table.

The version should now be:

2.0.9
Enter fullscreen mode Exit fullscreen mode

The upgrade from v2.0.0 to v2.0.9 is now complete.

4. Upgrade from v2.0.9 to v3.0.0

Apache DolphinScheduler v3.0.0 introduces a new database upgrade mechanism. Unlike previous releases, schema migrations are executed using the upgrade-schema.sh utility located under the tools directory.

Step 1. Back Up the Database

Export the v2.0.9 database.

mysqldump -uroot -pmysql dolphinscheduler_2_0_9 > .mysql_bak/dolphinscheduler_2_0_9.sql
Enter fullscreen mode Exit fullscreen mode

Step 2. Restore the Backup

Import the backup into the new database:

dolphinscheduler_3_0_0
Enter fullscreen mode Exit fullscreen mode

Step 3. Extract the Binary Package 

tar -zxvf apache-dolphinscheduler-3.0.0-bin.tar.gz \
-C ./upgrade_dolphin/
Enter fullscreen mode Exit fullscreen mode

Step 4. Run the Upgrade

Update the database configuration.

Navigate to:

apache-dolphinscheduler-3.0.0-bin
Enter fullscreen mode Exit fullscreen mode

Edit:

bin/env/dolphinscheduler_env.sh
Enter fullscreen mode Exit fullscreen mode

Configure the database connection to:

dolphinscheduler_3_0_0
Enter fullscreen mode Exit fullscreen mode

Next:

  • Copy mysql-connector-java-8.0.16.jar into the lib directory.
  • Modify the database upgrade script by removing the SQL statement that adds the alert_type column.

File location:

tools/sql/sql/upgrade/3.0.0_schema/mysql/dolphinscheduler_ddl.sql
Enter fullscreen mode Exit fullscreen mode

After updating the SQL file, execute the schema upgrade:

sh tools/bin/upgrade-schema.sh
Enter fullscreen mode Exit fullscreen mode

Review the output carefully and fix any reported errors before rerunning the script.

After a successful upgrade, verify that the t_ds_version table reports:

3.0.0
Enter fullscreen mode Exit fullscreen mode

The migration to v3.0.0 is now complete.

5. Upgrade from v3.0.0 to v3.1.0

The upgrade procedure for v3.1.0 follows the same workflow as previous releases, with one additional adjustment to the database migration script.

Step 1. Back Up the Database

Export the upgraded v3.0.0 database.

mysqldump -uroot -pmysql dolphinscheduler_3_0_0 > .mysql_bak/dolphinscheduler_3_0_0.sql
Enter fullscreen mode Exit fullscreen mode

Step 2. Restore the Backup

Import the backup into:

dolphinscheduler_3_1_0
Enter fullscreen mode Exit fullscreen mode

Step 3. Extract the Binary Package 

tar -zxvf apache-dolphinscheduler-3.1.0-bin.tar.gz \
-C ./upgrade_dolphin/
Enter fullscreen mode Exit fullscreen mode

Step 4. Run the Upgrade

Navigate to:

apache-dolphinscheduler-3.1.0-bin
Enter fullscreen mode Exit fullscreen mode

Update the database connection in:

bin/env/dolphinscheduler_env.sh
Enter fullscreen mode Exit fullscreen mode

Configure it to use:

dolphinscheduler_3_1_0
Enter fullscreen mode Exit fullscreen mode

Then:

  • Copy mysql-connector-java-8.0.16.jar into the lib directory.
  • Edit the schema upgrade script and remove the SQL statement that adds the other_params_json column.

File location:

tools/sql/sql/upgrade/3.1.0_schema/mysql/dolphinscheduler_ddl.sql
Enter fullscreen mode Exit fullscreen mode

Run the upgrade:

sh tools/bin/upgrade-schema.sh
Enter fullscreen mode Exit fullscreen mode

Monitor the execution logs for any errors.

After the migration completes successfully, verify the schema version in t_ds_version.

Expected version:

3.1.0
Enter fullscreen mode Exit fullscreen mode

The upgrade from v3.0.0 to v3.1.0 is now complete.

6. Upgrade from v3.1.0 to v3.1.8

The upgrade from v3.1.0 to v3.1.8 follows the standard Apache DolphinScheduler schema migration workflow.

Before upgrading, always create a complete backup of the current database to ensure rollback capability in case of unexpected issues.

Step 1. Back Up the Database

Export the upgraded v3.1.0 database:

mysqldump -uroot -pmysql dolphinscheduler_3_1_0 > .mysql_bak/dolphinscheduler_3_1_0.sql
Enter fullscreen mode Exit fullscreen mode

Step 2. Restore the Backup

Import the backup into the target database:

dolphinscheduler_3_1_8
Enter fullscreen mode Exit fullscreen mode

Step 3. Extract the Binary Package

Extract the Apache DolphinScheduler v3.1.8 distribution:

tar -zxvf apache-dolphinscheduler-3.1.8-bin.tar.gz \
-C ./upgrade_dolphin/
Enter fullscreen mode Exit fullscreen mode

Step 4. Run the Upgrade

Navigate to:

apache-dolphinscheduler-3.1.8-bin
Enter fullscreen mode Exit fullscreen mode

Update the database connection configuration in:

bin/env/dolphinscheduler_env.sh
Enter fullscreen mode Exit fullscreen mode

Configure the database as:

dolphinscheduler_3_1_8
Enter fullscreen mode Exit fullscreen mode

Next:

  • Copy mysql-connector-java-8.0.16.jar into the lib directory.
  • Execute the schema upgrade script: 
sh tools/bin/upgrade-schema.sh
Enter fullscreen mode Exit fullscreen mode

Carefully monitor the upgrade process and check the logs for any errors.

After successful execution, verify the schema version:

SELECT * FROM t_ds_version;
Enter fullscreen mode Exit fullscreen mode

The expected version should be:

3.1.8
Enter fullscreen mode Exit fullscreen mode

The upgrade from v3.1.0 to v3.1.8 is now complete.

7. Upgrade from v3.1.8 to v3.2.2

Apache DolphinScheduler v3.2.2 is the final target version in this upgrade path.

The migration process is similar to the previous upgrades.

Step 1. Back Up the Database

Export the upgraded v3.1.8 database:

mysqldump -uroot -pmysql dolphinscheduler_3_1_8 > .mysql_bak/dolphinscheduler_3_1_8.sql
Enter fullscreen mode Exit fullscreen mode

Step 2. Restore the Backup

Import the backup into:

dolphinscheduler_3_2_2
Enter fullscreen mode Exit fullscreen mode

Step 3. Extract the Binary Package 

tar -zxvf apache-dolphinscheduler-3.2.2-bin.tar.gz \
-C ./upgrade_dolphin/
Enter fullscreen mode Exit fullscreen mode

Step 4. Run the Upgrade

Navigate to:

apache-dolphinscheduler-3.2.2-bin
Enter fullscreen mode Exit fullscreen mode

Update the database configuration file:

bin/env/dolphinscheduler_env.sh
Enter fullscreen mode Exit fullscreen mode

Configure it to connect to:

dolphinscheduler_3_2_2
Enter fullscreen mode Exit fullscreen mode

Next:

  • Copy mysql-connector-java-8.0.16.jar into the lib directory.
  • Execute the schema migration script: 
sh tools/bin/upgrade-schema.sh
Enter fullscreen mode Exit fullscreen mode

Monitor the output carefully and resolve any migration errors if necessary.

After successful completion, verify the version:

3.2.2
Enter fullscreen mode Exit fullscreen mode

by checking the t_ds_version table.

The upgrade from v3.1.8 to v3.2.2 is now complete.

8. Post-Upgrade Configuration and Troubleshooting

After completing the upgrade, additional configuration adjustments may be required to ensure all existing workflows and data sources continue working correctly.

8.1 Update Script Dependencies

After upgrading to v3.2.2, update the script dependency configuration.

Edit:

bin/env/dolphinscheduler_env.sh
Enter fullscreen mode Exit fullscreen mode

Update paths for dependencies such as:

  • DataX
  • Python
  • Other external execution environments

Example:

Make sure all environment variables point to the correct installation paths after the upgrade.

8.2 Fix MySQL Data Source Configuration

After upgrading, clicking Edit Data Source for an existing MySQL data source may result in errors.

Root Cause

The issue is caused by an incompatible format in the:

t_ds_datasource.connection_params
Enter fullscreen mode Exit fullscreen mode

field.

The connection parameter structure changed between versions.

Existing records created in older versions may not match the format expected by Apache DolphinScheduler v3.2.2.

Solution

Manually update the connection_params field in the t_ds_datasource table according to the new v3.2.2 format.

Example:

{
  "user": "root",
  "password": "doris",
  "address": "jdbc:mysql://127.0.0.1:9030",
  "database": "PROD_ODS",
  "jdbcUrl": "jdbc:mysql://127.0.0.1:9030/PROD_ODS",
  "driverClassName": "com.mysql.cj.jdbc.Driver",
  "validationQuery": "select 1"
}
Enter fullscreen mode Exit fullscreen mode

After updating the connection parameters, the MySQL data source can be edited normally from the DolphinScheduler web interface.

Upgrade Completed Successfully 🎉

Following this guide, you can safely migrate Apache DolphinScheduler from:

v1.3.6
      ↓
v2.0.0
      ↓
v2.0.9
      ↓
v3.0.0
      ↓
v3.1.0
      ↓
v3.1.8
      ↓
v3.2.2
Enter fullscreen mode Exit fullscreen mode

The key lessons from this upgrade journey:

  • Always perform database backups before every version migration.
  • Use intermediate upgrade versions instead of skipping major releases.
  • Review schema migration scripts carefully when moving across major versions.
  • Be prepared to patch migration logic for customized workflows or legacy metadata.
  • Validate data source configurations after completing the upgrade.

A carefully planned migration process can significantly reduce risks when upgrading production Apache DolphinScheduler environments across multiple major releases.

Top comments (0)