DEV Community

Chen Debra
Chen Debra

Posted on

Running Apache DolphinScheduler with Spark on Kubernetes: A Practical Deployment Guide

#apachedolphinscheduler #spark #kubernetes #opensource

As enterprise data volumes continue to grow, deploying Apache DolphinScheduler on traditional physical servers or virtual machines is becoming increasingly challenging. Complex environment setup, low resource utilization, and limited scalability often make operations difficult. Running Spark and other big data workloads further amplifies these issues, requiring teams to maintain multiple runtime environments, resolve dependency conflicts, and handle unpredictable traffic spikes.

Deploying DolphinScheduler on Kubernetes provides a cloud-native approach to solving these challenges. Containerization ensures consistent runtime environments, shared storage simplifies the distribution of Spark binaries and other dependencies, and Kubernetes enables elastic resource scaling to maximize infrastructure efficiency.

In this guide, you'll learn how to deploy Apache DolphinScheduler on Kubernetes and integrate Spark workloads step by step. By the end, you'll have a highly available, scalable, and cloud-native workflow orchestration platform ready for production.

Environment Preparation

Prerequisites

Before getting started, ensure your environment meets the following requirements:

  • Kubernetes cluster (v1.19 or later)
  • kubectl command-line tool
  • Helm 3.x
  • A storage class that supports the ReadWriteMany (RWX) access mode (required for shared storage)

Deployment Architecture

Basic Deployment

Step 1. Add the Helm Repository 

helm repo add dolphinscheduler https://dolphinscheduler.apache.org/helm
helm repo update
Enter fullscreen mode Exit fullscreen mode

Step 2. Create a Namespace 

kubectl create namespace dolphinscheduler
Enter fullscreen mode Exit fullscreen mode

Step 3. Configure values.yaml

Create a custom values.yaml file with the following key configurations.

# Shared storage configuration
# Required for storing Spark binaries and other shared resources
common:
  sharedStoragePersistence:
    enabled: true
    mountPath: "/opt/soft"
    accessModes:
      - "ReadWriteMany"
    storageClassName: "your-storage-class"   # Replace with your storage class
    storage: "20Gi"

  configmap:
    # Spark environment variables
    SPARK_HOME: "/opt/soft/spark"
    HADOOP_HOME: "/opt/soft/hadoop"
    HADOOP_CONF_DIR: "/opt/soft/hadoop/etc/hadoop"
    JAVA_HOME: "/opt/java/openjdk"

# External database configuration
externalDatabase:
  enabled: true
  host: "your-mysql-host"
  port: "3306"
  database: "dolphinscheduler"
  username: "root"
  password: "your-password"
  type: "mysql"

# Worker configuration
worker:
  replicas: 2
  env:
    WORKER_EXEC_THREADS: "10"
Enter fullscreen mode Exit fullscreen mode

Key configuration notes

  • Shared Storage must be enabled to support Spark integration.
  • The shared directory is mounted at /opt/soft, where Spark and Hadoop binaries will be stored.
  • Ensure your Kubernetes storage class supports the ReadWriteMany (RWX) access mode so that all Worker Pods can access the same files.
  • Using an external MySQL database is recommended for production deployments.

Step 4. Deploy Apache DolphinScheduler

Install DolphinScheduler using Helm.

helm install dolphinscheduler dolphinscheduler/dolphinscheduler \
  --namespace dolphinscheduler \
  --values values.yaml \
  --version 3.2.0
Enter fullscreen mode Exit fullscreen mode

It is recommended to specify an explicit chart version to ensure deployment consistency.

Step 5. Verify the Deployment

After installation, verify that all components are running correctly.

Check the Pod status:

kubectl get pods -n dolphinscheduler
Enter fullscreen mode Exit fullscreen mode

Check the services:

kubectl get svc -n dolphinscheduler
Enter fullscreen mode Exit fullscreen mode

Check the Persistent Volume Claims:

kubectl get pvc -n dolphinscheduler
Enter fullscreen mode Exit fullscreen mode

If all Pods are in the Running state and the PVC has been successfully bound, your Apache DolphinScheduler deployment is ready for Spark integration.

Spark Integration

Once Apache DolphinScheduler has been successfully deployed, the next step is to integrate Apache Spark so that Spark applications can be scheduled and executed directly from DolphinScheduler.

Since the shared storage has already been mounted at /opt/soft, Spark only needs to be installed once. All Worker Pods can then access the same binaries through the shared volume.

Step 1. Download the Spark Binary Package

In this example, we'll use Apache Spark 3.2.1.

wget https://archive.apache.org/dist/spark/spark-3.2.1/spark-3.2.1-bin-hadoop2.7.tgz
Enter fullscreen mode Exit fullscreen mode

Step 2. Copy Spark to a Worker Pod

Because the shared storage is mounted under /opt/soft, you only need to copy the Spark package into one Worker Pod. The extracted files will automatically become available to all Workers through the shared volume.

Copy the Spark package:

kubectl cp spark-3.2.1-bin-hadoop2.7.tgz dolphinscheduler-worker-0:/opt/soft -n dolphinscheduler
Enter fullscreen mode Exit fullscreen mode

Log in to the Worker container and extract the package:

kubectl exec -it dolphinscheduler-worker-0 -n dolphinscheduler bash

cd /opt/soft

tar zxf spark-3.2.1-bin-hadoop2.7.tgz

rm -f spark-3.2.1-bin-hadoop2.7.tgz

ln -s spark-3.2.1-bin-hadoop2.7 spark
Enter fullscreen mode Exit fullscreen mode

Verify that Spark has been installed successfully:

$SPARK_HOME/bin/spark-submit --version
Enter fullscreen mode Exit fullscreen mode

If the Spark version information is displayed correctly, the installation has completed successfully.

Step 3. Configure Hadoop (Optional for Spark on YARN)

If you plan to run Spark jobs in YARN mode, Hadoop also needs to be installed and configured.

Download the Hadoop binary package:

wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz
Enter fullscreen mode Exit fullscreen mode

Copy it into the Worker Pod:

kubectl cp hadoop-3.3.1.tar.gz dolphinscheduler-worker-0:/opt/soft -n dolphinscheduler
Enter fullscreen mode Exit fullscreen mode

Extract and configure Hadoop:

kubectl exec -it dolphinscheduler-worker-0 -n dolphinscheduler bash

cd /opt/soft

tar zxf hadoop-3.3.1.tar.gz

rm -f hadoop-3.3.1.tar.gz

ln -s hadoop-3.3.1 hadoop
Enter fullscreen mode Exit fullscreen mode

Set the Hadoop environment variables (these have already been configured in values.yaml):

export HADOOP_HOME=/opt/soft/hadoop

export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop
Enter fullscreen mode Exit fullscreen mode

After the symbolic link is created and the environment variables are configured, DolphinScheduler Workers can locate the Hadoop installation automatically when submitting Spark jobs to YARN.

Verify Spark Jobs

Once Spark has been installed, it's recommended to verify the environment before running production workloads.

Method 1. Verify with a Shell Task

The quickest way to validate your Spark environment is by creating a Shell Task in the DolphinScheduler Web UI.

Run the following command:

$SPARK_HOME/bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  $SPARK_HOME/examples/jars/spark-examples_2.12-3.2.1.jar
Enter fullscreen mode Exit fullscreen mode

After the task finishes, review the execution logs.

If you see output similar to:

Pi is roughly 3.14...
Enter fullscreen mode Exit fullscreen mode

your Spark runtime has been configured successfully.

Method 2. Verify with a Spark Task

Upload the Example JAR

Before creating a Spark task, upload the example JAR through the DolphinScheduler Resource Center.

Upload:

spark-examples_2.12-3.2.1.jar
Enter fullscreen mode Exit fullscreen mode

Create a Spark Task

Configure the Spark task with the following settings.

Parameter Value
Program Type Java
Main Class org.apache.spark.examples.SparkPi
Main Program Package spark-examples_2.12-3.2.1.jar
Deploy Mode local
Master local[*]

Execute the workflow and confirm that the task completes successfully.

Method 3. Verify with a Spark SQL Task

You can also validate the Spark SQL engine by creating a Spark SQL task.

Use the following SQL statements:

-- Create a test table
CREATE TABLE IF NOT EXISTS test_table (
  id INT,
  name STRING
) USING parquet;

-- Insert sample data
INSERT INTO test_table VALUES
(1, 'test'),
(2, 'dolphinscheduler');

-- Query the data
SELECT * FROM test_table;
Enter fullscreen mode Exit fullscreen mode

Configure the task using these parameters:

Parameter Value
Program Type SQL
Deploy Mode local
Master local[*]

If the SQL statements execute successfully and return the expected results, the Spark SQL environment has been configured correctly.

Supported Spark Deployment Modes

According to the current Apache DolphinScheduler implementation, Kubernetes deployments support different Spark execution modes to varying degrees.

Deployment Mode Support Status Notes
Spark Local (client) Supported (via shared storage) Requires Spark binaries to be installed in the shared volume
Spark on YARN (cluster) Supported (with additional configuration) Requires a properly configured Hadoop environment
Spark Standalone (cluster) Not Supported Currently unavailable
Spark on Kubernetes (cluster) Not Supported Native Kubernetes deployment mode has not yet been implemented

Although native Spark on Kubernetes is not currently available, the shared-storage approach provides an effective solution for running Spark Local and Spark on YARN workloads within a Kubernetes-based DolphinScheduler deployment.

Production-Ready Configuration

While manually copying Spark binaries into the shared volume is sufficient for evaluation and testing, production deployments typically package Spark directly into the Worker image. This approach simplifies operations, shortens deployment time, and guarantees a consistent runtime environment across every Worker Pod.

Build a Custom Worker Image

Create a custom Worker image with Spark pre-installed.

FROM dolphinscheduler.docker.scarf.sh/apache/dolphinscheduler-worker:3.2.0

# Install Apache Spark
RUN wget https://archive.apache.org/dist/spark/spark-3.2.1/spark-3.2.1-bin-hadoop2.7.tgz && \
    tar zxf spark-3.2.1-bin-hadoop2.7.tgz -C /opt/soft && \
    rm -f spark-3.2.1-bin-hadoop2.7.tgz && \
    ln -s /opt/soft/spark-3.2.1-bin-hadoop2.7 /opt/soft/spark

# Configure environment variables
ENV SPARK_HOME=/opt/soft/spark
Enter fullscreen mode Exit fullscreen mode

Build and push the image to your container registry.

docker build -t your-registry/dolphinscheduler-worker:spark .
docker push your-registry/dolphinscheduler-worker:spark
Enter fullscreen mode Exit fullscreen mode

Then update your values.yaml file to use the custom Worker image.

worker:
  image:
    repository: your-registry/dolphinscheduler-worker
    tag: spark
Enter fullscreen mode Exit fullscreen mode

With this approach, every newly created Worker Pod comes with Spark pre-installed, eliminating manual setup after deployment.

Configure Resource Requests and Limits

Proper resource allocation is essential for maintaining stable task execution, especially when multiple Spark jobs are running concurrently.

Configure CPU and memory limits for Worker Pods as follows:

worker:
  resources:
    limits:
      memory: "8Gi"
      cpu: "4"
    requests:
      memory: "4Gi"
      cpu: "2"
Enter fullscreen mode Exit fullscreen mode

In production environments, it's recommended to configure resource requests and limits based on your workload characteristics. This helps Kubernetes schedule Pods more efficiently while preventing individual tasks from consuming excessive cluster resources.

Enable Worker Auto Scaling with KEDA

Workload volumes often fluctuate throughout the day. Instead of provisioning a fixed number of Workers, Apache DolphinScheduler supports KEDA (Kubernetes Event-driven Autoscaling), allowing Worker Pods to scale automatically based on workload demand.

Install KEDA 

kubectl create namespace keda

helm install keda kedacore/keda \
  --namespace keda \
  --version v2.0.0
Enter fullscreen mode Exit fullscreen mode

Enable Auto Scaling

After KEDA is installed, enable auto scaling when upgrading your DolphinScheduler deployment.

helm upgrade dolphinscheduler dolphinscheduler/dolphinscheduler \
  --namespace dolphinscheduler \
  --set worker.keda.enabled=true \
  --set worker.keda.minReplicaCount=1 \
  --set worker.keda.maxReplicaCount=10
Enter fullscreen mode Exit fullscreen mode

In this example:

  • The Worker deployment scales down to 1 replica during idle periods.
  • It can automatically scale up to 10 replicas when task volume increases.

This elasticity helps reduce infrastructure costs while ensuring sufficient computing capacity during peak workloads.

Troubleshooting Common Issues

If a Spark task fails to execute, the following checks can help identify the root cause.

Check Worker Logs

Worker logs are usually the first place to investigate execution failures.

View the live logs:

kubectl logs dolphinscheduler-worker-0 \
  -n dolphinscheduler \
  -f
Enter fullscreen mode Exit fullscreen mode

Search for logs related to a specific task:

kubectl logs dolphinscheduler-worker-0 \
  -n dolphinscheduler | grep "task"
Enter fullscreen mode Exit fullscreen mode

Verify the Shared Storage

Ensure the shared volume has been mounted correctly and is accessible from the Worker Pods.

Check the Persistent Volume Claim:

kubectl get pvc -n dolphinscheduler
Enter fullscreen mode Exit fullscreen mode

Verify the mounted directory inside the container:

kubectl exec -it dolphinscheduler-worker-0 \
  -n dolphinscheduler bash

df -h | grep opt/soft

ls -la /opt/soft
Enter fullscreen mode Exit fullscreen mode

Confirm that the Spark and Hadoop directories are present and accessible.

Verify Environment Variables

Check whether the required runtime environment variables have been configured correctly.

kubectl exec -it dolphinscheduler-worker-0 \
  -n dolphinscheduler bash

echo $SPARK_HOME

echo $HADOOP_HOME

$SPARK_HOME/bin/spark-submit --version
Enter fullscreen mode Exit fullscreen mode

If the environment variables return the expected paths and spark-submit reports the installed Spark version, the runtime environment has been configured successfully.

Common Issues

Spark Commands Cannot Be Found

Symptoms

The Worker reports errors such as spark-submit: command not found.

Possible Causes

  • SPARK_HOME is not configured correctly.
  • Spark was not extracted into the shared storage directory.
  • The symbolic link to the Spark installation is missing or incorrect.

Resolution

  • Verify that SPARK_HOME points to the correct installation path.
  • Confirm that Spark has been extracted under /opt/soft.
  • Recreate the symbolic link if necessary.

Shared Storage Fails to Mount

Symptoms

Worker Pods cannot access files stored in /opt/soft.

Possible Causes

  • The selected StorageClass does not support the ReadWriteMany (RWX) access mode.
  • The PersistentVolumeClaim is not bound successfully.
  • The storage configuration in values.yaml is incorrect.

Resolution

  • Verify that your storage backend supports RWX.
  • Check the status of the PVC and PersistentVolume.
  • Review the storage configuration in your Helm values file.

Spark on YARN Jobs Fail

Symptoms

Spark applications fail during submission or cannot connect to the YARN cluster.

Possible Causes

  • Hadoop is not installed correctly.
  • HADOOP_CONF_DIR points to an incorrect location.
  • Network connectivity to the YARN cluster is unavailable.

Resolution

  • Verify the Hadoop installation.
  • Confirm that HADOOP_CONF_DIR contains the correct configuration files.
  • Check connectivity between the Worker Pods and the YARN cluster.

Best Practices for Production Deployments

The following recommendations can help improve reliability, scalability, and operational efficiency in production environments.

Use High-Performance Shared Storage

Since Spark binaries and shared resources are accessed by all Worker Pods, choose a high-performance storage backend—such as SSD-backed network storage—to minimize startup latency and improve overall job performance.

Define Appropriate Resource Quotas

Configure CPU and memory requests and limits according to your workload profile. Proper resource management improves cluster utilization and reduces resource contention between concurrent jobs.

Implement Monitoring and Alerting

Monitor both the Kubernetes infrastructure and DolphinScheduler workloads. Collect metrics for Pod health, resource consumption, task execution status, and workflow success rates, and configure alerts for abnormal conditions.

Deploy for High Availability

Run multiple replicas for critical components to eliminate single points of failure. In production, it's generally recommended to deploy at least two Master replicas and two Worker replicas to improve service availability.

Establish a Backup Strategy

Regularly back up the DolphinScheduler metadata database as well as workflow resources and configuration files. A well-defined backup and recovery strategy helps minimize downtime during unexpected failures.

Notes

This guide is based on Apache DolphinScheduler 3.2.0. Configuration options and supported features may vary across releases.

At the time of writing, native Spark on Kubernetes (cluster mode) is not yet supported by DolphinScheduler. If your workload depends on this capability, keep an eye on future project releases for updates.

Finally, remember to tailor the deployment to your own infrastructure. In particular, review environment-specific settings such as the storage class, external database configuration, networking, and cluster resource allocation before moving into production.

Top comments (0)