DEV Community: Chen Debra

⚡ Explore how Apache DolphinScheduler powers enterprise data warehouses with seamless orchestration, data processing, BI analytics, and AI applications. Discover the full data journey! 🌐 #ApacheDolphinScheduler #DataEngineering #BigData

Chen Debra — Tue, 22 Sep 2026 09:40:11 +0000

Chen Debra

Sep 22

From Data Pipelines to Intelligent Applications: Building Enterprise Data Warehouses with Apache DolphinScheduler

#apachedolphinscheduler #datawarehouse #datascience #opensource

10 min read

🔎 A failed task with no logs? Learn how Apache DolphinScheduler troubleshooting traces the issue from TaskInstance metadata to Master dispatch and tenant configuration. #ApacheDolphinScheduler #DataEngineering #OpenSource

Chen Debra — Tue, 22 Sep 2026 09:39:43 +0000

Chen Debra

Sep 22

The Task Failed, but Where Did the Logs Go? Troubleshooting an Apache DolphinScheduler Failed Task

#apachedolphinscheduler #opensource #datascience #bigdata

9 min read

The Task Failed, but Where Did the Logs Go? Troubleshooting an Apache DolphinScheduler Failed Task

Chen Debra — Tue, 22 Sep 2026 09:39:20 +0000

A failure with no logs: tracing a task execution issue from metadata to the Master node

One of the most confusing scenarios in production scheduling systems is when a task is clearly marked as "failed" in the UI and an alert has already been generated, but critical execution information such as the task start time, execution host, execution directory, and log path are all empty.

There is no Worker log, no YARN Application ID, and no Spark Driver, Executor, or MapReduce Container logs.

When facing this situation, engineers often start troubleshooting from the Worker node, script paths, YARN environment, or network connectivity. However, this case revealed a different root cause:

The task had never reached the Worker node. The failure occurred while the Master was constructing the task execution context. The actual issue was neither the script nor the virtual IP, but an invalid tenant configuration in the workflow.

Key takeaway:
When a task_instance is already in the FAILED state, but host, start_time, log_path, and execute_path are all NULL, the first place to investigate should be the Master-side task submission and dispatch process, rather than searching for task execution logs that were never generated.

1. Incident Overview: Failed Task, Failed Alert, and Empty Logs

In this case, a batch workflow was scheduled to run every day at 14:15. The scheduling UI showed that the first Shell task failed, and subsequent tasks were not executed.

Meanwhile, the alert table already contained a "scheduler failed" record. However, the alert delivery logs also showed another issue:

no bind plugin instance

The latest failed alert can be retrieved from t_ds_alert:

SELECT id, title, content, alert_status, warning_type,
log, alertgroup_id, create_time,
process_instance_id
FROM t_ds_alert
ORDER BY create_time DESC
LIMIT 1;

The alert content contains the workflow instance ID, task Code, task name, task type, and failure status, making it a useful entry point for reconstructing the entire troubleshooting chain.

However, two issues must be distinguished:

Task execution failure
Alert delivery failure

These are independent problems and should not be treated as the same failure.

Problem	Direct Evidence	Impact
Task failure	`taskState=FAILURE`	The task itself did not execute successfully.
Alert failure	`alertgroup_id=0; no bind plugin instance`	The failure event was generated, but notifications (e.g., email/DingTalk) were not sent.

Troubleshooting principle:
Do not treat "alert delivery failure" as the cause of task failure. First determine which stage of the task lifecycle failed, then separately troubleshoot alert routing.

2. First Evidence: Metadata Shows the Task Never Reached the Worker

Using the process_instance_id and taskCode obtained from the alert, we can query t_ds_task_instance and locate the exact task execution record.

The result shows that the task instance was created and its state was already 6 (FAILURE). However, except for submit_time, all other runtime-related fields were empty.

Query the task lifecycle fields:

SELECT id AS task_instance_id, name, state, host,
submit_time, start_time, end_time,
log_path, execute_path, app_link
FROM t_ds_task_instance
WHERE process_instance_id = <process_instance_id>
AND task_code = <task_code>
ORDER BY id DESC;

Field	Query Result	Diagnostic Meaning
`state`	6	The task has been marked as failed by the Master.
`submit_time`	Has value	The task instance has been created in the metadata database.
`start_time`	NULL	The Worker never started execution.
`host`	NULL	No Worker has been assigned yet.
`execute_path`	NULL	The execution directory has not been created yet.
`log_path`	NULL	No task execution log will be generated.
`app_link`	NULL	Not submitted to YARN.

At this point, we can make the first critical conclusion:

The task failed before Worker execution began.

Continuing to SSH into Worker nodes, searching for logs, running yarn logs, or analyzing Spark Executor issues would not provide useful information, because the execution entity itself was never created.

3. A Misleading Clue: Why Does the Master Show a Different IP?

During troubleshooting, the primary IP address of the Master node was found to be inconsistent with the Master address displayed in the scheduling frontend. After SSH-ing into the address shown in the frontend, both the hostname and shell prompt still displayed the primary IP, which initially raised concerns about incorrect routing or an address registration issue.

The SSH connection tuple, hostname, and all network interface addresses were then verified:

echo "$SSH_CONNECTION"
hostname -f
ip -br addr

The output of ip -br addr showed that the primary IP was bound with a /24 network mask, while another address was bound with /32 on the same eth0 interface.

The latter resolved to an EMR virtual hostname, indicating that it was not another machine, but an auxiliary or virtual service IP on the same node.

Testing SSH connectivity to a virtual IP on the Master node only proves that the address is reachable locally.

To verify whether scheduling communication was actually working, the service port needed to be tested from a real Worker node.

Run the following commands from an actual Worker node:

nc -vz -w 3 <master-primary-ip> 5678
nc -vz -w 3 <master-virtual-ip> 5678

Both Master addresses successfully accepted connections on port 5678.

Meanwhile, Master heartbeat logs continued to update successfully under:

/nodes/master/:5678

At this point, the possibility of an unreachable Master registration address could be ruled out.

Lesson learned:
A mismatch between the IP displayed in the UI and the host's primary IP does not necessarily indicate a configuration error. First determine whether it is a secondary IP or VIP on the same machine, then verify service connectivity from the actual communication endpoint. Avoid relying only on local self-connectivity tests.

4. Decisive Evidence: "Tenant does not exists" in Master Logs

Since the task never reached the Worker node, the real evidence had to be found in the Master logs.

By searching the Master logs with multiple stable identifiers, including the workflow instance ID, task instance ID, task Code, and task name, the complete failure chain was reconstructed.

Use multiple identifiers to locate the same scheduling event:

grep -R -nE 'WorkflowInstance-<id>|TaskInstance-<id>|<task_code>|<task_name>' /var/log/.../master-server/

The failure occurred within milliseconds, before the task was actually dispatched to a Worker:

14:15:00.650 Task is ready to dispatch to worker
14:15:00.651 Tenant does not exists
14:15:00.651 Task state changes to FAILURE
14:15:00.652 Get taskExecutionContext fail
14:15:00.653 Dispatch standby task failed
14:15:00.690 Workflow state changes to FAILURE

The log output provided the decisive evidence:

processDefinition.tenantId=-1
processDefinition.tenantCode=null

processInstance.tenantId=-1
processInstance.tenantCode=null
processInstance.queue=null

When constructing the TaskExecutionContext, the Master node must determine the execution tenant information. However, the current workflow did not have a valid tenant configuration.

As a result, DolphinScheduler could not generate the task execution context, and the task was marked as FAILURE before Worker allocation.

Root cause:

The workflow definition and workflow instance both had tenantId=-1, with an empty tenantCode.

The Master failed to construct the TaskExecutionContext, causing the task to fail before it was dispatched to a Worker node.

5. Why Are There No Logs? Understanding the DolphinScheduler Task Lifecycle

In DolphinScheduler, task logs are not generated when a TaskInstance record is created in the database.

Instead, logs become available only after the following steps are completed:

The Master successfully constructs the task execution context.
The task is assigned to an available Worker.
The Worker creates the execution directory and starts running the task.

Understanding this sequence is essential when troubleshooting the situation where a task fails but no logs exist.

Stage	Typical Field Changes	Where to Check
1. Create instance	`submit_time` has value; `state` = submit successful	Metadata database, Master logs
2. Build context	Populate `tenant`, `queue`, environment, etc.	Master logs
3. Assign Worker	`host` has value	Master dispatch logs, registration center
4. Worker startup	`start_time`, `execute_path`, `log_path` have values	Worker task logs
5. Submit big-data job	`app_link` / Application ID has value	YARN and Container logs

These lifecycle fields also provide a practical troubleshooting strategy:

Field status	Investigation direction
`host` is empty	Check Master-side task creation and dispatch
`host` exists but `start_time` is empty	Check Worker assignment and Worker acceptance
`log_path` exists	Check Worker execution logs
Application ID exists	Proceed with YARN log investigation

In other words, the absence of logs is itself a valuable signal.

6. What Role Does Tenant Play in DolphinScheduler?

In DolphinScheduler, a Tenant is not merely a classification field displayed in the UI.

It is closely related to task execution identity and resource allocation.

When the Master constructs a task execution context, it needs to resolve information such as:

tenantCode
execution user
resource queue

This information tells the Worker:

Which user identity should execute the task?
Which resource queue should be used?
Which permissions should be applied?

Without valid tenant information, the Master cannot safely create an execution environment for the task.

In environments using:

Linux user switching
HDFS permission control
Kerberos authentication
YARN queue isolation

tenant configuration can further affect:

Operating system user identity
HDFS directory permissions
Authentication tickets
Resource queue assignment

Therefore, even after repairing the tenant configuration, engineers should still verify the corresponding OS user permissions and big data platform access control.

7. Confirm Tenant Relationships Through Metadata

To verify the relationship between workflows, users, and tenants, query the following metadata:

SELECT pd.id, pd.code, pd.name, pd.version,
pd.user_id, u.user_name,
pd.tenant_id AS workflow_tenant_id,
u.tenant_id AS user_tenant_id,
t.id AS matched_tenant_id,
t.tenant_code, t.queue_id
FROM t_ds_process_definition pd
LEFT JOIN t_ds_user u ON u.id = pd.user_id
LEFT JOIN t_ds_tenant t ON t.id = pd.tenant_id
WHERE pd.code = <process_definition_code>;

Confirm that the tenant exists and check historical workflow versions:

SELECT * FROM t_ds_tenant ORDER BY id;

SELECT * FROM t_ds_user
WHERE id = <user_id>
OR user_name = '<user_name>';

SELECT code, name, version, tenant_id, release_state
FROM t_ds_process_definition_log
WHERE code = <process_definition_code>
ORDER BY version DESC;

The expected abnormal result in this case was:

workflow_tenant_id = -1

and no corresponding record could be found in t_ds_tenant.

If a user already has a tenant but the workflow still shows tenant_id=-1, it indicates that the workflow did not correctly inherit tenant information during creation, import, or version migration.

Simply modifying the user configuration does not necessarily update existing workflows automatically.

8. Fix Procedure: Avoid Direct Updates to Metadata Tables

The recommended approach is to repair the configuration through the DolphinScheduler UI instead of directly modifying metadata tables.

Direct database updates may bypass:

workflow version management
cache synchronization
permission relationships
audit information

The standard repair process is:

1. Create or select a valid tenant

In Security Center → Tenant Management, create or select a valid tenant and associate the correct YARN Queue.

2. Bind the tenant to the workflow owner

In User Management, assign the tenant to the workflow owner.

3. Update the workflow configuration

Edit the failed workflow, select a valid tenant, save the workflow, and generate a new workflow version.

4. Publish the updated workflow

Release the new workflow version and verify that:

t_ds_process_definition.tenant_id > 0

5. Start a new execution

Trigger a completely new manual run instead of recovering the old failed instance.

6. Verify runtime permissions

After the task reaches the Worker node, verify:

Linux user permissions
script execution permissions
HDFS access
Kerberos authentication
YARN queue permissions

Why not rerun the old instance?
A failed instance stores a snapshot of the previous workflow version, where tenantId is still -1. Recovering the old instance may continue using the invalid configuration. After publishing a new workflow version, create a new instance for validation.

9. How to Validate After Fixing the Issue

After the repair is completed, validation should not stop at confirming that the workflow status turns green.

A complete verification process should confirm that the task has successfully passed the original failure point.

It is recommended to verify the workflow from three layers:

Definition layer

The latest workflow version has a valid tenant_id greater than 0.
The tenant can be correctly mapped to a record in t_ds_tenant.

Instance layer

The new workflow instance contains valid tenantCode.
The resource queue information is no longer empty.

Dispatch layer

TaskInstance.host is populated.
Worker nodes receive the task successfully.

Runtime layer

start_time, execute_path, and log_path are generated correctly.

Big data execution layer

For tasks submitting Spark or MapReduce jobs:

The Application ID can be retrieved.
YARN logs can be queried successfully.

Acceptance Field	When Fault Occurs	Expected After Fix
`tenant_id`	-1	> 0
`tenantCode` / `queue`	NULL	Valid value
`host`	NULL	Worker address
`start_time`	NULL	Actual start time
`execute_path` / `log_path`	NULL	Actual execution directory and log path
`app_link`	NULL	Application ID or link appears after submission to YARN

Conclusion: No Logs Are Logs

In scheduling systems, "no logs" does not mean there is no evidence.

On the contrary, when host, start_time, execute_path, log_path, and app_link are all empty, these fields already provide a strong indication:

The task never entered the execution layer.

Following this clue back to the Master node, the millisecond-level failure timeline eventually pointed to:

Tenant does not exists

A high-quality troubleshooting process is not about checking every component blindly.

Instead, it requires:

Using metadata to define the investigation boundary.
Using logs to reconstruct the execution timeline.
Using network tests to eliminate unrelated possibilities.
Separately closing the loop on task failures and alert delivery failures.

This approach transforms troubleshooting from experience-based debugging into a systematic, explainable, and reusable diagnostic methodology.

One-line summary:

When a TaskInstance enters the FAILED state but all runtime fields are NULL, check the Master first. In this case, the direct cause was tenantId=-1, which prevented the Master from constructing the TaskExecutionContext.

From Data Pipelines to Intelligent Applications: Building Enterprise Data Warehouses with Apache DolphinScheduler

Chen Debra — Tue, 22 Sep 2026 08:30:28 +0000

As enterprises accelerate their digital transformation, data platforms are evolving from traditional storage and analytics systems into foundational infrastructures that support business decision-making, intelligent applications, and data asset management.

Building a stable, efficient, and scalable data production system has become a key challenge in modern enterprise data warehouse (DW) initiatives.

At the Apache DolphinScheduler September Meetup, community practitioner Zan Liu shared practical experiences in building enterprise-grade data warehouses. The session explored how Apache DolphinScheduler can be used as the orchestration core, working together with data integration, data processing, data applications, and operational management capabilities to build a complete data engineering ecosystem.

Based on the Meetup session, this article provides an in-depth look at how Apache DolphinScheduler supports enterprise data warehouse practices across four key stages: environment setup, data processing, data applications, and operational upgrades.

👉🏻 Replay the Meetup session here

About the Speaker

Liu Zan：Data Analytics Engineer & Senior Big Data Architect

With extensive experience in data warehousing and big data technologies, Liu Zan specializes in BI development and Dify-based low-code Agent development. He has deep expertise in building enterprise data platforms and implementing data-driven applications.

Environment Setup: Building a Data Foundation for Enterprise Data Warehouses

The first step in building an enterprise data warehouse is selecting a technology architecture that can support long-term business growth and establishing a stable, reliable data infrastructure.

During the early stages of big data development, the Hadoop ecosystem became the foundation for many enterprise data platforms due to its distributed storage and computing capabilities. However, as real-time data requirements continue to increase, traditional architectures have gradually revealed limitations in certain scenarios.

On one hand, traditional Hadoop-based architectures are primarily designed for batch processing workloads. Data jobs often experience minute-level latency, making them less suitable for real-time analytics and fast decision-making scenarios.

On the other hand, traditional architectures tightly couple computing and storage resources. When enterprises need to scale a specific type of resource, such as computing capacity or storage capacity, the other resource is often affected as well, resulting in lower resource utilization.

In addition, the complexity of the big data ecosystem introduces higher costs in terms of component management, version compatibility, system maintenance, and technical learning.

Therefore, modern enterprise data warehouse practices are increasingly adopting lightweight and high-performance OLAP architectures.

1. Technology Selection: Using OLAP Databases to Support High-Performance Analytics

During the early phase of data warehouse development, the team adopted a traditional big data warehouse architecture. However, as data volume increased and business analytics requirements became more complex, several limitations gradually emerged in real-world usage.

First, traditional data warehouse architectures are mainly designed for offline batch processing scenarios. Job execution usually involves relatively high latency, with many tasks taking several minutes to complete, making it difficult to meet the low-latency requirements of certain business analytics scenarios.

Second, traditional architectures suffer from tightly coupled storage and compute resources. Since computing nodes are responsible for both data storage and processing, scaling either compute or storage independently becomes difficult. This can lead to inefficient resource utilization and increase the complexity of platform expansion.

Furthermore, traditional big data ecosystems usually rely on multiple components working together. The large number of components, combined with version compatibility requirements, increases the complexity of platform deployment, learning, and long-term operations.

To address these challenges in performance, scalability, and operations, the team adopted a distributed OLAP database architecture in its next-generation data warehouse implementation, improving analytics capabilities and overall platform efficiency.

In the new architecture, the orchestration layer adopts the open-source project Apache DolphinScheduler for workflow orchestration, task scheduling, dependency management, retry and alerting, resource management, and visual operations.

The data integration layer adopts another open-source project, Apache SeaTunnel, which provides data source connectivity, batch and streaming data synchronization, data transformation, and cleansing capabilities.

The new architecture delivers several improvements.

First, it provides significantly improved query performance. Based on distributed OLAP architecture, it enables sub-second responses for large-scale data queries. In wide-table aggregation scenarios, performance can improve by 5 to 10 times compared with traditional solutions, better supporting fast business analytics.

Second, the new architecture provides stronger scalability. Compared with traditional Hadoop ecosystems that rely on multiple components such as HBase and Spark, the new solution reduces dependencies on numerous third-party components, resulting in a simpler architecture and lower maintenance complexity.

Meanwhile, based on distributed cluster technology, the architecture provides stronger concurrent processing capabilities, reduces single-node I/O pressure, and improves overall platform stability.

Through this technology selection, the team successfully upgraded from a traditional data warehouse architecture to a modern OLAP-based architecture, establishing a more reliable data foundation for subsequent data synchronization, processing, and business applications.

2. Environment Setup: Reducing the Complexity of Big Data Platform Deployment

After completing the architecture selection, the next step was to build the infrastructure required for running data processing workflows.

During the environment setup process, the team used a cluster management tool to deploy the big data environment. The tool provides fast deployment capabilities, compatibility with open-source ecosystems, and simplified operations and maintenance.

Through centralized management, users can quickly initialize large-scale big data clusters while covering multiple areas including data integration, storage, computing engines, task scheduling, and permission management.

From an operations perspective, the tool enables unified monitoring and management of clusters, nodes, and services, improving daily maintenance efficiency.

After completing the infrastructure setup, the team further deployed Apache DolphinScheduler to provide unified scheduling capabilities for subsequent data workflows.

3. Deployment Process: Setting Up Apache DolphinScheduler

Within the overall data warehouse architecture, Apache DolphinScheduler is responsible for data workflow orchestration and scheduling management.

By deploying DolphinScheduler, the team was able to integrate subsequent data synchronization and data processing tasks into a unified workflow management system, providing visual orchestration and automated execution capabilities.

Data Processing: Connecting the Data Production Pipeline with Apache DolphinScheduler

After completing the infrastructure setup, data processing becomes the core stage of enterprise data warehouse development. In this practice, the team focused on data synchronization, data transformation, and task scheduling, using Apache DolphinScheduler to orchestrate and manage data workflows.

Within the overall data processing pipeline, DolphinScheduler is responsible for workflow management and task scheduling, while Apache SeaTunnel handles data synchronization. Together, the two projects enable an automated process from data ingestion to data processing.

This practice mainly covers Apache DolphinScheduler deployment, SeaTunnel integration, mirror table synchronization, model-layer processing, exchange table applications, and exploration of real-time data synchronization solutions.

1. Apache DolphinScheduler Application: Enabling Unified Data Task Scheduling

During the data processing phase, the team adopted Apache DolphinScheduler to centrally manage data workflows.

Compared with traditional approaches that rely on manually maintained scripts to manage task execution dependencies, DolphinScheduler provides visual workflow orchestration capabilities.

Users can define workflows through a drag-and-drop interface and organize different data processing tasks into DAG workflows based on their dependencies, significantly reducing workflow management complexity.

At the same time, DolphinScheduler supports multiple task types and provides cross-language extensibility, enabling it to adapt to diverse data processing scenarios. In enterprise data warehouse projects, different types of data tasks can be integrated into a unified scheduling system, simplifying workflow management and maintenance.

In terms of reliability, DolphinScheduler adopts a decentralized architecture, improving scheduling system stability and ensuring continuous execution of data workflows.

By introducing DolphinScheduler, the team connected data synchronization and data processing steps into a unified workflow, enabling centralized scheduling throughout the data production pipeline.

2. Apache SeaTunnel Integration: Building Data Synchronization Pipelines

For data ingestion and synchronization, the team introduced Apache SeaTunnel as the data integration component responsible for transferring data between different data sources.

Based on its Connector architecture, SeaTunnel supports multiple data synchronization scenarios, including offline synchronization, real-time synchronization, full synchronization, and incremental synchronization, reducing the complexity of managing data integration tasks.

At the execution engine layer, SeaTunnel uses the Zeta engine by default while also supporting Flink and Spark as Connector execution engines, providing greater flexibility for different business scenarios.

In addition, SeaTunnel improves data transfer efficiency through parallel reading and writing mechanisms, delivering stable, high-throughput, and low-latency data synchronization capabilities.

Within the overall data processing architecture, DolphinScheduler and SeaTunnel work together: DolphinScheduler handles workflow orchestration and execution scheduling, while SeaTunnel performs data synchronization operations. Together, they provide the foundation for efficient data movement across the data warehouse pipeline.

3. Mirror Table Synchronization: Improving Data Synchronization Flexibility

During the data synchronization process, the team adopted a mirror table synchronization approach to complete data migration.

Schema synchronization, flexible configuration, and simple operations

This solution provides a configuration experience similar to DataX while offering more comprehensive data synchronization capabilities.

In practical applications, the team used the tool to synchronize table schemas and then leveraged DolphinScheduler to centrally schedule synchronization tasks, enabling automated data migration workflows.

Data synchronization (DolphinScheduler)

By integrating synchronization tasks into the scheduling system, data migration workflows can be automatically executed according to predefined processes. Combined with task status management, this approach improves the controllability and reliability of data processing operations.

4. Model Layer Processing: Simplifying Data Transformation Workflows

After completing data synchronization, the team further processed data at the model layer.

During the model-layer data cleansing stage, Shell scripts were used for business logic processing. This scripting approach improved development efficiency while reducing future maintenance costs.

In the actual workflow, model-layer processing tasks were also managed through DolphinScheduler, forming a complete data processing pipeline together with upstream and downstream synchronization tasks.

5. Exchange Table Application: Enabling Seamless Data Switching

To improve data reliability and stability during updates, the team adopted an exchange table mechanism for data switching.

Through this approach, table data can be switched seamlessly without affecting business users, minimizing the impact of data update operations.

Combined with DolphinScheduler’s workflow orchestration capabilities, data processing and table switching operations can be executed automatically according to predefined sequences, further improving the stability of the data production pipeline.

6. Real-Time Solution Exploration: Investigating CDC-Based Real-Time Synchronization

Beyond offline data processing workflows, the team also explored solutions for real-time data synchronization scenarios.

This practice investigated a CDC-based real-time synchronization architecture without relying on Flink. The solution features a simpler architecture, strong performance, and easier operations and maintenance.

Through exploring real-time synchronization solutions, the team further enhanced its data processing capabilities and prepared the platform to support a wider range of future data application scenarios.

Data Applications: From Data Analytics to Intelligent Data Services

After data ingestion, synchronization, and transformation are completed, data needs to be further leveraged to support business analytics and application scenarios.

In this data warehouse implementation, the team focused on three key areas: BI applications, AI agent applications, and data governance, expanding the platform’s data service capabilities.

1. BI Applications: Supporting Data Analytics and Business Visualization

At the data application layer, the team connected databases with BI tools through database interfaces to enable analytics and visualization.

Based on built-in JDBC drivers, the database can be connected directly to BI platforms, providing strong concurrency capabilities and fast query response times. Meanwhile, compatibility with the MySQL protocol enables integration with existing data analytics tools.

Reporting application (FineReport)

For practical applications, the team used FineReport for reporting scenarios and DataEase for dashboard visualization.

Dashboard application (DataEase)

Through the previous data processing workflow, data synchronization and transformation tasks are centrally scheduled by Apache DolphinScheduler, ensuring that data updates are completed according to predefined processes and providing a stable foundation for BI analytics and visualization.

2. AI Agent Applications: Exploring Data Querying Scenarios

In traditional data analytics scenarios, users typically rely on reports or query tools to access data. To further expand data application methods, the team developed a data query AI agent based on Dify.

By building a data query AI agent, users can access data through a more convenient interaction method, providing new possibilities for data applications.

This practice demonstrates that a data platform needs to provide not only reliable data processing capabilities but also strong data support for upper-layer applications. The data consumed by intelligent applications still relies on continuous data generation through underlying data warehouse synchronization and processing workflows.

3. Data Governance: Enhancing Metadata Management Capabilities

As enterprise data assets continue to grow, data governance has become an essential component of modern data warehouse development.

In this practice, the team used OpenMetadata to support metadata management for Doris databases.

Through metadata management, data asset information stored in databases can be centrally managed, providing better support for data usage, maintenance, and future expansion.

Operations and Maintenance Upgrades: Ensuring Stable Data Warehouse Operations

After completing data warehouse construction and application implementation, long-term platform stability still requires a comprehensive operations and maintenance system.

In this practice, the team enhanced platform operations capabilities in three key areas: log monitoring, upgrade management, and failure alerting.

1. Log Monitoring: Improving System Observability

During daily operations, the team uses log monitoring to gain real-time visibility into the status of different platform components.

By monitoring component logs in real time and combining metric collection with customized alert mechanisms, the team can quickly identify and address potential issues.

In practice, the team uses Apache DolphinScheduler to view task execution logs while integrating Prometheus and Grafana for daily monitoring.

Among these tools, DolphinScheduler provides visibility into task execution logs, while Prometheus and Grafana monitor platform metrics. Together, they improve overall system observability and operational efficiency.

2. Upgrade Management: Simplifying Doris Cluster Operations

For database cluster maintenance, the team adopted DorisManager for Doris cluster management.

Through this all-in-one Doris cluster management tool, cluster upgrades and maintenance operations can be simplified, improving overall platform management efficiency.

3. Failure Alerting: Establishing Task Failure Notification Mechanisms

To detect data task failures in a timely manner, the team configured alert instances in Apache DolphinScheduler and integrated them with Feishu bots for failure notifications.

When a task fails during execution, the system can automatically send notifications through the alert mechanism, helping relevant teams quickly identify and resolve issues.

Conclusion: Apache DolphinScheduler Enables End-to-End Enterprise Data Warehouse Development

At the Apache DolphinScheduler September Meetup, the community shared a practical enterprise data warehouse implementation covering the complete lifecycle from environment setup and data processing to data applications and operational management.

Within this architecture, Apache DolphinScheduler runs throughout the entire data production workflow, connecting data synchronization, data transformation, and operational management through workflow orchestration and task scheduling capabilities.

Apache SeaTunnel handles data synchronization, OLAP databases provide analytical capabilities, while BI tools, AI agents, and metadata management systems further extend data application scenarios.

Through this implementation, the team built a comprehensive data platform covering both data processing and data applications, providing reliable support for stable enterprise data warehouse operations.

🔍 Ever wondered how Apache DolphinScheduler’s Master turns workflow commands into reliable task execution? Dive into the source code and trace its startup, scheduling, RPC, and failover flow. ⚙️ #ApacheDolphinScheduler #OpenSource #DataEngineering

Chen Debra — Fri, 18 Sep 2026 08:08:07 +0000

Chen Debra

Sep 18

Inside DolphinScheduler’s Master Startup Process: A Source Code Walkthrough

#apachedolphinscheduler #opensource #github #datascience

7 min read

Inside DolphinScheduler’s Master Startup Process: A Source Code Walkthrough

Chen Debra — Fri, 18 Sep 2026 08:06:11 +0000

Apache DolphinScheduler is a distributed, highly extensible, and visual workflow scheduler designed for creating, scheduling, and monitoring enterprise-scale big data workflows. As a core scheduling component, the Master node receives task commands, schedules DAG workflows, dispatches tasks to Workers, and handles fault tolerance and high availability across the cluster.

This article walks through the Master startup process in DolphinScheduler 3.2.0 from the source-code level. By tracing the startup sequence and examining the key modules involved, you can quickly build an understanding of how the Master service initializes and how its scheduling engine works.

1. How a Manually Triggered Workflow Starts

When you click “Run” in the Web UI to manually trigger a workflow, the workflow does not start executing immediately. Instead, DolphinScheduler first writes a Command to the database.

The entry point is ExecutorController:

@PostMapping(value = "start-process-instance")
public Result startProcessInstance(@RequestBody StartProcessInstanceCommand command) {
    // Parameter validation and permission checks...
    executorService.execProcessInstance(command);
    return Result.success();
}

The subsequent call chain is:

ExecutorServiceImpl#execProcessInstance(...)
→ createCommand(...)
→ CommandServiceImpl#createCommand(Command)
→ CommandMapper#insert(Command)

At this stage, DolphinScheduler only inserts a record into the t_ds_command table. After startup, the Master continuously polls this table and consumes pending commands, after which it actually restores and executes the corresponding workflow instances.

2. MasterServer Startup Entry Point and Overall Flow

The Master startup entry point is the org.apache.dolphinscheduler.server.master.MasterServer class. Its @PostConstruct-annotated run() method initializes and starts the major components in sequence:

@PostConstruct
public void run() throws SchedulerException {
    // 1. Start the RPC services (Server + Client)
    masterRPCServer.start();
    masterRpcClient.start();

    // 2. Load Task plugins
    taskPluginManager.loadPlugin();

    // 3. Start the registry client (register the Master and monitor cluster changes)
    masterRegistryClient.start();
    masterRegistryClient.setRegistryStoppable(this);

    // 4. Start the core scheduling engine
    masterSchedulerBootstrap.start();

    // 5. Start the asynchronous event processing service
    eventExecuteService.start();

    // 6. Start the failover thread
    failoverExecuteThread.start();

    // 7. Start the Quartz scheduler
    schedulerApi.start();

    // Add a JVM shutdown hook
    Runtime.getRuntime().addShutdownHook(new Thread(() -> {
        if (!ServerLifeCycleManager.isStopped()) {
            close("MasterServer shutdownHook");
        }
    }));
}

Let's walk through each step to understand the implementation details and the design behind them.

3. Starting the Master RPC Services

Inside DolphinScheduler, the Master uses Netty to implement RPC communication. The RPC layer consists of an RPC Server, which receives heartbeats and status updates from Workers, and an RPC Client, which sends task commands to Workers.

3.1 Starting the RPC Server

The MasterRPCServer.start() method initializes the RPC server as follows:

public void start() {
    log.info("Starting Master RPC Server...");
    NettyServerConfig serverConfig = masterConfig.getMasterRpcServerConfig();
    serverConfig.setListenPort(masterConfig.getListenPort());
    this.nettyRemotingServer = new NettyRemotingServer(serverConfig);
    // Register all MasterRpcProcessors to handle incoming messages
    for (MasterRpcProcessor processor : masterRpcProcessors) {
        this.nettyRemotingServer.registerProcessor(processor);
    }
    this.nettyRemotingServer.start();
    log.info("Started Master RPC Server...");
}

When NettyRemotingServer is constructed, DolphinScheduler selects either Epoll or NIO based on the runtime environment and initializes the bossGroup and workerGroup:

public NettyRemotingServer(NettyServerConfig config) {
    if (Epoll.isAvailable()) {
        bossGroup = new EpollEventLoopGroup(1, bossFactory);
        workGroup = new EpollEventLoopGroup(config.getWorkerThread(), workerFactory);
    } else {
        bossGroup = new NioEventLoopGroup(1, bossFactory);
        workGroup = new NioEventLoopGroup(config.getWorkerThread(), workerFactory);
    }
    // Initialize ServerBootstrap and register handlers
}

During start(), ServerBootstrap.bind(...) binds the server to the configured port. The default RPC port is 5678:

serverBootstrap
    .group(bossGroup, workGroup)
    .channel(NettyUtils.getServerSocketChannelClass())
    .childHandler(new ChannelInitializer<SocketChannel>() {
        protected void initChannel(SocketChannel ch) {
            initNettyChannel(ch);
        }
    });
ChannelFuture future = serverBootstrap.bind(serverConfig.getListenPort()).sync();

Once the server successfully binds to the port, the Master can receive RPC requests from Workers and other components.

3.2 Starting the RPC Client

MasterRpcClient.start() initializes the NettyRemotingClient, but it does not proactively establish connections to Workers:

public void start() {
    client = new NettyRemotingClient(masterConfig.getMasterRpcClientConfig());
    log.info("Success initialized MasterRPCClient...");
}

The NettyRemotingClient constructor also selects Epoll or NIO depending on the environment. It initializes the workerGroup, callbackExecutor, and responseFutureExecutor, and starts the response-future scanning mechanism:

bootstrap
    .group(workerGroup)
    .channel(NettyUtils.getSocketChannelClass())
    .handler(new ChannelInitializer<SocketChannel>() {
        public void initChannel(SocketChannel ch) {
            ch.pipeline()
              .addLast(new IdleStateHandler(...))
              .addLast(new NettyDecoder(), clientHandler, encoder);
        }
    });
responseFutureExecutor.scheduleWithFixedDelay(ResponseFuture::scanFutureTable, 0, 1, TimeUnit.SECONDS);

In other words, the Master initializes the client-side RPC infrastructure during startup. Actual connections to Workers are established when the Master needs to dispatch tasks.

4. Plugin Loading

DolphinScheduler uses Java SPI to dynamically load Task plugins, enabling integration with multiple execution engines such as Hive, Spark, and Flink.

The implementation of TaskPluginManager.loadPlugin() is:

public void loadPlugin() {
    PrioritySPIFactory<TaskChannelFactory> factory = new PrioritySPIFactory<>(TaskChannelFactory.class);
    for (Map.Entry<String, TaskChannelFactory> entry : factory.getSPIMap().entrySet()) {
        String name = entry.getKey();
        TaskChannelFactory plugin = entry.getValue();
        taskChannelFactoryMap.put(name, plugin);
        taskChannelMap.put(name, plugin.create());
    }
}

Internally, PrioritySPIFactory uses ServiceLoader.load(spiClass) to scan implementations registered under META-INF/services and handles conflicts between implementations with the same name:

for (T impl : ServiceLoader.load(spiClass)) {
    String key = impl.getIdentify().getName();
    if (map.containsKey(key)) resolveConflict(impl);
    else map.put(key, impl);
}

This SPI-based architecture allows Task-related capabilities to be extended without tightly coupling the core scheduling engine to every specific execution technology.

5. Registry Client Initialization and Heartbeat Maintenance

The Master communicates with ZooKeeper, or another supported registry, through masterRegistryClient. This component handles several key responsibilities:

Register the current Master node: Create an ephemeral node under /dolphinscheduler/master and store heartbeat information.
Maintain heartbeats: Periodically update node information to prevent the cluster from treating the Master as unavailable.
Monitor cluster changes: Subscribe to /dolphinscheduler/servers so the Master can dynamically detect Master and Worker nodes joining or leaving the cluster.

The startup logic is:

public void start() {
    this.masterHeartBeatTask = new MasterHeartBeatTask(masterConfig, registryClient);
    registry(); // Register the Master and start the heartbeat
    registryClient.addConnectionStateListener(new MasterConnectionStateListener(...));
    registryClient.subscribe(RegistryNodeType.ALL_SERVERS.getRegistryPath(), new MasterRegistryDataListener());
}

The core registration logic is:

void registry() {
    registryClient.remove(masterPath);
    registryClient.persistEphemeral(masterPath, JSONUtils.toJsonString(heartbeat));
    while (!registryClient.checkNodeExists(host, MASTER)) {
        ThreadUtils.sleep(3000);
    }
    masterHeartBeatTask.start();
}

The MasterRegistryDataListener handles registry events through handleMasterEvent() and handleWorkerEvent(), triggering failover processing or resource cleanup when cluster membership changes.

6. Starting the Core Scheduling Engine

The core scheduling engine is started by MasterSchedulerBootstrap, which brings together three major components:

Command recovery: Queries all pending Commands, creates the corresponding WorkflowExecuteRunnable instances, and adds them to the execution cache and event queue.
Event loop: WorkflowEventLooper continuously consumes events from workflowEventQueue and invokes the appropriate handler based on the event type.
Task executor: MasterTaskExecutorBootstrap starts the thread pools and queues responsible for consuming pending tasks and dispatching them to Workers through RPC.

The startup sequence is:

public synchronized void start() {
    super.start();              // part1: recover and submit Commands
    workflowEventLooper.start(); // part2: start the event loop
    masterTaskExecutorBootstrap.start(); // part3: dispatch tasks
}

6.1 Recovering Commands

The recovery process can be illustrated as follows:

List<Command> commands = findCommands();
commands.parallelStream().forEach(cmd -> {
    Optional<WorkflowExecuteRunnable> opt = factory.create(cmd);
    if (opt.isPresent()) {
        cacheManager.cache(id, runnable);
        workflowEventQueue.addEvent(new WorkflowEvent(START_WORKFLOW, id));
    }
});

The Master retrieves pending Commands, creates the corresponding workflow execution objects, caches them, and then pushes START_WORKFLOW events into the workflow event queue.

This is the key transition from a command stored in the database to an actual workflow execution process.

6.2 The Event Loop

WorkflowEventLooper implements Runnable:

public void run() {
    while (RUNNING) {
        WorkflowEvent event = queue.poolEvent();
        try (MDCAutoClosableContext ctx = setWorkflowIdMDC(...)) {
            handlerMap.get(event.getType()).handle(event);
        }
    }
}

When a START_WORKFLOW event is received, it is routed to WorkflowStartHandler, which invokes WorkflowExecuteRunnable.call():

public WorkflowStartStatus startWorkflow() {
    initTaskQueue();     // Initialize the DAG task queue
    submitPostNode(null); // Submit the first node
    return SUCCESS;
}

At this point, the workflow execution engine begins processing the DAG and submitting its executable tasks.

6.3 Task Dispatch

MasterTaskExecutorBootstrap starts three processing loops:

globalTaskDispatchLooper.start();
masterDelayTaskLooper.start();
asyncMasterTaskDelayLooper.start();

Among them, globalTaskDispatchLooper retrieves DefaultTaskExecuteRunnable instances from globalTaskDispatchWaitingQueue and invokes taskDispatcher.dispatch():

Message msg = taskDispatchRequest.convert2Command();
masterRpcClient.sendSyncCommand(host, msg, timeout);

The task dispatch request is converted into an RPC message and sent synchronously to the target Worker.

This is the point where the Master moves from workflow-level orchestration to actual task execution on a Worker node.

7. Event Processing Service

Event processing is divided into two main categories:

Workflow events: Events such as task status changes and workflow blocking.
Stream events: Custom events associated with streaming tasks.

EventExecuteService.start() starts the corresponding thread pools and continuously consumes events from the stateEvents and taskEvents queues:

workflowEventHandler(); // Submit to workflowExecuteThreadPool
streamTaskEventHandler(); // Submit to streamTaskExecuteThreadPool

By processing events asynchronously through dedicated thread pools, the Master can separate event production from event handling and avoid coupling the execution of different types of events too tightly.

8. Failover Processing

The failoverExecuteThread periodically checks the health of Master and Worker nodes. When a node becomes unavailable, DolphinScheduler handles two types of failover scenarios: Master Failover and Worker Failover.

8.1 Master Failover

When a Master node goes down, the registry removes its node. Other Master nodes detect the change and trigger failover processing:

failoverService.failoverServerWhenDown(serverHost, MASTER);

doFailoverMaster then:

Queries the ProcessInstances that require failover.
Calls processService.processNeedFailoverProcessInstances(processInstance) for each instance.
Writes the required execution commands back to t_ds_command.

This allows another available Master to take over the affected workflow instances and continue the scheduling process.

8.2 Worker Failover

When a Worker node goes down:

failoverService.failoverServerWhenDown(workerHost, WORKER);

failoverWorker queries the TaskInstance records for tasks that were running on the failed Worker. For each unfinished task, it:

Sets state = NEED_FAULT_TOLERANCE.
Updates the database through taskInstanceDao.upsert.
Submits a TaskStateEvent to workflowExecuteThreadPool, allowing the task to be dispatched again to another available Worker.

Through this mechanism, task execution can recover from Worker failures without requiring the workflow itself to be restarted from scratch.

9. Starting the Quartz Scheduler

In addition to manually triggered workflows, the Master uses Quartz to support scheduled execution.

SchedulerApi injects an org.quartz.Scheduler instance:

@Override
public void start() throws SchedulerException {
    scheduler.start();
}

The Quartz Scheduler is used to trigger scheduled jobs periodically, including tasks such as workflow dependency handling and scheduled workflow execution.

This complements the command-driven execution path used for manually triggered workflows.

10. Summary and Key Takeaways

This article has walked through the complete startup and execution flow of the Master node in DolphinScheduler 3.2.0, from manually writing a Command to the database, to MasterServer initialization, RPC service setup, plugin loading, registry integration, core scheduling engine startup, event processing, failover handling, and Quartz-based scheduled execution.

At the architectural level, the Master demonstrates a modular and decoupled design:

Network communication: RPC communication is implemented with Netty.
Plugin extensibility: Java SPI is used to dynamically load TaskChannel implementations.
High availability: Registry-based heartbeats and the Failover service provide fault-tolerance capabilities.
Scheduling engine: Parallel processing and an event-driven architecture work together to drive workflow execution.
Scheduled execution: Quartz provides additional scheduling capabilities for time-based tasks.

At the source-code level, the core logic is built around database operations, queues, event processing, and RPC messaging. Developers can continue tracing the call chain to explore the implementation details that are not covered here, and combine the source code with the DolphinScheduler community documentation for a deeper understanding of how the Master works.

Once you follow the startup path from MasterServer.run() through command recovery, event handling, task dispatch, and failover, the Master is no longer a black box: it becomes a clearly connected execution pipeline that turns workflow commands into reliable task execution across the cluster.

🚀 Apache DolphinScheduler 3.4.3 is here! Enhanced security, faster queries, stronger recovery, and more. Explore the latest release! #ApacheDolphinScheduler #DataOrchestration #OpenSource

Chen Debra — Fri, 18 Sep 2026 07:48:30 +0000

Chen Debra

Sep 18

Apache DolphinScheduler 3.4.3 Released: Stronger Security, Stability, and Scheduling Reliability

#apachedolphinecheduler #java #github #bigdata

5 min read

Apache DolphinScheduler 3.4.3 Released: Stronger Security, Stability, and Scheduling Reliability

Chen Debra — Fri, 18 Sep 2026 07:48:09 +0000

Apache DolphinScheduler 3.4.3 has officially been released. This release delivers continued improvements across scheduling, permissions, security, performance, and stability, with a focus on real-world production scenarios such as task and workflow instance queries, API access control, sensitive information protection, and task failure recovery. In addition, new missed-scheduling recovery strategies are planned for Apache DolphinScheduler 3.5.0.

As an open-source data orchestration platform designed for complex data workflow scenarios, Apache DolphinScheduler continues to strengthen its scheduling and orchestration capabilities through community collaboration. Version 3.4.3 further improves reliability in areas such as scheduling recovery, enterprise-grade access control, and large-scale task execution, providing a stronger foundation for stable operation in production environments.

View the Apache DolphinScheduler 3.4.3 Release

Stronger Access Control for Improved Platform Security

One of the key areas of improvement in 3.4.3 is access control.

Enterprise data platforms typically involve multiple users, projects, data sources, and workflows. Clear and well-defined permission boundaries are therefore essential to maintaining a secure operating environment.

Version 3.4.3 further improves access control across multiple APIs, covering scenarios such as user list access, data source authorization lists, cluster queries, workflow modifications, and sub-workflow references. Related permissions have also been further standardized and refined.

The release also further restricts access to audit logs for non-administrator users, ensuring that users can only access audit information within the scope of their permissions. In addition, task-level data source access now includes corresponding permission checks, providing stronger controls over data access.

Together, these improvements establish clearer permission boundaries across users, projects, workflows, and data sources, providing more comprehensive security for enterprise environments with multiple users and collaborative workflows.

Enhanced Protection for Sensitive Information

Beyond access control, Apache DolphinScheduler 3.4.3 also introduces targeted improvements to sensitive information protection.

The release removes plaintext passwords from Worker logs and prevents raw passwords from being exposed through parameter validation error messages. These changes reduce the risk of sensitive information leakage in both log output and error messages.

The release also fixes a Sonar Token exposure issue and removes some unused code, further improving the project's overall security and engineering quality.

More Efficient Instance Queries at Scale

As the number of workflows and task instances continues to grow, querying task and workflow instances has become a frequent operation in production environments.

To address this scenario, version 3.4.3 introduces several query optimizations. New indexes, including idx_project_submit_time and idx_project_start_time, have been added for task instance and workflow instance queries. The Mapper query logic has also been optimized to exclude unnecessary large text fields from list queries.

These improvements reduce data retrieval during queries and help lower database load, improving query efficiency and the overall user experience in environments with large numbers of task instances.

Improved Task Failure Recovery and Retry Handling

The ability to recover from task failures directly affects the continuity of data processing. Version 3.4.3 includes multiple fixes covering task failures, retries, and workflow recovery.

The release adds application termination handling during task failover, preventing related applications from continuing to run after a task failure. It also adjusts retry time calculation so that the next execution is scheduled based on endTime + retryInterval.

For scenarios where failed tasks are recreated, 3.4.3 fixes an issue where runtime states were not properly reset. During workflow recovery, the release also ensures that forced-success states are correctly preserved.

In addition, this version fixes issues where Worker Group parameters were not applied as specified when rerunning workflows, as well as abnormal TaskGroup slot handling when tasks were paused or terminated. These fixes further strengthen task recovery in exceptional execution scenarios.

More Robust Workflow Dependencies and Execution Logic

At the workflow execution layer, version 3.4.3 includes multiple fixes related to DAGs and dependent tasks.

The release fixes an issue where DAG.addEdge could accept an edge that creates a cycle in path-convergence scenarios, further strengthening DAG validation and ensuring workflow correctness.

For dependent tasks, 3.4.3 fixes an issue with parsing ALL-type dependencies. It also resolves a problem where a task instance could fail while the workflow instance remained in a running state when using the Continue strategy, enabling workflow status to more accurately reflect actual execution states under complex dependency conditions.

In addition, the release addresses issues related to SQL task data source field persistence, DataX tasks reading Job Definitions from associated resource files, and S3 resource lists containing more than 1,000 records.

Continued Improvements to Kubernetes, Alerts, and Other Capabilities

For cloud-native deployments, version 3.4.3 fixes an issue in the Helm Chart that could result in duplicate app.kubernetes.io/name labels in ConfigMaps and updates the MySQL Helm Chart version.

Alerting capabilities have also been improved. These changes include removing redundant plugin definition table checks during AlertServer startup and adjusting the behavior of Alert Script test notifications. The release also fixes an issue that could cause Kubernetes Alert HTTP tests to fail.

In addition, the release removes some deprecated APIs and unused code, while further improving documentation related to parameter precedence, upgrades, data sources, and configuration. These changes contribute to better maintainability and a smoother user experience.

Missed-Scheduling Recovery Strategies Planned for 3.5.0

Beyond the improvements included in this release, the community continues to advance Apache DolphinScheduler's scheduling capabilities.

The missed-scheduling recovery strategy proposed in DSIP-107 has completed the relevant implementation and has been merged into the development branch. It is planned for inclusion in version 3.5.0.

The feature supports three strategies for handling missed schedules:

SKIP_MISSED
FIRE_ONCE_NOW
FIRE_ALL_MISSED

These strategies provide different ways to handle schedules that were missed, giving users greater control over how scheduled workflows should be recovered after interruptions or other exceptional circumstances.

Continuously Strengthening Enterprise-Grade Data Orchestration

From missed-scheduling recovery and access control to query performance and task failure recovery, Apache DolphinScheduler 3.4.3 focuses closely on the challenges encountered in real-world production environments.

Rather than simply adding new features, this release continues to improve the core qualities required for long-running data workflows, including scheduling reliability, security, query performance, and task stability. The upcoming missed-scheduling recovery strategies will further strengthen scheduling behavior in exceptional scenarios. More granular access controls establish stronger security boundaries for multi-user enterprise environments, while improvements to task recovery, dependency handling, and query performance further enhance platform reliability in complex production environments.

For the complete list of updates, see the Release Notes.

Thanks to Our Contributors

The Apache DolphinScheduler 3.4.3 release would not have been possible without the continued participation and contributions of the community. We would like to thank the following 20 contributors for contributing code, submitting issues, participating in testing and reviews, and helping move the community forward:

det101, wcmolin, liang-wenjie, njnu-seafish, qiuyanjun888, destinyoooo, HomminLee, ruanwenjun, kittimzhe, SbloodyS, eye-gu, vlaborie, nkuprins, nanxiuzi, SEPURI-SAI-KRISHNA, hiSandog, hellodml, zhang-arvin, nikhiln64, yan9651688

Thank you to every contributor for your time, expertise, and continued support. The ongoing collaboration of the global Apache DolphinScheduler community enables the project to continuously improve its scheduling, orchestration, security, and stability capabilities while serving an ever-growing range of data engineering use cases.

We invite you to try Apache DolphinScheduler 3.4.3, explore the latest improvements, and join the community to help advance the open-source data orchestration ecosystem.

Modern Data Stack was built for humans 🚀 Harness Engineering makes agent‑driven data work production‑safe. #DataEngineering #AgenticAI #HarnessEngineering

Chen Debra — Fri, 11 Sep 2026 09:25:39 +0000

Chen Debra

Sep 11

Rebuilding Data Engineering with Harness Engineering: A New Paradigm for the Agent Era

#dataengineering #harnessengineering #agents #ai

21 min read

Rebuilding Data Engineering with Harness Engineering: A New Paradigm for the Agent Era

Chen Debra — Fri, 11 Sep 2026 09:23:45 +0000

As data platforms evolve from serving primarily human users to serving AI agents, the biggest change may not be the tools themselves, but the way data engineering is organized and delivered.

Over the past decade, data engineering has gone through a major wave of specialization. Large, monolithic data platforms have gradually evolved into what is now commonly known as the Modern Data Stack, a composable ecosystem of databases, compute engines, data integration and transformation tools, governance platforms, orchestration systems, and BI solutions. This specialization has dramatically improved engineering efficiency and shifted the industry from building massive, tightly coupled systems toward assembling flexible, modular capabilities.

But as Agentic AI enters the data engineering workflow, a fundamental limitation is becoming increasingly visible: the modern data stack was designed for people, not agents.

The next generation of data platforms will need to solve a different problem. It is no longer enough to help people operate increasingly sophisticated tools. The challenge is to enable agents to execute engineering work within the right business context, technical boundaries, security controls, and governance framework.

This is where Harness Engineering comes in.

The core idea is simple: AI can generate SQL, code, pipelines, and workflows at unprecedented speed. But generating engineering artifacts is not the same as delivering reliable engineering outcomes. A production-ready system needs to make those outputs trusted, verifiable, controlled, recoverable, and accountable.

In other words, the real opportunity in the Agentic AI era is not simply to make AI generate more data engineering. It is to build the engineering system that allows AI-generated work to safely reach production.

1. Data Platforms Are Entering the Agentic Era

Two Different Paths, One Destination

The evolution of major data platforms points toward the same fundamental shift.

Snowflake is moving from Data Warehouse → Data Cloud → AI Work Interface → Enterprise Agent Platform.

Databricks is evolving from Data Lake → Lakehouse → Data + AI Engineering → Agent-ready Execution Platform.

Although their approaches differ, both are converging around four core capabilities:

Context. Capability. Governance. Execution.

The data platform of the future will not simply store and process enterprise data. It will increasingly serve as the infrastructure through which agents understand enterprise context and take action.

Snowflake: The Data Platform Becomes an AI Entry Point

Snowflake's evolution is not simply about adding AI features to a data warehouse. It is about reorganizing data, semantics, governance, applications, and agents around AI.

Its trajectory can be broadly viewed in three stages:

Cloud Data Warehouse
Storage, compute, sharing, and governance

→ AI + Data Platform
AI, data, semantics, and governance

→ Agentic Enterprise Infrastructure
Coco, CoWork, Skills, and Agents

This shift has three important implications.

First, the value of a data platform is expanding from managing data to enabling agents to act on data.

Second, the enterprise AI interface is moving beyond SQL and BI toward natural language and agent-driven workflows.

Third, data is increasingly being understood as AI context, rather than simply something to be stored and queried.

The question is no longer just whether an enterprise can access its data. It is whether its data can provide the context an agent needs to make the right decision and take the right action.

Databricks: Turning Data and AI Engineering into an Agent Runtime

Databricks is taking a similar direction from a different starting point.

The goal is not simply to add AI capabilities to the Lakehouse. Instead, data, models, notebooks, pipelines, governance, and applications are becoming part of an agent-ready engineering environment.

For a data platform to be truly agent-ready, five capabilities matter:

Data must be discoverable.
Schemas must be understandable.
Metrics must have clear business definitions.
Workflows must be executable.
Actions must be auditable.

Being "agent-ready" is therefore much more than allowing a model to access data. It means making data, semantics, compute, governance, and execution work together for agents.

The User Is Changing: From Humans to Agents

The more fundamental change is happening on the user side.

Traditional data platforms were built primarily for:

Data Engineers
Data Analysts
BI Users
Platform Engineers

The next generation will increasingly serve:

Coding Agents
Data Agents
Business Agents
Operations Agents

These users have fundamentally different needs.

Human users need UIs, documentation, and guided workflows.

Agents need APIs, Skills, Context, Policies, and structured Feedback.

That difference has architectural consequences. A platform designed around human interaction cannot simply expose more APIs and expect to become agent-native. The underlying engineering model needs to change.

The Core Question Is Changing

The contrast between the past decade and the next one is becoming increasingly clear.

Over the past ten years, we built data platforms that helped people operate tools: write SQL, configure pipelines, build DAGs, inspect logs, and troubleshoot failed jobs.

Over the next decade, the goal will be to build data engineering platforms where people define the outcome and agents orchestrate the work.

The workflow shifts from:

Write SQL → Build Pipeline → Configure DAG → Monitor → Fix

to:

Understand Intent → Plan → Invoke Capabilities → Execute → Validate → Learn

The central question for data platforms is therefore changing from:

How do we make tools easier for people to operate?

to:

How do we enable agents to execute engineering work within the right context and security boundaries?

Looking back, this shift follows a familiar pattern.

The Database Era focused on storage, queries, and transactions.

The Big Data Platform Era focused on scale, distributed computing, and large-scale data processing.

The Modern Data Stack Era introduced cloud infrastructure, modular architectures, and standardized tools.

Now, with agents becoming a new class of data platform user, we are entering the Agentic Data Stack Era, where Context, Skills, Control, and Harness become first-class engineering concerns.

2. The Limits of the Modern Data Stack

First, Recognize What It Got Right

Before talking about what comes next, it is important to recognize how successful the Modern Data Stack has been.

Over the past decade, it addressed many of the biggest challenges in enterprise data infrastructure.

The old model relied on heavy projects, specialized hardware, extensive customization, long delivery cycles, and tightly coupled systems.

The Modern Data Stack introduced modular engineering, cloud resources, standardized tools, and composability.

It transformed data engineering from building one large system into assembling a set of specialized capabilities.

Instead of procurement, deployment, customization, and tightly coupled integration, teams could combine Cloud, Open Source, SaaS, and Standard APIs like building blocks.

That brought something the traditional data platform could not offer at the same scale: freedom to choose, evolve, replace, and recombine individual components.

Its most important contribution was arguably tool standardization.

Data engineering was decomposed into a professional toolchain:

Source
Business systems, SaaS, APIs

→ Ingestion
Synchronization, CDC, files

→ Storage
Warehouses, lakehouses, compute

→ Transformation
SQL, models, testing

→ Orchestration
DAGs, scheduling, retries

→ Governance
Catalog, lineage, permissions

→ BI
Dashboards and self-service analytics

Each layer became more specialized, and that specialization dramatically improved engineering productivity.

But There Is a Fundamental Limitation

The problem is also hidden in that success:

The Modern Data Stack was designed for humans, not agents.

When humans use data tools, they fill in missing context almost automatically.

They read documentation. They understand business terminology. They resolve ambiguity. They recognize risk. They know when something looks suspicious. And, critically, they understand who is accountable for the outcome.

Agents do not automatically have that context.

When an agent encounters SQL, documentation, DAGs, logs, business rules, and data catalogs, it cannot simply assume that the missing information is obvious.

This reveals an uncomfortable truth about today's supposedly automated data platforms:

Much of their "automation" still depends on humans silently filling in the gaps.

AI Is Amplifying the Problem

Generative AI is changing the economics of software generation.

The cost of producing SQL, code, DAGs, and configuration is rapidly approaching zero.

As a result, the scarce part of data engineering is shifting.

SQL, code, DAGs, and configuration are becoming commodities.

What remains scarce is:

Context. Verification. Governance. Controlled Execution. Accountability.

The old scarce skills were:

Write SQL → Build Pipelines → Configure DAGs

The emerging scarce skills are:

Provide Context → Verify Results → Govern Actions → Control Execution

The real challenge in the AI era is therefore not generating data engineering artifacts.

It is safely delivering those artifacts into production.

This is why the conversation is shifting.

The question is no longer simply:

Can AI generate code?

It is increasingly:

How does AI-generated work become production-ready?

For engineering teams, the real concern is not that AI can write SQL. It is what happens after the SQL has been written.

Who verifies it?

Who approves it?

Who owns the outcome?

Who is responsible if it is wrong?

Three problems become particularly important.

Complexity. There are already enough tools. Will agents create even more hidden dependencies, one-off scripts, and temporary workflows?

Accountability. If an agent generates SQL, ETL, or a DAG, who confirms that it is correct? Who approves it? Who owns the result?

Production risk. The most dangerous scenario is not necessarily that an agent writes incorrect SQL. It is that the incorrect SQL runs successfully.

And beneath these concerns are six additional engineering requirements:

Validation, Ownership, Lineage, Security, Rollback, and Audit.

AI can dramatically reduce the cost of generation. But without engineering controls, it can also dramatically increase operational complexity.

Data engineering has no meaningful concept of "close enough."

In many AI applications, an inaccurate answer may simply result in a poor user experience.

In data engineering, an incorrect result can flow directly into financial reports, business operations, customer decisions, and automated systems.

A missed CDC event, incorrect metric definition, incomplete dataset, or untraceable transformation can propagate through an entire chain:

Wrong Data → Wrong Decision → Wrong Action

The most dangerous failure is therefore not an agent that fails to execute.

It is an agent that produces the wrong result and successfully executes it in production.

3. Harness Engineering: Turning Generation into Delivery

The answer is an engineering layer between agents and the underlying tools: the Harness.

A Harness provides the controls, context, verification, and recovery mechanisms required to turn AI-generated work into production engineering.

Three Types of Evidence for Production Delivery

A production-grade Harness should provide three types of evidence.

Outcome Evidence

First, prove that the system actually improves delivery rather than simply adding another AI interface.

Does it reduce delivery time rather than simply moving work from execution to review?

Are errors discovered earlier?

Are rework, context switching, and waiting reduced?

Does the team actually complete engineering work faster?

Process Evidence

Next, every step should be explainable, traceable, and recoverable.

Can the boundaries between input, generation, approval, and execution be traced?

When something goes wrong, can the team determine whether the problem originated in Context, Skill, Runtime, or Policy?

Can the system retry, roll back, or hand control back to a human without forcing the team to start over?

Governance Evidence

Finally, high-risk actions must be explicitly constrained rather than implicitly delegated to the model.

Which actions can run automatically?

Which require approval?

Are those rules defined by Policy?

Does human intervention happen only at meaningful decision points?

Can the audit trail answer:

Who did what, when, why, and what happened as a result?

Without these three types of evidence, an agent simply generates content faster.

With them, the agent begins to deliver engineering outcomes.

Seven Sign-off Gates for Production

Before an agent-driven workflow reaches production, organizations should be able to verify seven critical sign-off gates.

These are not product features. They are production-readiness checkpoints.

Missing even one of them can turn a promising demo into an operational risk.

1. Intent can be verified

Goals, boundaries, and acceptance criteria must be captured as structured inputs rather than relying on informal instructions.

2. Context is complete

Business, data, permission, and execution context must be available. The model should not be expected to guess what the enterprise means.

3. The plan can be reviewed

SQL, DAGs, and task steps should be readable by humans, verifiable by systems, and reviewable from a risk perspective.

4. Execution is controlled

Permissions, target environments, execution paths, and Skill selection must operate within defined Policies.

5. Results can be validated

Results should pass data-quality checks, business-definition checks, reconciliation, or lineage validation rather than being accepted simply because execution succeeded.

6. Failures can be recovered

The system must know whether to retry, roll back, or escalate to a human. Failures cannot simply remain unresolved.

7. Actions are auditable

Plans, approvals, executions, results, and version changes should be recorded across the full lifecycle.

These seven gates are not designed to make agents smarter.

They are designed to give organizations a clear answer to a more important question:

When is it safe to delegate?

Three Foundations: Correctness, Capability, and Context

A Harness is only as effective as the three foundations underneath it.

SQL That Runs Is Not Necessarily Business-Correct

In data engineering, successful execution is only the lowest bar.

There are at least four levels of correctness:

Syntactic correctness → Execution correctness → Data correctness → Business correctness

A query can be technically correct and still produce the wrong business result.

Consider revenue.

Should "Revenue" mean:

Order Amount?
Paid Amount?
Recognized Revenue?
Net Revenue?

The query engine can determine whether the SQL is valid.

Only enterprise context can determine whether the calculation is business-correct.

Common failure modes include:

Choosing the wrong data source
Using the wrong metric definition
Applying the wrong time window
Creating duplicates through joins
Ignoring business rules

Without Capability, Agents Generate More Temporary Scripts

There is another risk.

If agents can only interact with raw SQL, Python, Shell, or low-level APIs, they may simply generate more one-off scripts.

Temporary scripts are typically:

One-time implementations
Difficult to standardize
Difficult to audit
Difficult to roll back
Poor at providing structured feedback

Engineering Capabilities are different.

A Capability is a reusable, structured Skill with defined inputs, outputs, policies, validation, and recovery behavior.

The difference is fundamental:

Without Capability, AI simply upgrades "humans writing temporary scripts" into "AI generating temporary scripts."

Without Context, Agents Guess What the Enterprise Means

Enterprise data is not simply a collection of tables and columns.

It is a context system containing business semantics, technical relationships, historical rules, and organizational ownership.

Consider a seemingly simple request:

Calculate revenue from high-value customers over the last 30 days.

An agent needs to know:

How are high-value customers defined?
What revenue definition should be used?
What exactly counts as the last 30 days?
Which data source is authoritative?
How should refunds be handled?
Who owns and approves the result?

These questions correspond to four types of context:

Business Context
Data Context
Execution Context
Organizational Context

Without Context, an agent is not understanding the enterprise.

It is guessing.

Software Itself Needs to Be Rebuilt for Agents

AI is also forcing software companies to answer a broader question:

What does software look like when agents, rather than humans, are its primary users?

Traditional software is UI-first.

A person opens an interface, finds a feature, fills in configuration, clicks Run, and handles exceptions.

Agent-native software needs to become Skill-first.

That means:

Skills are discoverable
Context can be injected
Policies can constrain actions
Feedback can be consumed programmatically
Results can be verified

Established vendors have significant legacy systems to work with. New companies have more freedom to rethink their architectures.

In this environment, the speed at which organizations recognize and respond to the shift may become a major competitive advantage.

Every piece of software that matters to data engineering will need to reconsider how an agent interacts with it.

The conclusion is straightforward:

Agents do not primarily lack intelligence. They lack the engineering system required to turn intelligence into reliable outcomes.

Today's models can already work with:

SQL, ETL, DAGs, Logs, Fixes, and Plans.

But they cannot independently take responsibility for:

Context, Permissions, Validation, Impact, Rollback, and Accountability.

Between Generation Capability and Production Capability sits an entire Engineering System.

The limiting factor for agents is therefore no longer intelligence alone.

It is engineering.

4. The Five-Layer Agentic Data Stack

A next-generation data platform can be organized into five layers:

The architectural principle is critical:

Agents should not directly call underlying tools.

Instead, they should access those capabilities through the Harness, while Context from L3 and Controls from L4 define what the agent is allowed to do.

Breaking Down the Five Layers

L1: Deterministic Execution

Intelligence should not be responsible for determinism.

The Runtime is.

It executes SQL, synchronizes data, runs batch and CDC workloads, schedules jobs, and produces logs and status information.

Typical components include:

Databases and warehouses
Lakehouses and Iceberg
Apache Spark
Apache Flink
Apache SeaTunnel
Apache DolphinScheduler
SQL engines
Data quality engines

L2: The Data Engineering Harness

After an agent understands the goal, generates a plan, and initiates a request, that request should pass through a controlled engineering layer.

A typical Harness includes:

Skill Definition → Permission Check → Context Injection → Validation → Observability → Rollback

Only then should the request become a controlled Skill that can interact with databases, operating systems, development platforms, and cloud services.

Without a Harness, the pattern is:

Direct Scripts → Direct APIs → Difficult Verification → Difficult Auditing

With a Harness, it becomes:

Standardized Skills → Clear Boundaries → Structured Feedback → Human Takeover

The value of the Harness is therefore to transform an agent's ability to generate content into an enterprise's ability to reliably deliver engineering work.

L3: Semantic & Knowledge Layer

The Semantic Layer answers questions such as:

Which table is trusted?
What does this metric actually mean?
Is this field sensitive?
Where did this data come from?
Who will be affected downstream?

It combines:

Business Rules, Metadata, Lineage, Metrics, Glossary, Ontology, Data Contracts, and Execution Memory.

In the BI era, the Semantic Layer primarily helped people understand data.

In the Agentic era, it becomes a context layer that agents must consult before taking action.

Without a Semantic Layer, an agent may understand column names.

It will not necessarily understand the enterprise behind them.

L4: Agentic Orchestration Control Plane

The orchestration layer must evolve beyond simply running DAGs.

Traditional orchestration looks like:

Human defines DAG → Scheduler executes Tasks

Agentic orchestration looks like:

Human defines Goal → Agent plans → Control Plane enforces boundaries → Human intervenes when necessary

The control plane introduces checkpoints such as:

Goal Description
Skill Checking
Policy Checking
Human Gate
Audit
Multi-task Configuration

L5: Business Intent

The biggest shift happens at the top of the stack.

The traditional approach asks people to describe steps:

Synchronize table A to table B.
Write this SQL.
Configure the DAG.
Run it every day at 8 AM.

The Agentic approach asks people to define outcomes:

Generate a daily dataset containing revenue from high-value customers, use the approved revenue definition, do not overwrite production tables, and require human approval if the result changes by more than 5%.

Business intent should therefore cover eight dimensions:

Goal, Data Scope, Time Window, Quality Requirements, Cost Constraints, Risk Boundaries, Approval Conditions, and Acceptance Criteria.

The future of data engineering is not about people describing every step.

It is about people defining goals, boundaries, and acceptance criteria.

Why Re-Layer the Stack?

The next-generation data platform is not simply the old platform with more plugins.

It requires a new division of responsibilities.

The traditional stack is organized around:

Storage → Compute → Orchestration → Governance → BI

The Agentic Data Stack is organized around:

Intent → Control → Semantic → Harness → Runtime

Together, these layers answer four fundamental questions:

Where does the agent get its business goals?

How does it understand enterprise capabilities and meaning?

Which capabilities can it invoke?

Who controls execution, validation, and rollback?

The Agentic era therefore requires data platforms to rethink the relationship between intent, context, capability, control, and execution.

The Minimum Viable Loop

The five-layer architecture comes together through a simple Harness loop:

Intent → Context → Plan → Skill → Execute → Validate → Review → Feedback

Intent defines the business goal, constraints, and acceptance criteria.

Context supplies business, data, execution, and organizational information.

Plan breaks the goal into synchronization, transformation, quality, orchestration, and other tasks.

Skill invokes standardized engineering capabilities.

Execute runs the work deterministically through the Runtime.

Validate checks data results, quality, and business rules.

Review brings humans into critical decisions.

Feedback sends logs, metrics, exceptions, and approval outcomes back to the agent.

This creates a continuous loop rather than a one-shot generation process.

Without the loop, an agent generates content.

With the loop, an agent starts delivering engineering work.

Three Design Principles

A Skill Is Not a Prompt. It Is a Controlled Execution Unit.

A Prompt is an instruction.

A Skill is an engineering capability composed of:

Input, Context, Policy, Execution, Validation, Rollback, and Output.

A Tool API exposes a low-level operation and defines parameters, leaving failure handling to the caller.

An Engineering Skill encapsulates an end-to-end engineering intent, defines its Context and Policy, and includes validation and recovery mechanisms.

The distinction is fundamental:

A Prompt determines how an agent responds. A Skill determines whether an agent can execute safely.

CLI for Agents, GUI for Humans

The interface model also needs to change.

CLI/API should serve execution and feedback:

Structured inputs and outputs
Easy programmatic invocation
Testability
Version control
Skills
MCP
SDKs
Declarative configuration

GUI should serve understanding, review, and governance:

Inspect agent plans and generated artifacts
Review SQL, DAGs, logs, and results
Monitor permissions and risk
Take control when uncertainty or exceptions arise

A complete workflow looks like this:

Human defines the goal through the GUI → Agent invokes Skills through CLI/API → Runtime executes → GUI presents DAGs, SQL, logs, and risk information → Human approves or takes over

Human-in-the-Loop Should Protect Risk Boundaries, Not Approve Everything

Human-in-the-loop does not mean humans should approve every agent action.

Every action should first pass through Policy-based automated screening and then be handled according to its risk level.

A practical model is:

Low risk → Automatic execution

Medium risk → Execute and notify

High risk → Human approval

Critical risk → Block or require dual approval

This leads to three principles:

Intervene based on risk, not every step.

Approve critical decisions, not mechanical actions.

Humans define the Policy; agents operate within it.

For example:

Low risk: reading metadata, querying development environments, generating documentation.

Medium risk: creating development tasks, running low-cost validation.

High risk: writing to production, changing schemas, modifying critical metrics.

Critical risk: deleting core tables, bulk overwrites, or high-risk operations involving sensitive data.

The goal is not to turn humans into approval bottlenecks.

Humans should design the risk boundaries within which agents operate.

Clear Ownership and Feedback-Driven Recovery

Production ownership also needs to be explicit.

Business Sign-off: Business owners define goals, constraints, and acceptance criteria.

Platform Governance: Platform owners define Policies, permissions, approvals, rollback mechanisms, and environment boundaries.

Automated Execution: Agents and the Harness supply Context, generate Plans, invoke Skills, execute through the Runtime, and return logs, validation results, and exceptions.

Audit: Reviewers and audit systems step in for high-risk, uncertain, or acceptance-conflicting decisions and maintain records of critical decisions.

Human intervention should generally be reserved for three situations:

A high-risk action could modify critical data or business state.
An exception cannot be safely recovered and the retry or rollback boundary is unclear.
The result conflicts with the acceptance criteria and requires a final business decision.

The product organizations ultimately need is not a demo that can write SQL.

It is a system where goals, permissions, execution, and outcomes can be managed as one accountable loop.

A truly Agentic system must also be able to learn from execution feedback.

A practical Feedback Loop is:

Plan → Execute → Observe → Diagnose → Repair → Validate → Continue / Rollback / Escalate

This loop is supported by Execution Memory, which continuously captures context, execution history, and operational experience.

Feedback can come from:

Execution state
Logs
Metrics
Data quality
Lineage impact
Human feedback

The agent can then:

Automatically repair configuration, parameters, SQL, or workflow issues when the root cause is clear.

Adjust the plan by changing resources, sequencing, or execution strategies.

Stop or escalate when the system cannot safely resolve the problem.

The key to Agentic systems is therefore not simply autonomous execution.

It is knowing what happened after execution.

5. From Concept to Practice

Case Study: A Complete Data Engineering Loop

A meaningful Agentic data engineering demo should prove more than the ability to generate SQL.

The real test is whether an agent can complete an end-to-end engineering workflow.

Starting from a business goal, the agent should be able to:

Discover data → Create an integration task → Execute synchronization → Generate SQL transformations → Build a workflow DAG → Execute the workflow → Read logs and diagnose issues → Repair and retry → Present the result for human review

In this model:

Agent Planning handles planning.

Harness Control manages permissions, policies, validation, and execution boundaries.

Runtime Execution performs deterministic engineering work.

The value of the demo is therefore not proving that a model can generate SQL.

It is proving that an agent can use a Harness to orchestrate multiple deterministic engineering systems into a complete delivery workflow.

Two Practical Harness Implementations

The theory becomes meaningful when it is reflected in real engineering systems.

Apache SeaTunnel and Apache DolphinScheduler provide two complementary examples of how Harness principles can be applied in open source data engineering.

SeaTunnel represents the data integration capability side: how a foundational data integration engine can evolve into a Skill that agents can discover, invoke, validate, and recover.

DolphinScheduler represents the engineering execution and orchestration side: how agent-generated work can become a real, executable, observable, and reviewable engineering asset.

One manages how data moves.

The other manages how engineering workflows run reliably.

Together, they illustrate how the L1 Runtime and L2 Harness layers can work together in real-world data engineering.

Apache SeaTunnel CLI: Turning Data Integration into a Skill

The Apache SeaTunnel CLI provides capabilities such as:

Data source discovery, including Source, Schema, Table, and Field
Automatic SeaTunnel Job generation
Batch synchronization and CDC execution
Structured execution feedback, including status, logs, row counts, and errors
Error-driven repair and retry

An agent's intent can be translated into a set of SeaTunnel Skills:

DiscoverSource()

InspectSchema()

CreateBatchSync()

CreateCDC()

ValidateMapping()

RunSyncJob()

The Data Integration Runtime then handles the execution loop:

Execute → Logs → Repair → Retry

This is an important architectural shift.

Instead of asking an agent to generate another temporary data integration script, SeaTunnel exposes reusable, structured capabilities that can be incorporated into an agent-driven engineering workflow.

The future direction for SeaTunnel CLI includes:

Context-aware Mapping

Expanding from Batch and CDC operations into a broader Data Flow Skill

Self-healing Data Integration

AI-ready Data Pipelines

The long-term destination of SeaTunnel CLI is therefore not simply a better command-line interface.

It is a Data Integration Skill that agents can reliably discover and use.

Apache DolphinScheduler: Turning Generated Work into Engineering Order

Apache DolphinScheduler provides another critical part of the Harness architecture.

Its capabilities include:

Generating Workflow DAGs
Establishing task dependencies automatically
Creating real engineering assets such as Definitions, Instances, and Versions
Executing and monitoring workflows through status, timing, retries, and logs
Repairing failed workflows and rerunning nodes
Providing a GUI for human review

The shift can be summarized as:

Traditional Workflow

Human defines every Task → Human configures dependencies → Scheduler executes DAG

Agentic Workflow

Human defines business goal → Agent generates execution plan → Policy checks boundaries → DolphinScheduler executes Workflow → Agent repairs issues / Human reviews

This is where orchestration becomes more than task scheduling.

DolphinScheduler provides the engineering structure required to turn generated plans into persistent, observable, and manageable workflow assets.

Its future direction can include:

Policy-aware Orchestration

Human Review Gates

Self-healing Workflows

Multi-Agent Coordination

Data Engineers Are Not Disappearing. Their Role Is Expanding.

The rise of agents does not eliminate data engineering.

It changes where data engineers create value.

The role is evolving through five stages:

SQL Writer
Focused on writing SQL to solve individual problems

→ Pipeline Builder
Configuring data flows and pipelines

→ Workflow Operator
Running and monitoring data workflows

→ Platform Engineer
Building platforms and infrastructure

→ Agent Capability Designer
Designing agent capabilities and the engineering systems behind them

Traditional data engineering work includes writing SQL, Python, and Spark code, configuring connectors and ETL jobs, building DAGs, managing schedules, troubleshooting failures, and documenting data.

The next generation of data engineers will increasingly focus on five types of design:

Context Designer
Design the business and data context that allows agents to understand the enterprise.

Skill Designer
Create reusable data capabilities that agents can reliably invoke and combine.

Policy Designer
Define rules and constraints that make agent actions controlled and trustworthy.

Evaluation Designer
Design evaluation criteria and mechanisms that make agent performance measurable and sustainable.

Agent Engineering Commander
Coordinate the broader engineering system and delivery process to amplify the capabilities of data teams.

Six core capabilities will remain essential:

Data modeling, business abstraction, architecture design, data governance, risk judgment, and accountability.

These are not becoming less important.

They are becoming the foundation for designing Context, Skills, and Policies that agents can actually use.

The most valuable data engineers of the future will therefore not necessarily be the people who build the most pipelines.

They will be the people who know how to organize data engineering capabilities so that agents can use them safely and effectively.

Start with High-Frequency, Low-Risk Work

Organizations should not begin Harness adoption by attempting to automate an entire data pipeline.

The better approach is to start with high-frequency, low-risk tasks where boundaries are clear and outcomes can be verified.

Prove that the system can deliver reliably within a well-defined scope, then gradually expand its autonomy.

Four categories are particularly suitable for the first phase:

Data discovery, metadata enrichment, and schema understanding
These are generally read-heavy tasks with limited write risk.

SQL drafting, rule validation, and DAG assembly
These follow a "generate first, review second" model.

Integration task creation, parameter orchestration, and environment checks
These are repetitive engineering tasks that can be standardized and templated.

Log diagnosis, repair recommendations, and retry orchestration
These form operational loops that can be replayed and verified.

By contrast, organizations should avoid fully autonomous execution for high-risk tasks such as:

Deleting, overwriting, or bulk-modifying production data
Changing critical metric definitions or restructuring cross-domain master data
Schema changes or high-cost writes without approval and rollback mechanisms
Complex cross-team workflows where ownership is unclear or results cannot be automatically verified

A practical Harness adoption path can be divided into three stages.

Collaborative Assistance

Agents discover metadata, generate drafts, and provide recommendations while humans review the results.

The goal is to teach the system how to operate under verification.

Controlled Execution

Harness-managed agents execute non-critical, reversible, and verifiable tasks.

Governed Autonomy

Only after Policies, auditing, validation, and rollback mechanisms are mature should organizations expand the agent's autonomous authority.

The principle is simple:

Start small. Prove reliability. Expand the boundary of autonomy.

Conclusion: The Future Is Trusted Agentic Data Engineering

The future of data engineering is not about removing humans from the loop.

It is about changing what humans do.

Humans define the goal.
Agents execute the work.
Harness Engineering governs the delivery.

The model can be summarized as:

Business Intent + Agent Intelligence + Enterprise Context + Engineering Skills + Policy & Control + Human Review = Trusted Agentic Data Engineering

The fundamental shift is not simply about better prompts, larger context windows, or more capable models.

It is about building an engineering system around those models that enables agents to operate continuously, safely, and accountably.

The agent generates.
The Runtime executes.
The Harness makes the outcome trustworthy.

That is the deeper transformation taking place in data engineering.

The next generation of data platforms will not be defined simply by how many tools they integrate or how much code their AI can generate. They will be defined by how effectively they connect business intent, enterprise context, engineering capabilities, execution controls, verification, and human accountability into one continuous delivery loop.

And that may be the real architecture of data engineering in the Agentic AI era.

🚀 Apache DolphinScheduler’s August updates bring stronger security, smarter missed-fire handling, performance gains, and critical bug fixes. Explore the changes! #DolphinScheduler #Apache #DataOps

Chen Debra — Fri, 11 Sep 2026 09:13:12 +0000

Chen Debra

Sep 11

What’s New in Apache DolphinScheduler This August: Stronger Security, Smarter Scheduling, and Better Stability

#apachedolphinscheduler #datascience #dataengineering #github

10 min read

What’s New in Apache DolphinScheduler This August: Stronger Security, Smarter Scheduling, and Better Stability

Chen Debra — Fri, 11 Sep 2026 09:12:54 +0000

The Apache DolphinScheduler August Monthly Report is here! Over the past month, the community continued to move the project forward with new improvements across functionality, performance, stability, and ecosystem development. Let’s take a closer look at the updates that stood out in August. And as always, a huge thank-you to everyone who contributed code, shared feedback, and supported the community. Every contribution helps DolphinScheduler continue to evolve!

📊 August at a Glance

Metric	Value
🚀 PRs Merged	23
👥 Contributors	10
➕ Lines Added	+3,702
➖ Lines Deleted	-1,327
🔀 Net Lines Changed	+2,375
📁 Modules Touched	8
📝 Documentation Files Touched	5
🧪 Test Files Touched	28

🏆 Top Contributors

Rank	GitHub Username	Primary Contribution Area	PRs	+Lines	-Lines	Overall Score
🥇	@ruanwenjun	Tests	10	1602	1001	88.59
🥈	@njnu-seafish	Performance	3	818	288	29.09
🥉	@SEPURI-SAI-KRISHNA	Debugging & Fixes	2	229	3	18.77
4.	@kittimzhe	Documentation	2	4	4	14.02
5.	@nikhiln64	Debugging & Fixes	1	344	8	10.16
6.	@hellodml	Debugging & Fixes	1	136	1	9.45
7.	@zhang-arvin	Debugging & Fixes	1	10	3	9.04
8.	@liang-wenjie	Tests	1	542	7	7.82
9.	@hiSandog	Tests	1	17	1	6.06
10.	@SbloodyS	Architecture & Engineering	1	0	11	6.01

🔄 Code Changes

Breakdown by Category: Features / Performance / Bug Fixes / Architecture

Category	PRs	Share
Features	6	26.1%
Performance Improvements	2	8.7%
Bug Fixes	9	39.1%
Architecture Improvements	6	26.1%

Bug fixes accounted for the largest share of this month’s changes, at 39.1% (9 PRs). Combined with 6 feature PRs and 2 performance improvements, the overall development rhythm was clear: stability first, with a steady stream of new capabilities.

📦 Key Modules

Ranked by the number of PR touches.

Rank	Module	PR Touches	Lines Added	Lines Deleted	Net Change
1.	`dolphinscheduler-api`	82	+2,132	-1,016	+1,116
2.	`dolphinscheduler-dao`	25	+307	-201	+106
3.	`docs`	14	+68	-48	+20
4.	`dolphinscheduler-ui`	10	+74	-27	+47
5.	`dolphinscheduler-master`	9	+374	-12	+362
6.	`dolphinscheduler-common`	6	+64	-6	+58
7.	`dolphinscheduler-scheduler-plugin`	6	+229	-4	+225
8.	`dolphinscheduler-task-plugin`	4	+305	-2	+303
9.	`misc`	3	+149	-11	+138

What the module breakdown tells us:

🏛️ API (15 touches) was the clear focus of this month’s changes, with 15/23 PRs involving the API module. Most of the work centered on strengthening authorization and removing obsolete APIs.
🏗️ DAO (7 touches) mainly supported the API changes, including query optimization by excluding large text fields and updates to authorization query logic.
🎨 UI (5 touches) saw several touchpoints, but most were relatively small changes (+49 net lines), serving primarily as supporting updates.
🔧 Master (4 touches) focused on scheduling stability, including task retries and failure recovery.

🎯 8 Changes Users Will Notice Most

The following updates are ranked by their potential user impact and focus on the changes that matter most in real-world deployments.

1. 🔐 Stronger Permissions and Security — 7 PRs

Representative PR: #18561 — [Fix-18559][API] Align workflow mutations with project write permissions (#18561)

Author: @ruanwenjun
Change size: +426 / -128 lines
PRs in this category this month: 7

Why it matters: Tenant isolation is now stricter, sensitive operations are better protected, and the overall security and compliance posture is stronger. Seven PRs this month focused on strengthening the permission model, covering project write-permission checks, cross-project authorization for sub-workflows, datasource and cluster authorization, Actuator endpoint authentication, user-list data masking, and authorization API optimization. These changes are particularly important for enterprise and multi-tenant deployments.

Key scenarios to verify:

✅ When creating, updating, or deleting a workflow, does the system strictly verify project-level write permissions?
✅ When referencing a sub-workflow, can the system verify that the user also has permission to access the referenced workflow, preventing unauthorized access across projects?
✅ When a task definition references a datasource, does the system verify that the current user has access to that datasource?
✅ Do cluster query APIs enforce permission checks consistently?
✅ Are user-list responses properly masked, with permissions appropriately restricted?
✅ Do sensitive Actuator endpoints require authentication before they can be accessed?

Full list of related PRs:

PR #	Title	Author	Diff
#18561	[Fix-18559][API] Align workflow mutations with project write permissions (#18561)	@ruanwenjun	+426/-128
#18597	[Fix-18596][API] Enforce permission checks for sub-workflow references (#18597)	@ruanwenjun	+450/-0
#18566	[Fix-18565][API] Validate datasource access for task definitions (#18566)	@ruanwenjun	+342/-17
#18564	[Improvement-18563][API] Refine datasource authorization list APIs (#18564)	@ruanwenjun	+102/-62
#18583	[Fix-18582][Authentication] Align actuator endpoint matching (#18583)	@ruanwenjun	+149/-11
#18560	[Improvement-18558][API] Harden user list access and responses (#18560)	@ruanwenjun	+65/-28
#18590	[Improvement-18589][API] Align cluster query permissions (#18590)	@ruanwenjun	+59/-20

2. 🛡️ Stability and Bug Fixes — 4 PRs

Representative PR: #18573 — [Fix-18570][Master] Detect wrapped CommandDuplicateHandleException in bootstrapError (#18570) (#18573)

Author: @hellodml
Change size: +136 / -1 lines
PRs in this category this month: 4

Why it matters: These fixes address abnormal workflow and task states, directly improving reliability in production. There were 9 bug-fix PRs in total this month, making bug fixing the largest category. The other bug-fix PRs are covered in dedicated sections for permissions, K8s, DataX, and documentation. The four core fixes below focus on issues outside those categories.

Key scenarios to verify:

✅ Failed task retries: When recreating a failed task instance, is the runtime state correctly reset to prevent stale state from causing retry failures?
✅ Retry timing: Is the retry scheduled based on endTime + retryInterval rather than startTime?
✅ Master startup failures: Can bootstrapError correctly identify a CommandDuplicateHandleException after it has been wrapped?
✅ Log messages: Has the typo in DataSourceServiceImpl log messages been corrected to avoid confusion during troubleshooting?

Full list of related PRs:

PR #	Title	Author	Diff
#18573	[Fix-18570][Master] Detect wrapped CommandDuplicateHandleException in bootstrapError (#18570) (#18573)	@hellodml	+136/-1
#18541	[Fix-18540][Master] Reset the runtime state when recreating a failed task instance (#18541)	@SEPURI-SAI-KRISHNA	+132/-0
#18539	[Fix-18538][Master] Schedule task retry at endTime + retryInterval (#18539)	@SEPURI-SAI-KRISHNA	+97/-3
#18581	[Improvement-18580][api] Fix typo in DataSourceServiceImpl log message (#18581)	@kittimzhe	+1/-1

3. ⏰ Scheduling Policies / Missed-Fire Handling — 1 PR

Representative PR: #18464 — [DSIP-18454][Scheduler] Add schedule missed fire policy (#18464)

Author: @liang-wenjie
Change size: +542 / -7 lines
PRs in this category this month: 1

Policy	Behavior	Best For
Skip	Skip missed triggers and wait for the next cron trigger	When missed runs should simply be skipped to avoid putting additional load on the system
FireOnceNow	Trigger only once immediately and discard the remaining missed runs	When only the most recent missed run needs to be executed
FireAll	Execute all missed triggers sequentially according to their original scheduled times	Financial, reconciliation, and other scenarios where every scheduled run must be executed

Why it matters: This is the biggest new feature of the month. If the Master is down or the scheduling thread is blocked and cron triggers are missed, you can now configure one of three policies to determine how DolphinScheduler handles those missed triggers.

Key scenarios to verify:

✅ When creating or editing a schedule, does the UI display the Missed Fire Policy dropdown?
✅ Simulate a two-hour Master outage by stopping the Master process and starting it again. Do the different policies behave as expected?
✅ Does the database upgrade script (3.5.0_schema) execute correctly, and does the t_ds_schedule table contain the newly added field?

4. ⚡ Performance Improvements — 2 PRs

Representative PR: #18444 — [Improvement-18443][API&DAO] Optimize WorkflowInstanceMapper to exclude large text fields from list queries (#18444)

Author: @njnu-seafish
Change size: +790 / -268 lines
PRs in this category this month: 2

Why it matters: Workflow-instance and task-instance list queries no longer fetch large TEXT fields such as global_params and process_instance_json unnecessarily. This can significantly improve list-page response times, database I/O, and memory usage, with particularly noticeable benefits in large-scale deployments.

Key scenarios to verify:

✅ Compare workflow-instance list-page response times before and after the optimization, especially with 1,000+ instances.
✅ Open an individual workflow instance and verify that large fields such as global parameters are still displayed correctly. The detail API continues to retrieve them.
✅ Verify that list-page export, search, and other functions continue to work as expected.

Full list of related PRs:

PR #	Title	Author	Diff
#18444	[Improvement-18443][API&DAO] Optimize WorkflowInstanceMapper to exclude large text fields from list queries (#18444)	@njnu-seafish	+790/-268
#18442	[Improvement-18441][API&DAO] Optimize TaskInstanceMapper to exclude large text fields from list queries (#18442)	@njnu-seafish	+17/-5

5. 🔌 Task Types and New Data Sources — 1 PR

Representative PR: #18434 — [Fix-18389][DataX] Read job definition from attached resource file when custom json is empty (#18434)

Author: @nikhiln64
Change size: +344 / -8 lines
PRs in this category this month: 1

Why it matters: Previously, DataX users had to paste JSON content into the custom JSON field in the UI and could not reuse files from the Resource Center. The new behavior allows DolphinScheduler to automatically read the job definition from a resource file when the custom JSON field is empty, bringing the experience in line with how SQL tasks load SQL from resource files.

6. 🧹 API Cleanup and Engineering Improvements — 5 PRs

Representative PR: #18569 — [Improvement-18568][API] Remove obsolete task update-with-upstream API (#18569)

Author: @ruanwenjun
Change size: +2 / -507 lines
PRs in this category this month: 5

Why it matters: This month, the community removed three groups of obsolete APIs covering cluster query-by-code, task update-with-upstream, and dynamic sub-workflow functionality, while also updating incompatible.md.

For users upgrading from an earlier version, this is an important area to review. If your organization has a custom frontend or automation scripts that rely on any of these legacy APIs, make sure to update them before upgrading.

Full list of related PRs:

PR #	Title	Author	Diff
#18569	[Improvement-18568][API] Remove obsolete task update-with-upstream API (#18569)	@ruanwenjun	+2/-507
#18557	[Improvement-18556][API] Remove obsolete dynamic sub-workflow API (#18557)	@ruanwenjun	+2/-138
#18584	[Chore][API] Remove obsolete cluster query-by-code API (#18584)	@ruanwenjun	+5/-90
#18408	[Chore][Common] Handle parentless paths in FileUtils (#18408)	@hiSandog	+17/-1
#18599	[Chore] Remove unused code- #18599 (#18599)	@SbloodyS	+0/-11

7. ☸️ K8s and Cloud-Native Deployment — 1 PR

Representative PR: #18574 — [Fix-17883] Fix K8s Alert HTTP test sending failed by using IP for non-StatefulSet pods (#17883) (#18574)

Author: @zhang-arvin
Change size: +10 / -3 lines
PRs in this category this month: 1

Why it matters: For non-StatefulSet Pods in K8s, such as Pods managed by a Deployment, HTTP alert-instance testing could previously fail because the endpoint was resolved using the hostname. The fix switches to IP-based addressing, improving the reliability of alert channels in K8s deployments.

8. 📚 Documentation and Example Improvements — 2 PRs

Representative PR: #18478 — [Doc-18474][Upgrade] Fix zh/en incompatible upgrade docs out of sync (#18478)

Author: @njnu-seafish
Change size: +11 / -15 lines
PRs in this category this month: 2

Why it matters: Broken links in the datasource and configuration documentation were fixed, while the Chinese and English upgrade-incompatibility documentation was brought back into alignment. These updates help reduce confusion and prevent issues caused by outdated or inconsistent documentation.

Full list of related PRs:

PR #	Title	Author	Diff
#18478	[Doc-18474][Upgrade] Fix zh/en incompatible upgrade docs out of sync (#18478)	@njnu-seafish	+11/-15
#18578	[Doc-18579] Fix malformed links in datasource and configuration docs (#18578)	@kittimzhe	+3/-3

⚠️ Upgrade and Validation Recommendations

Risk Assessment

Overall Risk Level: 🟠 Medium-High

Risk Area	Assessment	Details
Core Module Changes	⚠️ Yes	The API module was touched by 15/23 PRs; permission-related APIs require particular attention during regression testing
Number of Bug Fixes	9	A relatively high number of bug fixes; pay close attention to task retry and failure-recovery scenarios
Permission/Security Changes	7	⚠️ Significant changes to the permission model require comprehensive authorization testing
K8s/Deployment Changes	1	⚠️ Deployment-related logic was changed; Helm/Docker deployments should be verified
UI Changes	Yes	⚠️ Frontend changes require smoke testing of key pages
Database Schema	⚠️ Yes	DSIP-18464 adds new DDL; verify that the upgrade script is executed correctly

Before You Upgrade

📦 Back up the database: Before upgrading, create a full backup of the DolphinScheduler metadata database using mysqldump or pg_dump.
📋 Check the DDL scripts: Verify that the new scripts under dolphinscheduler-dao/src/main/resources/sql/upgrade/3.5.0_schema/ are included in the upgrade process.
📝 Review incompatible changes: Pay particular attention to the four groups of API removals marked this month in docs/docs/en/guide/upgrade/incompatible.md.
🧪 Check custom integrations: If you use OpenAPI, search your codebase to make sure none of these three removed APIs are still being called:

cluster/query-by-code (PR #18584)
task/update-with-upstream (PR #18569)
Dynamic sub-workflow APIs (PR #18557)

💾 Back up configuration files: Back up all configuration files under the conf/ directory.

Common Issues and Quick Checks

Symptom	Possible Cause	What to Check
A sudden 403 when saving a workflow	PR #18561 introduced stricter permission checks	Verify that the current user has write permission for the target project and that the upstream workflow referenced by the sub-workflow is also within the user's authorized scope
A sub-workflow cannot be referenced	PR #18597 added cross-project permission checks	Confirm that the user has the required permissions for the project containing the sub-workflow
Binding a datasource returns 403	PR #18566 added datasource authorization checks	Grant the user access to the corresponding datasource under Datasource Authorization
HTTP alert testing fails in a K8s deployment	Logic changed by PR #18574	Check Pod network policies and verify in the logs whether the connection is being made using a hostname or an IP address
Missed-fire behavior is not what you expected after upgrading	DSIP-18464 defaults to the Skip policy	Check the value of `missed_fire_policy`; switch the policy manually if missed runs need to be executed
Task retries trigger immediately instead of respecting the interval	PR #18539 changed the calculation to use `endTime`	Verify that retry time is calculated as task end time + retryInterval, rather than start time
Workflow-instance lists load slowly or return errors after upgrading	VO fields no longer match after large-field exclusions	Check whether the frontend version was upgraded at the same time, or roll back temporarily to isolate the changes from #18444/#18442

🙏 Thank You to Our Contributors

A huge thank-you to the 10 contributors who contributed code to Apache DolphinScheduler in August 2026:

@ruanwenjun, @njnu-seafish, @SEPURI-SAI-KRISHNA, @kittimzhe, @nikhiln64, @hellodml, @zhang-arvin, @liang-wenjie, @hiSandog, and @SbloodyS

Every PR helps DolphinScheduler continue to evolve in stability, usability, and ecosystem growth. 💪