DEV Community: Chen Debra

Modern Data Stack was built for humans 🚀 Harness Engineering makes agent‑driven data work production‑safe. #DataEngineering #AgenticAI #HarnessEngineering

Chen Debra — Fri, 11 Sep 2026 09:25:39 +0000

Chen Debra

Sep 11

Rebuilding Data Engineering with Harness Engineering: A New Paradigm for the Agent Era

#dataengineering #harnessengineering #agents #ai

21 min read

Rebuilding Data Engineering with Harness Engineering: A New Paradigm for the Agent Era

Chen Debra — Fri, 11 Sep 2026 09:23:45 +0000

As data platforms evolve from serving primarily human users to serving AI agents, the biggest change may not be the tools themselves, but the way data engineering is organized and delivered.

Over the past decade, data engineering has gone through a major wave of specialization. Large, monolithic data platforms have gradually evolved into what is now commonly known as the Modern Data Stack, a composable ecosystem of databases, compute engines, data integration and transformation tools, governance platforms, orchestration systems, and BI solutions. This specialization has dramatically improved engineering efficiency and shifted the industry from building massive, tightly coupled systems toward assembling flexible, modular capabilities.

But as Agentic AI enters the data engineering workflow, a fundamental limitation is becoming increasingly visible: the modern data stack was designed for people, not agents.

The next generation of data platforms will need to solve a different problem. It is no longer enough to help people operate increasingly sophisticated tools. The challenge is to enable agents to execute engineering work within the right business context, technical boundaries, security controls, and governance framework.

This is where Harness Engineering comes in.

The core idea is simple: AI can generate SQL, code, pipelines, and workflows at unprecedented speed. But generating engineering artifacts is not the same as delivering reliable engineering outcomes. A production-ready system needs to make those outputs trusted, verifiable, controlled, recoverable, and accountable.

In other words, the real opportunity in the Agentic AI era is not simply to make AI generate more data engineering. It is to build the engineering system that allows AI-generated work to safely reach production.

1. Data Platforms Are Entering the Agentic Era

Two Different Paths, One Destination

The evolution of major data platforms points toward the same fundamental shift.

Snowflake is moving from Data Warehouse → Data Cloud → AI Work Interface → Enterprise Agent Platform.

Databricks is evolving from Data Lake → Lakehouse → Data + AI Engineering → Agent-ready Execution Platform.

Although their approaches differ, both are converging around four core capabilities:

Context. Capability. Governance. Execution.

The data platform of the future will not simply store and process enterprise data. It will increasingly serve as the infrastructure through which agents understand enterprise context and take action.

Snowflake: The Data Platform Becomes an AI Entry Point

Snowflake's evolution is not simply about adding AI features to a data warehouse. It is about reorganizing data, semantics, governance, applications, and agents around AI.

Its trajectory can be broadly viewed in three stages:

Cloud Data Warehouse
Storage, compute, sharing, and governance

→ AI + Data Platform
AI, data, semantics, and governance

→ Agentic Enterprise Infrastructure
Coco, CoWork, Skills, and Agents

This shift has three important implications.

First, the value of a data platform is expanding from managing data to enabling agents to act on data.

Second, the enterprise AI interface is moving beyond SQL and BI toward natural language and agent-driven workflows.

Third, data is increasingly being understood as AI context, rather than simply something to be stored and queried.

The question is no longer just whether an enterprise can access its data. It is whether its data can provide the context an agent needs to make the right decision and take the right action.

Databricks: Turning Data and AI Engineering into an Agent Runtime

Databricks is taking a similar direction from a different starting point.

The goal is not simply to add AI capabilities to the Lakehouse. Instead, data, models, notebooks, pipelines, governance, and applications are becoming part of an agent-ready engineering environment.

For a data platform to be truly agent-ready, five capabilities matter:

Data must be discoverable.
Schemas must be understandable.
Metrics must have clear business definitions.
Workflows must be executable.
Actions must be auditable.

Being "agent-ready" is therefore much more than allowing a model to access data. It means making data, semantics, compute, governance, and execution work together for agents.

The User Is Changing: From Humans to Agents

The more fundamental change is happening on the user side.

Traditional data platforms were built primarily for:

Data Engineers
Data Analysts
BI Users
Platform Engineers

The next generation will increasingly serve:

Coding Agents
Data Agents
Business Agents
Operations Agents

These users have fundamentally different needs.

Human users need UIs, documentation, and guided workflows.

Agents need APIs, Skills, Context, Policies, and structured Feedback.

That difference has architectural consequences. A platform designed around human interaction cannot simply expose more APIs and expect to become agent-native. The underlying engineering model needs to change.

The Core Question Is Changing

The contrast between the past decade and the next one is becoming increasingly clear.

Over the past ten years, we built data platforms that helped people operate tools: write SQL, configure pipelines, build DAGs, inspect logs, and troubleshoot failed jobs.

Over the next decade, the goal will be to build data engineering platforms where people define the outcome and agents orchestrate the work.

The workflow shifts from:

Write SQL → Build Pipeline → Configure DAG → Monitor → Fix

to:

Understand Intent → Plan → Invoke Capabilities → Execute → Validate → Learn

The central question for data platforms is therefore changing from:

How do we make tools easier for people to operate?

to:

How do we enable agents to execute engineering work within the right context and security boundaries?

Looking back, this shift follows a familiar pattern.

The Database Era focused on storage, queries, and transactions.

The Big Data Platform Era focused on scale, distributed computing, and large-scale data processing.

The Modern Data Stack Era introduced cloud infrastructure, modular architectures, and standardized tools.

Now, with agents becoming a new class of data platform user, we are entering the Agentic Data Stack Era, where Context, Skills, Control, and Harness become first-class engineering concerns.

2. The Limits of the Modern Data Stack

First, Recognize What It Got Right

Before talking about what comes next, it is important to recognize how successful the Modern Data Stack has been.

Over the past decade, it addressed many of the biggest challenges in enterprise data infrastructure.

The old model relied on heavy projects, specialized hardware, extensive customization, long delivery cycles, and tightly coupled systems.

The Modern Data Stack introduced modular engineering, cloud resources, standardized tools, and composability.

It transformed data engineering from building one large system into assembling a set of specialized capabilities.

Instead of procurement, deployment, customization, and tightly coupled integration, teams could combine Cloud, Open Source, SaaS, and Standard APIs like building blocks.

That brought something the traditional data platform could not offer at the same scale: freedom to choose, evolve, replace, and recombine individual components.

Its most important contribution was arguably tool standardization.

Data engineering was decomposed into a professional toolchain:

Source
Business systems, SaaS, APIs

→ Ingestion
Synchronization, CDC, files

→ Storage
Warehouses, lakehouses, compute

→ Transformation
SQL, models, testing

→ Orchestration
DAGs, scheduling, retries

→ Governance
Catalog, lineage, permissions

→ BI
Dashboards and self-service analytics

Each layer became more specialized, and that specialization dramatically improved engineering productivity.

But There Is a Fundamental Limitation

The problem is also hidden in that success:

The Modern Data Stack was designed for humans, not agents.

When humans use data tools, they fill in missing context almost automatically.

They read documentation. They understand business terminology. They resolve ambiguity. They recognize risk. They know when something looks suspicious. And, critically, they understand who is accountable for the outcome.

Agents do not automatically have that context.

When an agent encounters SQL, documentation, DAGs, logs, business rules, and data catalogs, it cannot simply assume that the missing information is obvious.

This reveals an uncomfortable truth about today's supposedly automated data platforms:

Much of their "automation" still depends on humans silently filling in the gaps.

AI Is Amplifying the Problem

Generative AI is changing the economics of software generation.

The cost of producing SQL, code, DAGs, and configuration is rapidly approaching zero.

As a result, the scarce part of data engineering is shifting.

SQL, code, DAGs, and configuration are becoming commodities.

What remains scarce is:

Context. Verification. Governance. Controlled Execution. Accountability.

The old scarce skills were:

Write SQL → Build Pipelines → Configure DAGs

The emerging scarce skills are:

Provide Context → Verify Results → Govern Actions → Control Execution

The real challenge in the AI era is therefore not generating data engineering artifacts.

It is safely delivering those artifacts into production.

This is why the conversation is shifting.

The question is no longer simply:

Can AI generate code?

It is increasingly:

How does AI-generated work become production-ready?

For engineering teams, the real concern is not that AI can write SQL. It is what happens after the SQL has been written.

Who verifies it?

Who approves it?

Who owns the outcome?

Who is responsible if it is wrong?

Three problems become particularly important.

Complexity. There are already enough tools. Will agents create even more hidden dependencies, one-off scripts, and temporary workflows?

Accountability. If an agent generates SQL, ETL, or a DAG, who confirms that it is correct? Who approves it? Who owns the result?

Production risk. The most dangerous scenario is not necessarily that an agent writes incorrect SQL. It is that the incorrect SQL runs successfully.

And beneath these concerns are six additional engineering requirements:

Validation, Ownership, Lineage, Security, Rollback, and Audit.

AI can dramatically reduce the cost of generation. But without engineering controls, it can also dramatically increase operational complexity.

Data engineering has no meaningful concept of "close enough."

In many AI applications, an inaccurate answer may simply result in a poor user experience.

In data engineering, an incorrect result can flow directly into financial reports, business operations, customer decisions, and automated systems.

A missed CDC event, incorrect metric definition, incomplete dataset, or untraceable transformation can propagate through an entire chain:

Wrong Data → Wrong Decision → Wrong Action

The most dangerous failure is therefore not an agent that fails to execute.

It is an agent that produces the wrong result and successfully executes it in production.

3. Harness Engineering: Turning Generation into Delivery

The answer is an engineering layer between agents and the underlying tools: the Harness.

A Harness provides the controls, context, verification, and recovery mechanisms required to turn AI-generated work into production engineering.

Three Types of Evidence for Production Delivery

A production-grade Harness should provide three types of evidence.

Outcome Evidence

First, prove that the system actually improves delivery rather than simply adding another AI interface.

Does it reduce delivery time rather than simply moving work from execution to review?

Are errors discovered earlier?

Are rework, context switching, and waiting reduced?

Does the team actually complete engineering work faster?

Process Evidence

Next, every step should be explainable, traceable, and recoverable.

Can the boundaries between input, generation, approval, and execution be traced?

When something goes wrong, can the team determine whether the problem originated in Context, Skill, Runtime, or Policy?

Can the system retry, roll back, or hand control back to a human without forcing the team to start over?

Governance Evidence

Finally, high-risk actions must be explicitly constrained rather than implicitly delegated to the model.

Which actions can run automatically?

Which require approval?

Are those rules defined by Policy?

Does human intervention happen only at meaningful decision points?

Can the audit trail answer:

Who did what, when, why, and what happened as a result?

Without these three types of evidence, an agent simply generates content faster.

With them, the agent begins to deliver engineering outcomes.

Seven Sign-off Gates for Production

Before an agent-driven workflow reaches production, organizations should be able to verify seven critical sign-off gates.

These are not product features. They are production-readiness checkpoints.

Missing even one of them can turn a promising demo into an operational risk.

1. Intent can be verified

Goals, boundaries, and acceptance criteria must be captured as structured inputs rather than relying on informal instructions.

2. Context is complete

Business, data, permission, and execution context must be available. The model should not be expected to guess what the enterprise means.

3. The plan can be reviewed

SQL, DAGs, and task steps should be readable by humans, verifiable by systems, and reviewable from a risk perspective.

4. Execution is controlled

Permissions, target environments, execution paths, and Skill selection must operate within defined Policies.

5. Results can be validated

Results should pass data-quality checks, business-definition checks, reconciliation, or lineage validation rather than being accepted simply because execution succeeded.

6. Failures can be recovered

The system must know whether to retry, roll back, or escalate to a human. Failures cannot simply remain unresolved.

7. Actions are auditable

Plans, approvals, executions, results, and version changes should be recorded across the full lifecycle.

These seven gates are not designed to make agents smarter.

They are designed to give organizations a clear answer to a more important question:

When is it safe to delegate?

Three Foundations: Correctness, Capability, and Context

A Harness is only as effective as the three foundations underneath it.

SQL That Runs Is Not Necessarily Business-Correct

In data engineering, successful execution is only the lowest bar.

There are at least four levels of correctness:

Syntactic correctness → Execution correctness → Data correctness → Business correctness

A query can be technically correct and still produce the wrong business result.

Consider revenue.

Should "Revenue" mean:

Order Amount?
Paid Amount?
Recognized Revenue?
Net Revenue?

The query engine can determine whether the SQL is valid.

Only enterprise context can determine whether the calculation is business-correct.

Common failure modes include:

Choosing the wrong data source
Using the wrong metric definition
Applying the wrong time window
Creating duplicates through joins
Ignoring business rules

Without Capability, Agents Generate More Temporary Scripts

There is another risk.

If agents can only interact with raw SQL, Python, Shell, or low-level APIs, they may simply generate more one-off scripts.

Temporary scripts are typically:

One-time implementations
Difficult to standardize
Difficult to audit
Difficult to roll back
Poor at providing structured feedback

Engineering Capabilities are different.

A Capability is a reusable, structured Skill with defined inputs, outputs, policies, validation, and recovery behavior.

The difference is fundamental:

Without Capability, AI simply upgrades "humans writing temporary scripts" into "AI generating temporary scripts."

Without Context, Agents Guess What the Enterprise Means

Enterprise data is not simply a collection of tables and columns.

It is a context system containing business semantics, technical relationships, historical rules, and organizational ownership.

Consider a seemingly simple request:

Calculate revenue from high-value customers over the last 30 days.

An agent needs to know:

How are high-value customers defined?
What revenue definition should be used?
What exactly counts as the last 30 days?
Which data source is authoritative?
How should refunds be handled?
Who owns and approves the result?

These questions correspond to four types of context:

Business Context
Data Context
Execution Context
Organizational Context

Without Context, an agent is not understanding the enterprise.

It is guessing.

Software Itself Needs to Be Rebuilt for Agents

AI is also forcing software companies to answer a broader question:

What does software look like when agents, rather than humans, are its primary users?

Traditional software is UI-first.

A person opens an interface, finds a feature, fills in configuration, clicks Run, and handles exceptions.

Agent-native software needs to become Skill-first.

That means:

Skills are discoverable
Context can be injected
Policies can constrain actions
Feedback can be consumed programmatically
Results can be verified

Established vendors have significant legacy systems to work with. New companies have more freedom to rethink their architectures.

In this environment, the speed at which organizations recognize and respond to the shift may become a major competitive advantage.

Every piece of software that matters to data engineering will need to reconsider how an agent interacts with it.

The conclusion is straightforward:

Agents do not primarily lack intelligence. They lack the engineering system required to turn intelligence into reliable outcomes.

Today's models can already work with:

SQL, ETL, DAGs, Logs, Fixes, and Plans.

But they cannot independently take responsibility for:

Context, Permissions, Validation, Impact, Rollback, and Accountability.

Between Generation Capability and Production Capability sits an entire Engineering System.

The limiting factor for agents is therefore no longer intelligence alone.

It is engineering.

4. The Five-Layer Agentic Data Stack

A next-generation data platform can be organized into five layers:

The architectural principle is critical:

Agents should not directly call underlying tools.

Instead, they should access those capabilities through the Harness, while Context from L3 and Controls from L4 define what the agent is allowed to do.

Breaking Down the Five Layers

L1: Deterministic Execution

Intelligence should not be responsible for determinism.

The Runtime is.

It executes SQL, synchronizes data, runs batch and CDC workloads, schedules jobs, and produces logs and status information.

Typical components include:

Databases and warehouses
Lakehouses and Iceberg
Apache Spark
Apache Flink
Apache SeaTunnel
Apache DolphinScheduler
SQL engines
Data quality engines

L2: The Data Engineering Harness

After an agent understands the goal, generates a plan, and initiates a request, that request should pass through a controlled engineering layer.

A typical Harness includes:

Skill Definition → Permission Check → Context Injection → Validation → Observability → Rollback

Only then should the request become a controlled Skill that can interact with databases, operating systems, development platforms, and cloud services.

Without a Harness, the pattern is:

Direct Scripts → Direct APIs → Difficult Verification → Difficult Auditing

With a Harness, it becomes:

Standardized Skills → Clear Boundaries → Structured Feedback → Human Takeover

The value of the Harness is therefore to transform an agent's ability to generate content into an enterprise's ability to reliably deliver engineering work.

L3: Semantic & Knowledge Layer

The Semantic Layer answers questions such as:

Which table is trusted?
What does this metric actually mean?
Is this field sensitive?
Where did this data come from?
Who will be affected downstream?

It combines:

Business Rules, Metadata, Lineage, Metrics, Glossary, Ontology, Data Contracts, and Execution Memory.

In the BI era, the Semantic Layer primarily helped people understand data.

In the Agentic era, it becomes a context layer that agents must consult before taking action.

Without a Semantic Layer, an agent may understand column names.

It will not necessarily understand the enterprise behind them.

L4: Agentic Orchestration Control Plane

The orchestration layer must evolve beyond simply running DAGs.

Traditional orchestration looks like:

Human defines DAG → Scheduler executes Tasks

Agentic orchestration looks like:

Human defines Goal → Agent plans → Control Plane enforces boundaries → Human intervenes when necessary

The control plane introduces checkpoints such as:

Goal Description
Skill Checking
Policy Checking
Human Gate
Audit
Multi-task Configuration

L5: Business Intent

The biggest shift happens at the top of the stack.

The traditional approach asks people to describe steps:

Synchronize table A to table B.
Write this SQL.
Configure the DAG.
Run it every day at 8 AM.

The Agentic approach asks people to define outcomes:

Generate a daily dataset containing revenue from high-value customers, use the approved revenue definition, do not overwrite production tables, and require human approval if the result changes by more than 5%.

Business intent should therefore cover eight dimensions:

Goal, Data Scope, Time Window, Quality Requirements, Cost Constraints, Risk Boundaries, Approval Conditions, and Acceptance Criteria.

The future of data engineering is not about people describing every step.

It is about people defining goals, boundaries, and acceptance criteria.

Why Re-Layer the Stack?

The next-generation data platform is not simply the old platform with more plugins.

It requires a new division of responsibilities.

The traditional stack is organized around:

Storage → Compute → Orchestration → Governance → BI

The Agentic Data Stack is organized around:

Intent → Control → Semantic → Harness → Runtime

Together, these layers answer four fundamental questions:

Where does the agent get its business goals?

How does it understand enterprise capabilities and meaning?

Which capabilities can it invoke?

Who controls execution, validation, and rollback?

The Agentic era therefore requires data platforms to rethink the relationship between intent, context, capability, control, and execution.

The Minimum Viable Loop

The five-layer architecture comes together through a simple Harness loop:

Intent → Context → Plan → Skill → Execute → Validate → Review → Feedback

Intent defines the business goal, constraints, and acceptance criteria.

Context supplies business, data, execution, and organizational information.

Plan breaks the goal into synchronization, transformation, quality, orchestration, and other tasks.

Skill invokes standardized engineering capabilities.

Execute runs the work deterministically through the Runtime.

Validate checks data results, quality, and business rules.

Review brings humans into critical decisions.

Feedback sends logs, metrics, exceptions, and approval outcomes back to the agent.

This creates a continuous loop rather than a one-shot generation process.

Without the loop, an agent generates content.

With the loop, an agent starts delivering engineering work.

Three Design Principles

A Skill Is Not a Prompt. It Is a Controlled Execution Unit.

A Prompt is an instruction.

A Skill is an engineering capability composed of:

Input, Context, Policy, Execution, Validation, Rollback, and Output.

A Tool API exposes a low-level operation and defines parameters, leaving failure handling to the caller.

An Engineering Skill encapsulates an end-to-end engineering intent, defines its Context and Policy, and includes validation and recovery mechanisms.

The distinction is fundamental:

A Prompt determines how an agent responds. A Skill determines whether an agent can execute safely.

CLI for Agents, GUI for Humans

The interface model also needs to change.

CLI/API should serve execution and feedback:

Structured inputs and outputs
Easy programmatic invocation
Testability
Version control
Skills
MCP
SDKs
Declarative configuration

GUI should serve understanding, review, and governance:

Inspect agent plans and generated artifacts
Review SQL, DAGs, logs, and results
Monitor permissions and risk
Take control when uncertainty or exceptions arise

A complete workflow looks like this:

Human defines the goal through the GUI → Agent invokes Skills through CLI/API → Runtime executes → GUI presents DAGs, SQL, logs, and risk information → Human approves or takes over

Human-in-the-Loop Should Protect Risk Boundaries, Not Approve Everything

Human-in-the-loop does not mean humans should approve every agent action.

Every action should first pass through Policy-based automated screening and then be handled according to its risk level.

A practical model is:

Low risk → Automatic execution

Medium risk → Execute and notify

High risk → Human approval

Critical risk → Block or require dual approval

This leads to three principles:

Intervene based on risk, not every step.

Approve critical decisions, not mechanical actions.

Humans define the Policy; agents operate within it.

For example:

Low risk: reading metadata, querying development environments, generating documentation.

Medium risk: creating development tasks, running low-cost validation.

High risk: writing to production, changing schemas, modifying critical metrics.

Critical risk: deleting core tables, bulk overwrites, or high-risk operations involving sensitive data.

The goal is not to turn humans into approval bottlenecks.

Humans should design the risk boundaries within which agents operate.

Clear Ownership and Feedback-Driven Recovery

Production ownership also needs to be explicit.

Business Sign-off: Business owners define goals, constraints, and acceptance criteria.

Platform Governance: Platform owners define Policies, permissions, approvals, rollback mechanisms, and environment boundaries.

Automated Execution: Agents and the Harness supply Context, generate Plans, invoke Skills, execute through the Runtime, and return logs, validation results, and exceptions.

Audit: Reviewers and audit systems step in for high-risk, uncertain, or acceptance-conflicting decisions and maintain records of critical decisions.

Human intervention should generally be reserved for three situations:

A high-risk action could modify critical data or business state.
An exception cannot be safely recovered and the retry or rollback boundary is unclear.
The result conflicts with the acceptance criteria and requires a final business decision.

The product organizations ultimately need is not a demo that can write SQL.

It is a system where goals, permissions, execution, and outcomes can be managed as one accountable loop.

A truly Agentic system must also be able to learn from execution feedback.

A practical Feedback Loop is:

Plan → Execute → Observe → Diagnose → Repair → Validate → Continue / Rollback / Escalate

This loop is supported by Execution Memory, which continuously captures context, execution history, and operational experience.

Feedback can come from:

Execution state
Logs
Metrics
Data quality
Lineage impact
Human feedback

The agent can then:

Automatically repair configuration, parameters, SQL, or workflow issues when the root cause is clear.

Adjust the plan by changing resources, sequencing, or execution strategies.

Stop or escalate when the system cannot safely resolve the problem.

The key to Agentic systems is therefore not simply autonomous execution.

It is knowing what happened after execution.

5. From Concept to Practice

Case Study: A Complete Data Engineering Loop

A meaningful Agentic data engineering demo should prove more than the ability to generate SQL.

The real test is whether an agent can complete an end-to-end engineering workflow.

Starting from a business goal, the agent should be able to:

Discover data → Create an integration task → Execute synchronization → Generate SQL transformations → Build a workflow DAG → Execute the workflow → Read logs and diagnose issues → Repair and retry → Present the result for human review

In this model:

Agent Planning handles planning.

Harness Control manages permissions, policies, validation, and execution boundaries.

Runtime Execution performs deterministic engineering work.

The value of the demo is therefore not proving that a model can generate SQL.

It is proving that an agent can use a Harness to orchestrate multiple deterministic engineering systems into a complete delivery workflow.

Two Practical Harness Implementations

The theory becomes meaningful when it is reflected in real engineering systems.

Apache SeaTunnel and Apache DolphinScheduler provide two complementary examples of how Harness principles can be applied in open source data engineering.

SeaTunnel represents the data integration capability side: how a foundational data integration engine can evolve into a Skill that agents can discover, invoke, validate, and recover.

DolphinScheduler represents the engineering execution and orchestration side: how agent-generated work can become a real, executable, observable, and reviewable engineering asset.

One manages how data moves.

The other manages how engineering workflows run reliably.

Together, they illustrate how the L1 Runtime and L2 Harness layers can work together in real-world data engineering.

Apache SeaTunnel CLI: Turning Data Integration into a Skill

The Apache SeaTunnel CLI provides capabilities such as:

Data source discovery, including Source, Schema, Table, and Field
Automatic SeaTunnel Job generation
Batch synchronization and CDC execution
Structured execution feedback, including status, logs, row counts, and errors
Error-driven repair and retry

An agent's intent can be translated into a set of SeaTunnel Skills:

DiscoverSource()

InspectSchema()

CreateBatchSync()

CreateCDC()

ValidateMapping()

RunSyncJob()

The Data Integration Runtime then handles the execution loop:

Execute → Logs → Repair → Retry

This is an important architectural shift.

Instead of asking an agent to generate another temporary data integration script, SeaTunnel exposes reusable, structured capabilities that can be incorporated into an agent-driven engineering workflow.

The future direction for SeaTunnel CLI includes:

Context-aware Mapping

Expanding from Batch and CDC operations into a broader Data Flow Skill

Self-healing Data Integration

AI-ready Data Pipelines

The long-term destination of SeaTunnel CLI is therefore not simply a better command-line interface.

It is a Data Integration Skill that agents can reliably discover and use.

Apache DolphinScheduler: Turning Generated Work into Engineering Order

Apache DolphinScheduler provides another critical part of the Harness architecture.

Its capabilities include:

Generating Workflow DAGs
Establishing task dependencies automatically
Creating real engineering assets such as Definitions, Instances, and Versions
Executing and monitoring workflows through status, timing, retries, and logs
Repairing failed workflows and rerunning nodes
Providing a GUI for human review

The shift can be summarized as:

Traditional Workflow

Human defines every Task → Human configures dependencies → Scheduler executes DAG

Agentic Workflow

Human defines business goal → Agent generates execution plan → Policy checks boundaries → DolphinScheduler executes Workflow → Agent repairs issues / Human reviews

This is where orchestration becomes more than task scheduling.

DolphinScheduler provides the engineering structure required to turn generated plans into persistent, observable, and manageable workflow assets.

Its future direction can include:

Policy-aware Orchestration

Human Review Gates

Self-healing Workflows

Multi-Agent Coordination

Data Engineers Are Not Disappearing. Their Role Is Expanding.

The rise of agents does not eliminate data engineering.

It changes where data engineers create value.

The role is evolving through five stages:

SQL Writer
Focused on writing SQL to solve individual problems

→ Pipeline Builder
Configuring data flows and pipelines

→ Workflow Operator
Running and monitoring data workflows

→ Platform Engineer
Building platforms and infrastructure

→ Agent Capability Designer
Designing agent capabilities and the engineering systems behind them

Traditional data engineering work includes writing SQL, Python, and Spark code, configuring connectors and ETL jobs, building DAGs, managing schedules, troubleshooting failures, and documenting data.

The next generation of data engineers will increasingly focus on five types of design:

Context Designer
Design the business and data context that allows agents to understand the enterprise.

Skill Designer
Create reusable data capabilities that agents can reliably invoke and combine.

Policy Designer
Define rules and constraints that make agent actions controlled and trustworthy.

Evaluation Designer
Design evaluation criteria and mechanisms that make agent performance measurable and sustainable.

Agent Engineering Commander
Coordinate the broader engineering system and delivery process to amplify the capabilities of data teams.

Six core capabilities will remain essential:

Data modeling, business abstraction, architecture design, data governance, risk judgment, and accountability.

These are not becoming less important.

They are becoming the foundation for designing Context, Skills, and Policies that agents can actually use.

The most valuable data engineers of the future will therefore not necessarily be the people who build the most pipelines.

They will be the people who know how to organize data engineering capabilities so that agents can use them safely and effectively.

Start with High-Frequency, Low-Risk Work

Organizations should not begin Harness adoption by attempting to automate an entire data pipeline.

The better approach is to start with high-frequency, low-risk tasks where boundaries are clear and outcomes can be verified.

Prove that the system can deliver reliably within a well-defined scope, then gradually expand its autonomy.

Four categories are particularly suitable for the first phase:

Data discovery, metadata enrichment, and schema understanding
These are generally read-heavy tasks with limited write risk.

SQL drafting, rule validation, and DAG assembly
These follow a "generate first, review second" model.

Integration task creation, parameter orchestration, and environment checks
These are repetitive engineering tasks that can be standardized and templated.

Log diagnosis, repair recommendations, and retry orchestration
These form operational loops that can be replayed and verified.

By contrast, organizations should avoid fully autonomous execution for high-risk tasks such as:

Deleting, overwriting, or bulk-modifying production data
Changing critical metric definitions or restructuring cross-domain master data
Schema changes or high-cost writes without approval and rollback mechanisms
Complex cross-team workflows where ownership is unclear or results cannot be automatically verified

A practical Harness adoption path can be divided into three stages.

Collaborative Assistance

Agents discover metadata, generate drafts, and provide recommendations while humans review the results.

The goal is to teach the system how to operate under verification.

Controlled Execution

Harness-managed agents execute non-critical, reversible, and verifiable tasks.

Governed Autonomy

Only after Policies, auditing, validation, and rollback mechanisms are mature should organizations expand the agent's autonomous authority.

The principle is simple:

Start small. Prove reliability. Expand the boundary of autonomy.

Conclusion: The Future Is Trusted Agentic Data Engineering

The future of data engineering is not about removing humans from the loop.

It is about changing what humans do.

Humans define the goal.
Agents execute the work.
Harness Engineering governs the delivery.

The model can be summarized as:

Business Intent + Agent Intelligence + Enterprise Context + Engineering Skills + Policy & Control + Human Review = Trusted Agentic Data Engineering

The fundamental shift is not simply about better prompts, larger context windows, or more capable models.

It is about building an engineering system around those models that enables agents to operate continuously, safely, and accountably.

The agent generates.
The Runtime executes.
The Harness makes the outcome trustworthy.

That is the deeper transformation taking place in data engineering.

The next generation of data platforms will not be defined simply by how many tools they integrate or how much code their AI can generate. They will be defined by how effectively they connect business intent, enterprise context, engineering capabilities, execution controls, verification, and human accountability into one continuous delivery loop.

And that may be the real architecture of data engineering in the Agentic AI era.

🚀 Apache DolphinScheduler’s August updates bring stronger security, smarter missed-fire handling, performance gains, and critical bug fixes. Explore the changes! #DolphinScheduler #Apache #DataOps

Chen Debra — Fri, 11 Sep 2026 09:13:12 +0000

Chen Debra

Sep 11

What’s New in Apache DolphinScheduler This August: Stronger Security, Smarter Scheduling, and Better Stability

#apachedolphinscheduler #datascience #dataengineering #github

10 min read

What’s New in Apache DolphinScheduler This August: Stronger Security, Smarter Scheduling, and Better Stability

Chen Debra — Fri, 11 Sep 2026 09:12:54 +0000

The Apache DolphinScheduler August Monthly Report is here! Over the past month, the community continued to move the project forward with new improvements across functionality, performance, stability, and ecosystem development. Let’s take a closer look at the updates that stood out in August. And as always, a huge thank-you to everyone who contributed code, shared feedback, and supported the community. Every contribution helps DolphinScheduler continue to evolve!

📊 August at a Glance

Metric	Value
🚀 PRs Merged	23
👥 Contributors	10
➕ Lines Added	+3,702
➖ Lines Deleted	-1,327
🔀 Net Lines Changed	+2,375
📁 Modules Touched	8
📝 Documentation Files Touched	5
🧪 Test Files Touched	28

🏆 Top Contributors

Rank	GitHub Username	Primary Contribution Area	PRs	+Lines	-Lines	Overall Score
🥇	@ruanwenjun	Tests	10	1602	1001	88.59
🥈	@njnu-seafish	Performance	3	818	288	29.09
🥉	@SEPURI-SAI-KRISHNA	Debugging & Fixes	2	229	3	18.77
4.	@kittimzhe	Documentation	2	4	4	14.02
5.	@nikhiln64	Debugging & Fixes	1	344	8	10.16
6.	@hellodml	Debugging & Fixes	1	136	1	9.45
7.	@zhang-arvin	Debugging & Fixes	1	10	3	9.04
8.	@liang-wenjie	Tests	1	542	7	7.82
9.	@hiSandog	Tests	1	17	1	6.06
10.	@SbloodyS	Architecture & Engineering	1	0	11	6.01

🔄 Code Changes

Breakdown by Category: Features / Performance / Bug Fixes / Architecture

Category	PRs	Share
Features	6	26.1%
Performance Improvements	2	8.7%
Bug Fixes	9	39.1%
Architecture Improvements	6	26.1%

Bug fixes accounted for the largest share of this month’s changes, at 39.1% (9 PRs). Combined with 6 feature PRs and 2 performance improvements, the overall development rhythm was clear: stability first, with a steady stream of new capabilities.

📦 Key Modules

Ranked by the number of PR touches.

Rank	Module	PR Touches	Lines Added	Lines Deleted	Net Change
1.	`dolphinscheduler-api`	82	+2,132	-1,016	+1,116
2.	`dolphinscheduler-dao`	25	+307	-201	+106
3.	`docs`	14	+68	-48	+20
4.	`dolphinscheduler-ui`	10	+74	-27	+47
5.	`dolphinscheduler-master`	9	+374	-12	+362
6.	`dolphinscheduler-common`	6	+64	-6	+58
7.	`dolphinscheduler-scheduler-plugin`	6	+229	-4	+225
8.	`dolphinscheduler-task-plugin`	4	+305	-2	+303
9.	`misc`	3	+149	-11	+138

What the module breakdown tells us:

🏛️ API (15 touches) was the clear focus of this month’s changes, with 15/23 PRs involving the API module. Most of the work centered on strengthening authorization and removing obsolete APIs.
🏗️ DAO (7 touches) mainly supported the API changes, including query optimization by excluding large text fields and updates to authorization query logic.
🎨 UI (5 touches) saw several touchpoints, but most were relatively small changes (+49 net lines), serving primarily as supporting updates.
🔧 Master (4 touches) focused on scheduling stability, including task retries and failure recovery.

🎯 8 Changes Users Will Notice Most

The following updates are ranked by their potential user impact and focus on the changes that matter most in real-world deployments.

1. 🔐 Stronger Permissions and Security — 7 PRs

Representative PR: #18561 — [Fix-18559][API] Align workflow mutations with project write permissions (#18561)

Author: @ruanwenjun
Change size: +426 / -128 lines
PRs in this category this month: 7

Why it matters: Tenant isolation is now stricter, sensitive operations are better protected, and the overall security and compliance posture is stronger. Seven PRs this month focused on strengthening the permission model, covering project write-permission checks, cross-project authorization for sub-workflows, datasource and cluster authorization, Actuator endpoint authentication, user-list data masking, and authorization API optimization. These changes are particularly important for enterprise and multi-tenant deployments.

Key scenarios to verify:

✅ When creating, updating, or deleting a workflow, does the system strictly verify project-level write permissions?
✅ When referencing a sub-workflow, can the system verify that the user also has permission to access the referenced workflow, preventing unauthorized access across projects?
✅ When a task definition references a datasource, does the system verify that the current user has access to that datasource?
✅ Do cluster query APIs enforce permission checks consistently?
✅ Are user-list responses properly masked, with permissions appropriately restricted?
✅ Do sensitive Actuator endpoints require authentication before they can be accessed?

Full list of related PRs:

PR #	Title	Author	Diff
#18561	[Fix-18559][API] Align workflow mutations with project write permissions (#18561)	@ruanwenjun	+426/-128
#18597	[Fix-18596][API] Enforce permission checks for sub-workflow references (#18597)	@ruanwenjun	+450/-0
#18566	[Fix-18565][API] Validate datasource access for task definitions (#18566)	@ruanwenjun	+342/-17
#18564	[Improvement-18563][API] Refine datasource authorization list APIs (#18564)	@ruanwenjun	+102/-62
#18583	[Fix-18582][Authentication] Align actuator endpoint matching (#18583)	@ruanwenjun	+149/-11
#18560	[Improvement-18558][API] Harden user list access and responses (#18560)	@ruanwenjun	+65/-28
#18590	[Improvement-18589][API] Align cluster query permissions (#18590)	@ruanwenjun	+59/-20

2. 🛡️ Stability and Bug Fixes — 4 PRs

Representative PR: #18573 — [Fix-18570][Master] Detect wrapped CommandDuplicateHandleException in bootstrapError (#18570) (#18573)

Author: @hellodml
Change size: +136 / -1 lines
PRs in this category this month: 4

Why it matters: These fixes address abnormal workflow and task states, directly improving reliability in production. There were 9 bug-fix PRs in total this month, making bug fixing the largest category. The other bug-fix PRs are covered in dedicated sections for permissions, K8s, DataX, and documentation. The four core fixes below focus on issues outside those categories.

Key scenarios to verify:

✅ Failed task retries: When recreating a failed task instance, is the runtime state correctly reset to prevent stale state from causing retry failures?
✅ Retry timing: Is the retry scheduled based on endTime + retryInterval rather than startTime?
✅ Master startup failures: Can bootstrapError correctly identify a CommandDuplicateHandleException after it has been wrapped?
✅ Log messages: Has the typo in DataSourceServiceImpl log messages been corrected to avoid confusion during troubleshooting?

Full list of related PRs:

PR #	Title	Author	Diff
#18573	[Fix-18570][Master] Detect wrapped CommandDuplicateHandleException in bootstrapError (#18570) (#18573)	@hellodml	+136/-1
#18541	[Fix-18540][Master] Reset the runtime state when recreating a failed task instance (#18541)	@SEPURI-SAI-KRISHNA	+132/-0
#18539	[Fix-18538][Master] Schedule task retry at endTime + retryInterval (#18539)	@SEPURI-SAI-KRISHNA	+97/-3
#18581	[Improvement-18580][api] Fix typo in DataSourceServiceImpl log message (#18581)	@kittimzhe	+1/-1

3. ⏰ Scheduling Policies / Missed-Fire Handling — 1 PR

Representative PR: #18464 — [DSIP-18454][Scheduler] Add schedule missed fire policy (#18464)

Author: @liang-wenjie
Change size: +542 / -7 lines
PRs in this category this month: 1

Policy	Behavior	Best For
Skip	Skip missed triggers and wait for the next cron trigger	When missed runs should simply be skipped to avoid putting additional load on the system
FireOnceNow	Trigger only once immediately and discard the remaining missed runs	When only the most recent missed run needs to be executed
FireAll	Execute all missed triggers sequentially according to their original scheduled times	Financial, reconciliation, and other scenarios where every scheduled run must be executed

Why it matters: This is the biggest new feature of the month. If the Master is down or the scheduling thread is blocked and cron triggers are missed, you can now configure one of three policies to determine how DolphinScheduler handles those missed triggers.

Key scenarios to verify:

✅ When creating or editing a schedule, does the UI display the Missed Fire Policy dropdown?
✅ Simulate a two-hour Master outage by stopping the Master process and starting it again. Do the different policies behave as expected?
✅ Does the database upgrade script (3.5.0_schema) execute correctly, and does the t_ds_schedule table contain the newly added field?

4. ⚡ Performance Improvements — 2 PRs

Representative PR: #18444 — [Improvement-18443][API&DAO] Optimize WorkflowInstanceMapper to exclude large text fields from list queries (#18444)

Author: @njnu-seafish
Change size: +790 / -268 lines
PRs in this category this month: 2

Why it matters: Workflow-instance and task-instance list queries no longer fetch large TEXT fields such as global_params and process_instance_json unnecessarily. This can significantly improve list-page response times, database I/O, and memory usage, with particularly noticeable benefits in large-scale deployments.

Key scenarios to verify:

✅ Compare workflow-instance list-page response times before and after the optimization, especially with 1,000+ instances.
✅ Open an individual workflow instance and verify that large fields such as global parameters are still displayed correctly. The detail API continues to retrieve them.
✅ Verify that list-page export, search, and other functions continue to work as expected.

Full list of related PRs:

PR #	Title	Author	Diff
#18444	[Improvement-18443][API&DAO] Optimize WorkflowInstanceMapper to exclude large text fields from list queries (#18444)	@njnu-seafish	+790/-268
#18442	[Improvement-18441][API&DAO] Optimize TaskInstanceMapper to exclude large text fields from list queries (#18442)	@njnu-seafish	+17/-5

5. 🔌 Task Types and New Data Sources — 1 PR

Representative PR: #18434 — [Fix-18389][DataX] Read job definition from attached resource file when custom json is empty (#18434)

Author: @nikhiln64
Change size: +344 / -8 lines
PRs in this category this month: 1

Why it matters: Previously, DataX users had to paste JSON content into the custom JSON field in the UI and could not reuse files from the Resource Center. The new behavior allows DolphinScheduler to automatically read the job definition from a resource file when the custom JSON field is empty, bringing the experience in line with how SQL tasks load SQL from resource files.

6. 🧹 API Cleanup and Engineering Improvements — 5 PRs

Representative PR: #18569 — [Improvement-18568][API] Remove obsolete task update-with-upstream API (#18569)

Author: @ruanwenjun
Change size: +2 / -507 lines
PRs in this category this month: 5

Why it matters: This month, the community removed three groups of obsolete APIs covering cluster query-by-code, task update-with-upstream, and dynamic sub-workflow functionality, while also updating incompatible.md.

For users upgrading from an earlier version, this is an important area to review. If your organization has a custom frontend or automation scripts that rely on any of these legacy APIs, make sure to update them before upgrading.

Full list of related PRs:

PR #	Title	Author	Diff
#18569	[Improvement-18568][API] Remove obsolete task update-with-upstream API (#18569)	@ruanwenjun	+2/-507
#18557	[Improvement-18556][API] Remove obsolete dynamic sub-workflow API (#18557)	@ruanwenjun	+2/-138
#18584	[Chore][API] Remove obsolete cluster query-by-code API (#18584)	@ruanwenjun	+5/-90
#18408	[Chore][Common] Handle parentless paths in FileUtils (#18408)	@hiSandog	+17/-1
#18599	[Chore] Remove unused code- #18599 (#18599)	@SbloodyS	+0/-11

7. ☸️ K8s and Cloud-Native Deployment — 1 PR

Representative PR: #18574 — [Fix-17883] Fix K8s Alert HTTP test sending failed by using IP for non-StatefulSet pods (#17883) (#18574)

Author: @zhang-arvin
Change size: +10 / -3 lines
PRs in this category this month: 1

Why it matters: For non-StatefulSet Pods in K8s, such as Pods managed by a Deployment, HTTP alert-instance testing could previously fail because the endpoint was resolved using the hostname. The fix switches to IP-based addressing, improving the reliability of alert channels in K8s deployments.

8. 📚 Documentation and Example Improvements — 2 PRs

Representative PR: #18478 — [Doc-18474][Upgrade] Fix zh/en incompatible upgrade docs out of sync (#18478)

Author: @njnu-seafish
Change size: +11 / -15 lines
PRs in this category this month: 2

Why it matters: Broken links in the datasource and configuration documentation were fixed, while the Chinese and English upgrade-incompatibility documentation was brought back into alignment. These updates help reduce confusion and prevent issues caused by outdated or inconsistent documentation.

Full list of related PRs:

PR #	Title	Author	Diff
#18478	[Doc-18474][Upgrade] Fix zh/en incompatible upgrade docs out of sync (#18478)	@njnu-seafish	+11/-15
#18578	[Doc-18579] Fix malformed links in datasource and configuration docs (#18578)	@kittimzhe	+3/-3

⚠️ Upgrade and Validation Recommendations

Risk Assessment

Overall Risk Level: 🟠 Medium-High

Risk Area	Assessment	Details
Core Module Changes	⚠️ Yes	The API module was touched by 15/23 PRs; permission-related APIs require particular attention during regression testing
Number of Bug Fixes	9	A relatively high number of bug fixes; pay close attention to task retry and failure-recovery scenarios
Permission/Security Changes	7	⚠️ Significant changes to the permission model require comprehensive authorization testing
K8s/Deployment Changes	1	⚠️ Deployment-related logic was changed; Helm/Docker deployments should be verified
UI Changes	Yes	⚠️ Frontend changes require smoke testing of key pages
Database Schema	⚠️ Yes	DSIP-18464 adds new DDL; verify that the upgrade script is executed correctly

Before You Upgrade

📦 Back up the database: Before upgrading, create a full backup of the DolphinScheduler metadata database using mysqldump or pg_dump.
📋 Check the DDL scripts: Verify that the new scripts under dolphinscheduler-dao/src/main/resources/sql/upgrade/3.5.0_schema/ are included in the upgrade process.
📝 Review incompatible changes: Pay particular attention to the four groups of API removals marked this month in docs/docs/en/guide/upgrade/incompatible.md.
🧪 Check custom integrations: If you use OpenAPI, search your codebase to make sure none of these three removed APIs are still being called:

cluster/query-by-code (PR #18584)
task/update-with-upstream (PR #18569)
Dynamic sub-workflow APIs (PR #18557)

💾 Back up configuration files: Back up all configuration files under the conf/ directory.

Common Issues and Quick Checks

Symptom	Possible Cause	What to Check
A sudden 403 when saving a workflow	PR #18561 introduced stricter permission checks	Verify that the current user has write permission for the target project and that the upstream workflow referenced by the sub-workflow is also within the user's authorized scope
A sub-workflow cannot be referenced	PR #18597 added cross-project permission checks	Confirm that the user has the required permissions for the project containing the sub-workflow
Binding a datasource returns 403	PR #18566 added datasource authorization checks	Grant the user access to the corresponding datasource under Datasource Authorization
HTTP alert testing fails in a K8s deployment	Logic changed by PR #18574	Check Pod network policies and verify in the logs whether the connection is being made using a hostname or an IP address
Missed-fire behavior is not what you expected after upgrading	DSIP-18464 defaults to the Skip policy	Check the value of `missed_fire_policy`; switch the policy manually if missed runs need to be executed
Task retries trigger immediately instead of respecting the interval	PR #18539 changed the calculation to use `endTime`	Verify that retry time is calculated as task end time + retryInterval, rather than start time
Workflow-instance lists load slowly or return errors after upgrading	VO fields no longer match after large-field exclusions	Check whether the frontend version was upgraded at the same time, or roll back temporarily to isolate the changes from #18444/#18442

🙏 Thank You to Our Contributors

A huge thank-you to the 10 contributors who contributed code to Apache DolphinScheduler in August 2026:

@ruanwenjun, @njnu-seafish, @SEPURI-SAI-KRISHNA, @kittimzhe, @nikhiln64, @hellodml, @zhang-arvin, @liang-wenjie, @hiSandog, and @SbloodyS

Every PR helps DolphinScheduler continue to evolve in stability, usability, and ecosystem growth. 💪

🚨 Empty Host, failed task? A DolphinScheduler incident reveals how memory protection can stop task dispatch. Learn how to trace the root cause and fix it. #DolphinScheduler #Apache #DataOps

Chen Debra — Fri, 11 Sep 2026 09:00:32 +0000

Chen Debra

Sep 11

When a DolphinScheduler Task Has No Host: How Memory Protection Can Halt Scheduling

#apachedolphinscheduler #datascience #dataengineering

6 min read

When a DolphinScheduler Task Has No Host: How Memory Protection Can Halt Scheduling

Chen Debra — Fri, 11 Sep 2026 09:00:13 +0000

1. What Happened?

1.1 Alert

At around 4 a.m., a series of scheduling failure alerts started coming in:

scheduler failed
projectName: 数仓平台
processName: 【ods】同步MySQL「每小时」
taskName: OPTIMIZE table_records
taskType: SQL
taskState: FAILURE
taskEndTime: 2026-03-13 04:34:01

1.2 What We Saw in the UI

After logging in to DolphinScheduler and checking the task instances, two things immediately stood out:

Task status: Failed
Task instance Host: Empty

So, what does an empty Host field actually mean?

No Worker node was willing to take the task.

The task never even got the chance to execute. It was rejected before it could be dispatched to a Worker.

2. Troubleshooting

2.1 Check System Resources First

The first question was straightforward: Was the machine running out of resources?

CPU usage was low, so CPU was not the bottleneck.

Memory usage, however, was a different story.

The immediate priority in production is to stop the bleeding, so the Worker was restarted first:

docker restart dolphinscheduler-worker

After the restart, the task was rerun successfully and the business recovered.

But a restart only restores service. It does not explain why the problem happened in the first place. So the investigation continued.

2.2 Check the Worker Logs

Next, we searched the Worker logs for memory-related warnings and errors:

grep -i "memory\|error\|exception" /data/dolphin/worker/logs/dolphinscheduler-worker.xxx.log | head -20

The logs told us exactly where to look:

[WARN] current cpu load average 0.03 is higher than 1.0
       or available memory 0.295 is lower than 0.3
[WARN] current cpu load average 0.03 is higher than 1.0
       or available memory 0.294 is lower than 0.3

The Worker heartbeat runs a resource check every 10 seconds. The available memory ratio had repeatedly dropped to 29.5%, below the configured 30% threshold.

2.3 Check the Master Logs

The Master logs showed a similar picture:

grep -i "overload\|memory\|dispatch" /data/dolphin/master/logs/dolphinscheduler-master.xxx.log | head -20

[WARN] Current available memory percentage 0.297 is too low, reserved.memory=0.3
[WARN] The current server is overload, cannot consumes commands.
[WARN] worker 10.0.1.100:1234 current cpu load average 0.0 is too high
       or available memory 4.57G is too low

The Master-side behavior was now clear:

The Master itself did not have enough available memory and stopped consuming scheduling commands.
Workers were considered overloaded, so the Master stopped dispatching tasks to them.

In other words, the task was not failing because the SQL itself could not run. It was never successfully dispatched to a Worker in the first place.

2.4 Confirm the Memory Situation

We then checked the actual memory usage:

$ free -h
               total   used    free    shared  buff/cache  available
Mem:           15Gi    8.9Gi   482Mi   1.1Gi   5.9Gi       4.9Gi

Available memory was 4.9 GiB out of 15 GiB, or roughly 32%.

That was already dangerously close to the 30% threshold. Once the overnight batch workload caused even a small increase in memory usage, the available memory ratio could easily fall below the protection line.

2.5 Find Out What Was Consuming the Memory

The next step was to identify the biggest memory consumers:

ps -aux --sort=-%mem | head -n 11

The result was straightforward:

Master 4G + Worker 4G + API 1G + Alert 1G + Flink + MySQL + ZooKeeper

Everything was competing for memory on a machine with only 15 GiB of RAM.

3. Root Cause

3.1 What Actually Happened?

DolphinScheduler has a built-in memory protection mechanism:

Metric	Threshold	Result
Available Memory Ratio	< `reserved.memory` (default: 0.3, i.e. 30%)	Master stops consuming commands; Worker rejects tasks
CPU Load	> `max.cpu.load.avg` (default: 1.0)	Same

When the available memory ratio drops below 30%, the protection mechanism kicks in:

Master: "There isn't enough memory, so I won't consume scheduling commands."
Worker: "Memory is too low. Don't send me any more tasks."
Task instance: Host = empty
Result: Task fails

This explains the seemingly strange combination we saw in the UI: a failed task with an empty Host field.

The task was blocked during scheduling and dispatching because both the Master and Worker sides detected insufficient available memory.

3.2 Why Was Memory Running Low?

The machine had only 15 GiB of memory, while the four DolphinScheduler components alone were configured with a combined 10 GiB of JVM heap:

Component	Heap Configuration	Description
Master	`-Xmx4g`	Responsible for scheduling decisions; 4G is far more than it needs
Worker	`-Xmx4g`	Tasks are executed in forked processes, so the Worker itself does not need 4G
API	`-Xmx1g`	Reasonable
Alert	`-Xmx1g`	Reasonable

The default 4 GiB heap settings for the Master and Worker were simply too aggressive for this machine.

In practice, their actual RSS usage was only around 1–2 GiB each, meaning a significant amount of memory was reserved without being actively used.

On a resource-constrained host, those JVM heap reservations left too little headroom for the rest of the stack and made the system much more likely to trigger DolphinScheduler's memory protection mechanism.

4. How to Fix It

There are three possible approaches, listed in recommended order.

Option 1: Reduce the Master and Worker JVM Heap Size — Recommended

Add the following settings to the environment section of docker-compose.yml:

dolphinscheduler-master:
  environment:
    - JAVA_OPTS=-Xms2g -Xmx2g -Xmn1g

dolphinscheduler-worker:
  environment:
    - JAVA_OPTS=-Xms2g -Xmx2g -Xmn1g

This is the preferred approach because it addresses the underlying resource allocation issue instead of simply weakening the protection mechanism.

Option 2: Lower the Memory Protection Threshold

The default reserved.memory is 0.3, meaning DolphinScheduler reserves 30% of available memory as a safety margin.

It can be lowered to 10%:

dolphinscheduler-master:
  environment:
    - MASTER_RESERVED_MEMORY=0.1

dolphinscheduler-worker:
  environment:
    - WORKER_RESERVED_MEMORY=0.1

However, this is more of a workaround than a fundamental fix.

If the machine is genuinely short on memory, lowering the threshold does not create more memory. It simply allows the system to continue running under tighter resource conditions, which increases the risk of an OOM.

Option 3: Set `mem_limit` for Containers

You can also place explicit memory limits on individual containers to prevent one service from consuming too much memory and starving other components:

dolphinscheduler-master:
  mem_limit: 3g

dolphinscheduler-worker:
  mem_limit: 3g

This provides stronger resource isolation between containers, but it should be configured carefully according to the actual workload and memory requirements of each component.

5. Verification

In this case, we chose Option 1 and reduced the Master and Worker heap size from 4 GiB to 2 GiB.

After restarting the services, we verified the results.

5.1 Memory Availability Improved

$ free -h
               total   used    free    shared  buff/cache  available
Mem:           15Gi    6.8Gi   2.6Gi   1.1Gi   5.9Gi       7.1Gi

Available memory increased from 4.9 GiB to 7.1 GiB.

The available-memory ratio increased from approximately 32% to 47%, giving the system a much larger safety margin above the 30% protection threshold.

5.2 Check Actual Container Memory Usage

$ docker stats --no-stream dolphinscheduler-master dolphinscheduler-worker
CONTAINER   NAME                     MEM USAGE / LIMIT    MEM %
aec302...   dolphinscheduler-master  1.098GiB / 15.34GiB  7.16%
cd6df5...   dolphinscheduler-worker  1.049GiB / 15.34GiB  6.84%

The numbers looked much healthier:

Master: 1.1 GiB actual memory usage, with a 2 GiB heap limit
Worker: 1.05 GiB actual memory usage, with a 2 GiB heap limit

Both components had sufficient headroom for normal workloads.

5.3 Verify the JVM Parameters

Finally, we confirmed that the new JVM parameters were actually applied:

$ docker exec dolphinscheduler-master ps -ef | grep java | head -1
/opt/java/openjdk/bin/java -Xms2g -Xmx2g -Xmn1g ... MasterServer

The parameters were in effect, and the logs were back to normal. The previous memory warnings were no longer appearing.

6. Takeaways

Item	Before Optimization	After Optimization
Master/Worker Heap	4G each	2G each
Available System Memory	4.9G (32%)	7.1G (47%)
Memory Protection Threshold	Frequently Triggered	Well Above the Threshold
Scheduling Status	Nighttime Downtime	Operating Normally

There are four practical lessons from this incident:

DolphinScheduler's default JVM heap settings may be too large for resource-constrained machines. Always size the heap according to the available hardware and actual workload.
reserved.memory=0.3 is a double-edged sword. It protects the system from running completely out of memory, but it can also prevent tasks from being dispatched when available memory falls below the safety threshold.
An empty Host field does not necessarily mean a network problem. In this case, it was a sign that the task could not be dispatched because DolphinScheduler's memory protection mechanism rejected the overloaded nodes.
In production, restore service first, then investigate, and finally fix the underlying issue. Restart to stop the immediate impact, inspect the logs to identify the root cause, and adjust the resource configuration to prevent the problem from happening again.

🚀 This September, the Apache DolphinScheduler Meetup is back! Build an enterprise data warehouse with Apache DolphinScheduler! Join us Sept 15 at 8 PM (UTC+8).

Chen Debra — Thu, 03 Sep 2026 09:43:05 +0000

Chen Debra

Sep 3

More Than Workflow Scheduling: How Apache DolphinScheduler Powers Enterprise Data Warehouse Projects

#apachedolphinscheduler #datawarehouse #datascience #opensource

4 min read

More Than Workflow Scheduling: How Apache DolphinScheduler Powers Enterprise Data Warehouse Projects

Chen Debra — Thu, 03 Sep 2026 08:09:27 +0000

Building a data warehouse is about far more than simply storing data.

As enterprise data volumes continue to grow and business scenarios become increasingly complex, traditional big data architectures are facing mounting challenges around latency, performance, resource utilization, and operational costs. At the same time, the entire data lifecycle—from business requirements and data synchronization to data cleansing, processing, production, analytics, and operations—still involves a wide range of complex engineering challenges.

How can open source technologies help enterprises build a modern data warehouse that is easy to use, highly reliable, and built to scale?

This September, Apache DolphinScheduler Meetup is back!

This edition of the community meetup features Zan Liu, Data Analytics Engineer and Senior Big Data Architect, who will share “Building an Enterprise-Grade Data Warehouse with Apache DolphinScheduler.” Drawing on real-world engineering experience, Zan will break down how to use Apache DolphinScheduler as the core foundation for end-to-end workflow orchestration and production operations, combined with a distributed, high-performance OLAP data warehouse, to build a modern enterprise data warehouse.

If you are working on data warehousing, data development, workflow orchestration, OLAP, data operations, or enterprise data platforms, this is a session worth saving the date for.

One Session, the Full Story of Building an Enterprise Data Warehouse

Traditional big data solutions often face a number of practical challenges when deployed in enterprise environments.

Offline batch processing typically comes with minute-level latency, making it increasingly difficult to meet the real-time requirements of modern business scenarios. Tightly coupled compute and storage also make it difficult for enterprises to scale resources independently based on actual needs. Adding more storage or compute capacity separately can even result in lower utilization of the other resource.

At the same time, the complexity of the modern big data ecosystem means more components to manage, higher learning costs, and stricter requirements for version compatibility. For enterprise data teams, connecting these technologies into a stable and efficient data platform is often more important than simply choosing any individual component.

Even when an enterprise has accumulated massive volumes of business data, there is still a long list of challenges between a business requirement being raised and a data task running reliably in production. These include data synchronization, data cleansing, workflow orchestration, application integration, and end-to-end operations.

So, what does it really take to build a modern enterprise data warehouse?

That is the question this session will explore.

Based on hands-on engineering experience, Zan Liu will walk through the complete implementation of an enterprise-grade modern data warehouse, with Apache DolphinScheduler serving as the core foundation for end-to-end workflow orchestration and production operations, combined with a distributed, high-performance OLAP data warehouse.

Starting with business requirements and continuing through data synchronization, data production, data applications, task scheduling, and operations, the session will show how open source technologies can be brought together across the entire data lifecycle to create a complete data production loop.

Even more importantly, the entire solution is built on an open source ecosystem. Beyond the architecture itself, the session will focus on the practical challenges that engineers actually encounter when putting such a solution into production.

By bringing workflow orchestration, data processing, and data analytics together, the goal is to lower the barriers to data development and operations, improve the reliability and scalability of enterprise data platforms, and explore a more flexible approach to enterprise data operations.

🎤 Meet the Speaker

Zan Liu，Data Analytics Engineer | Senior Big Data Architect

With extensive experience in data warehousing and big data, Zan specializes in BI development and low-code Agent development with Dify. He has extensive hands-on experience in enterprise data platform development and data application implementation.

📌 Session Topic

“Building an Enterprise-Grade Data Warehouse with Apache DolphinScheduler”

In this session, you’ll learn:

What challenges do traditional big data architectures face in real-world enterprise deployments?
How can Apache DolphinScheduler be used to build an end-to-end workflow orchestration system?
How can a distributed, high-performance OLAP data warehouse be combined with workflow orchestration to build a modern enterprise data warehouse?
How can the entire process—from business requirements to data production and data applications—be connected end to end?
How can data synchronization, cleansing, scheduling, and operations come together to form a unified data production workflow?
How can enterprises build a data platform that is easy to use, highly reliable, and scalable with an open source ecosystem?

From architecture design to real-world implementation, this time we’re going beyond “which technologies should you choose?” and focusing on “how do you actually make them work in production?”

🗓️ Livestream Details

Date & Time: September 15, 2026, at 8:00 PM(China Standard Time, UTC+8)
Save Your Spot: https://meeting.tencent.com/dm/A0ISviQSgbjM

If you’re building an enterprise data warehouse or looking for a more efficient and reliable workflow solution for your data platform, join us for the livestream and see how a modern enterprise data warehouse can be built step by step with an open source technology stack.

🎁 Exclusive Giveaways During the Livestream

Along with the technical insights, this edition of Meetup also comes with exclusive Apache DolphinScheduler swag.

Join the livestream not only for practical insights and real-world experience, but also for a chance to take home a DolphinScheduler keychain!

September 15 at 8:00 PM. See you in the livestream!

🚨 Think DolphinScheduler is down? Prometheus + Grafana can alert you before your manager does. ☕📊 #DolphinScheduler #Prometheus #Grafana

Chen Debra — Thu, 03 Sep 2026 07:59:08 +0000

Chen Debra

Sep 3

Think DolphinScheduler Is Down? You’ll Know with Prometheus + Grafana

#apachedolphinscheduler #prometheus #grafana #ai

13 min read

Think DolphinScheduler Is Down? You’ll Know with Prometheus + Grafana

Chen Debra — Thu, 03 Sep 2026 07:58:06 +0000

Introduction

Ever had this happen? DolphinScheduler quietly goes down, your workflows stop running, and you’re sitting at your desk enjoying your coffee—until your manager suddenly asks, “Why hasn’t the data been updated yet?”

At that moment, your face probably looks just like the production service: completely stuck.

In our previous article, When Workflows Get Stuck: Troubleshooting and Preventing Task Deadlocks in Apache DolphinScheduler, we discussed an uncomfortable reality: if DolphinScheduler goes down because of blocking or a deadlock, it may not be able to alert you about the problem itself. After all, you can’t exactly expect a scheduler that has already “gone offline” to send you a farewell message.

We looked around and found surprisingly few complete, practical guides for integrating DolphinScheduler with Prometheus monitoring.

So this article fills that gap.

We’ll walk through a lightweight DolphinScheduler + Prometheus + Grafana monitoring setup that has been validated in production, covering the entire process from metric collection and alerting rules to dashboard visualization.

By the end, you should be able to find out when DolphinScheduler is in trouble—before someone else does.

1. Architecture Overview

Let’s start with the big picture. The entire monitoring pipeline can be broken down into three simple steps:

Step	Component	Responsibility
1	DolphinScheduler	Exposes the `/actuator/prometheus` metrics endpoint
2	Prometheus	Periodically scrapes metrics and evaluates alerting rules
3	Grafana	Provides visual dashboards for an at-a-glance view of scheduler health

In simple terms:

DolphinScheduler collects the vital signs, Prometheus monitors them, and Grafana turns the data into something you can actually see.

With the three working together, you can finally enjoy your coffee with a little more peace of mind.

This time, for real.

2. Integrating Prometheus

2.1 Verify That DolphinScheduler Exposes Metrics

DolphinScheduler 2.0.0 and later include a built-in Prometheus metrics endpoint, so no additional plugin is required.

You can verify it directly with curl:

curl http://<dolphin-host>:12345/dolphinscheduler/actuator/prometheus

If the response contains a bunch of metrics beginning with ds_, congratulations—DolphinScheduler is ready to be monitored.

If you get a 404, your version may be too old. Upgrade DolphinScheduler first and then try again.

2.2 Configure Prometheus to Scrape DolphinScheduler

Add the following scrape configuration to prometheus.yml:

scrape_configs:
  # Host monitoring (optional; used to monitor the server running DolphinScheduler)
  - job_name: 'node-exporter'
    static_configs:
      - targets:
          - '<dolphin-host>:9100'
        labels:
          instance: 'dolphin-server'
          nodename: 'dolphin-server'

  # DolphinScheduler metrics
  - job_name: 'dolphinscheduler'
    static_configs:
      - targets: ['<dolphin-host>:12345']
        labels:
          service: dolphinscheduler
    metrics_path: '/dolphinscheduler/actuator/prometheus'
    metric_relabel_configs:
      - source_labels: [application]
        target_label: app
      - regex: 'application'
        action: labeldrop

Tip: Replace <dolphin-host> with the actual address of your DolphinScheduler service. If you’re running DolphinScheduler in a cluster, add all relevant nodes.

2.3 Configure Alerting Rules

This is where things get serious.

What’s the point of monitoring if nobody gets notified when something goes wrong?

Create an alert rule file named dolphinscheduler-rules.yml. The following 10 rules have been validated in production and cover scenarios ranging from “DolphinScheduler is down” to “a workflow has gone off the rails.”

# DolphinScheduler monitoring alert rules
groups:
  - name: dolphinscheduler-alerts
    rules:

      # ============ Core rule: service availability ============

      # Rule 1: No successful tasks in the past hour (likely unavailable)
      - alert: DolphinSchedulerNoSuccessfulTasksForOneHour
        expr: increase(ds_task_instance_count_total{state="success"}[1h]) == 0
        for: 5m
        labels:
          severity: critical
          group: dolphinscheduler
        annotations:
          summary: "No successful DolphinScheduler tasks in the past hour"
          description: "Instance {{ $labels.application }} has produced no successful tasks in the past hour and may be unavailable. Check the service immediately."

      # Rule 2: No successful tasks in the past 30 minutes (early warning)
      - alert: DolphinSchedulerNoSuccessfulTasksFor30Minutes
        expr: increase(ds_task_instance_count_total{state="success"}[30m]) == 0
        for: 5m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "No successful DolphinScheduler tasks in the past 30 minutes"
          description: "Instance {{ $labels.application }} has produced no successful tasks in the past 30 minutes. Check the service."

      # ============ Task health checks ============

      # Rule 3: High task failure rate
      - alert: DolphinSchedulerHighTaskFailureRate
        expr: |
          (
            increase(ds_task_instance_count_total{state="fail"}[10m])
            /
            (increase(ds_task_instance_count_total{state="finish"}[10m]) + 1)
          ) > 0.1
        for: 5m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "High DolphinScheduler task failure rate"
          description: "Instance {{ $labels.application }} has a task failure rate above 10% over the past 10 minutes. Current failure rate: {{ $value | humanizePercentage }}"

      # Rule 4: Task dispatch failures
      - alert: DolphinSchedulerTaskDispatchFailure
        expr: increase(ds_task_dispatch_failure_count_total[10m]) > 0
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "DolphinScheduler task dispatch failure"
          description: "Instance {{ $labels.application }} has experienced {{ $value }} task dispatch failures in the past 10 minutes."

      # ============ Worker health checks ============

      # Rule 5: No active Worker threads (tasks have been submitted, but nothing is executing)
      - alert: DolphinSchedulerWorkerNoActiveThreads
        expr: ds_worker_active_execute_thread == 0 and increase(ds_task_instance_count_total{state="submit"}[5m]) > 0
        for: 10m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "DolphinScheduler Worker has no active execution threads"
          description: "The Worker for instance {{ $labels.application }} has no active execution threads, while tasks have been submitted. Tasks may not be executing."

      # Rule 6: High Worker memory utilization
      - alert: DolphinSchedulerWorkerHighMemoryUsage
        expr: ds_worker_memory_usage > 0.85
        for: 5m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "High DolphinScheduler Worker memory usage"
          description: "The Worker for instance {{ $labels.application }} is using {{ $value | humanizePercentage }} memory, exceeding the 85% threshold."

      # Rule 7: High Worker CPU utilization
      - alert: DolphinSchedulerWorkerHighCPUUsage
        expr: ds_worker_cpu_usage > 0.85
        for: 5m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "High DolphinScheduler Worker CPU usage"
          description: "The Worker for instance {{ $labels.application }} is using {{ $value | humanizePercentage }} CPU, exceeding the 85% threshold."

      # ============ Workflow & Master checks ============

      # Rule 8: Long-running workflow instances (potentially blocked)
      - alert: DolphinSchedulerLongRunningWorkflows
        expr: ds_workflow_instance_running > 10
        for: 30m
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "Long-running DolphinScheduler workflows"
          description: "Instance {{ $labels.application }} has {{ $value }} workflow instances running for more than 30 minutes. Possible blocking detected."

      # Rule 9: Master failover check failures
      - alert: DolphinSchedulerMasterFailoverCheckFailure
        expr: increase(ds_master_scheduler_failover_check_count_total{result="fail"}[10m]) > 0
        labels:
          severity: critical
          group: dolphinscheduler
        annotations:
          summary: "DolphinScheduler Master failover check failure"
          description: "Instance {{ $labels.application }} has recorded {{ $value }} Master failover check failures in the past 10 minutes."

      # ============ Alert channel self-check ============

      # Rule 10: Alert delivery failures (monitoring the monitoring system)
      - alert: DolphinSchedulerAlertDeliveryFailure
        expr: increase(ds_alert_send_count_total{status="fail"}[10m]) > 5
        labels:
          severity: warning
          group: dolphinscheduler
        annotations:
          summary: "DolphinScheduler alert delivery failures"
          description: "Instance {{ $labels.application }} has experienced {{ $value }} alert delivery failures in the past 10 minutes."

The 10 rules above can be grouped into four categories:

Group	Rule	Severity	Trigger Condition
Service Availability	No successful tasks for 1h	Critical	0 successful tasks within 1 hour
Service Availability	No successful tasks for 30min	Warning	0 successful tasks within 30 minutes
Task Health	High task failure rate	Warning	Task failure rate > 10% over 10 minutes
Task Health	Task dispatch failure	Warning	Any task dispatch failure within 10 minutes
Worker	No active threads	Warning	Active thread count = 0
Worker	Memory > 85%	Warning	Threshold exceeded for 5 consecutive minutes
Worker	CPU > 85%	Warning	Threshold exceeded for 5 consecutive minutes
Workflow	Long-running workflows	Warning	Running instances > 10 for 30 consecutive minutes
Master	Abnormal failover	Critical	Failover check fails
Self-Monitoring	Alert delivery failure	Warning	More than 5 failures within 10 minutes

2.4 Verify That Alerts Work

When DolphinScheduler stops running or workflows become stalled for some reason, the corresponding alerts will be triggered automatically.

Now, when the service goes down, you can find out immediately instead of waiting until your manager finds out first.

That’s the whole point of monitoring.

3. Building a Grafana Dashboard

Alerts alone are not enough. You also need a dashboard that gives you a quick visual overview of the scheduler's health.

That’s where Grafana comes in.

Prerequisite: Grafana must already be connected to Prometheus as a data source. If it isn’t, go to Grafana → Configuration → Data Sources and add Prometheus.

3.1 Import the Dashboard

Go to:

Grafana → Dashboards → New → Import

Option 1: Import by Dashboard ID

This is the recommended approach.

On the import page, enter the dashboard ID:

Then click Load.

This is the easiest option. One number is all it takes, and you can skip the hassle of manually configuring the dashboard JSON.

Option 2: Import from a JSON Template

If your Grafana instance cannot access the public internet—for example, in a corporate network environment—you can manually import the following JSON template.

The JSON is quite long, so the compressed version is provided below. Copy it directly into Grafana’s JSON import field:

{"__inputs":[{"name":"DS_PROMETHEUS","label":"Prometheus","description":"","type":"datasource","pluginId":"prometheus","pluginName":"Prometheus"}],"__requires":[{"type":"grafana","id":"grafana","name":"Grafana","version":"9.0.0"},{"type":"datasource","id":"prometheus","name":"Prometheus","version":"1.0.0"},{"type":"panel","id":"stat","name":"Stat","version":""},{"type":"panel","id":"gauge","name":"Gauge","version":""},{"type":"panel","id":"timeseries","name":"Time series","version":""},{"type":"panel","id":"piechart","name":"Pie chart","version":""},{"type":"panel","id":"bargauge","name":"Bar gauge","version":""}],"annotations":{"list":[{"builtIn":1,"datasource":{"type":"datasource","uid":"grafana"},"enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"editable":true,"fiscalYearStartMonth":0,"graphTooltip":0,"id":null,"links":[],"liveNow":false,"panels":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"short"},"overrides":[]},"gridPos":{"h":4,"w":4,"x":0,"y":0},"id":1,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"textMode":"auto"},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_task_instance_count_total{state=\"success\"}","refId":"A"}],"title":"Task Success Total","type":"stat"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":1}]},"unit":"short"},"overrides":[]},"gridPos":{"h":4,"w":4,"x":4,"y":0},"id":2,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"textMode":"auto"},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_task_instance_count_total{state=\"fail\"}","refId":"A"}],"title":"Task Failure Total","type":"stat"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"blue","value":null}]},"unit":"short"},"overrides":[]},"gridPos":{"h":4,"w":4,"x":8,"y":0},"id":3,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"textMode":"auto"},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_workflow_instance_running","refId":"A"}],"title":"Running Workflows","type":"stat"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"max":1,"min":0,"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"yellow","value":0.7},{"color":"red","value":0.85}]},"unit":"percentunit"},"overrides":[]},"gridPos":{"h":4,"w":6,"x":12,"y":0},"id":4,"options":{"orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"showThresholdLabels":false,"showThresholdMarkers":true},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_worker_memory_usage","refId":"A"}],"title":"Worker Memory Usage","type":"gauge"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"max":1,"min":0,"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"yellow","value":0.7},{"color":"red","value":0.85}]},"unit":"percentunit"},"overrides":[]},"gridPos":{"h":4,"w":6,"x":18,"y":0},"id":5,"options":{"orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"showThresholdLabels":false,"showThresholdMarkers":true},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_worker_cpu_usage","refId":"A"}],"title":"Worker CPU Usage","type":"gauge"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"short"},"overrides":[{"matcher":{"id":"byName","options":"success"},"properties":[{"id":"color","value":{"fixedColor":"green","mode":"fixed"}}]},{"matcher":{"id":"byName","options":"fail"},"properties":[{"id":"color","value":{"fixedColor":"red","mode":"fixed"}}]}]},"gridPos":{"h":8,"w":12,"x":0,"y":4},"id":6,"options":{"legend":{"calcs":["last","max"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_task_instance_count_total{state=\"success\"}[5m])","legendFormat":"success","refId":"A"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_task_instance_count_total{state=\"fail\"}[5m])","legendFormat":"fail","refId":"B"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_task_instance_count_total{state=\"timeout\"}[5m])","legendFormat":"timeout","refId":"C"}],"title":"Task Execution Trend (5-Minute Increment)","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"hideFrom":{"tooltip":false,"viz":false,"legend":false}},"mappings":[],"unit":"short"},"overrides":[]},"gridPos":{"h":8,"w":6,"x":12,"y":4},"id":7,"options":{"legend":{"displayMode":"table","placement":"right","showLegend":true,"values":["value","percent"]},"pieType":"pie","tooltip":{"mode":"single","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_task_instance_count_total","legendFormat":"{{state}}","refId":"A"}],"title":"Task Status Distribution","type":"piechart"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"max":1,"min":0,"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"yellow","value":0.05},{"color":"red","value":0.1}]},"unit":"percentunit"},"overrides":[]},"gridPos":{"h":8,"w":6,"x":18,"y":4},"id":8,"options":{"orientation":"auto","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"showThresholdLabels":false,"showThresholdMarkers":true},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"rate(ds_task_instance_count_total{state=\"fail\"}[10m]) / (rate(ds_task_instance_count_total{state=\"finish\"}[10m]) + 0.001)","refId":"A"}],"title":"Task Failure Rate (10 Minutes)","type":"gauge"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"blue","value":null}]},"unit":"short"},"overrides":[]},"gridPos":{"h":8,"w":12,"x":0,"y":12},"id":9,"options":{"displayMode":"gradient","minVizHeight":10,"minVizWidth":0,"orientation":"horizontal","reduceOptions":{"values":false,"calcs":["lastNotNull"],"fields":""},"showUnfilled":true},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_task_execution_count_by_type_total > 0","legendFormat":"{{task_type}}","refId":"A"}],"title":"Task Type Distribution","type":"bargauge"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"short"},"overrides":[]},"gridPos":{"h":8,"w":12,"x":12,"y":12},"id":10,"options":{"legend":{"calcs":["last"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_workflow_instance_running","legendFormat":"Running","refId":"A"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_workflow_instance_count_total{state=\"success\"}[5m])","legendFormat":"Success-{{process_definition_code}}","refId":"B"}],"title":"Workflow Instance Trend","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"short"},"overrides":[{"matcher":{"id":"byName","options":"Memory Usage"},"properties":[{"id":"unit","value":"percentunit"},{"id":"custom.axisPlacement","value":"right"}]},{"matcher":{"id":"byName","options":"CPU Usage"},"properties":[{"id":"unit","value":"percentunit"},{"id":"custom.axisPlacement","value":"right"}]}]},"gridPos":{"h":8,"w":12,"x":0,"y":20},"id":11,"options":{"legend":{"calcs":["last","max"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_worker_active_execute_thread","legendFormat":"Active Threads","refId":"A"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_worker_memory_usage","legendFormat":"Memory Usage","refId":"B"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"ds_worker_cpu_usage","legendFormat":"CPU Usage","refId":"C"}],"title":"Worker Resource Monitoring","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"short"},"overrides":[{"matcher":{"id":"byName","options":"Failure"},"properties":[{"id":"color","value":{"fixedColor":"red","mode":"fixed"}}]}]},"gridPos":{"h":8,"w":12,"x":12,"y":20},"id":12,"options":{"legend":{"calcs":["last","max"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_alert_send_count_total{status=\"success\"}[5m])","legendFormat":"Success","refId":"A"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"increase(ds_alert_send_count_total{status=\"fail\"}[5m])","legendFormat":"Failure","refId":"B"}],"title":"Alert Delivery Trend (5-Minute Increment)","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"s"},"overrides":[]},"gridPos":{"h":8,"w":12,"x":0,"y":28},"id":13,"options":{"legend":{"calcs":["mean","max"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"rate(http_server_requests_seconds_sum{uri=~\"/projects.*\"}[5m]) / rate(http_server_requests_seconds_count{uri=~\"/projects.*\"}[5m])","legendFormat":"{{uri}} - {{method}}","refId":"A"}],"title":"Average API Response Time","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"${datasource}"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":10,"gradientMode":"none","hideFrom":{"tooltip":false,"viz":false,"legend":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"reqps"},"overrides":[]},"gridPos":{"h":8,"w":12,"x":12,"y":28},"id":14,"options":{"legend":{"calcs":["mean","max"],"displayMode":"table","placement":"bottom","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"pluginVersion":"9.5.0","targets":[{"datasource":{"type":"prometheus","uid":"${datasource}"},"expr":"rate(http_server_requests_seconds_count{uri=~\"/projects.*\"}[5m])","legendFormat":"{{uri}} - {{method}}","refId":"A"}],"title":"API Request Rate","type":"timeseries"}],"refresh":"30s","schemaVersion":38,"style":"dark","tags":["dolphinscheduler","big-data"],"templating":{"list":[{"current":{"selected":false,"text":"Prometheus","value":"Prometheus"},"hide":0,"includeAll":false,"label":"Data Source","multi":false,"name":"datasource","options":[],"query":"prometheus","refresh":1,"regex":"","skipUrlSync":false,"type":"datasource"}]},"time":{"from":"now-6h","to":"now"},"timepicker":{"refresh_intervals":["10s","30s","1m","5m","15m","30m","1h","2h","1d"]},"timezone":"browser","title":"DolphinScheduler Monitoring Dashboard","uid":"dolphinscheduler-overview","version":1,"weekStart":""}

3.2 Dashboard Preview

Once the import is complete, you’ll have a full monitoring dashboard containing the following panels:

Panel	Type	Monitoring Content
Total Success/Failed Tasks	Stat Card	Overview of global task counts
Running Workflows	Stat Card	Number of currently active workflows
Worker Memory/CPU Usage	Gauge	Red/Yellow/Green indicators; turns red above 85%
Task Execution Trend	Time-series Line Chart	Success/Failure/Timeout 5-minute increments
Task Status Distribution	Pie Chart	Proportion of each task status
Task Failure Rate	Gauge	10-minute rolling failure rate
Task Type Distribution	Bar Chart	Task count by type (SQL / Shell / Python)
Workflow Instance Trend	Time-series Line Chart	Workflow execution trends
Worker Resource Monitoring	Time-series Line Chart	Integrated view of Thread Count + Memory + CPU
Alert Notification Trend	Time-series Line Chart	Count of successful/failed alert notifications
API Response Time & Request Rate	Time-series Line Chart	DolphinScheduler API performance

Here’s what the final dashboard looks like:

4. Conclusion

Let’s recap what we built in this article.

Metric collection: Prometheus scrapes DolphinScheduler’s built-in /actuator/prometheus endpoint.
Alerting: 10 alert rules cover service availability, task health, Worker status, and Master failover.
Visualization: Grafana dashboard ID 24841 lets you import the monitoring dashboard with just a few clicks.

If you’re running DolphinScheduler, it’s worth setting up monitoring like this.

After all, being woken up by an alert at 3 a.m. is still better than being woken up by your manager the next morning.

If this guide helps you avoid even one “DolphinScheduler went down and nobody noticed” incident, consider giving it a like or saving it for later.

And if you have questions or improvements, drop them in the comments. Let’s make data engineering a little less painful—one production pitfall at a time.

🤖 From workflow orchestration to AI-native data operations! See how Apache DolphinScheduler powers governed, auditable Data Agents for safer enterprise automation. 🚀 #ApacheDolphinScheduler #DataAgents #AI

Chen Debra — Thu, 03 Sep 2026 07:55:53 +0000

Chen Debra

Sep 3

From Workflow Orchestration to Natural Language: How Apache DolphinScheduler Can Power Enterprise Data Agents

#nlp #agents #ai #apachedolphinscheduler

14 min read

From Workflow Orchestration to Natural Language: How Apache DolphinScheduler Can Power Enterprise Data Agents

Chen Debra — Thu, 03 Sep 2026 07:48:36 +0000

As large language models (LLMs) and AI agents make their way into enterprise data environments, the way people interact with data platforms is changing. Instead of navigating complex tools and workflows, users can describe a business goal in natural language. An agent can then understand the intent, gather the necessary context, plan the required steps, and invoke platform capabilities to carry out the task.

For enterprise data platforms, however, the real challenge is not simply teaching AI to “have a conversation.” The bigger question is how to enable AI to execute safely in production, validate results reliably, and maintain a complete audit trail throughout the process.

At a recent session hosted by the Apache DolphinScheduler community, we invited Li Qingwang, a Data Engineer from Cisco, Webex to share an enterprise Data Agent implementation based on Apache DolphinScheduler. His presentation explored how enterprise data platforms can evolve from traditional workflow orchestration backends into natural-language interfaces for data operations.

Video Replay

About the Speaker

Li Qingwang is a Big Data Platform Development Engineer at Cisco Webex and a Committer of Apache DolphinScheduler.

The presentation focused on four key areas: evolving from a workflow orchestration platform into a data interface, defining the role and architecture of an Agent platform, establishing a governance, execution, and validation loop, and preparing the platform for capability expansion and production adoption.

1. From Workflow Orchestration Platform to Data Interface

1.1 Starting with Apache DolphinScheduler Customization

Apache DolphinScheduler provides a powerful set of capabilities, but enterprise adoption often requires customization to meet internal business and governance requirements. Through platform-level customization and extension, its general-purpose workflow orchestration capabilities can be packaged into an enterprise data platform that provides a unified production entry point for different teams.

The first challenge this approach addresses is the lack of consistency in how different data teams access and use the platform.

At the access layer, capabilities such as Projects / Namespaces, tenants, permissions, and resource groups can be standardized to provide different teams with a unified production entry point.

At the task layer, the platform can support different task types, including SQL, Spark, Flink, and ETL, while establishing consistent standards for custom tasks and parameters.

Once tasks enter production, the platform also needs to support release governance, including environment management, version control, dependency management, approvals, scheduling, and backfills.

At the same time, comprehensive operational capabilities are required to manage task status, logs, alerts, reruns, troubleshooting, and auditing.

With these platform-level capabilities in place, the role of Apache DolphinScheduler can be extended beyond workflow scheduling: it becomes the unified orchestration and execution foundation for production tasks across SQL, Spark, Flink, ETL, and other workloads.

1.2 Once the Platform Is Shared, New Bottlenecks Emerge

After workflow orchestration capabilities have been standardized and shared across the organization, the next bottlenecks often move beyond scheduling itself. They emerge around task development, release, operations, and maintenance.

Bringing a production task to life involves multiple stages, from discovery and development to deployment and troubleshooting. These stages often remain heavily dependent on domain experts.

The first challenge is data discovery.

Enterprise data assets may be scattered across tables, Topics, and different data systems, while ownership and data definitions may reside with different teams. Users may know what data they need without knowing which table or Topic contains it—or which team owns it and can clarify the relevant business definition.

The second challenge is development.

SQL, code, and configuration can involve a significant technical learning curve, requiring support from data engineers or platform specialists. For business users who are not familiar with the underlying technologies, turning a business requirement into a runnable data workflow is a barrier in itself.

The third challenge is production release.

Whether a task is ready for production is not simply a question of whether the code runs. Environment configuration, permissions, parameters, and potential operational risks must also be evaluated. These decisions often require platform or domain expertise.

The fourth challenge is troubleshooting.

When something goes wrong, engineers may need to correlate logs, metrics, source systems, and target systems to identify the root cause. Troubleshooting is no longer about checking the status of a single task; it requires evidence from multiple systems.

As platforms become more open and accessible, reliance on experts can therefore become a new bottleneck: users know what they want to achieve, but still depend on experts for data discovery, development, production release, and operations.

This is where Data Agents can make a meaningful difference.

2. Defining the Role and Architecture of an Agent Platform

Traditional data platforms focus primarily on how to execute data tasks. Data Agents address a different question: how users can interact with and operate data platforms more naturally.

At its core, a Data Agent can be understood through three fundamental actions:

Understand business intent, execute within governance boundaries, and validate outcomes with traceable evidence.

2.1 The Agent Platform as a Governance-First AI Operations Layer

An enterprise Data Agent is not simply a chatbot added to an existing data platform. It is an AI interaction and operations layer built on top of the organization's existing production governance framework.

Its core positioning can be summarized as:

Natural-language data operations + production-grade governance.

The platform needs to provide four capabilities simultaneously.

The first is Intent-driven.

Users no longer need to learn specific APIs or platform procedures before they can get started. They simply describe the outcome they want. The Agent interprets the goal, plans the required steps, and selects the appropriate tools to carry them out.

The second is Governed.

Every production action taken by AI must be constrained by identity, permissions, risk controls, and approval policies. The Agent can plan an action, but it must not bypass the governance boundaries already established by the enterprise.

The third is Auditable.

An Agent interaction should not leave behind only a final result. The platform should retain the session, tool calls, decisions, approvals, and final artifacts, creating a complete and traceable record of what happened.

The fourth is Verifiable.

A task should not be considered successful simply because the Agent generated valid SQL or configuration. It needs to be validated before release, and its runtime behavior and final results should be verified against logs and data evidence after execution.

This leads to a clear principle: AI can handle planning, but every production action must pass through a unified governance and validation layer—the AI Harness.

2.2 Self-Service Data Agents: Six Areas of Transformation

With this positioning, a Data Agent is not limited to generating SQL. It can support multiple stages of data work.

For pipeline development, users traditionally need to write code, SQL, and configuration manually. With an Agent, they can describe their requirements in natural language.

For troubleshooting, engineers traditionally search through logs and metrics manually. An Agent can automatically gather relevant evidence and assist with diagnosis.

For platform operations, users have traditionally relied on experts to operate the UI or APIs. Agents shift this model toward intent-driven, controlled execution.

For data discovery, users traditionally search separately for jobs, tables, and owners. An Agent can retrieve metadata through natural-language queries.

For code review, the context required for review is often distributed across multiple systems. An Agent can combine code diffs, sandbox results, risk information, and audit records to provide contextual assistance.

For knowledge reuse, experts often answer the same questions repeatedly and provide the same operational support. Agents can turn this accumulated experience into domain knowledge and reusable Agent capabilities.

The transformation is therefore not simply about replacing human operations with AI. It is about moving from manual construction, search, review, and support toward governed AI-assisted collaboration.

2.3 From a Single Question to a Complete Production Workflow

From the user's perspective, the goal of a Data Agent is to provide a unified interface covering the entire data workflow.

A user might first ask:

“What does this workflow do? What are the inputs, filters, aggregations, and outputs?”

This represents discovery and understanding.

The user could then ask:

“Read data from Kafka, build an ETL pipeline, and configure it to run as a daily workflow.”

This moves into development and release.

When a task fails, the user might ask:

“Latency suddenly increased last night. Help me identify the cause and recommend a fix.”

This corresponds to monitoring and troubleshooting.

In an analytics scenario, the user could also ask:

“Find trusted tables, analyze the metrics, and generate an explainable dashboard.”

This covers analysis and visualization.

The Agent can also examine resource utilization and determine whether a workload is over-provisioned.

In other words, a single interface can eventually cover:

Data Engineering · Data Discovery · DataOps · Insight

2.4 Overall Architecture: An Enterprise AI Harness

To bring these capabilities into production, the Agent itself is not enough. A complete enterprise-grade AI Harness is also required.

At the top is the interaction layer, which provides Web, CLI, and MCP Client interfaces and can connect to AI clients such as Codex and Claude.

Below that is the Agent orchestration layer, responsible for intent understanding, task planning, context discovery, tool orchestration, and evidence summarization.

The key layer is the AI Harness.

This layer handles identity and access control, sandbox validation, risk policies, approvals, idempotency, auditing, and tracing. It does not replace the Agent. Instead, it establishes the governance boundaries within which the Agent can perform production actions.

Below the Harness is the tool and execution layer, which includes Discovery, SQL / Spark / Flink / ETL, DS Workflow, and Observability.

At the bottom are enterprise production systems, including Kafka, Catalog, Lakehouse, OLAP, and Compute.

Within this architecture, Apache DolphinScheduler continues to provide workflow orchestration, scheduling, and execution capabilities.

The platform also needs dedicated state and audit storage to record information such as Session, Approval, Sandbox Result, Artifact, and Trace.

The architecture therefore does not allow an Agent to directly operate production systems. Instead, a governance and validation layer is placed between the Agent and production systems.

2.5 Agent Platform: Building an Ecosystem of Domain Agents

Once a unified platform foundation is in place, it can evolve into an ecosystem of domain-specific Agents.

These can include Data Engineering Agents responsible for Flink SQL, Spark, ETL, and Workflow; Approval and Policy Agents responsible for approval orchestration, policy evaluation, and risk blocking; and Sandbox Validation Agents responsible for local syntax checks, Kafka samples, and remote sandbox validation. Human approval remains part of the process for high-risk scenarios.

There can also be Workflow Observability Agents responsible for status, logs, checkpoints, and source/target evidence; Discovery and Lineage Agents responsible for metadata, owners, resources, and lineage; and platform-support Agents responsible for Sessions, Approvals, Traces, and failure evidence.

These Agents do not each need to build their own governance framework. Instead, they share the same foundation:

MCP Entry Point, Identity, Sandbox, Policy, Approval, Audit, Trace, and Idempotency.

This is one of the key differences between an enterprise-grade Agent platform and a collection of standalone AI applications: domain capabilities can continue to expand, while governance should remain as unified as possible.

3. Governance, Execution, and the Validation Loop

If the second section addresses what a Data Agent should be, the third focuses on how a Data Agent can execute safely.

The approach can be summarized in three principles:

A unified governance model with three execution paths; risk-based approval combined with pre-release validation; and a post-release feedback loop based on runtime and data evidence.

3.1 Governance and Execution Model: One Governance Framework, Three Paths

A complete Agent request can be broken down into six steps.

The first step is User Intent: the user submits a business request in natural language.

The second step is Agent Planning: the Agent generates the required steps, SQL, or configuration based on the request.

The third step is Context Discovery: the Agent retrieves additional information such as metadata, ownership, and historical context.

The fourth step is Harness Checks: the system verifies identity, sandbox results, and risk, and determines what type of approval is required.

The fifth step is Governed Execution: the Agent invokes APIs through controlled wrappers.

The sixth step is Audit and Result: execution evidence is recorded and the final conclusion is returned to the user.

On this basis, execution can be divided into three paths.

Read Path

The Read Path is primarily used for discovery, status, logs, metadata, and other read-only information. It does not modify production state. The system records the operation for auditing before returning the result.

Change Path

Changes require stricter controls. A typical flow is:

Sandbox → Risk Policy → Automatic / Human Approval / Block → Idempotent Execution After Approval

In other words, an operation that changes production state cannot be executed simply because the Agent has generated a technically correct API request.

Observe Path

The Observe Path collects runtime, source, target, and log evidence after a task has been released or whenever the user requests additional information.

This creates a straightforward operating principle:

Every request has a defined path, every high-risk action has a gate, and every result is auditable.

3.2 Risk-Based Approval: Automatic, Human, or Blocked

Production operations should not all follow the same approval process. Instead, they can be classified according to risk.

For low-risk operations, if the user has permission to operate on the target resource, sandbox validation has passed, and the user has confirmed the intended action, the operation can be automatically approved and proceed to governed execution.

For high-risk or high-impact operations, such as modifying another user's resources or shared resources, changing a shared cluster, or operating where ownership boundaries are unclear, execution should pause and require human confirmation.

Some situations should be blocked outright, including invalid identities, failed syntax or data validation, attempts to deploy directly to production without validation, or operations that could potentially cause widespread or dangerous impact.

The goal is not to introduce manual approval for every operation. Instead, it is to match the control mechanism and execution path to the level of risk.

Low-risk operations can remain efficient, high-risk operations can retain human oversight, and clearly unsafe operations can be stopped immediately.

3.3 Pre-Release Validation: Catch Errors Before Production

A traditional data workflow may follow this pattern:

Create → Release → Discover Error → Production Incident

The problem is that many errors only become visible after a task reaches production. Agent-driven workflows aim to move validation earlier in the process.

The first step is Plan, where the task plan is generated.

Next comes Local Sandbox, where syntax and logic are validated.

The workflow can then proceed to Remote Sample, using Kafka samples or a cluster sandbox for additional validation.

This is followed by Result Analysis, where schema and business risks are evaluated.

Only after these checks does the workflow move to Create & Release, allowing the task to be safely deployed.

The key is not to rely on a single validation step. Instead, the platform combines local execution, realistic sample data, remote sandbox validation, and structured result analysis to identify errors and risks before they reach production.

3.4 Post-Release Validation: From Submission to Verified Results

Even when a task has passed pre-release validation and has been successfully deployed, the workflow is not necessarily complete.

A true production feedback loop needs to track the process from task creation all the way to the final business outcome.

The complete process includes:

Create / Release → Check Runtime Status → Review Logs → Diagnose Source / Target → Verify Business Results → Return Conclusion

The evidence collected during verification can include runtime status such as Running or Failed, recent logs and error messages, Kafka consumption status, and whether the target table has been populated successfully.

The final response from a Data Agent should therefore go beyond “the task has been submitted” or “the API call succeeded.” Wherever possible, it should answer the questions that actually matter:

Did the task run successfully? Did the data actually flow? Did the target system produce the expected result?

This moves Agent validation beyond simply confirming that an API call succeeded toward verifying that the intended outcome was actually achieved.

4. Capability Expansion and Production Readiness

Once governance, execution, and validation are unified on a common foundation, Data Agent capabilities no longer need to be limited to a single scenario. They can continuously expand across different areas of the data ecosystem.

The core idea at this stage is straightforward: domain Agents can continue to evolve while reusing the same governance and audit foundation, turning individual expert knowledge into reusable platform capabilities.

4.1 Platform Expansion: Every Domain Can Become an Agent

A Predictive Operations Agent can proactively identify potential failures, recommend resource adjustments, and execute remediation under defined policies.

For data discovery, a Universal Discovery Agent can search across jobs, Topics, tables, owners, lineage, and data-quality signals.

For data development, a Pipeline Generation Agent can generate pipelines from documentation, create workflows from schemas, and support cross-engine transformations.

For analytics, a Business Insights Agent can identify trusted data, generate queries and visualizations, and provide explanations.

For data governance, a Data Quality Agent can identify schema drift, distribution anomalies, and late-arriving data while helping improve governance tags.

A Cost Optimization Agent can analyze resource utilization and recommend runtime configurations, parallelism, and resource reservation strategies.

These capabilities may belong to different domains, but adding another Agent should not require building another security and governance framework from scratch.

The North Star for the entire system is:

Domain capabilities can continue to expand, while production safety remains protected by a unified platform Harness.

4.2 Business Value: Turning Expert Knowledge into Platform Capabilities

From a business perspective, this architecture delivers value in four key areas.

First is efficiency. Natural-language interaction and Agents can accelerate pipeline development while reducing repetitive manual platform operations.

Second is quality. By moving validation earlier in the process, issues can be identified before they reach production, improving workflow reliability.

Third is governance. Every AI action is tied to an identity, while policy decisions can be audited throughout the entire process.

Fourth is scale. Domain Agents can be reused across teams and scenarios, reducing an organization's reliance on experts for repetitive support.

Ultimately, the data platform itself may evolve:

Today: a UI / API-driven platform for experts.
Tomorrow: an AI-native Agent platform.

The key is not simply to add an AI chat interface. It is to use governed, reusable Agents that understand production semantics to capture, standardize, and scale capabilities that previously depended on individual experts.

Conclusion: From Workflow Orchestration to a Natural-Language Data Interface

At the heart of the presentation is a simple idea: Data Agents are not designed to replace Apache DolphinScheduler. Instead, they change how users access and interact with the production capabilities already provided by DolphinScheduler.

First, the orchestration foundation remains unchanged. Apache DolphinScheduler continues to provide core capabilities for workflow orchestration, task execution, monitoring, and operations.

Second, the entry point changes. Users no longer need to begin with a specific UI, API, or task configuration. They can start by describing a business goal, while the Agent connects that intent to data discovery, development, release, and troubleshooting.

More importantly, governance and control mechanisms provide the foundation for safe execution. The four principles—Intent-driven, Governed, Auditable, and Verifiable—work together to constrain production actions and keep Agent capabilities within enterprise governance boundaries.

The real question for enterprise Data Agents, therefore, is not simply whether “AI can perform a data operation.” The question is whether that operation can be carried out in a real production environment with the right permissions, clear boundaries, reliable validation, traceable evidence, and the ability to run continuously and safely.

As the presentation concluded:

The value of a Data Agent is not that it can “have a conversation,” but that it enables the right data work to run safely, continuously, and verifiably.

As Data Agents become increasingly embedded in enterprise data platforms, Apache DolphinScheduler's core orchestration and execution capabilities will remain an important foundation for the entire architecture. On top of that foundation, an AI Harness can connect natural-language interaction, domain Agents, governance policies, approvals, sandboxes, auditing, and result validation—helping data platforms evolve from traditional expert-driven platforms toward AI-native Agent platforms.