As large language models (LLMs) and AI agents make their way into enterprise data environments, the way people interact with data platforms is changing. Instead of navigating complex tools and workflows, users can describe a business goal in natural language. An agent can then understand the intent, gather the necessary context, plan the required steps, and invoke platform capabilities to carry out the task.
For enterprise data platforms, however, the real challenge is not simply teaching AI to “have a conversation.” The bigger question is how to enable AI to execute safely in production, validate results reliably, and maintain a complete audit trail throughout the process.
At a recent session hosted by the Apache DolphinScheduler community, we invited Li Qingwang, a Data Engineer from Cisco, Webex to share an enterprise Data Agent implementation based on Apache DolphinScheduler. His presentation explored how enterprise data platforms can evolve from traditional workflow orchestration backends into natural-language interfaces for data operations.
About the Speaker
Li Qingwang is a Big Data Platform Development Engineer at Cisco Webex and a Committer of Apache DolphinScheduler.
The presentation focused on four key areas: evolving from a workflow orchestration platform into a data interface, defining the role and architecture of an Agent platform, establishing a governance, execution, and validation loop, and preparing the platform for capability expansion and production adoption.
1. From Workflow Orchestration Platform to Data Interface
1.1 Starting with Apache DolphinScheduler Customization
Apache DolphinScheduler provides a powerful set of capabilities, but enterprise adoption often requires customization to meet internal business and governance requirements. Through platform-level customization and extension, its general-purpose workflow orchestration capabilities can be packaged into an enterprise data platform that provides a unified production entry point for different teams.
The first challenge this approach addresses is the lack of consistency in how different data teams access and use the platform.
At the access layer, capabilities such as Projects / Namespaces, tenants, permissions, and resource groups can be standardized to provide different teams with a unified production entry point.
At the task layer, the platform can support different task types, including SQL, Spark, Flink, and ETL, while establishing consistent standards for custom tasks and parameters.
Once tasks enter production, the platform also needs to support release governance, including environment management, version control, dependency management, approvals, scheduling, and backfills.
At the same time, comprehensive operational capabilities are required to manage task status, logs, alerts, reruns, troubleshooting, and auditing.
With these platform-level capabilities in place, the role of Apache DolphinScheduler can be extended beyond workflow scheduling: it becomes the unified orchestration and execution foundation for production tasks across SQL, Spark, Flink, ETL, and other workloads.
1.2 Once the Platform Is Shared, New Bottlenecks Emerge
After workflow orchestration capabilities have been standardized and shared across the organization, the next bottlenecks often move beyond scheduling itself. They emerge around task development, release, operations, and maintenance.
Bringing a production task to life involves multiple stages, from discovery and development to deployment and troubleshooting. These stages often remain heavily dependent on domain experts.
The first challenge is data discovery.
Enterprise data assets may be scattered across tables, Topics, and different data systems, while ownership and data definitions may reside with different teams. Users may know what data they need without knowing which table or Topic contains it—or which team owns it and can clarify the relevant business definition.
The second challenge is development.
SQL, code, and configuration can involve a significant technical learning curve, requiring support from data engineers or platform specialists. For business users who are not familiar with the underlying technologies, turning a business requirement into a runnable data workflow is a barrier in itself.
The third challenge is production release.
Whether a task is ready for production is not simply a question of whether the code runs. Environment configuration, permissions, parameters, and potential operational risks must also be evaluated. These decisions often require platform or domain expertise.
The fourth challenge is troubleshooting.
When something goes wrong, engineers may need to correlate logs, metrics, source systems, and target systems to identify the root cause. Troubleshooting is no longer about checking the status of a single task; it requires evidence from multiple systems.
As platforms become more open and accessible, reliance on experts can therefore become a new bottleneck: users know what they want to achieve, but still depend on experts for data discovery, development, production release, and operations.
This is where Data Agents can make a meaningful difference.
2. Defining the Role and Architecture of an Agent Platform
Traditional data platforms focus primarily on how to execute data tasks. Data Agents address a different question: how users can interact with and operate data platforms more naturally.
At its core, a Data Agent can be understood through three fundamental actions:
Understand business intent, execute within governance boundaries, and validate outcomes with traceable evidence.
2.1 The Agent Platform as a Governance-First AI Operations Layer
An enterprise Data Agent is not simply a chatbot added to an existing data platform. It is an AI interaction and operations layer built on top of the organization's existing production governance framework.
Its core positioning can be summarized as:
Natural-language data operations + production-grade governance.
The platform needs to provide four capabilities simultaneously.
The first is Intent-driven.
Users no longer need to learn specific APIs or platform procedures before they can get started. They simply describe the outcome they want. The Agent interprets the goal, plans the required steps, and selects the appropriate tools to carry them out.
The second is Governed.
Every production action taken by AI must be constrained by identity, permissions, risk controls, and approval policies. The Agent can plan an action, but it must not bypass the governance boundaries already established by the enterprise.
The third is Auditable.
An Agent interaction should not leave behind only a final result. The platform should retain the session, tool calls, decisions, approvals, and final artifacts, creating a complete and traceable record of what happened.
The fourth is Verifiable.
A task should not be considered successful simply because the Agent generated valid SQL or configuration. It needs to be validated before release, and its runtime behavior and final results should be verified against logs and data evidence after execution.
This leads to a clear principle: AI can handle planning, but every production action must pass through a unified governance and validation layer—the AI Harness.
2.2 Self-Service Data Agents: Six Areas of Transformation
With this positioning, a Data Agent is not limited to generating SQL. It can support multiple stages of data work.
For pipeline development, users traditionally need to write code, SQL, and configuration manually. With an Agent, they can describe their requirements in natural language.
For troubleshooting, engineers traditionally search through logs and metrics manually. An Agent can automatically gather relevant evidence and assist with diagnosis.
For platform operations, users have traditionally relied on experts to operate the UI or APIs. Agents shift this model toward intent-driven, controlled execution.
For data discovery, users traditionally search separately for jobs, tables, and owners. An Agent can retrieve metadata through natural-language queries.
For code review, the context required for review is often distributed across multiple systems. An Agent can combine code diffs, sandbox results, risk information, and audit records to provide contextual assistance.
For knowledge reuse, experts often answer the same questions repeatedly and provide the same operational support. Agents can turn this accumulated experience into domain knowledge and reusable Agent capabilities.
The transformation is therefore not simply about replacing human operations with AI. It is about moving from manual construction, search, review, and support toward governed AI-assisted collaboration.
2.3 From a Single Question to a Complete Production Workflow
From the user's perspective, the goal of a Data Agent is to provide a unified interface covering the entire data workflow.
A user might first ask:
“What does this workflow do? What are the inputs, filters, aggregations, and outputs?”
This represents discovery and understanding.
The user could then ask:
“Read data from Kafka, build an ETL pipeline, and configure it to run as a daily workflow.”
This moves into development and release.
When a task fails, the user might ask:
“Latency suddenly increased last night. Help me identify the cause and recommend a fix.”
This corresponds to monitoring and troubleshooting.
In an analytics scenario, the user could also ask:
“Find trusted tables, analyze the metrics, and generate an explainable dashboard.”
This covers analysis and visualization.
The Agent can also examine resource utilization and determine whether a workload is over-provisioned.
In other words, a single interface can eventually cover:
Data Engineering · Data Discovery · DataOps · Insight
2.4 Overall Architecture: An Enterprise AI Harness
To bring these capabilities into production, the Agent itself is not enough. A complete enterprise-grade AI Harness is also required.
At the top is the interaction layer, which provides Web, CLI, and MCP Client interfaces and can connect to AI clients such as Codex and Claude.
Below that is the Agent orchestration layer, responsible for intent understanding, task planning, context discovery, tool orchestration, and evidence summarization.
The key layer is the AI Harness.
This layer handles identity and access control, sandbox validation, risk policies, approvals, idempotency, auditing, and tracing. It does not replace the Agent. Instead, it establishes the governance boundaries within which the Agent can perform production actions.
Below the Harness is the tool and execution layer, which includes Discovery, SQL / Spark / Flink / ETL, DS Workflow, and Observability.
At the bottom are enterprise production systems, including Kafka, Catalog, Lakehouse, OLAP, and Compute.
Within this architecture, Apache DolphinScheduler continues to provide workflow orchestration, scheduling, and execution capabilities.
The platform also needs dedicated state and audit storage to record information such as Session, Approval, Sandbox Result, Artifact, and Trace.
The architecture therefore does not allow an Agent to directly operate production systems. Instead, a governance and validation layer is placed between the Agent and production systems.
2.5 Agent Platform: Building an Ecosystem of Domain Agents
Once a unified platform foundation is in place, it can evolve into an ecosystem of domain-specific Agents.
These can include Data Engineering Agents responsible for Flink SQL, Spark, ETL, and Workflow; Approval and Policy Agents responsible for approval orchestration, policy evaluation, and risk blocking; and Sandbox Validation Agents responsible for local syntax checks, Kafka samples, and remote sandbox validation. Human approval remains part of the process for high-risk scenarios.
There can also be Workflow Observability Agents responsible for status, logs, checkpoints, and source/target evidence; Discovery and Lineage Agents responsible for metadata, owners, resources, and lineage; and platform-support Agents responsible for Sessions, Approvals, Traces, and failure evidence.
These Agents do not each need to build their own governance framework. Instead, they share the same foundation:
MCP Entry Point, Identity, Sandbox, Policy, Approval, Audit, Trace, and Idempotency.
This is one of the key differences between an enterprise-grade Agent platform and a collection of standalone AI applications: domain capabilities can continue to expand, while governance should remain as unified as possible.
3. Governance, Execution, and the Validation Loop
If the second section addresses what a Data Agent should be, the third focuses on how a Data Agent can execute safely.
The approach can be summarized in three principles:
A unified governance model with three execution paths; risk-based approval combined with pre-release validation; and a post-release feedback loop based on runtime and data evidence.
3.1 Governance and Execution Model: One Governance Framework, Three Paths
A complete Agent request can be broken down into six steps.
The first step is User Intent: the user submits a business request in natural language.
The second step is Agent Planning: the Agent generates the required steps, SQL, or configuration based on the request.
The third step is Context Discovery: the Agent retrieves additional information such as metadata, ownership, and historical context.
The fourth step is Harness Checks: the system verifies identity, sandbox results, and risk, and determines what type of approval is required.
The fifth step is Governed Execution: the Agent invokes APIs through controlled wrappers.
The sixth step is Audit and Result: execution evidence is recorded and the final conclusion is returned to the user.
On this basis, execution can be divided into three paths.
Read Path
The Read Path is primarily used for discovery, status, logs, metadata, and other read-only information. It does not modify production state. The system records the operation for auditing before returning the result.
Change Path
Changes require stricter controls. A typical flow is:
Sandbox → Risk Policy → Automatic / Human Approval / Block → Idempotent Execution After Approval
In other words, an operation that changes production state cannot be executed simply because the Agent has generated a technically correct API request.
Observe Path
The Observe Path collects runtime, source, target, and log evidence after a task has been released or whenever the user requests additional information.
This creates a straightforward operating principle:
Every request has a defined path, every high-risk action has a gate, and every result is auditable.
3.2 Risk-Based Approval: Automatic, Human, or Blocked
Production operations should not all follow the same approval process. Instead, they can be classified according to risk.
For low-risk operations, if the user has permission to operate on the target resource, sandbox validation has passed, and the user has confirmed the intended action, the operation can be automatically approved and proceed to governed execution.
For high-risk or high-impact operations, such as modifying another user's resources or shared resources, changing a shared cluster, or operating where ownership boundaries are unclear, execution should pause and require human confirmation.
Some situations should be blocked outright, including invalid identities, failed syntax or data validation, attempts to deploy directly to production without validation, or operations that could potentially cause widespread or dangerous impact.
The goal is not to introduce manual approval for every operation. Instead, it is to match the control mechanism and execution path to the level of risk.
Low-risk operations can remain efficient, high-risk operations can retain human oversight, and clearly unsafe operations can be stopped immediately.
3.3 Pre-Release Validation: Catch Errors Before Production
A traditional data workflow may follow this pattern:
Create → Release → Discover Error → Production Incident
The problem is that many errors only become visible after a task reaches production. Agent-driven workflows aim to move validation earlier in the process.
The first step is Plan, where the task plan is generated.
Next comes Local Sandbox, where syntax and logic are validated.
The workflow can then proceed to Remote Sample, using Kafka samples or a cluster sandbox for additional validation.
This is followed by Result Analysis, where schema and business risks are evaluated.
Only after these checks does the workflow move to Create & Release, allowing the task to be safely deployed.
The key is not to rely on a single validation step. Instead, the platform combines local execution, realistic sample data, remote sandbox validation, and structured result analysis to identify errors and risks before they reach production.
3.4 Post-Release Validation: From Submission to Verified Results
Even when a task has passed pre-release validation and has been successfully deployed, the workflow is not necessarily complete.
A true production feedback loop needs to track the process from task creation all the way to the final business outcome.
The complete process includes:
Create / Release → Check Runtime Status → Review Logs → Diagnose Source / Target → Verify Business Results → Return Conclusion
The evidence collected during verification can include runtime status such as Running or Failed, recent logs and error messages, Kafka consumption status, and whether the target table has been populated successfully.
The final response from a Data Agent should therefore go beyond “the task has been submitted” or “the API call succeeded.” Wherever possible, it should answer the questions that actually matter:
Did the task run successfully? Did the data actually flow? Did the target system produce the expected result?
This moves Agent validation beyond simply confirming that an API call succeeded toward verifying that the intended outcome was actually achieved.
4. Capability Expansion and Production Readiness
Once governance, execution, and validation are unified on a common foundation, Data Agent capabilities no longer need to be limited to a single scenario. They can continuously expand across different areas of the data ecosystem.
The core idea at this stage is straightforward: domain Agents can continue to evolve while reusing the same governance and audit foundation, turning individual expert knowledge into reusable platform capabilities.
4.1 Platform Expansion: Every Domain Can Become an Agent
A Predictive Operations Agent can proactively identify potential failures, recommend resource adjustments, and execute remediation under defined policies.
For data discovery, a Universal Discovery Agent can search across jobs, Topics, tables, owners, lineage, and data-quality signals.
For data development, a Pipeline Generation Agent can generate pipelines from documentation, create workflows from schemas, and support cross-engine transformations.
For analytics, a Business Insights Agent can identify trusted data, generate queries and visualizations, and provide explanations.
For data governance, a Data Quality Agent can identify schema drift, distribution anomalies, and late-arriving data while helping improve governance tags.
A Cost Optimization Agent can analyze resource utilization and recommend runtime configurations, parallelism, and resource reservation strategies.
These capabilities may belong to different domains, but adding another Agent should not require building another security and governance framework from scratch.
The North Star for the entire system is:
Domain capabilities can continue to expand, while production safety remains protected by a unified platform Harness.
4.2 Business Value: Turning Expert Knowledge into Platform Capabilities
From a business perspective, this architecture delivers value in four key areas.
First is efficiency. Natural-language interaction and Agents can accelerate pipeline development while reducing repetitive manual platform operations.
Second is quality. By moving validation earlier in the process, issues can be identified before they reach production, improving workflow reliability.
Third is governance. Every AI action is tied to an identity, while policy decisions can be audited throughout the entire process.
Fourth is scale. Domain Agents can be reused across teams and scenarios, reducing an organization's reliance on experts for repetitive support.
Ultimately, the data platform itself may evolve:
Today: a UI / API-driven platform for experts.
Tomorrow: an AI-native Agent platform.
The key is not simply to add an AI chat interface. It is to use governed, reusable Agents that understand production semantics to capture, standardize, and scale capabilities that previously depended on individual experts.
Conclusion: From Workflow Orchestration to a Natural-Language Data Interface
At the heart of the presentation is a simple idea: Data Agents are not designed to replace Apache DolphinScheduler. Instead, they change how users access and interact with the production capabilities already provided by DolphinScheduler.
First, the orchestration foundation remains unchanged. Apache DolphinScheduler continues to provide core capabilities for workflow orchestration, task execution, monitoring, and operations.
Second, the entry point changes. Users no longer need to begin with a specific UI, API, or task configuration. They can start by describing a business goal, while the Agent connects that intent to data discovery, development, release, and troubleshooting.
More importantly, governance and control mechanisms provide the foundation for safe execution. The four principles—Intent-driven, Governed, Auditable, and Verifiable—work together to constrain production actions and keep Agent capabilities within enterprise governance boundaries.
The real question for enterprise Data Agents, therefore, is not simply whether “AI can perform a data operation.” The question is whether that operation can be carried out in a real production environment with the right permissions, clear boundaries, reliable validation, traceable evidence, and the ability to run continuously and safely.
As the presentation concluded:
The value of a Data Agent is not that it can “have a conversation,” but that it enables the right data work to run safely, continuously, and verifiably.
As Data Agents become increasingly embedded in enterprise data platforms, Apache DolphinScheduler's core orchestration and execution capabilities will remain an important foundation for the entire architecture. On top of that foundation, an AI Harness can connect natural-language interaction, domain Agents, governance policies, approvals, sandboxes, auditing, and result validation—helping data platforms evolve from traditional expert-driven platforms toward AI-native Agent platforms.
Top comments (0)