As enterprises accelerate their digital transformation, data platforms are evolving from traditional storage and analytics systems into foundational infrastructures that support business decision-making, intelligent applications, and data asset management.
Building a stable, efficient, and scalable data production system has become a key challenge in modern enterprise data warehouse (DW) initiatives.
At the Apache DolphinScheduler September Meetup, community practitioner Zan Liu shared practical experiences in building enterprise-grade data warehouses. The session explored how Apache DolphinScheduler can be used as the orchestration core, working together with data integration, data processing, data applications, and operational management capabilities to build a complete data engineering ecosystem.
Based on the Meetup session, this article provides an in-depth look at how Apache DolphinScheduler supports enterprise data warehouse practices across four key stages: environment setup, data processing, data applications, and operational upgrades.
👉🏻 Replay the Meetup session here
About the Speaker
Liu Zan：Data Analytics Engineer & Senior Big Data Architect
With extensive experience in data warehousing and big data technologies, Liu Zan specializes in BI development and Dify-based low-code Agent development. He has deep expertise in building enterprise data platforms and implementing data-driven applications.
Environment Setup: Building a Data Foundation for Enterprise Data Warehouses
The first step in building an enterprise data warehouse is selecting a technology architecture that can support long-term business growth and establishing a stable, reliable data infrastructure.
During the early stages of big data development, the Hadoop ecosystem became the foundation for many enterprise data platforms due to its distributed storage and computing capabilities. However, as real-time data requirements continue to increase, traditional architectures have gradually revealed limitations in certain scenarios.
On one hand, traditional Hadoop-based architectures are primarily designed for batch processing workloads. Data jobs often experience minute-level latency, making them less suitable for real-time analytics and fast decision-making scenarios.
On the other hand, traditional architectures tightly couple computing and storage resources. When enterprises need to scale a specific type of resource, such as computing capacity or storage capacity, the other resource is often affected as well, resulting in lower resource utilization.
In addition, the complexity of the big data ecosystem introduces higher costs in terms of component management, version compatibility, system maintenance, and technical learning.
Therefore, modern enterprise data warehouse practices are increasingly adopting lightweight and high-performance OLAP architectures.
1. Technology Selection: Using OLAP Databases to Support High-Performance Analytics
During the early phase of data warehouse development, the team adopted a traditional big data warehouse architecture. However, as data volume increased and business analytics requirements became more complex, several limitations gradually emerged in real-world usage.
First, traditional data warehouse architectures are mainly designed for offline batch processing scenarios. Job execution usually involves relatively high latency, with many tasks taking several minutes to complete, making it difficult to meet the low-latency requirements of certain business analytics scenarios.
Second, traditional architectures suffer from tightly coupled storage and compute resources. Since computing nodes are responsible for both data storage and processing, scaling either compute or storage independently becomes difficult. This can lead to inefficient resource utilization and increase the complexity of platform expansion.
Furthermore, traditional big data ecosystems usually rely on multiple components working together. The large number of components, combined with version compatibility requirements, increases the complexity of platform deployment, learning, and long-term operations.
To address these challenges in performance, scalability, and operations, the team adopted a distributed OLAP database architecture in its next-generation data warehouse implementation, improving analytics capabilities and overall platform efficiency.
In the new architecture, the orchestration layer adopts the open-source project Apache DolphinScheduler for workflow orchestration, task scheduling, dependency management, retry and alerting, resource management, and visual operations.
The data integration layer adopts another open-source project, Apache SeaTunnel, which provides data source connectivity, batch and streaming data synchronization, data transformation, and cleansing capabilities.
The new architecture delivers several improvements.
First, it provides significantly improved query performance. Based on distributed OLAP architecture, it enables sub-second responses for large-scale data queries. In wide-table aggregation scenarios, performance can improve by 5 to 10 times compared with traditional solutions, better supporting fast business analytics.
Second, the new architecture provides stronger scalability. Compared with traditional Hadoop ecosystems that rely on multiple components such as HBase and Spark, the new solution reduces dependencies on numerous third-party components, resulting in a simpler architecture and lower maintenance complexity.
Meanwhile, based on distributed cluster technology, the architecture provides stronger concurrent processing capabilities, reduces single-node I/O pressure, and improves overall platform stability.
Through this technology selection, the team successfully upgraded from a traditional data warehouse architecture to a modern OLAP-based architecture, establishing a more reliable data foundation for subsequent data synchronization, processing, and business applications.
2. Environment Setup: Reducing the Complexity of Big Data Platform Deployment
After completing the architecture selection, the next step was to build the infrastructure required for running data processing workflows.
During the environment setup process, the team used a cluster management tool to deploy the big data environment. The tool provides fast deployment capabilities, compatibility with open-source ecosystems, and simplified operations and maintenance.
Through centralized management, users can quickly initialize large-scale big data clusters while covering multiple areas including data integration, storage, computing engines, task scheduling, and permission management.
From an operations perspective, the tool enables unified monitoring and management of clusters, nodes, and services, improving daily maintenance efficiency.
After completing the infrastructure setup, the team further deployed Apache DolphinScheduler to provide unified scheduling capabilities for subsequent data workflows.
3. Deployment Process: Setting Up Apache DolphinScheduler
Within the overall data warehouse architecture, Apache DolphinScheduler is responsible for data workflow orchestration and scheduling management.
By deploying DolphinScheduler, the team was able to integrate subsequent data synchronization and data processing tasks into a unified workflow management system, providing visual orchestration and automated execution capabilities.
Data Processing: Connecting the Data Production Pipeline with Apache DolphinScheduler
After completing the infrastructure setup, data processing becomes the core stage of enterprise data warehouse development. In this practice, the team focused on data synchronization, data transformation, and task scheduling, using Apache DolphinScheduler to orchestrate and manage data workflows.
Within the overall data processing pipeline, DolphinScheduler is responsible for workflow management and task scheduling, while Apache SeaTunnel handles data synchronization. Together, the two projects enable an automated process from data ingestion to data processing.
This practice mainly covers Apache DolphinScheduler deployment, SeaTunnel integration, mirror table synchronization, model-layer processing, exchange table applications, and exploration of real-time data synchronization solutions.
1. Apache DolphinScheduler Application: Enabling Unified Data Task Scheduling
During the data processing phase, the team adopted Apache DolphinScheduler to centrally manage data workflows.
Compared with traditional approaches that rely on manually maintained scripts to manage task execution dependencies, DolphinScheduler provides visual workflow orchestration capabilities.
Users can define workflows through a drag-and-drop interface and organize different data processing tasks into DAG workflows based on their dependencies, significantly reducing workflow management complexity.
At the same time, DolphinScheduler supports multiple task types and provides cross-language extensibility, enabling it to adapt to diverse data processing scenarios. In enterprise data warehouse projects, different types of data tasks can be integrated into a unified scheduling system, simplifying workflow management and maintenance.
In terms of reliability, DolphinScheduler adopts a decentralized architecture, improving scheduling system stability and ensuring continuous execution of data workflows.
By introducing DolphinScheduler, the team connected data synchronization and data processing steps into a unified workflow, enabling centralized scheduling throughout the data production pipeline.
2. Apache SeaTunnel Integration: Building Data Synchronization Pipelines
For data ingestion and synchronization, the team introduced Apache SeaTunnel as the data integration component responsible for transferring data between different data sources.
Based on its Connector architecture, SeaTunnel supports multiple data synchronization scenarios, including offline synchronization, real-time synchronization, full synchronization, and incremental synchronization, reducing the complexity of managing data integration tasks.
At the execution engine layer, SeaTunnel uses the Zeta engine by default while also supporting Flink and Spark as Connector execution engines, providing greater flexibility for different business scenarios.
In addition, SeaTunnel improves data transfer efficiency through parallel reading and writing mechanisms, delivering stable, high-throughput, and low-latency data synchronization capabilities.
Within the overall data processing architecture, DolphinScheduler and SeaTunnel work together: DolphinScheduler handles workflow orchestration and execution scheduling, while SeaTunnel performs data synchronization operations. Together, they provide the foundation for efficient data movement across the data warehouse pipeline.
3. Mirror Table Synchronization: Improving Data Synchronization Flexibility
During the data synchronization process, the team adopted a mirror table synchronization approach to complete data migration.
Schema synchronization, flexible configuration, and simple operations
This solution provides a configuration experience similar to DataX while offering more comprehensive data synchronization capabilities.
In practical applications, the team used the tool to synchronize table schemas and then leveraged DolphinScheduler to centrally schedule synchronization tasks, enabling automated data migration workflows.
Data synchronization (DolphinScheduler)
By integrating synchronization tasks into the scheduling system, data migration workflows can be automatically executed according to predefined processes. Combined with task status management, this approach improves the controllability and reliability of data processing operations.
4. Model Layer Processing: Simplifying Data Transformation Workflows
After completing data synchronization, the team further processed data at the model layer.
During the model-layer data cleansing stage, Shell scripts were used for business logic processing. This scripting approach improved development efficiency while reducing future maintenance costs.
In the actual workflow, model-layer processing tasks were also managed through DolphinScheduler, forming a complete data processing pipeline together with upstream and downstream synchronization tasks.
5. Exchange Table Application: Enabling Seamless Data Switching
To improve data reliability and stability during updates, the team adopted an exchange table mechanism for data switching.
Through this approach, table data can be switched seamlessly without affecting business users, minimizing the impact of data update operations.
Combined with DolphinScheduler’s workflow orchestration capabilities, data processing and table switching operations can be executed automatically according to predefined sequences, further improving the stability of the data production pipeline.
6. Real-Time Solution Exploration: Investigating CDC-Based Real-Time Synchronization
Beyond offline data processing workflows, the team also explored solutions for real-time data synchronization scenarios.
This practice investigated a CDC-based real-time synchronization architecture without relying on Flink. The solution features a simpler architecture, strong performance, and easier operations and maintenance.
Through exploring real-time synchronization solutions, the team further enhanced its data processing capabilities and prepared the platform to support a wider range of future data application scenarios.
Data Applications: From Data Analytics to Intelligent Data Services
After data ingestion, synchronization, and transformation are completed, data needs to be further leveraged to support business analytics and application scenarios.
In this data warehouse implementation, the team focused on three key areas: BI applications, AI agent applications, and data governance, expanding the platform’s data service capabilities.
1. BI Applications: Supporting Data Analytics and Business Visualization
At the data application layer, the team connected databases with BI tools through database interfaces to enable analytics and visualization.
Based on built-in JDBC drivers, the database can be connected directly to BI platforms, providing strong concurrency capabilities and fast query response times. Meanwhile, compatibility with the MySQL protocol enables integration with existing data analytics tools.
Reporting application (FineReport)
For practical applications, the team used FineReport for reporting scenarios and DataEase for dashboard visualization.
Dashboard application (DataEase)
Through the previous data processing workflow, data synchronization and transformation tasks are centrally scheduled by Apache DolphinScheduler, ensuring that data updates are completed according to predefined processes and providing a stable foundation for BI analytics and visualization.
2. AI Agent Applications: Exploring Data Querying Scenarios
In traditional data analytics scenarios, users typically rely on reports or query tools to access data. To further expand data application methods, the team developed a data query AI agent based on Dify.
By building a data query AI agent, users can access data through a more convenient interaction method, providing new possibilities for data applications.
This practice demonstrates that a data platform needs to provide not only reliable data processing capabilities but also strong data support for upper-layer applications. The data consumed by intelligent applications still relies on continuous data generation through underlying data warehouse synchronization and processing workflows.
3. Data Governance: Enhancing Metadata Management Capabilities
As enterprise data assets continue to grow, data governance has become an essential component of modern data warehouse development.
In this practice, the team used OpenMetadata to support metadata management for Doris databases.
Through metadata management, data asset information stored in databases can be centrally managed, providing better support for data usage, maintenance, and future expansion.
Operations and Maintenance Upgrades: Ensuring Stable Data Warehouse Operations
After completing data warehouse construction and application implementation, long-term platform stability still requires a comprehensive operations and maintenance system.
In this practice, the team enhanced platform operations capabilities in three key areas: log monitoring, upgrade management, and failure alerting.
1. Log Monitoring: Improving System Observability
During daily operations, the team uses log monitoring to gain real-time visibility into the status of different platform components.
By monitoring component logs in real time and combining metric collection with customized alert mechanisms, the team can quickly identify and address potential issues.
In practice, the team uses Apache DolphinScheduler to view task execution logs while integrating Prometheus and Grafana for daily monitoring.
Among these tools, DolphinScheduler provides visibility into task execution logs, while Prometheus and Grafana monitor platform metrics. Together, they improve overall system observability and operational efficiency.
2. Upgrade Management: Simplifying Doris Cluster Operations
For database cluster maintenance, the team adopted DorisManager for Doris cluster management.
Through this all-in-one Doris cluster management tool, cluster upgrades and maintenance operations can be simplified, improving overall platform management efficiency.
3. Failure Alerting: Establishing Task Failure Notification Mechanisms
To detect data task failures in a timely manner, the team configured alert instances in Apache DolphinScheduler and integrated them with Feishu bots for failure notifications.
When a task fails during execution, the system can automatically send notifications through the alert mechanism, helping relevant teams quickly identify and resolve issues.
Conclusion: Apache DolphinScheduler Enables End-to-End Enterprise Data Warehouse Development
At the Apache DolphinScheduler September Meetup, the community shared a practical enterprise data warehouse implementation covering the complete lifecycle from environment setup and data processing to data applications and operational management.
Within this architecture, Apache DolphinScheduler runs throughout the entire data production workflow, connecting data synchronization, data transformation, and operational management through workflow orchestration and task scheduling capabilities.
Apache SeaTunnel handles data synchronization, OLAP databases provide analytical capabilities, while BI tools, AI agents, and metadata management systems further extend data application scenarios.
Through this implementation, the team built a comprehensive data platform covering both data processing and data applications, providing reliable support for stable enterprise data warehouse operations.
Top comments (0)