In an enterprise data platform, synchronizing data from a business system to a target database is often only the first step in the data processing pipeline.
Before business data enters a data warehouse or analytics system, it often needs to go through a series of processing steps, including:
- Field selection;
- Format adjustments;
- Anomaly handling;
- Deduplication;
- Value mapping.
At the same time, a data task that runs continuously needs to answer several additional questions:
When should it run? Which execution engine should it use? Did the task complete successfully? How can issues be identified when something goes wrong?
Therefore, a complete ETL task is not simply:
Input → Transform → Output
It also needs to form a complete process:
Task Creation → Runtime Configuration → ETL Workflow Orchestration → Scheduled Execution → Run Result Monitoring
For this scenario, qData Open-Source Edition provides visual data integration capabilities.
Users can create data integration tasks, build data ingestion, transformation, and output workflows through drag-and-drop components, and combine execution engine and scheduling configurations to put the designed ETL workflow into continuous operation.
Using a real water level anomaly handling task as an example, let’s look at how an ETL workflow is created and executed step by step.
Step 1: Create a Data Integration Task
In qData, ETL workflows are first managed as data integration tasks.
When creating a task, in addition to basic information such as the task name, task category, and owner, users also need to determine how the task will be executed later.
qData currently supports different execution engines for different task scenarios.
For relatively lightweight data reading, writing, and synchronization tasks, users can select DataX and configure:
- Initial JVM memory;
- Maximum JVM memory;
- Channel concurrency;
- Byte throttling;
- Record throttling;
- Dirty data limit.
These parameters do not determine how the data is processed. Instead, they control how relevant resources are used while the task is running.
For larger-scale offline data processing scenarios, users can select Spark and configure parameters such as:
- Driver cores;
- Driver memory;
- Number of Executors;
- Executor cores;
- Executor memory;
- Yarn queue.
The current page also reserves an entry for the Flink execution engine.
However, in the version shown in the document, it is marked as not yet available, so the currently available execution methods remain the primary options.
Once the execution engine is determined, another question needs to be answered:When should the task run?
Therefore, the task creation process also allows users to configure the scheduling system and corresponding scheduling method.
It can be simply understood as:
- Execution Engine: How the task runs；
- Scheduling Configuration: When the task runs；
- ETL Workflow: What the task actually does during execution。
After completing these basic settings, select Save and Configure Workflow to enter the actual data processing workflow design stage.
Step 2: Build the ETL Workflow on a Visual Canvas
After entering the workflow configuration page, available data processing components are provided on the left, while the center is a visual workflow canvas.
Users can drag different components onto the canvas according to the data processing logic, then connect them to define the order of data flow.
For the water level anomaly handling example, the basic workflow consists of three core nodes:
Table Input → Transformation → Table Output
Each node has a different responsibility:
- Table Input: Where does the data come from?
- Transformation: What processing and cleansing should be performed after the data enters the workflow?
- Table Output: Where should the processed data be written?
The value of this approach is not simply turning scripts into several graphical components.
For subsequent maintenance, opening a task allows users to understand its data flow directly from the workflow structure:
Where does the data come from? Which nodes does it pass through? Where does it ultimately go?
When business rules need to be adjusted, users can also locate the corresponding processing node first and then modify its configuration, rather than searching through large amounts of scripts to find the relevant logic.
Step 3: Configure the Input — Where Does the Data Come From?
The first question an ETL workflow needs to solve is:
Where is the data that needs to be processed?
After opening the Table Input component, users can configure the data source for the current node.
In this example, the water level data comes from:
Water Resources Management System (Third-Party Database)
After selecting the corresponding data connection and then specifying the WATER_LEVEL table, the data source for this ETL task is defined.
The system then reads and displays the table structure, including fields such as:
ID、STATION_CODE、OBS_TIME、WATER_LEVEL
These fields are then passed to the subsequent transformation and output nodes.
In addition to selecting a database table directly, the current Table Input component also provides other integration methods, including:
- SQL；
- Data connections；
- Asset tables.
For this example, because the task only needs to read water level data from an existing business system database, the data connection and corresponding table can be selected directly.
At this point, the first question in the ETL workflow is clearly defined:Where does the data come from, and which data needs to be processed next?
Step 4: Data Transformation — What Happens After the Data Is Connected?
If a table is simply copied as-is from one table to another, the process is closer to data synchronization.
The more important part of ETL is:
What processing should be performed after the data is read?
Therefore, transformation nodes can be added between the Table Input and Table Output components.
qData currently provides multiple transformation components, including:
Sort Records,Field Derivation,Remove Duplicate Records,Add Constant,Field Selection and Modification,Value Mapping.
These components are designed for different data processing scenarios.
For example:
- If the source data contains fields that are not needed later, field selection can be used.
- If the same batch of data contains duplicate records, deduplication can be performed.
- If field values from different business systems use inconsistent definitions, value mapping can be used for conversion.
- If a new field needs to be calculated based on existing fields, field derivation can be used.
In the current water level data example, the key issue is:Handling anomalous water level values.
Why can’t the water level data simply be written directly into the downstream data warehouse?
Suppose the source system’s WATER_LEVEL field contains values that are clearly outside the normal business range.
If these anomalous values are written directly into the downstream data warehouse without processing, they may continue to participate in statistics, analysis, and metric calculations, ultimately affecting the results of data usage.
Therefore, an anomaly handling rule can be added for the WATER_LEVEL field in the transformation node, together with the corresponding data cleansing method.
At this point, the workflow no longer simply states:
“Perform data transformation here.”
It further defines:
Which field should be processed, and according to which rule?
This allows data cleansing logic to be applied to specific fields and specific rules.
Step 5: Turn Common Cleansing Logic into Configurable Rules
In enterprise data governance, many data quality issues tend to occur repeatedly.
If a new processing logic has to be developed every time a problem appears, maintenance costs will increase as the number of tasks grows.
Therefore, qData’s transformation components can also use existing, maintained data cleansing rules.
For example:
- Date Format Standardization：Convert dates from different sources into a specified format.
- Decimal Place Standardization：Standardize the decimal precision of data from different sources.
- Remove Field Spaces：Process unnecessary spaces before and after strings.
- Enumeration Value Mapping Standardization：Convert field values from different systems into a unified representation.
- Numeric Boundary Adjustment：Process data that falls outside a specified range.
Returning to the water level anomaly handling example, corresponding numeric processing rules can be configured for the WATER_LEVEL field according to actual business requirements.
In this way, the source data is not written directly into the target table.
Instead:
Read → Cleanse → Output
The role of ETL therefore extends beyond simple data movement to also include the necessary data processing and standardization during data flow.
Step 6: Configure the Output — Where Does the Processed Data Go?
After data transformation is complete, the endpoint of the entire workflow needs to be determined:
Where should the processed data be written?
This is handled by the Table Output component.
If Table Input answers:
“Where do we read from?”
then Table Output answers:
“Where do we write to?”
In this example, the processed water level data is ultimately written to:
DWD_WATER_LEVEL_CLEAN_OUTLIER
But selecting the target table is only the first step.
A mapping relationship also needs to be established between the source and target fields, for example:
ID → ID
STATION_CODE → STATION_CODE
OBS_TIME → OBS_TIME
WATER_LEVEL → WATER_LEVEL
Through field mapping, users can directly see where each source field will ultimately be written in the target table.
If the source and target table structures are not exactly the same, field selection, modification, or processing can first be completed through transformation nodes before the final field mapping.
At this point, the core data processing path of the ETL workflow has been established:
Data Reading → Data Transformation and Cleansing → Field Mapping → Data Output
Step 7: The Workflow Is Ready. How Does It Keep Running?
Completing the visual workflow configuration does not mean the data will automatically start being processed.
The previous steps answered:
“How should this task process data?”
The next step is the actual execution stage.
Because the execution engine and scheduling method were already configured when the task was created, once the scheduled time is reached, the scheduling system triggers the current data integration task.
The configured DataX or Spark execution engine then runs the corresponding data processing workflow.
The responsibilities of the three parts are clear:
- Scheduling System: When to execute；
- DataX / Spark: How to execute；
- ETL Workflow: What to process during execution.
Taking the water level task as an example, when the scheduled time arrives, the system reads the new WATER_LEVEL data, applies the configured anomaly handling rules, and writes the cleansed results to the target table.
The same ETL workflow does not need to be configured again each time.
It can continue running according to the predefined rules.
Step 8: Once the Task Is Running, You Also Need to Know How It Is Running
For an actual data platform, being able to execute a task is only the foundation.
During long-term operation, users also need to know:
- Is the task enabled?
- Has scheduling been enabled?
- What execution cycle is being used?
- Did the most recent execution succeed or fail?
Therefore, after execution, data integration tasks can still be viewed and managed centrally from the task list.
If the most recent task execution succeeded, the corresponding status can be viewed directly.
If an exception occurs, task runtime information can be used to further locate the problem.
This is especially important for data tasks that run over the long term.
A data integration task may continue running for months or even longer.
During that time, issues such as changes to source data structures, adjustments to business rules, or abnormal data may occur and require maintenance.
Therefore, from task creation and workflow configuration to scheduled execution and runtime monitoring, what is ultimately established is a complete data integration task management process, rather than a one-time ETL configuration.
Back to the Example: How Does the Water Level ETL Task Actually Run?
Connecting the previous steps together, the complete process of the water level anomaly handling task becomes clear:
① Create the Data Integration Task
Configure the task name, category, owner, and other information. Select DataX or Spark, and set the scheduling strategy.
② Build the Visual ETL Workflow
Use Table Input, Transformation, and Table Output components to establish the basic processing flow.
③ Connect the Source Data
Connect to the third-party Water Resources Management System and read the WATER_LEVEL table, including fields such as:ID、STATION_CODE、OBS_TIME、WATER_LEVEL.
④ Process Anomalous Water Level Values
Apply the corresponding anomaly handling rules to the WATER_LEVEL field.
⑤ Configure Output and Field Mapping
Map source fields to target fields and write the processed data to:DWD_WATER_LEVEL_CLEAN_OUTLIER。
⑥ Trigger Execution Through Scheduling
Once the scheduled time is reached, the scheduling system triggers the task, and DataX or Spark processes the data according to the configured ETL workflow.
⑦ Check the Runtime Status
Return to the data integration task list to view the latest execution result and task runtime status.
At this point, the complete data processing chain has been established—from raw water level data in a third-party business system, through anomaly handling, to the target data table.
The Value of Drag and Drop Is More Than “Writing Less Code”
When people talk about visual ETL, one of the first questions that comes to mind is:Can it reduce the amount of SQL or scripting required?
But for a data platform, the more important value of visual orchestration is that it organizes what was previously scattered across:
- Data reading;
- Data transformation;
- Data cleansing;
- Field mapping;
- Data output;
- Task execution.
into a data processing workflow that can be viewed directly.
By opening a task, users can answer several questions more intuitively:
- Where does the data come from?
- What processing does it go through?
- Where does it ultimately go?
- How does the workflow continue to run?
For users who are just beginning to work with data platforms, they can also start with the most basic:
Input → Transform → Output
and build their first ETL workflow, then gradually add data cleansing, transformation rules, and scheduling configurations based on actual business requirements.
Conclusion
Data integration is not simply about synchronizing data from one system to another.
A truly long-running ETL task needs to address:
- Data integration;
- Data processing and transformation;
- Data cleansing;
- Field mapping;
- Data output;
- Execution engines;
- Scheduled execution;
- Runtime management.
qData Open-Source Edition connects these stages through visual ETL orchestration:
Create the task → Configure the workflow → Bring the data in, process it, and write it out → Use execution engines and scheduling capabilities to keep the configured workflow running as planned
The water level anomaly handling example also shows that the key value of visual ETL is not simply reducing the amount of code required for workflow configuration.
It is making the source, processing steps, output direction, and execution method of a data processing task more explicit.
For data platform development, this complete management approach—from data flow to task execution—also provides a more intuitive foundation for continuously adding data cleansing rules, expanding processing workflows, and maintaining long-running data tasks.
Top comments (0)