DEV Community: Apache SeaTunnel

🔍 A successful pipeline doesn’t guarantee trusted data. See how Apache SeaTunnel enables reliable movement, recovery, and schema evolution.

Apache SeaTunnel — Thu, 27 Aug 2026 07:50:12 +0000

Apache SeaTunnel

Aug 27

Pipeline Green Isn’t Data Correct

#ai #datascience #apacheseatunnel #programming

11 min read

🔄 A financial tech company migrated 200+ ETL workflows from Informatica to Apache SeaTunnel, cutting infrastructure costs 60% and runtimes 40%!

Apache SeaTunnel — Thu, 27 Aug 2026 07:49:43 +0000

Apache SeaTunnel

Aug 27

From Informatica to Apache SeaTunnel: A Financial-Grade ETL Migration in Practice

#apacheseatunnel #etl #datascience #dataengineering

5 min read

From Informatica to Apache SeaTunnel: A Financial-Grade ETL Migration in Practice

Apache SeaTunnel — Thu, 27 Aug 2026 07:31:23 +0000

1. Project Background and Challenges

As the head of data architecture at a financial technology company, I led the migration of our core ETL system from Informatica PowerCenter to a domestic ETL platform last year. The migration covered more than 200 workflows and a core system processing terabytes of data every day. It took five months from start to finish, and we ultimately completed the transition smoothly with zero data incidents. Today, I’d like to share the key decisions, technical details, and lessons learned from the project.

As a long-standing leader in enterprise data integration, Informatica has dominated the traditional ETL market for more than 20 years. Its visual development environment, reliable scheduling engine, and comprehensive metadata management have made it a de facto standard in industries such as financial services and telecommunications. However, with the changing international landscape and the growing demand for domestic technology adoption, we had to confront three practical challenges:

High licensing costs: Annual maintenance fees running into millions of RMB were a significant burden for a mid-sized enterprise.
A closed technology stack: It was difficult to integrate deeply with emerging real-time computing and AI platforms.
Slow response to customization needs: Custom requirements often required cross-border collaboration, resulting in delivery cycles that could stretch to several months.

After years of development, domestic ETL platforms such as Kettle, DataX, and SeaTunnel have become viable alternatives for core ETL capabilities. Our technical evaluation showed that, in batch-processing scenarios, domestic platforms could cover approximately 85% of Informatica’s functionality while costing only one-third as much and offering the flexibility for secondary development.

2. Migration Strategy and Design

2.1 Technology Selection

We conducted an in-depth evaluation of three mainstream domestic ETL tools:

We ultimately selected Apache SeaTunnel as the primary migration platform for three main reasons:

Support for both Spark and Flink engines, making it a good fit for our future real-time data warehouse strategy.
A plugin-based architecture that makes it easier to extend support for custom data sources.
An active Chinese-language community that enables us to resolve technical issues quickly.

2.2 Migration Strategy

We adopted a hybrid approach combining phased migration with parallel-run validation:

Decouple the components: Break each Informatica workflow into three independent modules for extraction, transformation, and loading.
Map the functionality:

Source data extraction → SeaTunnel Source plugins
Complex transformation logic → Rebuild with Spark SQL
Scheduling dependencies → Orchestrate with Apache DolphinScheduler
1. Validate the data:

# Use a combination of CRC32 and sample-based comparison for validation
def verify_data(source_df, target_df):
    if source_df.count() != target_df.count():
        return False
    sample_ratio = 0.01
    source_sample = source_df.sample(sample_ratio)
    target_sample = target_df.sample(sample_ratio)
    return source_sample.exceptAll(target_sample).isEmpty()

Key lesson: Don’t try to replicate Informatica workflows 1:1. Use the migration as an opportunity to optimize the data flows. We redesigned 30% of the transformation logic that had performance bottlenecks.

3. Core Migration Implementation

3.1 Metadata Migration

The Informatica Repository contained thousands of metadata objects. We developed a metadata parsing tool to automate the migration:

Export XML metadata through the PowerCenter CLI.
Use XSLT to transform key attributes:

<!-- Example of mapping transformation -->
<xsl:template match="SOURCE">
  <connector type="jdbc">
    <property name="url" value="{@DBSERVER}"/>
    <property name="table" value="{@OBJECTNAME}"/>
  </connector>
</xsl:template>

Generate SeaTunnel configuration file templates.

3.2 Refactoring Complex Transformations

Informatica components such as Expression and Aggregator required special handling during the migration.

Conditional routing: The original workflows used the Router component.

-- Reimplemented with Spark SQL
df.createTempView("source");
spark.sql("""
  SELECT *, 
    CASE 
      WHEN amount > 10000 THEN 'VIP' 
      ELSE 'NORMAL' 
    END AS customer_level
  FROM source
""");

Slowly Changing Dimensions (SCD): The original implementation relied on the Slowly Changing Dimension wizard.

-- Implement Type 2 SCD using MERGE INTO
MERGE INTO dim_customer t
USING stage_customer s
ON t.customer_id = s.customer_id
WHEN MATCHED AND t.current_flag='Y' AND t.email <> s.email THEN
  UPDATE SET t.current_flag='N', t.end_date=CURRENT_DATE
  INSERT VALUES (s.customer_id, s.email, ..., 'Y', CURRENT_DATE, NULL)

3.3 Performance Tuning in Practice

The most challenging issue we encountered was an ETL job containing 20 joins that ran into an OOM error on SeaTunnel. We resolved it through the following optimizations:

Analyze the execution plan:

# Get the Spark physical execution plan
EXPLAIN EXTENDED 
SELECT * FROM fact f JOIN dim1 d1 ON f.id=d1.id ...

Optimization measures:

Enable dynamic partition pruning: spark.sql.optimizer.dynamicPartitionPruning=true
Adjust the broadcast threshold: spark.sql.autoBroadcastJoinThreshold=20MB
Force broadcast joins for dimension tables: /*+ BROADCAST(dim1) */

Parameter comparison:

4. Validation and Cutover

4.1 Ensuring Data Consistency

We established a three-level validation framework:

Record-level validation: Generate a CRC32 fingerprint for the entire table.

SELECT 
  SUM(CAST(CRC32(CONCAT_WS('|',col1,col2,...)) AS BIGINT)) AS checksum 
FROM table

Business metric comparison: Keep month-over-month fluctuations in key KPIs below 1%.
User acceptance testing: Have business teams validate the data in their reports.

4.2 Gradual Rollout

We migrated workloads incrementally by business line:

First, migrate non-core marketing analytics workloads.
Next, migrate the risk management system.
Finally, migrate the financial settlement system.

We monitored each phase for one week, with a particular focus on:

Data latency
Resource utilization
Error logs

5. Lessons Learned

5.1 Key Success Factors

Team training: Two months before the migration, we organized training sessions for Informatica developers to learn Spark and SeaTunnel.
A complete toolchain:

Developed an auxiliary workflow conversion tool.
Built an automated data comparison platform.
1. Vendor support: Established a direct communication channel with the SeaTunnel core team.

5.2 Lessons Learned the Hard Way

Time zone issues: Informatica uses the server's time zone by default, while Spark uses UTC. We therefore needed to explicitly handle time zone conversion for all relevant time fields:

FROM_UTC_TIMESTAMP(CAST(col AS TIMESTAMP), 'Asia/Shanghai')

Character encoding pitfalls: The ZHS16GBK encoding used by the Oracle source database had to be explicitly configured:

source:
  jdbc:
    connection_options: "oracle.jdbc.convertNlsStrings=true"

Transaction semantics: Informatica uses auto-commit by default, while Spark requires transaction behavior to be controlled explicitly:

df.write
  .option("isolationLevel", "READ_COMMITTED")
  .mode("overwrite")
  .saveAsTable("target")

Quantified Results After the Migration

60% reduction in infrastructure costs, from eight physical servers to a Kubernetes cluster
40% reduction in average job execution time
New capabilities for real-time data processing

The biggest takeaway from this migration is that domestic infrastructure software has reached a level where it can serve as a viable alternative to traditional enterprise platforms. But successful migration requires more than simply replacing one tool with another. It requires a shift in technical mindset.

Instead of treating migration as a straightforward tool replacement exercise, we used it as an opportunity to rethink and modernize our data architecture, laying the groundwork for the next stage of real-time processing and intelligent data operations.

Pipeline Green Isn’t Data Correct

Apache SeaTunnel — Thu, 27 Aug 2026 07:06:32 +0000

In data engineering, Fivetran is often seen as the go-to choice for simplicity and low operational overhead, while open-source projects such as Apache SeaTunnel offer greater flexibility and more control over the underlying data infrastructure. So, should a data team choose Managed ELT, or build and operate its own data integration platform? This has long been a point of debate among data teams.

A recent discussion in Reddit’s r/dataengineering community provides a representative example. A data engineer was scaling up data operations after primarily relying on Cloud Run Jobs and dbt Core. As data pipelines gradually became a core part of the business infrastructure, the team began evaluating two options: Fivetran + dbt, or an open-source stack + dbt Core. Their concern was not simply which tool to use, but whether Fivetran’s cost was justified and, beyond its large library of pre-built connectors, what Managed ELT actually delivers for an enterprise.

This is an important question because as data volumes grow, the real challenge is no longer simply “Can we move the data?” It becomes “Can we move the data reliably, and how can we prove that the data is correct once it gets there?”

That is why evaluating a data integration platform based solely on whether a pipeline succeeds is nowhere near enough.

Pipeline Green does not mean Data Correct.

When a job reports SUCCESS, it only means that the system considers the execution complete. It does not prove that the source and target data are complete and consistent. It does not prove that the data is fresh enough. And it certainly does not prove that business metrics have not deviated from expectations.

A mature data engineering architecture needs to address data movement, workflow orchestration, state management and recovery, runtime observability, and data quality validation as a whole.

From this perspective, instead of simply debating “Fivetran or Apache SeaTunnel,” it is more useful to break down the engineering principles behind these two approaches.

Managed ELT Solves Operational Overhead, Not Data Correctness

The value of Fivetran is easy to understand. For a large number of SaaS data sources, developing and maintaining connectors is itself a significant engineering investment. API authentication, pagination, rate limiting, incremental synchronization, schema changes, retries, and changes to third-party APIs all require ongoing effort.

For smaller companies without a dedicated data infrastructure team, outsourcing this work to a Managed Service can significantly lower the barrier to building data pipelines.

The Reddit discussion also included users who argued that Fivetran’s biggest value is reducing the infrastructure and connector maintenance burden, allowing data teams to onboard new SaaS sources quickly. That value is real.

But there is an important distinction:

Managed does not mean Observable.

Imagine an advertising data synchronization job completes normally at 2 a.m. The platform reports Pipeline Success. The next morning, however, the business team discovers that advertising spend is 20% lower than expected.

A number of things could have happened. The upstream API may have returned incomplete data. Data for a particular time window may have been delayed. A schema change may have caused some fields to be written incorrectly. Or the synchronization job itself may have worked perfectly while a downstream model introduced the problem.

From the pipeline’s perspective, these situations may result in different technical states. From the business perspective, they lead to the same outcome:

The data cannot be trusted.

This means data engineering needs to distinguish between at least three layers.

Pipeline Health focuses on whether a job executed successfully. Data Observability focuses on whether the data is fresh, complete, and stable. Data Quality and Business Reconciliation go one step further by validating whether the data conforms to business rules and whether key business metrics make sense.

These three layers are complementary. None can replace the others.

If you only look at Pipeline Status, a “green” pipeline can still produce incorrect data.

This is why SeaTunnel needs to be understood as part of a broader architecture. SeaTunnel does not need to position itself as a platform that solves every data quality problem. Its more appropriate role is to provide one of the most fundamental layers of the data infrastructure stack: reliable data movement and execution.

SeaTunnel’s First Job: Getting Data There Reliably

In a simple data synchronization task, the flow looks straightforward:

Source → Transform → Sink

But once the workload enters production, the real challenges quickly emerge.

What happens if a job has been running for several hours, processing tens or even hundreds of millions of records, and a Worker suddenly fails? Where should the job resume? If the Source has already read part of the data while the Sink has committed only part of it, how can the system avoid duplicate writes when the job restarts? And in a CDC pipeline, how does the system know which MySQL Binlog position, PostgreSQL LSN, Oracle SCN, or parallel-read Split has already been processed?

This is one of the most important differences between a data synchronization system and a simple script.

SeaTunnel Zeta Engine’s Checkpoint and State mechanisms are designed to address state management and failure recovery for continuously running workloads. A Checkpoint records the state of a pipeline at a specific point in time, allowing a failed job to recover from the latest valid state rather than simply starting over.

For CDC workloads, this state can also be associated with information such as MySQL Binlog positions, PostgreSQL LSNs, Oracle SCNs, and the processing progress of parallel-read Splits.

This means reliability in SeaTunnel is not simply about “retrying automatically after a failure.”

The fundamental difference is that Retry means running the work again, while Recovery means knowing what has already been completed and continuing from the correct position.

At large data volumes, that distinction becomes critical.

Suppose a 500-million-row synchronization job fails after reaching 80% completion. Restarting from the beginning means repeating a huge amount of computation and may also result in duplicate data on the target side. State-based recovery, by contrast, can limit the impact of the failure to a much smaller portion of the workload.

For large-scale data synchronization, therefore, Checkpoint is not an optional add-on. It is part of the infrastructure required for reliable data movement.

From Checkpoint to Schema Evolution: Building Reliability into the Runtime

Reliable data movement is not only about recovering from failures.

Another common challenge for modern data platforms is Schema Drift. This is particularly relevant to the SaaS API scenarios discussed in the Reddit thread, where the upstream data structure is not fully under the enterprise’s control.

A new API field, a changed data type, or even a renamed field can potentially affect the entire downstream data pipeline.

More importantly, schema changes do not necessarily cause a pipeline to fail immediately.

For example, a field that was previously numeric might later be returned as a string. Or conversion_value might be renamed to conversionValue. If the connector and downstream processing logic do not correctly detect and handle these changes, the pipeline may continue running normally while ultimately producing missing or incorrect data.

This is another classic example of why Pipeline Green does not mean Data Correct.

SeaTunnel’s Schema Evolution capabilities allow data synchronization to account for changes in data structures rather than simply moving the data itself. Through mechanisms such as Schema Mapping, DDL Propagation, Dynamic Application, and Compatibility Checks, schema changes can become part of the pipeline’s runtime processing rather than an unexpected source of failure.

This is particularly important for CDC workloads. When the schema of a source database table changes, a data synchronization system cannot simply assume that the schema will remain unchanged forever.

From this perspective, a truly reliable data pipeline needs to manage at least three types of state simultaneously: the data itself, the schema of that data, and the point up to which the data has been processed.

SeaTunnel’s runtime design is built around these three dimensions to provide reliable data synchronization.

And that is one of the biggest differences between SeaTunnel and a simple ETL script.

SeaTunnel Doesn’t Prove Business Data Is Correct. It Provides the Foundation for Proving It.

There is an important distinction here that is easy to overlook: SeaTunnel’s Checkpoint, Recovery, and Runtime Metrics capabilities are not the same thing as comprehensive Data Observability.

Suppose a SeaTunnel job completes successfully. The source system generated 500 million records, but only 497 million ultimately reached the target.

It would be inaccurate to simply call this a SeaTunnel pipeline failure. From the execution engine’s perspective, the job may have completed all of its configured operations successfully.

The problem belongs to a higher-level data validation layer.

A well-designed data platform should therefore establish a clear division of responsibilities:

SeaTunnel ensures that data can move reliably from Source to Target while providing task status, Checkpoints, execution metrics, and failure recovery.

Data Quality and Observability tools then validate metrics such as freshness, row count, schema, and distribution.

Finally, business reconciliation verifies whether critical business metrics are consistent with expectations.

For example, after an advertising data synchronization job completes, it is not enough to verify that the job succeeded. You should also check the source and target row counts, the latest data timestamp, and key metrics such as spend, clicks, impressions, and conversions.

If the source has data through 10 a.m. while the target still contains data only through 3 a.m., the pipeline should be considered to have a data freshness problem—even if its status is SUCCESS.

Therefore, SeaTunnel’s role is not to claim that one tool can solve every data quality problem. Its role is to provide a reliable data movement and runtime foundation for the data quality and observability layers above it.

This is ultimately more practical than trying to put every capability into a single platform, because reliable data movement and business data correctness are fundamentally two different problems.

From SeaTunnel to DolphinScheduler: Managing the Bigger Pipeline

As data operations scale, enterprises often encounter another challenge: a single synchronization task running reliably does not mean the entire data pipeline is reliable.

For example, Google Ads and Meta Ads data may need to be synchronized before a unified data model can run. Once the model is complete, business reports can be generated. Those reports may then need to be exported to other systems.

At this point, the real challenge becomes dependency management between tasks.

SeaTunnel handles data movement, while Apache DolphinScheduler can operate at the workflow orchestration layer. Together, they establish a clear division of responsibilities: SeaTunnel handles how data moves reliably, while DolphinScheduler handles how multiple tasks are organized, scheduled, and executed according to their dependencies.

This architecture is more practical than expecting a single data integration tool to handle connectors, workflows, scheduling, and the entire enterprise DataOps lifecycle.

For the scenario described in the Reddit discussion, different SaaS data sources can be synchronized through SeaTunnel, while DolphinScheduler organizes multiple synchronization jobs, dbt models, and downstream export tasks into a complete DAG.

If one node encounters a problem, the orchestration layer can determine whether downstream tasks should continue. SeaTunnel, meanwhile, handles the state and recovery of the individual synchronization job.

This creates two distinct layers of reliability in the data platform:

DolphinScheduler provides dependency-level reliability at the workflow layer, while SeaTunnel provides execution reliability at the data movement layer.

The Value of WhaleStudio ACP: Connecting the Pieces

Once an enterprise operates hundreds or even thousands of data pipelines, having SeaTunnel and DolphinScheduler alone is still not enough.

The data team eventually faces some very practical questions:

Who manages these pipelines? Who is authorized to modify them? How do you troubleshoot an incident? And when an AI Agent creates a pipeline, who is responsible for approving it?

This is where WhaleStudio ACP can play an important role.

If SeaTunnel is viewed as the data movement and execution layer, and DolphinScheduler as the workflow orchestration layer, WhaleStudio ACP—WhaleStudio’s commercial offering—moves closer to a control plane for data engineering.

It can bring pipeline creation, execution, monitoring, access control, approval, and governance into a unified interface.

This becomes particularly valuable as AI Agents begin taking part in data engineering.

In the future, a user may only need to tell an Agent:

“Sync Google Ads and Meta Ads data every day, and run the dbt model once all the data has arrived.”

The Agent can generate the pipeline, configure the data sources, establish dependencies, and execute the workflow.

But enterprises cannot simply give an Agent unlimited access to production environments.

What they need is a governed execution path:

The Agent understands requirements and generates the plan. WhaleStudio ACP manages permissions, policies, and human approval. DolphinScheduler handles workflow orchestration. SeaTunnel handles reliable data movement. Data Quality and Observability systems validate the final results.

In this model, AI Agents do not bypass the existing data infrastructure. They operate on top of it.

That is also what differentiates WhaleStudio ACP from a simple AI chat interface.

AI can determine what needs to be done, but enterprises still need to control who is authorized to do it, what exactly was done, which data was affected, and how every action can be traced when something goes wrong.

As a result, the future data platform will need to observe more than pipelines.

In addition to Pipeline Execution Traces, teams will need visibility into what an Agent did, what it changed, which data sources it accessed, which tasks it created, and which downstream systems were ultimately affected.

Data Observability is therefore evolving beyond simply “observing data pipelines” toward “observing both data pipelines and AI Agents.”

So, the Real Question Isn’t “Fivetran or SeaTunnel?”

Returning to the original discussion on Reddit, it is difficult to give a universal answer to the question “Is Fivetran worth the cost?”

For companies with relatively small data volumes, limited data infrastructure expertise, and a need to onboard large numbers of SaaS data sources quickly, Managed ELT clearly has value.

What enterprises are buying is not just a collection of connectors. They are also buying the maintenance effort, infrastructure, and operational responsibility behind those connectors.

But as data volumes increase and pipelines become a critical part of the business infrastructure, the questions change.

The focus is no longer simply on the number of connectors or how quickly a new source can be onboarded. Enterprises also need to consider how much control they want over their data infrastructure.

This is where SeaTunnel’s value goes beyond being “an open-source data synchronization tool.”

Through capabilities including Connectors, parallel processing, CDC, Checkpoint, State, Failover, Recovery, Schema Evolution, and Runtime Metrics, SeaTunnel provides a reliability foundation for the data movement layer.

DolphinScheduler addresses complex workflow orchestration at the layer above it.

Data Quality and Observability tools validate freshness, completeness, and business correctness.

WhaleStudio ACP then connects execution, orchestration, observability, access control, approval, and Agent governance into a broader data engineering system.

The result is a layered data engineering architecture rather than a single tool attempting to do everything.

This may be a more useful way to think about the Fivetran vs. SeaTunnel debate.

The real choice for an enterprise is not simply “Which tool is better?”

It is how much of its data infrastructure it wants to hand over to a Managed Service, and how much control it wants to retain over data, execution state, failure recovery, workflows, and governance.

Because once data becomes a core part of the business infrastructure, Pipeline Green is only the starting point.

What truly matters is whether data can arrive reliably, whether the system knows exactly how far processing has progressed, whether failures can be recovered from the right point, whether data anomalies can be detected in time, and ultimately whether the business can prove that the data can be trusted.

That is the real value of SeaTunnel in a modern data engineering stack: not simply replicating Fivetran, but providing enterprises with a controlled, recoverable, and scalable foundation for reliable data movement.

Solve HTTP memory limits in big data! 🚀 Switch from JSON to JSONL with Apache SeaTunnel for smooth streaming. 💡 #ApacheSeaTunnel #BigData #DataEngineering #JSONL

Apache SeaTunnel — Fri, 21 Aug 2026 03:04:56 +0000

Apache SeaTunnel

Aug 21

From JSON to JSONL: How Apache SeaTunnel Solves the HTTP Big Data Transfer Memory Dilemma

#json #http #apacheseatunnel #datascience

3 min read

From JSON to JSONL: How Apache SeaTunnel Solves the HTTP Big Data Transfer Memory Dilemma

Apache SeaTunnel — Fri, 21 Aug 2026 03:04:30 +0000

Apache SeaTunnel seamlessly extracts data via HTTP endpoints. Typically, an HTTP server returns responses in standard JSON format. SeaTunnel parses this JSON data, extracts the relevant schema and field types, and routes the records downstream to target systems.

While standard JSON handling works well for moderate payloads, massive HTTP datasets create severe memory pressure for both clients and servers. How can we overcome this bottleneck? The answer lies in streaming with JSONL (JSON Lines)—processing and transmitting data line-by-line as it generates. This streaming approach eliminates the need to load entire payloads into memory at once, drastically reducing overhead. For instance, when querying one month of order records totaling 1 million rows from a database, data can be fetched and processed in 5-day incremental batches rather than holding all 1 million records in memory simultaneously.

1. Data Output Formats: JSON vs. JSONL

JSON Standard JSON responses returned via HTTP follow an array structure like this:

[
  { "id": 1, "name": "a" },
  { "id": 2, "name": "b" }
]

The entire payload is treated as a single unified JSON document, requiring the system to process the full payload in one go.

JSONL JSONL responses returned via HTTP follow a line-delimited format like this:

{"id":1,"name":"a"}
{"id":2,"name":"b"}

Instead of a single enclosing JSON object or array, each row represents an independent JSON entity, separated by newline characters.

2. Key Parameter for Format Identification

To differentiate between these response formats, configure the parsing behavior using the key parameter below:

Parameter: enable_multi_lines
Default Value: false
Description: Determines whether to enable multi-line JSON parsing. When set to true, SeaTunnel reads newline-delimited JSON entries (JSONL/NDJSON) sequentially without loading the full payload as a single object.

Parameter	Value	Description
`enable_multi_lines`	`true`	Read JSONL format
`enable_multi_lines` (Default)	`false`	Read JSON format

3. Practical Configuration Example

Step 1: Set Up an HTTP Test Service Returning Both Formats

[root@localhost]# curl http://localhost:3000
[  {    "id": 1,    "name": "aaa"  },  {    "id": 2,    "name": "bbb"  }]

[root@localhost]# curl http://localhost:3000/jsonl
{"id":1,"name":"aaa"}
{"id":2,"name":"bbb"}

Step 2: Configuration for Standard JSON

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Http {
    plugin_output = "http"
    url = "http://localhost:3000"
    method = "GET"
    format = "json"
    enable_multi_lines = false
    schema = {
      fields {
        id = int
        name = string
      }
    }
  }
}

sink {
  Console {
    parallelism = 1
  }
}

Console Output:

2026-07-04 18:13:09,073 INFO  [.a.s.c.s.c.s.ConsoleSinkWriter] [st-multi-table-sink-writer-1] - subtaskIndex=0  rowIndex=1:  SeaTunnelRow#tableId=Optional[http] SeaTunnelRow#kind=INSERT : 1, aaa
2026-07-04 18:13:09,073 INFO  [.a.s.c.s.c.s.ConsoleSinkWriter] [st-multi-table-sink-writer-1] - subtaskIndex=0  rowIndex=2:  SeaTunnelRow#tableId=Optional[http] SeaTunnelRow#kind=INSERT : 2, bbb

Step 3: Configuration for JSONL Streaming

source {
  Http {
    plugin_output = "http"
    url = "http://localhost:3000/jsonl"
    method = "GET"
    format = "json"
    enable_multi_lines = true
    schema = {
      fields {
        id = int
        name = string
      }
    }
  }
}

When enable_multi_lines is set to true, the execution output matches the JSON source output above. However, if enable_multi_lines remains false on a JSONL stream, SeaTunnel processes only the first row and drops all subsequent records.

4. Key Takeaway

For standard HTTP services returning single JSON string payloads, leave the default setting unchanged as false. For streaming JSONL outputs, always set enable_multi_lines = true to guarantee complete, memory-efficient data ingestion.

Data Ingestion Must Never Be a "Black Box"!

Apache SeaTunnel — Fri, 21 Aug 2026 02:54:42 +0000

As corporate data pipelines evolve into mission-critical production systems, the primary risk is no longer "sync failure"—it's not knowing why it succeeded or why it failed.

Over the past five years, closed-source ELT tools such as Fivetran, Stitch, and Hevo have driven the adoption of the Modern Data Stack. Promising "no-code" setups and "data sync in 5 minutes," they significantly lowered the entry barrier for data integration. However, as enterprise data volumes explode, regulatory compliance tightens, and AI Agents begin consuming enterprise data directly, an increasing number of data teams are reconsidering a fundamental question:

Should Data Ingestion really be a black box?

A highly upvoted discussion on Reddit, titled "Beware of Fivetran and other ELT tools. : r/dataengineering - Reddit," laid bare this growing industry anxiety.

In the thread, dozens of frontline data engineers shared their production pain points: automatically modified schemas, incorrect primary key resolution, unverifiable sync logic, exorbitant rerun costs, and near-impossible migrations. Behind these complaints lies a deeper systemic issue—not just a flawed product, but the inherent observability and auditability defects of closed-source ingestion architectures.

Apache SeaTunnel was not created to be just another ELT tool. It introduces a completely different data integration philosophy: make Connectors, Pipelines, and Engines fully transparent, transforming data synchronization from a black-box service into a verifiable software system.

This article addresses three core questions:

Why is black-box ingestion becoming a major enterprise data risk?
Why do black-box issues worsen exponentially at scale?
How does Apache SeaTunnel build a truly trusted, auditable data integration architecture?

Why Data Ingestion Cannot Be a Black Box

Transitioning from "Sync Tool" to "Data Infrastructure"

A decade ago, data synchronization was merely the initial step of ETL.

Today, it carries far greater responsibilities:

Serving as the primary data entryway for Data Lakes and Data Warehouses
Providing context data sources for AI Agents
Acting as the real-time pipeline for CDC incremental synchronization
Laying the foundation for data governance and lineage tracking

This shift means that Ingestion no longer just moves data—it determines whether data can be trusted.

When synchronization logic remains opaque, enterprises effectively hand over their most critical data gateway to unverifiable software.

Seven Black-Box Pitfalls Highlighted on Reddit

Engineers in the Reddit thread echoed remarkably consistent frustration. While issues surfaced as bugs, pricing spikes, or SLA breaches, the root cause was always the same: unobservable internal implementation.

Surface Problem	Root Cause
Field names automatically modified	Connector internal mapping invisible
Schema automatically changed	Type inference algorithm closed-source
Primary key misidentification	CDC logic unverifiable
Rerun cost extremely high	Synchronization strategy uncontrollable
Lag reason unknown	Pipeline internal state invisible
Unable to migrate	Connector behavior platform-locked
Long bug fix cycle	Users cannot self-locate and fix

These issues share a common pattern:

Users see the outcome, but never the process.

Consider a typical scenario:

Suppose Salesforce's Account.OwnerId field is automatically mapped to owner_id in the target warehouse.

For business teams, it looks like a harmless field name tweak.

For data engineers, it triggers a cascade of failures:

Broken downstream dbt models
Failing BI Dashboards
AI Agent prompts failing to locate fields
Severed data lineage

The fundamental problem isn't just the break—it's the unanswered questions:

Why did it change? When did it change? Who decided to change it?

Closed-source tools usually offer little more than: "Connector updated."

That is simply insufficient for production systems.

Why the Black Box Becomes Exponentially Dangerous at Scale

Many teams initially adopt a mindset of:

"Let's use a SaaS tool first, and optimize later."

However, as data volume grows, risk doesn't increase linearly—it scales exponentially.

Level 1: Unverifiable Schema Evolution

Modern SaaS APIs are defined by continuous change.

For instance:

Shopify introduces new fields
Salesforce alters data types
HubSpot removes properties

Closed-source tools typically rely on automated Schema Evolution.

The flow generally operates like this:

It sounds intelligent on paper.

The underlying risk lies in the unknown:

Why did the schema change? How were data types inferred? What is the compatibility strategy?

Users have no way to verify it.

Consequently, many enterprises turn off automated schema evolution and revert to manual schema maintenance.

Intelligence, ironically, becomes liability.

Level 2: CDC Is More Than Data Replication

The core of Change Data Capture (CDC) isn't just binlog parsing—it's the consistency protocol.

A mature CDC pipeline must answer critical operational questions:

How does it switch between Snapshot and Incremental phases?
How are Checkpoints restored?
How are primary key modifications handled?
How are DDL changes propagated?

Closed-source tools claim:

"CDC Supported."

Yet the underlying logic governing data correctness remains hidden:

If these protocols are invisible, enterprises cannot prove:

Whether recovered data strictly guarantees Exactly-Once semantics.

In finance, healthcare, and government sectors, this represents a severe compliance risk.

Level 3: AI Amplifies Black-Box Vulnerabilities

In the era of AI Agents, data synchronization directly impacts model output quality for the first time.

Traditional BI reports can tolerate a 30-minute delay.

AI Agents cannot.

They require:

Real-time context
Explainable data sources
Traceable reasoning pathways

If data originates from a black-box pipeline, an Agent cannot explain "where this number came from."

Data Lineage shifts from a governance requirement into a prerequisite for AI trust.

How Apache SeaTunnel Builds Auditable Data Integration

SeaTunnel operates on a core design principle:

Every Record Has a Visible Journey.

It breaks Ingestion down into three fully transparent, auditable layers:

Auditable Connectors
Auditable Pipelines
Auditable Engine

Together, these layers form an end-to-end trusted data chain.

1. Fully Open-Source Connectors: Transparent Data Ingestion

The biggest risk of closed-source platforms isn't a lack of connectors—it's that their connectors cannot be audited.

SeaTunnel's connectors are 100% open-source, making every sync action fully auditable.

A MySQL CDC Connector, for example, features this architecture:

Developers can inspect every mechanism directly:

How snapshots are chunked
How binlogs are parsed
How offsets are persisted
How schema events are emitted

Zero hidden logic.

The result:

Bugs can be pinpointed and fixed immediately, without waiting for vendor support tickets.

2. Pipeline as Code: Configurations Function as Audit Documents

SeaTunnel utilizes declarative Pipelines.

A synchronization task itself acts as a complete audit record.

env {
  parallelism = 4
}

source {
  MySQL-CDC {
    table-names = ["orders"]
  }
}

transform {
  Sql {
    query = "SELECT * FROM orders WHERE status='paid'"
  }
}

sink {
  Iceberg {}
}

Compared to black-box GUIs, this code-driven model provides three distinct advantages:

Pipelines cease to be mere configurations—they become core software assets.

For enterprises, data synchronization can finally integrate seamlessly into standard DevOps workflows.

3. Engine Transparency: Verifiable Runtime Execution

SeaTunnel's core engineering strength lies in its runtime engine: Zeta Engine.

While traditional ELT depends heavily on external computation engines, SeaTunnel features its own dedicated, purpose-built execution runtime for data integration.

Core structure:

Alongside standard Data Records, Zeta flows three types of Control Events through the stream:

Because control events share the exact same pipeline stream as data records:

Ordering is guaranteed
State consistency is maintained
Recovery remains deterministic

This architecture enables SeaTunnel to deliver true Engine-level Exactly-Once processing guarantees.

4. Controlled Schema Evolution: Shift from "Auto-Modify" to "Governed Evolution"

SeaTunnel does not oppose schema evolution—it opposes unexplainable schema evolution. When schemas change, SeaTunnel explicitly generates a Schema Event.

The governed workflow proceeds as follows:

Administrators can define custom policies:

Whether new fields are permitted
How type conflicts are handled
Whether to pause pipelines for manual approval

Every schema modification leaves a complete audit log, providing essential control for finance, healthcare, and government data teams.

5. Native On-Premises Deployment & Full Data Sovereignty

The ultimate constraint of closed-source SaaS tools isn't pricing—it's deployment topology.

In banking, healthcare, government, and manufacturing, data cannot leave internal network boundaries. Designed for self-hosting from day one, SeaTunnel runs natively on Kubernetes, Yarn, Standalone clusters, and bare-metal environments, keeping full runtime control in enterprise hands.

Crucially, connectors operate independently of vendor cloud services. Engineering teams can build, audit, and deploy custom connectors without waiting on vendor API roadmaps.

This sovereignty drives large enterprises toward open-source data integration.

Why "Auditability" Trumps "No-Code"

The first phase of the Modern Data Stack focused on faster data ingestion.

The emerging Agentic Data Stack phase demands trusted data sources.

These goals are complementary, but priorities have evolved.

Architectural comparison:

Teams often select closed-source SaaS tools for speed of initial delivery.

Yet as data becomes a primary strategic asset, critical requirements shift:

Can I verify the provenance of every data record?
Can I prove data was not altered in transit?
Can I operate and maintain this architecture independently of a vendor?

If the answer to any of these is no, Data Ingestion remains a black box.

The Future of Data Platforms Requires "Trusted Synchronization"

Data engineering has progressed across three key generations:

ETL Era: Solved how to transport data.
ELT Era: Solved how to quickly land data into warehouses.
Trusted Data Ingestion Era: Driven by AI, governance, and compliance requirements.

Trust does not require unnecessary complexity—it requires reasoned decisions, verifiable sync processes, and clear data lineage.

Apache SeaTunnel's core value extends beyond its 100+ connectors, batch-stream unification, or CDC capabilities. Its true impact lies in transforming data synchronization from an opaque SaaS service into a transparent, verifiable, and extensible software system.

In the AI era, models demand reliable context, enterprises require trustworthy data, and trusted data begins with Data Ingestion that is no longer a black box.

See how @ASFSeaTunnel Zeta built a scientifically accurate benchmark! 👇 #SeaTunnel #BigData #DataEngineering #Performance #OpenSource

Apache SeaTunnel — Fri, 14 Aug 2026 07:35:51 +0000

Apache SeaTunnel

Aug 14

600,000 Records/Sec with Sub-100ms Latency: Inside SeaTunnel Zeta’s Rigorous Performance Test

#opensource #apacheseatunnel #datascience #programming

11 min read

600,000 Records/Sec with Sub-100ms Latency: Inside SeaTunnel Zeta’s Rigorous Performance Test

Apache SeaTunnel — Fri, 14 Aug 2026 07:35:26 +0000

Written by Niu Zhiwei

Figure 1: JMH Summary in a single run, including complete Pipelines and SeaTunnelRow hot path operations

Figure 2: Throughput, Latency, Growth Ratio, and Valid Samples across Five Pipelines under Identical Load

1. Interpreting the Test Results

This round of benchmarking was executed on Java 8, with a 4 GiB heap, 4 visible JVM processors, and a pipeline parallelism of 4. Each job run processed 1,000,000 records, targeting a planned input rate of 600,000 records/sec, with a payload size of 256 characters per record.

The results yield three primary insights:

Stable and Sustainable Processing: Across all five scenario pipelines, Sink throughput consistently held between 588,000 and 591,000 records/sec. The P99 latency registered between 100 and 101 ms, while the Growth Ratio hovered tightly between 0.99 and 1.00. Additionally, all 75 out of 75 samples were marked valid. These metrics confirm that under the tested workload, the system suffered no continuous backlog accumulation.
Feature Overhead Within Margin of Error: The maximum discrepancy in Pipeline JMH Scores across scenarios was roughly 0.87%, whereas the margins of error for this run spanned 0.88% to 1.38%. Statistically, this indicates no observable feature-induced overhead under this configuration—though it stops short of declaring any feature "zero-overhead."
Fixed Test Load vs. Capacity Limit: The 600,000 records/sec figure reflects a pre-set, fixed workload rather than the absolute performance ceiling. Evaluating maximum capacity would require systematically scaling up input rates while closely tracking throughput, P99 latency, and latency growth trends.

In summary, these data points demonstrate that the benchmark path is fully connected and sustainable under current loads. They provide a sound baseline for standardizing performance comparisons across various engine capabilities, rather than delivering an absolute performance verdict stripped of runtime context.

2. Why Micro-benchmarks and Complete Pipelines Co-exist

SeaTunnel employs a two-tier benchmarking strategy, with each level engineered to answer distinct operational questions.

SeaTunnelRow Micro-benchmarks: Pinpointing Hot Paths

The SeaTunnelRowBenchmark isolates basic operations along the critical execution paths of Sources, Transforms, and Sinks. This includes Row creation, field reads, cloning, projections, Options handling, and size calculations.

Operation	Score (ops/ms)	CV	Primary Scope & Meaning
`copyPlainRow`	22,328.74	1.01%	Full clone of a standard Row
`copyProjectedPlainRow`	59,769.89	1.57%	Projected clone (4 out of 8 fields)
`copyProjectedRowWithOptions`	49,282.32	0.59%	Projected clone with Options processing
`copyRowWithOptions`	20,789.92	0.22%	Full clone of a Row containing Options
`copyRowWithTracePayload`	20,499.68	0.27%	Full clone of a Row with Trace Payload
`copyThenMutateCopiedOptions`	7,305.63	1.01%	Clone followed by Options mutation
`createRowAndGetBytesSize`	5,837.52	0.68%	Row instantiation paired with size calculation
`createRowWithSetField`	7,668.90	0.59%	Field-by-field Row construction
`getBytesSizeCached`	254,677.57	1.13%	Reading pre-cached byte sizes
`readFields`	28,347.47	0.47%	Reading and consuming all fields

Higher ops/ms values indicate superior performance (e.g., copyPlainRow at 22,328.74 ops/ms corresponds to roughly 22.33 million operations per second).

These micro-metrics are primarily designed for cross-version comparisons of identical methods or for benchmarking functionally similar operations. Crucially, a high score in getBytesSizeCached relative to createRowAndGetBytesSize cannot be translated linearly to overall pipeline speedups: the former merely fetches cached state, whereas the latter encompasses both object instantiation and initial byte calculation.

Comparing structurally similar cloning operations reveals that full cloning with Options exhibits a ~6.9% drop in throughput compared to plain cloning, while adding a Trace Payload decreases throughput by ~8.2%. These granular variances offer useful optimization directives, though their real-world impact must always be validated through end-to-end Pipeline benchmarks.

Zeta Pipeline Benchmarks: End-to-End System Validation

The full Pipeline benchmark provisions live single-node Zeta Masters and Workers. Bounded jobs are executed through actual Client invocation, configuration parsing, task submission, and scheduling paths.

Cluster initialization occurs during the Trial Setup phase and is explicitly excluded from active timing intervals. Every benchmark invocation submits a bounded job carrying 1,000,000 records. The timed measurement window encompasses job configuration generation, submission, scheduling, Source generation, optional Transforms, Sink execution, and completion wait times.

Compared to isolated unit calls, this architecture mirrors production execution while eliminating external environment jitter via in-memory Sources and Blackhole Sinks—making it the ideal setup for assessing engine-level code updates.

3. Theoretical Foundations of the Benchmark Design

Benchmark credibility rests on two pillars: statistical reliability of the collected metrics, and accurate alignment between measured indicators and actual performance bottlenecks. Achieving the former requires handling JVM variability, isolating independent samples, and controlling experimental noise; achieving the latter demands clear boundaries between throughput and latency measurements.

Our benchmarking framework builds upon three foundational studies in systems evaluation:

Research Study	Core Problem Addressed	Key Findings & Recommendations
Statistically Rigorous Java Performance Evaluation (OOPSLA 2007)	Java execution varies per run. How do we extract statistically sound conclusions?	Distinguish startup from steady-state performance; treat independent JVM runs as critical experimental units; report both point estimates and uncertainty metrics.
Rigorous Benchmarking in Reasonable Time (ISMM 2013)	Builds, JVM instances, and iterations introduce variance. How should finite benchmark budgets be allocated?	Identify variance across execution layers; run calibration experiments to direct repetition budgets toward the primary sources of uncertainty.
Benchmarking Distributed Stream Data Processing Systems (ICDE 2018)	Where should throughput and latency timing begin in stream processing systems?	Employ open-loop workload generation and event-time tracking; evaluate sustainable throughput alongside backlog accumulation and tail latency.

Together, these studies form a continuous methodology: verifying sample validity, spending execution budgets efficiently, and establishing correct measurement boundaries.

JVM Performance Evaluation: From Raw Repetition to Statistically Sound Sampling

Java runtime performance is non-deterministic. Even with fixed codebases, parameters, and physical hardware, variables like JIT compilation timing, Garbage Collection (GC) pauses, thread scheduling, heap memory layout, and OS interruptions introduce performance drift across runs.

Relying strictly on the "best run out of N" distorts true operational metrics. Selecting the peak result answers only "how fast the system ran under optimal conditions." As N scales, the probability of sampling an abnormally lucky run increases, systematically skewing conclusions in favor of higher-variance implementations.

Furthermore, evaluation frameworks must differentiate between startup performance and steady-state capability. Startup metrics capture JVM spin-up, class loading, initialization overhead, and cold execution; steady-state metrics isolate sustained processing power post-initialization and JIT compilation. Conflating these two regimes compromises data integrity.

Because iterations within a single JVM instance share JIT compilation profiles, compiled code artifacts, heap state, and GC history, they cannot be treated as statistically independent trials. Independent JVM invocations (represented as Forks in JMH) constitute indispensable units of measurement. Reliable reporting must present point estimates accompanied by confidence intervals, rather than isolated averages or peak figures.

Allocating Experimental Budgets: Identifying Variance Layers

Increasing benchmark iterations blindly does not guarantee a proportional rise in data precision. Performance experiments operate across nested structural layers:

Build
 └─ JVM Execution / Fork
       └─ Measurement Iteration
            └─ Pipeline Invocation

Variance can emerge at every layer: rebuilding code alters binary layouts; launching new JVMs resets JIT and heap configurations; while iterations within the same JVM are susceptible to localized runtime jitter.

If primary variance stems from JVM Forks, increasing iteration counts within a single Fork from 5 to 50 will not mitigate inter-Fork discrepancies; budget is better spent allocating additional independent Forks. Conversely, if code compilation introduces non-trivial drift, repetitions must be elevated to the Build layer.

We begin by conducting calibration experiments: capturing raw samples across all layers, estimating variance contributions alongside execution costs for Builds, Forks, and Iterations, and directing experimental budgets toward dominant noise sources. Consequently, static Fork and Iteration counts serve merely as baselines rather than unchangeable configurations.

Equally critical is differentiating between random errors and systematic bias. Additional repetitions improve random error estimations, but cannot correct for structural biases like host background load or execution ordering. If a baseline runs on an idle machine while a candidate runs under system load, higher sample counts merely yield a more precise measurement of a flawed setup.

Stream Processing Metrics: Shifting from Stability to Accuracy

While the first two studies establish statistical rigor, the third addresses metric validity: whether measured parameters truly reflect stream processing capabilities.

Traditional closed-loop workload generators issue new batches only after previous ones finish processing. If the processing engine slows down, the generator throttles its issuance rate accordingly, obscuring queuing delays occurring prior to engine ingestion. This creates a misleading pattern where an overloaded system reports deceptively low internal processing latencies due to Coordinated Omission.

To prevent this, input rates must operate independently of downstream processing speed, and latency must be measured against an event's original scheduled creation time. Otherwise, queueing prior to Source ingestion is ignored, yielding metrics that capture processing speed post-ingestion while omitting end-to-end event delay.

This underscores the distinction between sustained throughput and transient peak processing rates. Sustained throughput represents the maximum input rate an engine can process indefinitely without accumulating persistent backlogs or inflating event-time latency. Throughput, data integrity, tail latency, and latency growth trends must be analyzed holistically; evaluating any single metric in isolation risks incorrect performance assessments.

Synthesis of the Three Methodologies

These three principles form a rigorous benchmark evaluation workflow: establish whether the scope targets startup, steady-state, sub-routine hot paths, or end-to-end architectures; identify independent sampling units and variance sources; and verify that throughput and latency boundaries accurately reflect production constraints.

Statistical analysis cannot salvage flawed boundary definitions, nor can valid boundaries compensate for uncalibrated sample variance. Only when both domains align can benchmark results evolve from isolated execution figures into actionable engineering evidence.

4. Operationalizing Theory in Zeta Pipeline Benchmarks

Translating these theoretical principles into the Zeta Pipeline benchmarking suite relies on three implementation choices: preserving independent JVM samples, allocating iteration budgets to primary variance layers, and employing open-loop generation to measure true queuing delays.

Plan-Time Scheduling with `BenchmarkSource`

The system's Source schedules planned generation times using absolute temporal offsets:

scheduledTime = startTime
              + wholeSeconds × 1000
              + remainder × 1000 / rate

Generation schedules remain decoupled from downstream processing delays. Upon reaching the Sink, event latency is computed as:

event-time latency = Sink Receipt Time - Planned Generation Time

Any processing bottlenecks within the engine immediately register in the P50, P95, and P99 percentiles, preventing Source-side throttling from masking true internal delays.

Isolating Engine Overhead via Deterministic Components

Test components are engineered to minimize stochastic noise:

Source: Generates a fixed volume of memory records with static payload sizes.
Transform: Executes deterministic hashing ops and outputs verifiable checksums.
Sink: Bypasses external IO; tracks row counts, throughput, latency distributions, and checksums.
Pipeline Uniformity: Every pipeline scenario retains identical record counts, parallelism settings, and resource allocations, altering only the target functional capabilities.

The five benchmark scenarios cover baseline data flows, Transforms, real-time busyness metrics, StainTrace tracking, and composite feature sets. This guarantees uniform workloads while isolating incremental engine overhead.

Data Validation as a Prerequisite for Sample Validity

High throughput paired with unverified or dropped data yields invalid metrics. Every completed job undergoes structural validation:

processed_rows must match expected_rows exactly.
Baseline Source -> Sink pipelines must output a checksum of 0.
Transform-enabled pipelines must produce non-zero checksums.
Latency percentiles must fit within tracked bounds without hitting overflow buckets.
P99 latencies and growth ratios must satisfy strict safety thresholds.

A sample is included in final reporting only when data completeness, functional execution, and latency metrics pass validation.

Fork Configurations, Warmups, and Machine-Readable Outputs

The default JMH setup runs 3 independent Forks, each including 3 warmup iterations and 5 measurement iterations. Warmup results are excluded from aggregated reporting; only measurement iterations populate final datasets.

Because a single measurement iteration executes multiple full Pipeline invocations, a reported 75/75 metric represents 75 validated, end-to-end job runs rather than 75 individual data rows or 15 JMH iterations.

Alongside Markdown summaries, the suite exports raw JMH JSON, per-run Pipeline JSON, normalized metrics, and environmental metadata. Version control commits, JDK builds, JVM arguments, container images, CPU topologies, kernel revisions, host memory, and load parameters are logged to guarantee run-to-run comparability.

The 3 Forks × 5 Measurement Iterations setup provides an initial baseline budget. As continuous integration collects historical performance trends across dedicated hardware, these repetition rates, execution rules, and regression thresholds will be adjusted dynamically.

5. Deciphering Dual-Layer Metrics

Differentiating JMH Scores from Pipeline Throughput

The Pipeline benchmark registers 1,000,000 logical operations per invocation. JMH converts total job execution time into an ops/s score:

JMH Score ≈ 1,000,000 / Total Job Duration

This total duration includes job setup, configuration parsing, submission, task scheduling, Source generation, Transform processing, Sink consumption, and teardown synchronization.

Conversely, Pipeline JSON records a narrower measurement window:

Pipeline Throughput
    = processed_rows / (Final Sink Receipt Time - Initial Sink Receipt Time)

This isolates the interval during which the Sink actively receives data, excluding setup and teardown overheads. Consequently, reported JMH Scores range between 454,000 and 458,000 ops/s, while direct Pipeline Throughput tracks between 588,000 and 591,000 records/sec. This variance reflects distinct measurement boundaries rather than metric divergence.

Metric Definitions: Score, Error, CV, and Units

Field	Description	Analytical Application
`Score`	Estimated throughput measured by JMH	Higher values denote greater processing throughput.
`Error`	Half-width of the confidence interval expressed as a percentage of the Score	Defines the score range: `Score × (1 ± Error%)`.
`CV`	Coefficient of Variation (Standard Deviation divided by Mean)	Lower values reflect higher sample stability within the run.
`Unit`	Measurement units	Reported as `ops/s` for Pipelines and `ops/ms` for Row micro-benchmarks.

Error and CV reflect internal variance within a single test run; they do not account for cross-node hardware variations, CPU scheduling noise, or system load fluctuations. Small score variations (~1%) between commits should not be interpreted as performance regressions without cross-run validation.

Latency Percentiles, Growth Ratios, and Valid Samples

P50: Median latency; establishes the baseline execution delay.
P95 / P99: Tail latency metrics; exposes queueing delays, GC pauses, and thread contention.
Max: Peak recorded latency; captures extreme outliers, evaluated alongside percentiles.
Growth: Ratio comparing late-stage P99 latency against early-stage P99 latency to identify backlog accumulation.
Valid: Count of fully validated, non-overflowing measurement samples completed during testing.

The Growth Ratio is calculated using the following formula:

Growth = (Late-Stage P99 + 1) / (Early-Stage P99 + 1)

Throughput tracks completed work volume, Latency quantifies record wait time, and the Growth Ratio tracks whether queueing delays are escalating over time. Evaluating all three together is essential for accurate performance diagnostics.

6. Zeta Engine Performance Analysis

Allocated 4 visible JVM processors and a 4 GiB heap, SeaTunnel Zeta instantiated single-node Master and Worker runtimes, executing end-to-end Source -> Transform -> Sink pipelines through standard submission and scheduling paths. The engine maintained Sink throughput between 588,000 and 591,000 records/sec, capped P99 latency around 100 ms, and successfully validated all 75 out of 75 measurement runs.

Growth Ratios across all five pipeline configurations remained close to 1.0, proving that the engine handled a steady 600,000 records/sec input stream without accumulating internal latency backlogs. Incorporating Transforms, real-time busyness metrics, and StainTrace features introduced no throughput degradation beyond standard error bounds. While this does not imply "zero computational cost," it confirms these features introduce no measurable performance bottlenecks along core data paths.

These results reflect performance under controlled test parameters rather than absolute system limits, omitting external connector IO, multi-node network transport, and Checkpoint persistence costs. Nevertheless, within these test bounds, SeaTunnel Zeta demonstrated high throughput, stable tail latencies, and efficient functional overhead management—establishing itself as an engine for unified batch and stream synchronization across databases, message queues, data warehouses, and data lakes.

7. Key Takeaways and Engineering Summary

Designing an effective benchmarking suite requires establishing clear evaluation goals before writing test routines. Applying concepts like startup vs. steady-state performance, sample independence, variance layer allocation, open-loop testing, and sustained throughput prevents generating precise yet uninformative metrics.

Once testing goals are defined, key execution variables must be held constant: timing boundaries, workload profiles, and hardware resource limits. Implementing strict output validation guarantees that throughput and latency figures are evaluated only on correct, fully processed datasets.

Finally, benchmark runs should separate warmup phases from measurement windows, retain independent iterations, and track throughput alongside tail latency, Growth Ratios, and variance metrics. Single runs on shared build runners highlight performance signals, but dedicated test environments are required to collect historical distributions and establish reliable performance regression thresholds.

The value of benchmarking lies not in generating a single optimized metric, but in building a repeatable, verifiable testing framework that tracks performance shifts and catches regressions over time.

💡 How a simple JDBC parameter forced an engine-level evolution! Discover how @ASFSeaTunnel Zeta introduced STIP-23 FlushSignal to solve the scheduled flush dilemma. ⚡️ #SeaTunnel #DataEngineering #BigData #OpenSource

Apache SeaTunnel — Fri, 14 Aug 2026 07:23:51 +0000

Apache SeaTunnel

Aug 14

How a Single JDBC Parameter Drove Engine Evolution in Apache SeaTunnel Zeta

#ai #apacheseatunnel #dataengineering #programming

21 min read

How a Single JDBC Parameter Drove Engine Evolution in Apache SeaTunnel Zeta

Apache SeaTunnel — Fri, 14 Aug 2026 07:23:13 +0000

Lead-in: In a data integration system, a simple configuration parameter often masks intricate underlying engineering design. In this Apache SeaTunnel Meetup recap, the speaker takes the batch_interval_ms parameter of the JDBC Sink as an entry point to deep-dive into the batch-processing mechanisms during data writing—scaling up the perspective from the Connector layer to the overarching architecture of the SeaTunnel Zeta Engine.

Meetup Video Playback: https://youtu.be/L2QZefyJP88?si=WRZmqn_SRE4Ba_lC

About the Speaker

Zhiwei Niu: Apache SeaTunnel Contributor (GitHub ID: nzw921rx). Currently focused on risk control, specializing in data synchronization and processing. Possesses deep expertise in databases and big data technology, with extensive hands-on experience in data integration and task stability.

In a data synchronization system, many issues initially appear isolated to a specific Connector. However, as you dig deeper, you often discover that they actually touch upon the core runtime mechanisms of the entire data processing engine.

This sharing stems from a seemingly simple parameter in JDBC Sink: batch_interval_ms.

The parameter’s goal is straightforward: when data in the buffer sits longer than a specified threshold, a flush should be triggered even if the batch size hasn't been met—thereby minimizing data synchronization latency.

Yet during practical implementation, the challenge quickly escalated beyond the bounds of the JDBC Connector itself.

The core issue was not how JDBC executes SQL, but rather a fundamental runtime challenge every data synchronization engine must solve:

Who manages scheduled background tasks?
On which thread should the flush be executed?
How are exceptions propagated to the main Task upon a flush failure?
How can lifecycle operations (such as Checkpoints, close, and cancel) avoid concurrency conflicts?

Going a step further: when the Source temporarily yields no incoming data, can the system still trigger a flush on a schedule?

Thus, what started as a JDBC Sink parameter, batch_interval_ms, gradually evolved into a design overhaul at the SeaTunnel Zeta Engine level, ultimately giving birth to the Engine-Level FlushSignal design in STIP-23.

01 The Root Cause: Addressing Data Latency

In batch writing scenarios for the JDBC Sink, two conditions typically trigger a flush: batch_size and batch_interval_ms.

The logic behind batch_size is intuitive. As data flows into the Sink, the system buffers the records and checks whether the buffered count has hit the set threshold. Once met, a batch write is executed.

For instance:

if (buffer.size >= batchSize) {
    flush();
}

This logic fits naturally inside writeRecord(), because the buffer size only increments when new data arrives.

However, the semantics of batch_interval_ms are entirely different.

It does not mean "when the next record arrives, check by the way if the elapsed time since the last flush exceeds the limit." Instead, it demands a true time-based trigger mechanism: even if no new data arrives, as long as the configured interval has elapsed since the last flush, a flush operation must be triggered.

In high-throughput scenarios, the difference between these two semantics might be negligible because records arrive continuously, driving regular execution of writeRecord() and time checks.

However, in real-world production environments, many workloads do not run at a continuous high throughput.

Consider Change Data Capture (CDC) streams, small table synchronizations, off-peak business periods, or temporary lulls in a specific Source partition. Unwritten data might be idling in the buffer, but because no new records arrive, the system never enters the next writeRecord() call.

Consequently, the core promise of batch_interval_ms—triggering flushes purely based on time—fails to hold up.

This is precisely why the issue could not be patched inside the JDBC Sink alone and called for a fundamental rethink from the Engine's execution mechanism.

02 Starting from a Single Parameter

Approach 1: Spawning Background Threads inside the Connector

When first tackling this problem, the natural instinct was to introduce a scheduled task directly inside the JDBC Connector.

For example, using a ScheduledExecutorService to spin up a background thread:

scheduledExecutor.scheduleAtFixedRate(() -> {
    flush();
}, batchIntervalMs, batchIntervalMs, TimeUnit.MILLISECONDS);

This approach superficially meets the requirement. Even when the Source produces no data, the background thread fires on schedule and calls flush().

However, once plugged into the SeaTunnel Task execution model, serious issues emerge.

In the standard data pipeline, writeRecord() and flush() run on the Task's main execution path. By spawning a background thread, flush() becomes a separate, uncoordinated path managed entirely by the Connector.

Thread execution splits as follows:

Executed on Task Thread:

writeRecord(record)

Executed on Connector Background Thread:

flush()

This means two threads can concurrently access the same buffer.

At the same time, a flush might execute concurrently with Checkpoints, schema evolution, or close routines. Furthermore, exceptions occurring on the background thread cannot naturally propagate to the main Task thread, and stopping this timer thread gracefully during task failure, cancellation, or shutdown becomes a major challenge.

If every Sink Connector were to implement scheduled flushes this way, each would end up reinventing the wheel to handle thread lifecycles and timer synchronization.

Thus, the fundamental flaw became clear: to support a single timing parameter, the Connector was forced to assume responsibilities that belong strictly to the runtime engine.

This was a clear boundary violation.

Approach 2: Checking Elapsed Time within writeRecord

Another line of thought was to avoid background threads altogether and simply check the timestamp inside writeRecord():

public void writeRecord(Row row) {
    buffer.add(row);

    if (buffer.size() >= batchSize) {
        flush();
        return;
    }

    if (System.currentTimeMillis() - lastFlushTime >= batchIntervalMs) {
        flush();
    }
}

This avoids extra threads. Flushing and writing remain strictly on the single-threaded execution path, exceptions propagate cleanly to the Task, and lifecycle management remains simple.

Yet, it misses the primary objective.

If no new data arrives, writeRecord() is never invoked.

Hence, what this actually implements is not "flush every 5 seconds," but rather "when the next record arrives, if more than 5 seconds have elapsed, execute a flush."

While this subtle difference is imperceptible under steady data flows, it completely falls apart in low-throughput, intermittent CDC, small table sync, or idle Source scenarios—leaving buffered data stranded indefinitely.

In short, neither approach fulfills the true semantics of batch_interval_ms.

Spawning background threads inside the Connector guarantees time-based execution but introduces severe concurrency, exception propagation, and lifecycle headaches. On the flip side, evaluating time within writeRecord() preserves the single-threaded execution model but fails when Sources go idle.

The core issue was never JDBC implementation—it was abstraction hierarchy.

batch_interval_ms fundamentally requires a core Engine capability:

Even when no incoming data exists, the Engine must generate a control event based on time. This event does not manipulate the Sink directly; instead, it flows down the standard data pipeline, allowing the Sink to execute the flush on its own consumer thread.

In short:

Engine Timer → FlushSignal → Sink flushAction

This is precisely the core problem STIP-23 was built to solve.

03 Zeta Task Execution Model: Integrating FlushSignal into the Engine

Before exploring how the Engine generates FlushSignals, we must first understand how tasks run inside the SeaTunnel Zeta Engine. FlushSignal is not an isolated trick; it must integrate seamlessly into Zeta's existing data processing framework and traverse the data pipeline managed by the Engine.

Without a firm grasp of task scheduling, data flow, and control event processing, it is hard to see why FlushSignal was designed as an Engine-Level Signal rather than a Connector-managed utility.

Task Execution Model: Scheduling and Dispatching

In the Zeta Engine, once a data synchronization job is submitted, it is not executed directly by a single thread. Instead, the Engine orchestrates, splits, and dispatches the job into distinct Tasks across the cluster.

The Engine oversees the complete job lifecycle—including job submission, resource coordination, Task instantiation, and runtime status monitoring. Based on the logical execution plan, the Engine maps different data processing stages to corresponding Tasks, which perform the actual work.

From an execution perspective, a Task is not an isolated bit of business logic, but a managed execution unit operating under the Engine. It wraps input, transformation, and output logic, passing data according to runtime rules defined by the Engine.

This architecture ensures Connectors do not need to worry about task scheduling or thread management. A Connector simply provides data processing logic, while task lifecycles, state management, and threading models are handled uniformly by the Engine.

This serves as the foundation for FlushSignal.

If a Connector spawns its own Timer to call flush() directly, it bypasses the Task execution model entirely, spawning an unmonitored side-channel that the Engine cannot manage.

Data Pipeline: How Records Flow

Beyond task execution, we must look at how regular data records traverse the Zeta Engine.

SeaTunnel follows a classic data flow topology:

Source → Transform → Sink

The Source extracts data from external systems and translates it into internal, standardized SeaTunnel records.

Next, records enter the Transform phase, where field mappings, filtering, and transformations are applied according to user configurations.

Finally, transformed records pass downstream to the Sink, which writes them to the target storage system.

Crucially, records are not directly handed off from Source to Sink via direct method calls; they flow through data channels managed by the Engine.

This implies that every stage in the data flow operates under unified Engine control, including memory buffer passing, thread models, and task status tracking.

For FlushSignal, this architectural trait is vital.

If the Engine already boasts a battle-tested data transmission channel, the most logical design for new control events is to reuse this existing pipeline rather than building a side-channel.

In other words, FlushSignals should not bypass the engine via Timer threads calling Sinks directly. They should enter the Engine's data channel just like ordinary records.

Control Events: How SchemaChange Synchronizes

Alongside standard data records, the SeaTunnel Engine manages a second vital class of data: control events.

SchemaChange represents a classic example.

In CDC or dynamic schema synchronization scenarios, the structure of source data can evolve dynamically (e.g., adding columns, altering data types).

These alterations are not standard data records; they are structural metadata events that must be synchronized downstream.

Consequently, SchemaChange events travel directly along the data pipeline.

They are not processed in isolation at the Source, nor do they bypass Transform or Sink modules. Instead, they flow downstream as special control events within the stream, recognized and passed along at each stage.

Transforms typically pass structural change events through without needing to interpret their business logic.

Ultimately, the Sink acts on the SchemaChange event, applying DDL updates to the target system.

This proves that the SeaTunnel Engine data stream carries both operational Data Records and runtime Control Events.

The design of FlushSignal leverages this exact pattern. If SchemaChange can travel seamlessly through the data pipeline as a control event, FlushSignal can do the same.

Checkpoint: The Barrier Processing Pipeline

Beside SchemaChange, Checkpoint Barriers represent another critical control event within the Zeta Engine.

In Exactly-Once processing scenarios, Checkpoints ensure state consistency and data delivery guarantees.

During a Checkpoint, the Engine generates a Barrier and injects it into the data channel.

The Barrier flows downstream alongside data records, establishing a consistent state boundary across task operators.

Upon receiving a Barrier, a Task executes corresponding routines—such as persisting operator states and coordinating Checkpoint completion—ensuring the overall pipeline can safely resume from a known checkpoint state in case of failure.

This underlines a fundamental design rule:

Control events do not need to bypass the data pipeline; they can natively leverage existing Engine channels.

Barriers are not dispatched via sideband thread calls; they propagate through the Task's data stream.

This directly aligns with the philosophy behind FlushSignal.

FlushSignal is not an out-of-band call, but a brand-new control event type. Like SchemaChange and Checkpoint Barriers, it is generated by the Engine and propagated via existing data pipelines.

By examining task scheduling, record flow, SchemaChange propagation, and Checkpoint Barrier management, we see that the SeaTunnel Zeta Engine already possesses a robust data and event processing framework.

Therefore, when JDBC Sink ran into the batch_interval_ms limitation, the right solution was not adding local timer threads to JDBC, but introducing a new control event into the Engine's native execution model.

This explains why STIP-23 ultimately opted for an Engine-Level FlushSignal. Rather than introducing a specialized patch for JDBC, it introduces a standardized runtime signal for time-driven flushes built atop SeaTunnel's existing Task and event architecture.

04 Engine-Level Flush Abstraction

During JDBC Sink development, the underlying challenge exposed by batch_interval_ms was not how to execute a flush, but how to elevate flushes to an Engine-level scheduled capability.

Traditional Connector designs treat flushes as localized buffer commits, evaluated inside writeRecord() as volume thresholds are met. However, time-driven triggers differ fundamentally from data-driven triggers. While batch_size relies on continuous incoming data, batch_interval_ms requires a flush to fire once an interval elapses—even if zero new records enter the pipeline.

Hence, flushing can no longer live as a localized Connector method call. It must be promoted to an Engine-managed runtime event.

The core philosophy of STIP-23 is to abstract Flush operations into Engine-level FlushSignals. Connectors no longer manage local Timers, nor do background threads invoke Sink flush routines directly. Instead, the Engine generates Signals based on configured time intervals, propagating them along SeaTunnel's existing data channels so Sink operators can handle them according to their own semantics.

Under this model, FlushSignal behaves identically to existing SeaTunnel control events. The data pipeline handles standard Data Records alongside Checkpoint Barriers and SchemaChange events. FlushSignal enters the record stream as a first-class control event, giving the Engine unified lifecycle and propagation control.

From Flush Method to Signal Event

Translating Flush operations from direct method calls into Signal events marks the turning point of this architecture.

Traditionally, flushes happened entirely inside Sink operators. For example, in the JDBC Sink, when internal buffers hit capacity, executeBatch() flushed data to the database. This pattern worked flawlessly for volume-based triggers since incoming records continually updated buffer states.

However, once timing triggers enter the picture, the model breaks down. batch_interval_ms implies: "After a specified duration, attempt a flush even if no new records arrive." If this logic remains trapped inside writeRecord(), the execution degrades to "check if timed out whenever the next record happens to arrive," which misses the mark for scheduled flushes.

To illustrate how different event types flow through SeaTunnel's data channel, we use specific identifiers:

R denotes DataRecord (standard operational data);
Ck denotes Checkpoint Barrier (marking checkpoint state boundaries);
Sc denotes SchemaChange (representing metadata evolution events);
F denotes FlushSignal (representing time-driven flush intent).

In a standard data stream, records and barriers propagate sequentially:

R → R → R → Ck → R → R → Ck

When SchemaChange occurs, it injects directly into the stream:

R → R → Sc → R → R → Ck

SchemaChange operates as a inline control event without disturbing the overarching delivery model.

FlushSignal follows this exact paradigm:

R → F → Sc → R → R → Ck → R

Rather than bypassing the data path to notify the Sink directly, it steps into the stream as a native event type.

Thus, FlushSignal does not introduce a ad-hoc invocation path; it elevates flushing into a runtime signal that the Engine can observe, route, and schedule.

How the Engine Triggers Signals

The creation of FlushSignals is handled exclusively by the Engine, completely freeing Connectors from managing background threads.

The complete Engine trigger flow proceeds as follows:

The process begins with job configuration.

When setting:

sink.flush.interval

to a non-zero value, the FlushSignal capability activates. Setting it to 0 disables the feature without affecting legacy job configurations.

During task startup, SourceFlowLifeCycle registers the corresponding Timer. Subsequently, the timerFlushWorker inside TaskExecutionService executes recurring tasks based on fixed-delay scheduling.

Here, the Timer's responsibility is strictly constrained: it only fires periodic notifications and triggers onTimerTick(). The Timer thread itself never executes Sink flushes or manipulates SinkWriter objects directly.

When a Timer tick occurs, the Engine calls:

collector.sendFlushSignal(jobId, taskId)

injecting a FlushSignal into the Source output.

Notably, injecting a FlushSignal relies on the exact same checkpointLock used by Checkpoints.

This design prevents race conditions between FlushSignals and Checkpoint Barriers, as both act as state-altering control events within the data processing path.

By generating FlushSignals at the Source according to standard execution models, the Engine avoids forcing direct executions on Sinks via unmanaged background threads.

Signal Propagation across the Pipeline

Once injected, the FlushSignal travels downstream via SeaTunnel's native Record channel.

The signal propagation chain proceeds as follows:

The signal may transit through internal Engine components like Queue / Disruptor buffers along the way.

The end-to-end responsibilities are divided cleanly:

Engine Timer generates the Signal;
Source injects the Signal into the stream;
Transform transparently passes the Signal through;
SinkFlowLifeCycle detects the Signal and executes the flush routine.

At the Source stage, FlushSignal is broadcast downstream via:

sendRecordToNext()

reaching all downstream consumers through:

output.received(record)

In short, FlushSignal relies on the exact same transmission mechanics as standard Data Records.

When passing through Transforms, no transformation or business logic is applied.

Upon recognizing a Signal, the Transform calls collector.collect(record) directly, skipping transform(). This guarantees FlushSignals bypass record-level processing while preserving their sequence in the stream.

Finally, the FlushSignal arrives at SinkFlowLifeCycle.

The Sink identifies the item as a Signal rather than a Data Record, routing it to:

processSignal()

which executes the registered flush action.

This highlights a core architectural principle:

The signal propagates down shared record channels, but only the Sink interprets and acts upon its semantic meaning.

How Signals Handle Backpressure

Once inside the data channel, FlushSignals must handle backpressure gracefully when buffer queues reach capacity.

SeaTunnel applies distinct queuing policies depending on event types:

For standard DataRecords:

put() / ringBuffer.next()

When queues fill up, records block and wait for available capacity to guarantee zero data loss.

For Checkpoint Barriers:

Barriers similarly execute:

put() / ringBuffer.next()

blocking until space frees up to ensure state integrity across Checkpoints.

However, FlushSignal uses a non-blocking strategy:

offer() / tryPublishEvent()

If the queue is full:

Immediately returns false

and:

Drops the current Signal

In other words, FlushSignal never blocks the core data pipeline when queues are saturated.

This design makes complete sense because FlushSignal carries no business payloads—it signals a transient flush intent.

If internal queues are already saturated, the system is actively processing a heavy volume of data. Under high load, prioritizing raw business records and checkpoint integrity over timed flushes is the correct trade-off.

Furthermore:

prepareClose:
Signal bypassed directly

and:

Queue Full:
FlushSignalQueueFailureTotal +1

These explicit safeguards ensure FlushSignals never block job shutdown, error recovery, or core data throughput under stress.

This keeps FlushSignal lightweight and highly resilient without risking stream stability.

How the Sink Executes Flushes

When a FlushSignal reaches SinkFlowLifeCycle, the Sink alone decides how to execute the operation.

Throughout this entire process, the Engine remains agnostic about JDBC-specific SQL flushing logic or database commands.

The Engine's job ends once the FlushSignal is delivered to the Sink.

Upon receiving the Signal, SinkFlowLifeCycle enters:

processSignal()

and routes execution based on signal classification.

For standard Records:

Record → sinkWriter.write()

continuing normal data ingestion.

For FlushSignals:

FlushSignal → flushAction

executing the specific flush behavior registered by the Connector.

This enforces a clean separation of concerns between Engine and Connector.

The Engine supplies FlushSignal infrastructure without dictating what concrete actions a Sink must take upon arrival.

Because flush semantics vary widely across storage systems:

For JDBC Sinks, a flush means executing executeBatch() SQL statements.
For other Sinks, it might trigger bulk HTTP uploads, stream loads, or memory buffer commits.

Hence, each Connector explicitly opts in by registering its own flushAction.

This makes FlushSignal an extensible Engine primitive rather than a hardcoded trick bound to a single Connector.

Ultimately, flushing evolved from an internal JDBC Sink parameter into a unified runtime control event across the SeaTunnel Engine. The Engine governs signal generation and propagation, while Connectors govern local execution—collaborating through clean interfaces.

This represents a key architectural step forward in STIP-23. It solves far more than batch_interval_ms—it equips the SeaTunnel Zeta Engine with a robust, generalized runtime control framework.

05 Exactly-Once Guarantees

Elevating scheduled flushing into an Engine-level FlushSignal solved the timing issue, but introduced a crucial challenge: how to preserve SeaTunnel's strict Exactly-Once delivery semantics when FlushSignals trigger mid-stream.

For Sinks leveraging XA transactions, a Flush does not equal a Transaction Commit. A FlushSignal can trigger localized batch updates or prepare transaction boundaries, but it must never bypass Checkpoints to commit transactions independently. Under SeaTunnel's Exactly-Once model, transaction states are tightly bound to Checkpoint State. Only transactions wrapped inside a persisted Checkpoint qualify for final commits.

Therefore, Engine-Level FlushSignals must adhere to a strict invariant: FlushSignals may advance internal transaction states, but they can never commit transactions independently of Checkpoints.

Constraints of Exactly-Once

Before FlushSignal was introduced, Sink transaction commits were managed entirely by Checkpoints. When a Checkpoint Barrier arrived, the system persisted transaction states, executing final commits only during the notifyCheckpointComplete phase.

If a FlushSignal were to trigger an XA COMMIT directly, this state model would break down.

The pattern on the left side of the diagram is dangerous because transactions are committed before the Checkpoint State is recorded.

If an XID executes XA COMMIT before its Checkpoint successfully persists, SeaTunnel's state manager has no record of that transaction being committed upon recovery.

If the Source fails shortly after and restarts from the previous Checkpoint, it will re-consume and re-send those exact records—causing duplicate writes in the target database.

Thus, neither Timers nor FlushSignals can ever commit transactions directly.

The correct approach is shown on the right side of the diagram. Here, the FlushSignal simply advances the active transaction into a PREPARE state, leaving the final COMMIT waiting for Checkpoint completion.

In short: PREPARE can occur early, but COMMIT must wait for Checkpoint.

This allows FlushSignal to adjust the cadence of transaction staging without compromising Exactly-Once boundaries.

Following this principle, the current JDBC XA Writer implementation does not register timer-based flushes, keeping transaction commits firmly anchored to the Checkpoint lifecycle.

Splitting XA Transactions

With direct commits off the table, the next question becomes: how does FlushSignal influence XA transaction demarcation?

The target design model functions as follows:

R denotes DataRecord;
F1, F2 denote FlushSignals;
CK denotes Checkpoint Barrier.

When a FlushSignal arrives, it does not finish the Checkpoint lifecycle. Instead, it helps the Sink split large transaction boundaries into manageable chunks.

Transactions are chunked as follows:

txn-1: R R R F1
txn-2: R R F2
txn-3: R R CK

When F1 arrives, the Sink executes:

executeBatch() → XA PREPARE → beginTx()

completing data staging for that specific segment.

Subsequent records flow into a fresh transaction scope until the next FlushSignal arrives.

When F2 arrives, it triggers another PREPARE sequence for that segment.

Crucially, these prepared transactions remain uncommitted in the database:

pendingCommitInfo and pendingStates are staged in memory, then aggregated and persisted during the next Checkpoint.

FlushSignal simply helps the Sink partition monolithic transactions into multiple prepared XA transactions, while leaving final execution rights to Checkpoints.

When the Checkpoint Barrier finally arrives:

prepare txn-3
snapshotState()
merge pending
    ↓
Checkpoint State: [txn-1, txn-2, txn-3]

The Checkpoint State records all prepared transactions (txn-1, txn-2, txn-3).

Finally, during the execution of:

notifyCheckpointComplete

the engine calls:

XA COMMIT ALL

committing all prepared segments in one atomic step.

This design allows FlushSignal and Exactly-Once semantics to coexist seamlessly. FlushSignal provides flexible transaction boundaries, while Checkpoint maintains absolute control over final commits.

Failover and Recovery Rules

In distributed execution environments, tasks can fail at any moment. Therefore, robust recovery rules are required to resolve transaction states during failovers.

The golden rule remains: Recovery depends strictly on whether an XID was persisted in a completed Checkpoint State.

FlushSignal itself plays no role during recovery; it merely influences when transactions enter the prepared state.

Depending on when a failure occurs, recovery falls into three distinct scenarios:

Scenario 1: Failure occurs before Checkpoint completion.

Though txn-A and txn-B were prepared via F1 and F2, their metadata never made it into a persisted Checkpoint State.

Recovery sequence:

State contains no txn-A / txn-B
    ↓
XA RECOVER
    ↓
ROLLBACK
    ↓
Source replayed from CK-N

Uncommitted prepared transactions are safely rolled back, and the Source replays stream data from the last valid Checkpoint.

Scenario 2: Failure occurs after Checkpoint State is saved, but before COMMIT.

Here, XIDs were successfully persisted inside the Checkpoint State.

Recovery sequence:

State contains txn-A / txn-B / txn-C
    ↓
restoreCommit
    ↓
COMMIT

The system reads transaction metadata from the Checkpoint State and commits them to the target storage.

Scenario 3: Failure occurs after Checkpoint commits complete.

Recovery sequence:

Transactions already committed
    ↓
XA RECOVER returns empty
    ↓
no-op

No orphan transactions remain; the system resumes standard execution.

Across all three failure modes, FlushSignal never compromises SeaTunnel's core Exactly-Once guarantees. It simply introduces finer control over transaction staging within the overarching Checkpoint framework.

FlushSignal handles staging, Checkpoint State handles persistence, and notifyCheckpointComplete handles execution.

This clean separation ensures Engine-Level flushing operates safely within XA transactions while establishing a extensible pattern for future runtime events.

06 System Boundaries and Architecture Roles

The journey from fixing JDBC's batch_interval_ms to introducing an Engine-Level FlushSignal forced a fundamental rethink of the architectural boundaries between SeaTunnel Engine and its Connectors.

Initially, looking at the problem purely through the lens of the JDBC Connector led to localized fixes.

Because the JDBC Sink needed timed flushes, the immediate thought was to spawn a local Timer inside the JDBC plugin.

However, deeper analysis revealed that Timer management, thread coordination, exception propagation, and task lifecycle management are not JDBC-specific problems at all. They are fundamental runtime capabilities that belong to the core execution engine.

If every Connector implemented its own Timer, code duplication would explode across the codebase.

Each Connector would need to reinvent how to launch, stop, and safeguard background threads against race conditions with Checkpoints, job cancellations, and teardowns.

Eventually, a simple configuration parameter would force Connectors to maintain increasingly complex runtime logic.

That is a classic anti-pattern.

Through the design of FlushSignal, SeaTunnel re-established clear domain boundaries between Engine and Connector.

Connectors focus strictly on external system semantics.

For instance, the JDBC Sink knows how to construct batch SQL statements, execute transactions, handle prepare/commit/rollback semantics, and perform flushes when requested.

These operational details belong exclusively to the Connector.

The Engine manages generic, system-wide runtime mechanics.

This includes Timer lifecycles, control event generation, stream signal routing, and ensuring Signals execute safely on the correct Task thread.

These capabilities belong to the platform engine, not individual plugins.

Therefore, instead of having JDBC manage local Timers, the cleaner architecture dictates:

Connector registers a flushAction.
Engine generates and routes the FlushSignal downstream to the Sink.
Sink executes the flush action safely on its dedicated processing thread.

This cleanly decouples timing mechanics from target storage behaviors.

In STIP-23, the Engine exposes this configuration capability cleanly:

sink.flush.interval = 5000

This setting defines scheduled behavior at the Engine level.

When enabled, the Engine periodically emits FlushSignals. Setting it to 0 disables signal emission, keeping legacy jobs operating as normal.

Yet, the Engine never invokes external flush actions directly.

The Engine remains intentionally ignorant of whether a JDBC Sink flushes via executeBatch(), or whether another Sink relies on bulk HTTP posts, stream loads, or memory buffer commits.

Connectors opt in by registering an action via the Sink Context:

context.registerFlushAction(() -> {
    flush();
});

Registration is entirely at the Connector's discretion.

The overarching win here is that Timers never invoke Sink operations directly out-of-band.

If a Timer thread executed out-of-band flushes directly:

Timer Thread → SinkWriter.flush()

the system would fall back into the original multi-threading trap, reintroducing:

Race conditions between flushes and normal writes;
Uncaught exceptions bypassing Task error channels;
Fragile thread lifecycle management during teardowns.

Instead, STIP-23 restricts the Timer to emitting FlushSignals. Once injected into standard record channels, the Signal flows downstream to SinkFlowLifeCycle, executing safely within the main consumer thread.

The end-to-end execution flow moves predictably:

Timer → FlushSignal → Data Path → SinkFlowLifeCycle → flushAction.run()

Flushing is successfully brought back into SeaTunnel's unified execution model.

07 Key Takeaways: What FlushSignal Teaches Us About Engine Evolution

The journey from a JDBC batch_interval_ms parameter to an Engine-Level FlushSignal started as a local patch, but ultimately revealed how modern data engines must evolve through proper abstraction. The longevity of an engine feature depends not on how many lines of code are written, but on whether domain boundaries are respected, existing architectures are reused, and extensible hooks are provided.

1. Define Semantics First: Clarify Capability Boundaries

Designing FlushSignal required establishing strict semantic boundaries for what a flush means. A FlushSignal does not guarantee a successful commit, nor does it imply data is immediately visible externally—it simply grants the Sink an opportunity to perform a flush.

Different storage engines define "success" in fundamentally different ways. At-Least-Once processing accepts replaying data upon failure, whereas Exactly-Once processing relies strictly on Checkpoints and transaction boundaries to eliminate duplicates. Thus, FlushSignal can only trigger execution opportunities; it can never replace a Connector's internal consistency mechanisms.

This explains why the Engine must never force all Sinks to execute flushes blindly. Across different Connectors, a flush might mean executing batch SQL, staging transactions, or triggering custom API calls. If the Engine ignored these nuances, it would risk corrupting transaction states and breaking Exactly-Once guarantees.

A reusable capability must clearly state what it delivers—and what it intentionally leaves to lower layers.

2. Evolve atop Existing Foundations: Reuse the Core Execution Model

FlushSignal avoids introducing ad-hoc sideband execution paths. Instead, it introduces a new event type directly into the existing Task pipeline.

Had the Timer invoked the Sink directly:

Timer Thread → SinkWriter.flush()

it might have seemed easier to code, but it would have introduced thread concurrency issues—causing race conditions with active writes, Checkpoints, and task shutdowns while masking background exceptions from the main Task thread.

Instead, FlushSignal leverages SeaTunnel's battle-tested data channels. The Engine Timer emits a Signal, injecting it at the Source to flow naturally along the Source → Transform → Sink path.

Along this chain, the Source injects the Signal, Transforms pass it through untouched, and SinkFlowLifeCycle detects it to run the local flush action. FlushSignals share the exact thread and queue models as Data Records, eliminating the need for unmanaged sideband threads.

This reflects a fundamental rule of engine design: new features should seamlessly build upon core abstractions rather than stacking special-case hacks.

3. Extension Points Dictate Evolution Costs: From Local Patches to Framework Abstractions

The value of FlushSignal extends far beyond resolving JDBC flush timeouts; it sets a precedent for how Engine and Connector responsibilities should be divided.

The Engine manages generic infrastructure—Timer scheduling, Signal generation, lifecycle coordination, and stream routing. Meanwhile, Connectors handle target-specific semantics—executing actual flushes, opting into timing features, and guaranteeing local transaction integrity.

By offering clean flushAction registration via Context APIs or SPIs, Connectors can freely opt into scheduled capabilities without the Engine needing to know lower-level implementation details.

This modular architecture delivers three distinct advantages: default behaviors preserve legacy Connector stability, new capabilities honor existing transactional semantics, and future runtime control requirements can leverage this exact same event-driven abstraction.

Looking back at where this started, batch_interval_ms was never just a missing parameter inside a JDBC Connector—it exposed a missing runtime control abstraction within the Engine itself. Moving from localized thread hacks to Engine-Level Signals represents a classic transition from quick-fix engineering to clean architectural abstraction.

A mature data engine does not attempt to hardcode every edge case up front. Instead, it establishes robust abstractions and extension points, allowing new capabilities to integrate cleanly, safely, and at minimal cost.

What Happens When Apache SeaTunnel Submits a Job?

Apache SeaTunnel — Fri, 07 Aug 2026 17:00:00 +0000

A SeaTunnel job submission may look like a simple submitJob request from the outside. However, inside the Server, the request goes through multiple stages, including Master node validation, job coordination, JobMaster initialization, physical execution plan construction, Pipeline resource allocation, and TaskGroup deployment.

Based on the submitJob sequence I analyzed, this article focuses on one core path: from the moment a job submission request enters the SeaTunnel Server to the point where TaskExecutionService.deployTask() is finally called to deploy the TaskGroup.

This article does not cover the internal thread model of TaskExecutionService, Task execution details, or data flow processing. Instead, it focuses on the job submission, scheduling, and deployment lifecycle.

Core Components

Before diving into the workflow, let’s first understand the responsibilities of several key components involved in the submitJob execution path.

Role	Responsibility
`SubmitJobServlet`	Receives external job submission requests and serves as one of the entry points on the Server side.
`JobInfoService`	Handles job submission logic and determines whether the current node is a Master or a Worker.
`MasterNode`	When the current node is not the Master, forwards the job submission request to the Master node.
`CoordinatorService`	Serves as the job coordination entry point, checks whether the job is already running, and creates/manages `JobMaster`.
`JobMaster`	Acts as the execution control center for a single job, responsible for initializing the runtime context, classloader, checkpoint configuration, and other settings.
`PhysicalPlan`	A physical execution plan built from the logical DAG, responsible for driving Job-level state transitions.
`SubPlan`	A Pipeline-level scheduling unit responsible for resource allocation and Pipeline state transitions.
`ResourceUtils`	Requests execution resources for the Pipeline.
`PhysicalVertex`	A finer-grained physical execution node responsible for deploying TaskGroups.
`TaskExecutionService`	The execution service that ultimately receives and deploys TaskGroups.

Overall Workflow

The following simplified flow diagram provides an overview of the complete lifecycle.

The entire flow can be summarized in one sentence:

SubmitJobServlet
  -> JobInfoService
  -> MasterNode / CoordinatorService
  -> JobMaster
  -> PhysicalPlan
  -> SubPlan
  -> PhysicalVertex
  -> TaskExecutionService

Now, let’s break down each stage.

Stage 1: Request Enters JobInfoService

The job submission entry point first reaches SubmitJobServlet, which then delegates the request to JobInfoService.

The key point here is not to start the job immediately, but to first determine:

Is the node currently receiving the request the Master node?

If the current node is the Master, JobInfoService can continue the submission process locally.

If the current node is a Worker, the request needs to be forwarded to the Master through MasterNode.submitJob().

This design ensures that job submission is always coordinated by the Master node, preventing multiple nodes from independently creating scheduling contexts for the same job.

Stage 2: CoordinatorService Takes Over the Job

After reaching the Master node, the request continues to CoordinatorService.submitJob().

At this stage, CoordinatorService mainly performs two tasks:

Check whether the job already exists or is currently running.
If it is a new job, create and initialize the corresponding JobMaster.

If the job is already running, SeaTunnel does not need to create another scheduling context and can directly return a successful submission response.

For a new job, the process enters the JobMaster initialization phase.

At this point, submitJob has moved from API request handling into scheduler-level processing.

Stage 3: JobMaster Initialization

JobMaster can be understood as the runtime control center for a job.

After creating JobMaster, SeaTunnel performs several preparation steps required before execution, including:

Building the classloader required for job execution.
Initializing checkpoint-related configurations.
Preparing the context required to build the physical execution plan from the logical DAG.

At this stage, Tasks have not been deployed yet. Instead, the system is preparing the runtime environment required for later scheduling.

Stage 4: From Logical DAG to PhysicalPlan

After JobMaster initialization, SeaTunnel builds a PhysicalPlan based on the logical DAG.

An important concept here is:

SeaTunnel does not start the entire job at once. Instead, it gradually progresses through different states using a state machine.

At the Job level, the core state transition can be simplified as:

CREATED -> SCHEDULED -> startSubPlanStateProcess

PhysicalPlan is responsible for Job-level state transitions, while actual Pipeline scheduling continues further down into SubPlan.

Stage 5: SubPlan Resource Allocation and Deployment

At the SubPlan layer, SeaTunnel shifts its focus from the entire Job to the Pipeline level.

SubPlan.stateProcess() executes different logic based on the current Pipeline state:

The key points at this stage are:

In the CREATED state, the Pipeline first transitions to SCHEDULED.
In the SCHEDULED state, SeaTunnel starts requesting resources through ResourceUtils.applyResourceForPipeline().
After resources are successfully allocated, the Pipeline enters the DEPLOYING state.
If resource allocation fails, the Pipeline enters makePipelineFailing(e).

Therefore, a Pipeline is not deployed immediately. It must first acquire the required execution resources.

Stage 6: PhysicalVertex Deploys TaskGroup

When the Pipeline enters the DEPLOYING state, the SubPlan starts launching internal PhysicalVertex components.

PhysicalVertex first updates the Task state to DEPLOYING, then performs deployment based on the assigned slotProfile.

During deployment, there is a key decision point:

Is the target Worker local or remote?

If the target Worker is a remote node, SeaTunnel sends a deployment request through DeployTaskOperation. The request is eventually handled on the target Worker through:

TaskExecutionService.deployTask(taskGroupInfo)

After successful deployment, PhysicalVertex updates the Task state to RUNNING.

If deployment fails, the system enters:

makeTaskGroupFailing()

Once all TaskGroups inside the Pipeline are successfully deployed and enter the running state, the SubPlan also transitions to RUNNING.

Failure, Cancellation, and Recovery Paths

In addition to the normal submission and deployment path, the SubPlan state machine also handles failures, cancellations, and recovery scenarios.

The process can be simplified as follows:

This is why the state machine design is important:

The normal path can continue through deployment and execution.
Failure paths can transition into failing / failed.
Cancellation paths can transition into canceling / canceled.
When recovery conditions are met, resources can be released, requested again, and the Pipeline can be restored.

In other words, the state machine is not designed to make the workflow complicated. It exists to make the entire job lifecycle controllable and reliable.

Complete Sequence Diagram

Finally, the complete sequence diagram connects the entire workflow and provides a clearer view of the execution order.

Summary

After a SeaTunnel job is submitted, the core process is not simply “receive the request and start the task.”

The complete lifecycle roughly follows this path:

SubmitJobServlet
  -> JobInfoService
  -> MasterNode / CoordinatorService
  -> JobMaster
  -> PhysicalPlan
  -> SubPlan
  -> PhysicalVertex
  -> TaskExecutionService

The responsibilities of each component are:

JobInfoService handles the submission entry point and determines whether the request needs to be forwarded to the Master node.
CoordinatorService manages job coordination, prevents duplicate submissions, and creates the JobMaster.
JobMaster initializes the runtime context required by the job.
PhysicalPlan manages Job-level state transitions.
SubPlan handles Pipeline-level resource allocation and scheduling.
PhysicalVertex manages TaskGroup deployment.
TaskExecutionService is the final entry point responsible for deploying TaskGroups.

Understanding this lifecycle makes it much easier to explore SeaTunnel’s Task execution model, data flow architecture, and checkpoint mechanism, because each module can be placed in its correct position within the overall architecture.

DEV Community: Apache SeaTunnel

🔍 A successful pipeline doesn’t guarantee trusted data. See how Apache SeaTunnel enables reliable movement, recovery, and schema evolution.

Pipeline Green Isn’t Data Correct

🔄 A financial tech company migrated 200+ ETL workflows from Informatica to Apache SeaTunnel, cutting infrastructure costs 60% and runtimes 40%!

From Informatica to Apache SeaTunnel: A Financial-Grade ETL Migration in Practice

From Informatica to Apache SeaTunnel: A Financial-Grade ETL Migration in Practice

1. Project Background and Challenges

2. Migration Strategy and Design

2.1 Technology Selection

2.2 Migration Strategy

3. Core Migration Implementation

3.1 Metadata Migration

3.2 Refactoring Complex Transformations

3.3 Performance Tuning in Practice

4. Validation and Cutover

4.1 Ensuring Data Consistency

4.2 Gradual Rollout

5. Lessons Learned

5.1 Key Success Factors

5.2 Lessons Learned the Hard Way

Quantified Results After the Migration

Pipeline Green Isn’t Data Correct

Managed ELT Solves Operational Overhead, Not Data Correctness

SeaTunnel’s First Job: Getting Data There Reliably

From Checkpoint to Schema Evolution: Building Reliability into the Runtime

SeaTunnel Doesn’t Prove Business Data Is Correct. It Provides the Foundation for Proving It.

From SeaTunnel to DolphinScheduler: Managing the Bigger Pipeline

The Value of WhaleStudio ACP: Connecting the Pieces

So, the Real Question Isn’t “Fivetran or SeaTunnel?”

Solve HTTP memory limits in big data! 🚀 Switch from JSON to JSONL with Apache SeaTunnel for smooth streaming. 💡 #ApacheSeaTunnel #BigData #DataEngineering #JSONL

From JSON to JSONL: How Apache SeaTunnel Solves the HTTP Big Data Transfer Memory Dilemma

From JSON to JSONL: How Apache SeaTunnel Solves the HTTP Big Data Transfer Memory Dilemma

1. Data Output Formats: JSON vs. JSONL

2. Key Parameter for Format Identification

3. Practical Configuration Example

4. Key Takeaway

Data Ingestion Must Never Be a "Black Box"!

Why Data Ingestion Cannot Be a Black Box

Transitioning from "Sync Tool" to "Data Infrastructure"

Seven Black-Box Pitfalls Highlighted on Reddit

Why the Black Box Becomes Exponentially Dangerous at Scale

Level 1: Unverifiable Schema Evolution

Level 2: CDC Is More Than Data Replication

Level 3: AI Amplifies Black-Box Vulnerabilities

How Apache SeaTunnel Builds Auditable Data Integration

1. Fully Open-Source Connectors: Transparent Data Ingestion

2. Pipeline as Code: Configurations Function as Audit Documents

3. Engine Transparency: Verifiable Runtime Execution

4. Controlled Schema Evolution: Shift from "Auto-Modify" to "Governed Evolution"

5. Native On-Premises Deployment & Full Data Sovereignty

Why "Auditability" Trumps "No-Code"

The Future of Data Platforms Requires "Trusted Synchronization"

See how @ASFSeaTunnel Zeta built a scientifically accurate benchmark! 👇 #SeaTunnel #BigData #DataEngineering #Performance #OpenSource

600,000 Records/Sec with Sub-100ms Latency: Inside SeaTunnel Zeta’s Rigorous Performance Test

600,000 Records/Sec with Sub-100ms Latency: Inside SeaTunnel Zeta’s Rigorous Performance Test

1. Interpreting the Test Results

2. Why Micro-benchmarks and Complete Pipelines Co-exist

SeaTunnelRow Micro-benchmarks: Pinpointing Hot Paths

Zeta Pipeline Benchmarks: End-to-End System Validation

3. Theoretical Foundations of the Benchmark Design

JVM Performance Evaluation: From Raw Repetition to Statistically Sound Sampling

Allocating Experimental Budgets: Identifying Variance Layers

Stream Processing Metrics: Shifting from Stability to Accuracy

Synthesis of the Three Methodologies

4. Operationalizing Theory in Zeta Pipeline Benchmarks

Plan-Time Scheduling with BenchmarkSource

Isolating Engine Overhead via Deterministic Components

Data Validation as a Prerequisite for Sample Validity

Fork Configurations, Warmups, and Machine-Readable Outputs

5. Deciphering Dual-Layer Metrics

Differentiating JMH Scores from Pipeline Throughput

Metric Definitions: Score, Error, CV, and Units

Latency Percentiles, Growth Ratios, and Valid Samples

6. Zeta Engine Performance Analysis

7. Key Takeaways and Engineering Summary

💡 How a simple JDBC parameter forced an engine-level evolution! Discover how @ASFSeaTunnel Zeta introduced STIP-23 FlushSignal to solve the scheduled flush dilemma. ⚡️ #SeaTunnel #DataEngineering #BigData #OpenSource

How a Single JDBC Parameter Drove Engine Evolution in Apache SeaTunnel Zeta

How a Single JDBC Parameter Drove Engine Evolution in Apache SeaTunnel Zeta

About the Speaker

01 The Root Cause: Addressing Data Latency

Plan-Time Scheduling with `BenchmarkSource`