DEV Community: Apache SeaTunnel

What Happens When Apache SeaTunnel Submits a Job?

Apache SeaTunnel — Fri, 07 Aug 2026 17:00:00 +0000

A SeaTunnel job submission may look like a simple submitJob request from the outside. However, inside the Server, the request goes through multiple stages, including Master node validation, job coordination, JobMaster initialization, physical execution plan construction, Pipeline resource allocation, and TaskGroup deployment.

Based on the submitJob sequence I analyzed, this article focuses on one core path: from the moment a job submission request enters the SeaTunnel Server to the point where TaskExecutionService.deployTask() is finally called to deploy the TaskGroup.

This article does not cover the internal thread model of TaskExecutionService, Task execution details, or data flow processing. Instead, it focuses on the job submission, scheduling, and deployment lifecycle.

Core Components

Before diving into the workflow, let’s first understand the responsibilities of several key components involved in the submitJob execution path.

Role	Responsibility
`SubmitJobServlet`	Receives external job submission requests and serves as one of the entry points on the Server side.
`JobInfoService`	Handles job submission logic and determines whether the current node is a Master or a Worker.
`MasterNode`	When the current node is not the Master, forwards the job submission request to the Master node.
`CoordinatorService`	Serves as the job coordination entry point, checks whether the job is already running, and creates/manages `JobMaster`.
`JobMaster`	Acts as the execution control center for a single job, responsible for initializing the runtime context, classloader, checkpoint configuration, and other settings.
`PhysicalPlan`	A physical execution plan built from the logical DAG, responsible for driving Job-level state transitions.
`SubPlan`	A Pipeline-level scheduling unit responsible for resource allocation and Pipeline state transitions.
`ResourceUtils`	Requests execution resources for the Pipeline.
`PhysicalVertex`	A finer-grained physical execution node responsible for deploying TaskGroups.
`TaskExecutionService`	The execution service that ultimately receives and deploys TaskGroups.

Overall Workflow

The following simplified flow diagram provides an overview of the complete lifecycle.

The entire flow can be summarized in one sentence:

SubmitJobServlet
  -> JobInfoService
  -> MasterNode / CoordinatorService
  -> JobMaster
  -> PhysicalPlan
  -> SubPlan
  -> PhysicalVertex
  -> TaskExecutionService

Now, let’s break down each stage.

Stage 1: Request Enters JobInfoService

The job submission entry point first reaches SubmitJobServlet, which then delegates the request to JobInfoService.

The key point here is not to start the job immediately, but to first determine:

Is the node currently receiving the request the Master node?

If the current node is the Master, JobInfoService can continue the submission process locally.

If the current node is a Worker, the request needs to be forwarded to the Master through MasterNode.submitJob().

This design ensures that job submission is always coordinated by the Master node, preventing multiple nodes from independently creating scheduling contexts for the same job.

Stage 2: CoordinatorService Takes Over the Job

After reaching the Master node, the request continues to CoordinatorService.submitJob().

At this stage, CoordinatorService mainly performs two tasks:

Check whether the job already exists or is currently running.
If it is a new job, create and initialize the corresponding JobMaster.

If the job is already running, SeaTunnel does not need to create another scheduling context and can directly return a successful submission response.

For a new job, the process enters the JobMaster initialization phase.

At this point, submitJob has moved from API request handling into scheduler-level processing.

Stage 3: JobMaster Initialization

JobMaster can be understood as the runtime control center for a job.

After creating JobMaster, SeaTunnel performs several preparation steps required before execution, including:

Building the classloader required for job execution.
Initializing checkpoint-related configurations.
Preparing the context required to build the physical execution plan from the logical DAG.

At this stage, Tasks have not been deployed yet. Instead, the system is preparing the runtime environment required for later scheduling.

Stage 4: From Logical DAG to PhysicalPlan

After JobMaster initialization, SeaTunnel builds a PhysicalPlan based on the logical DAG.

An important concept here is:

SeaTunnel does not start the entire job at once. Instead, it gradually progresses through different states using a state machine.

At the Job level, the core state transition can be simplified as:

CREATED -> SCHEDULED -> startSubPlanStateProcess

PhysicalPlan is responsible for Job-level state transitions, while actual Pipeline scheduling continues further down into SubPlan.

Stage 5: SubPlan Resource Allocation and Deployment

At the SubPlan layer, SeaTunnel shifts its focus from the entire Job to the Pipeline level.

SubPlan.stateProcess() executes different logic based on the current Pipeline state:

The key points at this stage are:

In the CREATED state, the Pipeline first transitions to SCHEDULED.
In the SCHEDULED state, SeaTunnel starts requesting resources through ResourceUtils.applyResourceForPipeline().
After resources are successfully allocated, the Pipeline enters the DEPLOYING state.
If resource allocation fails, the Pipeline enters makePipelineFailing(e).

Therefore, a Pipeline is not deployed immediately. It must first acquire the required execution resources.

Stage 6: PhysicalVertex Deploys TaskGroup

When the Pipeline enters the DEPLOYING state, the SubPlan starts launching internal PhysicalVertex components.

PhysicalVertex first updates the Task state to DEPLOYING, then performs deployment based on the assigned slotProfile.

During deployment, there is a key decision point:

Is the target Worker local or remote?

If the target Worker is a remote node, SeaTunnel sends a deployment request through DeployTaskOperation. The request is eventually handled on the target Worker through:

TaskExecutionService.deployTask(taskGroupInfo)

After successful deployment, PhysicalVertex updates the Task state to RUNNING.

If deployment fails, the system enters:

makeTaskGroupFailing()

Once all TaskGroups inside the Pipeline are successfully deployed and enter the running state, the SubPlan also transitions to RUNNING.

Failure, Cancellation, and Recovery Paths

In addition to the normal submission and deployment path, the SubPlan state machine also handles failures, cancellations, and recovery scenarios.

The process can be simplified as follows:

This is why the state machine design is important:

The normal path can continue through deployment and execution.
Failure paths can transition into failing / failed.
Cancellation paths can transition into canceling / canceled.
When recovery conditions are met, resources can be released, requested again, and the Pipeline can be restored.

In other words, the state machine is not designed to make the workflow complicated. It exists to make the entire job lifecycle controllable and reliable.

Complete Sequence Diagram

Finally, the complete sequence diagram connects the entire workflow and provides a clearer view of the execution order.

Summary

After a SeaTunnel job is submitted, the core process is not simply “receive the request and start the task.”

The complete lifecycle roughly follows this path:

SubmitJobServlet
  -> JobInfoService
  -> MasterNode / CoordinatorService
  -> JobMaster
  -> PhysicalPlan
  -> SubPlan
  -> PhysicalVertex
  -> TaskExecutionService

The responsibilities of each component are:

JobInfoService handles the submission entry point and determines whether the request needs to be forwarded to the Master node.
CoordinatorService manages job coordination, prevents duplicate submissions, and creates the JobMaster.
JobMaster initializes the runtime context required by the job.
PhysicalPlan manages Job-level state transitions.
SubPlan handles Pipeline-level resource allocation and scheduling.
PhysicalVertex manages TaskGroup deployment.
TaskExecutionService is the final entry point responsible for deploying TaskGroups.

Understanding this lifecycle makes it much easier to explore SeaTunnel’s Task execution model, data flow architecture, and checkpoint mechanism, because each module can be placed in its correct position within the overall architecture.

🔍 Ever wondered what happens after submitting a SeaTunnel job? Follow the journey from SubmitJobServlet to TaskExecutionService and discover the engine behind reliable data pipeline execution. #ApacheSeaTunnel #DataEngineering 🚦

Apache SeaTunnel — Fri, 07 Aug 2026 10:15:12 +0000

Apache SeaTunnel

Aug 7

What Happens When Apache SeaTunnel Submits a Job?

#apacheseatunnel #programming #datascience #opensource

5 min read

🌊 Apache SeaTunnel brings 5 sessions to Community Over Code Asia 2026! Explore data lakes, AI pipelines, Data Fabric, CDC, and next-gen data infrastructure with global open source experts. 🚀 #ApacheSeaTunnel #OpenSource #DataEngineering

Apache SeaTunnel — Fri, 07 Aug 2026 08:23:53 +0000

Apache SeaTunnel

Aug 7

Countdown: 1 Day to Community Over Code Asia 2026 — Apache SeaTunnel Brings 5 Technical Sessions Starting August 8

#techtalks #communityovercodeasia #apacheseatunnel #opensource

10 min read

Countdown: 1 Day to Community Over Code Asia 2026 — Apache SeaTunnel Brings 5 Technical Sessions Starting August 8

Apache SeaTunnel — Fri, 07 Aug 2026 08:14:51 +0000

Only 1 day to go! Apache SeaTunnel sessions at Community Over Code Asia 2026 will kick off on August 8 (China Standard Time).

As one of the Apache Software Foundation’s key technology events in Asia, Community Over Code Asia 2026 will bring together open source contributors, developers, and technology enthusiasts from around the world to explore the latest practices and future trends in open source technology.

Starting from August 8 (China Standard Time), Apache SeaTunnel will bring five technical sessions across two major tracks — Data Lake & Data Warehouse and DataOps — covering key topics including data integration, real-time synchronization, AI-driven data pipelines, and the evolution of modern data infrastructure.

Before the event begins, let’s take a closer look at the Apache SeaTunnel sessions and explore the technical insights behind each talk. We warmly invite you to join the event, connect with the Apache SeaTunnel community, and be part of this global open source gathering.

See you on August 8!

Data Lake & Data Warehouse Track

Data lakes and data warehouses are essential solutions for storing and managing data. They play a critical role in data management, analytics, and decision-making.

Within the Apache Software Foundation, many projects focus on data lakes and data warehouses, including Apache Hive, Apache Hudi, Apache Iceberg, Apache Paimon, Apache Cassandra, and Apache HBase.

In this track, attendees will learn about the latest developments in data lake and warehouse technologies, production best practices from companies, and the future roadmaps of these projects.

Track Producers

Lidong Dai

Lidong Dai is CTO of WhaleOps, Apache Incubator Mentor, Apache DolphinScheduler PMC Member, and Apache SeaTunnel PMC Member.

With 16 years of experience in data technologies, he focuses on AI-ready heterogeneous data integration, data processing orchestration, and data governance.

Based on the Apache SeaTunnel and Apache DolphinScheduler ecosystems, he leads the development of WhaleStudio, a commercial data platform solution that has served customers across industries including finance, automotive, gaming, manufacturing, retail, and internet services.

Shaofeng Shi

Shaofeng Shi is a member of the Apache Software Foundation and a Mentor of Apache Gravitino, Apache Gluten, Apache HoraeDB, and other projects.

He focuses on big data analytics and cloud computing technologies. Previously, he worked as a Senior Engineer in eBay’s Global Analytics Infrastructure team and as a Software Architect in IBM’s Cloud Computing division.

Zongtang Hu

Zongtang Hu is a technical expert in middleware and big data at China Mobile Cloud Center, where he leads the middleware and big data team.

He has more than eight years of experience in message middleware kernel development and architecture design. He participated in the development of multiple major middleware products, including China Mobile Cloud RocketMQ, MQTT, and Kafka, contributing to their core architecture and implementation from the ground up.

As a technical speaker, he has shared insights at ApacheCon Asia 2022/2023/2024, Apache RocketMQ Summit/Meetup, and Cloud Native Service conferences.

He has extensive open source community experience and serves as a Maintainer/Committer in communities including Apache RocketMQ, Nacos, openEuler message-middleware SIG, and OpenMessaging.

He was recognized as an Outstanding Contributor to Cloud Computing Open Source Standards by the China Academy of Information and Communications Technology (CAICT) in 2023, named an OSCAR Open Source Leader by CAICT in 2024, and received multiple open source community recognitions.

Huaxin Gao

Huaxin Gao is a Software Engineer at Snowflake, an Apache Spark Committer and PMC Member.

She is also a Committer of Apache Iceberg and Apache DataFusion Comet. Her contributions cover query engines, table formats, and distributed data systems.

Within this track, Apache SeaTunnel will bring one technical sessions. Let’s take a closer look.

Session: From Data Ingestion to Data Lake: Building a Modern Lakehouse with Apache SeaTunnel

Time: August 8, 14:30–15:00 (China Standard Time)

Session Overview

Building a data lake is no longer just about choosing Iceberg, Hudi, or Paimon. In real-world systems, the biggest challenge often lies one step earlier: how data reliably, efficiently, and continuously enters the lake. In this session, we will explore how Apache SeaTunnel serves as a unified data ingestion and integration layer for modern data lake architectures. Starting from common pain points—multi-source data, Batch + CDC coexistence, schema evolution, and operational complexity—we will walk through how SeaTunnel simplifies data movement into data lakes and lakehouse systems. Through real production scenarios, you will see how SeaTunnel connects transactional databases, message queues, and file systems into Iceberg- or lakehouse-based storage, enabling scalable, maintainable, and evolvable data platforms. The talk focuses on practical architecture decisions, not vendor-specific solutions.

The session will cover:

Typical data lake architecture evolution and common pitfalls
The role of data integration in lake and lakehouse systems
Apache SeaTunnel architecture and design principles
End-to-end ingestion examples: databases, CDC, and streaming data into data lakes
Operational considerations and best practices
Roadmap of SeaTunnel in the data lake ecosystem

Speaker

Lidong Dai | Co-founder of WhaleOps Technology

Apache Incubator Mentor, Apache DolphinScheduler PMC Member, and Apache SeaTunnel PMC Member.

Beyond the Data Lake & Data Warehouse track, Apache SeaTunnel will also bring four technical sessions to the DataOps track this year. Each session is packed with practical insights and cutting-edge exploration, making this one of the most anticipated parts of the event.

DataOps Track

This track focuses on some of the most innovative and forward-looking projects in the Apache ecosystem.

It brings together leading experts and contributors from projects including Apache DolphinScheduler, Apache Airflow, Apache SeaTunnel, Apache Flume, Apache Sqoop, Apache Griffin, Apache Atlas, and other DataOps-related communities to explore the latest advancements in data operations, automation, and orchestration.

Whether you are an experienced data professional or just getting started in the field, this track provides valuable insights across a wide range of topics, including data pipelines, ETL, orchestration, data quality, metadata management, and more.

Join us at Community Over Code Asia 2026 to explore the evolving world of DataOps with the Apache community.

Track Producers

Wei Guo

Community Over Code Asia 2026

CEO of WhaleOps, Apache Member, Apache Incubator Mentor

Wei Guo is the CEO of WhaleOps, a member of the Apache Software Foundation, an Apache Incubator Mentor, and currently serves as a member of the Open Source Technology Committee of the China Communications Society and Deputy Director of the Intelligent Application Services Branch of the China Software Industry Association.

He is also Vice President of the Global SME Entrepreneurship Union, President of the Beijing Chapter of TGO Kunpeng Club, and a recognized digital technology leader.

He serves as a member of the Apache DolphinScheduler PMC and an Apache SeaTunnel Mentor, and is the founder of the ClickHouse Chinese Community.

Wei Guo graduated from Peking University and has more than 20 years of experience in the big data field. He previously worked as a Senior Architect at IBM and Teradata, CTO of eBay’s China business, General Manager of Wanda E-commerce Data Department, and Director of Big Data at Lenovo Research Institute.

He has made significant contributions to research and innovation in emerging big data technologies.

Lifeng Nie

Community Over Code Asia 2026

COO of WhaleOps, Apache SeaTunnel PMC Member & Apache DolphinScheduler Committer

Lifeng Nie is COO of WhaleOps, an Apache SeaTunnel PMC Member, Apache DolphinScheduler Committer, and was recognized as one of the “33 Open Source Pioneers of China 2023.”

He also leads the volunteer team of the ClickHouse Chinese Community and actively contributes to open source ecosystem development.

1. Session: From Natural Language to Reliable Data Pipelines: Building an AI-Powered CLI for Apache SeaTunnel

Time: August 9, 13:30–14:00 (China Standard Time)

Session Overview

Modern DataOps is not only about moving data faster, but also about making pipeline development easier, safer, and more accessible to engineers and data teams.

In this talk, I will introduce seatunnel-cli, a new Python-based CLI for Apache SeaTunnel that generates HOCON pipeline configurations directly from natural language descriptions in both English and Chinese.

The tool is designed as an AI-powered multi-agent workflow: Planner, Config Generator, Validator, and Auto-fix. It combines a three-tier knowledge base, a connector catalog automatically generated from SeaTunnel Java source code, dry-run validation, and iterative repair to help users produce runnable pipeline configurations with less manual effort. It also supports multiple LLM providers, persistent session memory, interactive exploration, and single-shot scripting usage.I will share the technical design behind the CLI, including how we extract and resolve connector metadata at scale, how validation and auto-fix loops improve configuration quality, and how this approach can reduce the barrier to building SeaTunnel jobs in real-world DataOps scenarios.

The session will also cover practical lessons from testing across multilingual inputs, multiple model providers, and broken-config recovery cases.

Speaker

Xin Zhang | Solution Architect, Amazon Web Services

Xin Zhang is an AWS Solutions Architect, responsible for solution consulting and design based on the AWS Cloud platform. He has a rich experience in R&D and architecture practice in the fields of system architecture, data warehousing, and real-time computing.

2. Session: Building AI's Data Artery: Architecture and Practices of Unified Multimodal Data Pipelines

Time: August 9, 14:00–14:30 (China Standard Time)

Session Overview

In the GenAI era, the massive flow of multimodal data demands a robust infrastructure, yet fragmented data pipelines have become a critical bottleneck for enterprises. At Tongcheng Travel, we historically operated 4 disjointed data pipeline services (Offline Sync, Real-time Lake Ingestion, legacy Sqoop, and a standalone SeaTunnel service). This fragmentation caused extremely high maintenance costs and hindered unified data governance.

This session details how we successfully architected a unified “Data Artery” through platformization. We will explore how we consolidated the data entry points and built a true “Batch-Stream Unified” foundational architecture based on Apache SeaTunnel, comprehensively supporting data flows from traditional data warehouses to modern AI scenarios.

Key Content:

Breaking Data Silos: A deep dive into designing a unified multimodal data pipeline service powered by the Apache SeaTunnel engine, smoothly replacing and consolidating 4 legacy integration systems to achieve complete architectural standardization.
Compute Enhancement & AI Multimodal Empowerment: Exploring how to deeply integrate SeaTunnel’s Transform mechanism with real-time stream processing capabilities to efficiently execute complex data cleaning and dynamic transformations. We will highlight hardcore support for AI workloads, including real-time parsing of unstructured data and Embedding preprocessing for LLMs.
Zero-Downtime Migration & Strict Validation: Sharing enterprise-grade practices on migrating massive legacy tasks. We will detail the “Dynamic Task Conversion and Bi-directional Data Reconciliation Mechanism” we designed to ensure zero data loss and a seamless transition for the business during the underlying architecture upgrade.
Future Cloud-Native Evolution: Looking ahead at the blueprint for multimodal unified data pipelines. We will discuss cloud-native containerized deployments on Kubernetes for elastic scaling, and how to deeply integrate with the LLM ecosystem to build a robust Data+AI foundation.

Speaker

Xiaochen Zhou | Data Engineer at Tongcheng Travel | Apache SeaTunnel Committer

Xiaochen Zhou is a Data Engineer at Tongcheng Travel and an active Apache SeaTunnel Committer. In his current role, he specializes in designing, building, and optimizing high-performance data pipelines. Within the open-source community, he is deeply involved in the core development and technical evolution of Apache SeaTunnel. Recently, his focus has shifted to the intersection of Data and AI, where he is dedicated to architecting unified multimodal data pipelines for the GenAI era.

3.From 'Usable' to 'Governable': ClassLoader Lifecycle Governance Practice in Apache SeaTunnel

Time: August 9, 15:15–15:45 (China Standard Time)

Session Overview

ClassLoader leaks are among the most hidden and difficult-to-diagnose runtime problems in long-lived JVM systems, and a common challenge faced by long-running JVM workloads across the Apache ecosystem. Existing approaches largely remain at the “post-mortem investigation” stage: monitoring tools and heap dumps can tell developers “which ClassLoaders are still alive,” but struggle to clearly explain “why they cannot be reclaimed,” let alone translate governance intent into verifiable, reproducible engineering practices.

Through deep analysis of Apache SeaTunnel’s classloading mechanism, we identified a widely overlooked blind spot: a large number of seemingly correct lifecycle implementations never establish explicit resource-close semantics. Without enforced close and drain constraints, resource reclamation becomes highly unpredictable — and in certain scenarios, leads to ClassLoaders that can never be collected.

Drawing on our exploration of runtime governance for long-lived JVM systems, we proposed a systematic ClassLoader lifecycle governance improvement plan to the Apache SeaTunnel community. The core shift is from passive “post-hoc residual reference hunting” to proactive “building deterministic reclaimable semantics and lifecycle closure.” We incrementally introduced explicit lifecycle close mechanisms, enforced classloading boundary constraints, and active residual reference cleanup. The governance proposal is currently under community discussion, with the Phase 1 optimization PR in review.

This talk will explore real-world community practices in Apache SeaTunnel, covering:

Why long-lived systems cannot rely on implicit GC to manage underlying runtime resources
How to build a ClassLoader governance standard for long-lived JVM systems and progressively land it in Apache open source projects
How to complete a smooth, kernel-level architectural governance upgrade without breaking compatibility

Speaker

Jinxiang Yang | Creator of LingFrame & LingMirror | Apache SeaTunnel Community Member | JVM Runtime Governance

Jinxiang Yang is the creator of LingFrame (灵珑), an open source runtime governance framework for long-lived JVM single-process systems, and LingMirror, an IntelliJ IDEA plugin for static ClassLoader leak diagnosis. Focused on the governance of long-running JVM systems, he identified critical ClassLoader lifecycle gaps in Apache SeaTunnel through deep source code analysis, and proposed a systematic governance improvement plan that has been actively discussed in the SeaTunnel community. He believes a system is not complete when it runs — it is complete when its lifecycle can be observed, governed, and proven correct over time.

4. Session: Why SeaTunnel Engine Needs Clearer Distributed Abstractions

Time: August 9, 15:45–16:15 (China Standard Time)

Session Overview

In distributed data processing engines, responsibilities such as coordination, communication, and state management are often tightly coupled within a single framework. This can make the system difficult to evolve, extend, or replace individual components over time.

In this talk, I will share my experience working on the SeaTunnel engine, where we explored introducing clearer abstractions to reduce dependency on a single distributed framework. I will discuss why this kind of decoupling is necessary, the approach we considered, and the challenges we encountered along the way.

Rather than focusing on a specific technology choice, this talk will highlight the design questions and trade-offs involved in separating core responsibilities in a distributed engine.

Attendees will gain practical insights into how to think about abstraction boundaries and how to approach architectural evolution in real-world distributed systems.

Speaker

Doyeon Kim | Apache SeaTunnel Committer

Doyeon Kim is an Apache SeaTunnel Committer and a student with a strong interest in data engineering and distributed systems. She has contributed to SeaTunnel in areas such as connector development, engine improvements, and architectural discussions. Her recent work has focused on engine internals, dependency decoupling, and design trade-offs in distributed systems. Through this talk, she shares lessons learned from exploring clearer abstractions in the SeaTunnel engine.

💓 Don’t miss one of the biggest open source events of the year — Community Over Code Asia 2026. Join Apache SeaTunnel and connect with the global open source community!

We look forward to meeting you there!

🌟 Click Read More to register.

Limited seats are available. Register now and join us! 👆

🚀 Data is the fuel of AI, but reliable data movement is the foundation. Explore how SeaTunnel Zeta enables scalable sync, CDC, and resilient pipelines. 🌊 #ApacheSeaTunnel #DataEngineering #AIInfrastructure #AI

Apache SeaTunnel — Fri, 07 Aug 2026 07:00:42 +0000

Apache SeaTunnel

Aug 7

Why Open Source Data Integration Is Becoming the Foundation of AI-Era Data Infrastructure

#ai #programming #opensource #apacheseatunnel

6 min read

Why Open Source Data Integration Is Becoming the Foundation of AI-Era Data Infrastructure

Apache SeaTunnel — Fri, 07 Aug 2026 07:00:21 +0000

In the past, when enterprises built data platforms, their focus was usually concentrated on two areas: how to store more data and how to leverage computing capabilities to analyze that data. As a result, data warehouses, data lakes, and various computing engines became the core components driving the evolution of data architectures.

However, with the rise of real-time business requirements and AI applications, enterprises are facing a new challenge that is no longer simply “how to analyze data,” but rather “how to keep data flowing reliably and continuously.”

Today, a typical enterprise data environment includes multiple systems such as operational databases, messaging systems, real-time computing platforms, data lakes, and analytical databases. Data generated by business operations needs to continuously move from one system to another to support real-time analytics, business decisions, and intelligent applications.

Data synchronization has gradually evolved from a simple ETL extraction task into a fundamental capability that connects the entire data ecosystem.

However, traditional data synchronization approaches are facing new challenges. In the early stages, enterprises often relied on scripts to replicate data. As the number of data sources and synchronization tasks increased, these scripts gradually turned into difficult-to-maintain data pipelines. Later, enterprises attempted to use computing engines such as Flink and Spark to solve synchronization problems, but these frameworks were primarily designed for data processing rather than long-running, reliable data movement.

This is why enterprises are increasingly looking for dedicated execution engines designed specifically for data synchronization scenarios.

From Scripts to Compute Engines: Why Traditional Synchronization Approaches Fall Short

When data volumes are relatively small, a simple Python script can complete a data synchronization task. For example, it can read data from an operational database, perform basic transformations, and write the results into a data warehouse. This approach is simple to develop and can quickly satisfy initial business requirements.

However, as enterprises accumulate more data sources and target systems, problems with the script-based approach become increasingly apparent. Different data sources require different connection logic, and different business scenarios require different data processing methods. Eventually, enterprises are no longer managing just a few synchronization jobs, but rather a collection of data programs without unified management capabilities.

More importantly, scripts often struggle with failures in production environments. When a synchronization task that has been running for hours fails, the system needs to know which data has already been processed, which records need to be resent, and how to prevent duplicate writes. These challenges cannot be solved effectively through simple scripts.

As a result, enterprises began adopting distributed computing frameworks, hoping to improve synchronization reliability through better task management and fault tolerance capabilities.

However, a computing engine and a synchronization engine are designed to solve different problems.

A computing engine focuses on complex data processing, such as transformations, aggregations, and calculations. A synchronization system focuses on ensuring that data can move from Source to Sink reliably and efficiently.

For large-scale synchronization workloads, running simple data transfer pipelines on a complete computing framework introduces additional resource consumption and operational complexity. Enterprises therefore need a Runtime that focuses specifically on the data movement process.

SeaTunnel Zeta: A Data Execution Engine Designed for Synchronization Scenarios

The goal of the SeaTunnel Zeta Engine is not to become another general-purpose computing platform. Instead, it is designed specifically for data synchronization scenarios with a dedicated execution architecture.

In traditional data integration platforms, data connectors, task execution, and state management are often tightly coupled. When enterprises need to add new data sources or support new synchronization scenarios, system complexity continues to increase.

SeaTunnel adopts a decoupled architecture between Connectors and Runtime, allowing data connectivity capabilities and execution capabilities to evolve independently.

In this architecture, Connectors are responsible for connecting different data systems, such as databases, message queues, and data lake storage systems. They solve the problem of how to read and write data. The Zeta Engine is responsible for the actual execution process of synchronization jobs, including scheduling, execution, state management, and failure recovery.

The significance of this design is that expanding the data source ecosystem does not affect the execution engine itself. When enterprises integrate new data systems, they only need to add the corresponding Connector without redesigning the entire synchronization workflow.

How Distributed Execution Enables Large-Scale Data Synchronization

The biggest performance challenge in enterprise data synchronization is not simply increasing the number of threads. The real challenge is how to properly split large-scale jobs and allow multiple execution nodes to work together efficiently.

For example, when synchronizing a large table containing billions of records, if a single node is responsible for reading and writing all data, the overall execution time will be limited by factors such as data read speed, network bandwidth, and the write capability of the target system.

Through its distributed execution model, Zeta Engine breaks large synchronization jobs into multiple Tasks, which are executed in parallel by different Workers.

This approach allows synchronization jobs to fully utilize cluster resources and scale horizontally as data volumes grow.

More importantly, distributed execution does not only improve performance. It changes how enterprises manage synchronization workloads. In the past, a synchronization task was treated as an independent program. In the Zeta Runtime, synchronization tasks become data pipelines managed by a unified execution system, making scheduling, monitoring, and recovery much easier.

In CDC Scenarios, the Real Challenge Is Data Consistency

With the growing demand for real-time data, CDC has become a critical capability for enterprise data synchronization.

CDC continuously captures database change logs and synchronizes changes such as INSERT, UPDATE, and DELETE operations to target systems.

A typical workflow looks like this:

Many people believe the core challenge of CDC is reading Binlog or WAL. However, log capture is only the beginning.

The truly complex problems are ensuring event processing order while changes continue to arrive, and enabling correct recovery after failures.

For example, assume a database generates three change events:

Event A → Event B → Event C

If Event A and Event B have already been successfully processed, but Event C fails due to an unexpected issue during synchronization, the system must know the current processing position and continue correctly after recovery.

Without state management capabilities, duplicate data or missing data may occur.

This is one of the key differences between a synchronization engine and a simple data transfer tool.

How Checkpoint Ensures Reliable Recovery for Synchronization Jobs

For long-running data synchronization jobs, failures are not exceptional events. They are operational states that must be considered as part of system design.

A reliable synchronization system needs to record critical states during job execution. When a task is interrupted due to network issues, node failures, or pressure from the target system, the system should be able to resume execution based on the saved state instead of restarting the entire process from the beginning.

SeaTunnel Zeta uses state management and the Checkpoint mechanism to preserve the runtime state of synchronization jobs.

The value of Checkpoint is not only about recovering failed jobs. More importantly, it ensures data correctness after recovery. For continuously running data pipelines such as CDC workflows, this capability determines whether the system can operate reliably over the long term.

Self-Hosted Deployment Gives Enterprises Control Over Data Movement

As data becomes a core enterprise asset, more organizations are paying closer attention to how their data synchronization platforms are deployed.

Data synchronization systems connect multiple critical internal systems and manage the entire journey of data — where it is generated, how it is processed, and where it ultimately flows. Therefore, for industries such as finance, manufacturing, and large enterprises, deploying synchronization systems within their own infrastructure environments is highly important.

SeaTunnel supports self-hosted deployment, allowing enterprises to run synchronization clusters on their preferred infrastructure, including physical servers, private clouds, or Kubernetes environments.

This approach enables enterprises to maintain control over their data execution environments while adjusting resource allocation according to business scale, without relying entirely on external platforms.

From Data Transfer Tools to Data Movement Infrastructure

Data synchronization is undergoing a significant transformation.

In the past, enterprises viewed synchronization as just one step in the ETL process. Today, with the rapid growth of real-time data and AI applications, synchronization capabilities have become a critical infrastructure layer connecting business systems, data platforms, and intelligent applications.

The core value of SeaTunnel Zeta is not simply providing more data connectors. Instead, through a Runtime designed specifically for synchronization scenarios, it addresses the fundamental challenges enterprises face when operating long-running data pipelines.

Through Connector and Runtime decoupling, distributed execution models, CDC support, and state consistency mechanisms, SeaTunnel is helping data synchronization evolve from traditional data transfer jobs into a more reliable and scalable data infrastructure capability.

🔽 🔽
Learn more:

Apache SeaTunnel: https://github.com/apache/seatunnel
About WhaleOps: https://www.whaleops.io/ WhaleOps is the driving force behind AApache SeaTunnel’s commercial ecosystem, empowering enterprises with enterprise-grade data integration, synchronization, and DataOps solutions built on open source innovation.

🚀 Master MySQL PostgreSQL migration with Apache SeaTunnel 2.3.12! Explore auto-generated SQL, UPSERT, partitioning, and performance tuning for production data pipelines. ⚡

Apache SeaTunnel — Thu, 30 Jul 2026 09:54:08 +0000

Apache SeaTunnel

Jul 30

Building Production-Ready MySQL to PostgreSQL Data Pipelines with Apache SeaTunnel 2.3.12

#mysql #postgressql #apacheseatunnel #dataengineering

18 min read

Building Production-Ready MySQL to PostgreSQL Data Pipelines with Apache SeaTunnel 2.3.12

Apache SeaTunnel — Thu, 30 Jul 2026 09:53:18 +0000

This article provides a complete hands-on walkthrough of heterogeneous data synchronization with Apache SeaTunnel 2.3.12, covering everything from PostgreSQL container deployment and test data preparation to five commonly used production synchronization patterns.

Through these practical examples, you will learn how to leverage SeaTunnel’s core capabilities, including data transformation, batch optimization, and pre-execution SQL operations, enabling developers to quickly build MySQL-to-PostgreSQL synchronization pipelines—even without prior experience with data integration frameworks.

The default target database used in this demo is PostgreSQL 18.1.
Docker installation guide:
https://www.cnblogs.com/kakarotto-chen/p/19351495
The source database is MySQL.
SeaTunnel MySQL Source Connector documentation:
https://seatunnel.apache.org/zh-CN/docs/connector-v2/source/Mysql
The SeaTunnel sink connector used in this demo is PostgreSQL Sink.
Official documentation:
https://seatunnel.apache.org/zh-CN/docs/connector-v2/sink/PostgreSql

Preparing Source Test Data (MySQL)

Before running the synchronization demos, prepare the source dataset in MySQL.

The demo uses a sample table named t_8_100w, which contains approximately 1 million records and covers common data types, including:

Numeric fields
String fields
Timestamp fields
Large text fields
Nullable address fields

Example schema:

CREATE TABLE t_8_100w (
    id BIGINT PRIMARY KEY,
    user_name VARCHAR(2000),
    sex VARCHAR(20),
    decimal_f DECIMAL(32,6),
    phone_number VARCHAR(20),
    age VARCHAR(20),
    create_time TIMESTAMP,
    description TEXT,
    address VARCHAR(2000)
);

You can generate the sample dataset using your own test data generator or load the prepared SQL script before running the SeaTunnel synchronization jobs.

Demo 1: Full Load + Truncate Before Insert + Physical Delete

Suitable for extracting tables without primary keys. This mode takes effect only when the integration type is Full Load and the synchronization delete mode is configured as Physical Delete.
During each execution, the target table will be cleared first and then fully reloaded with all data from the source table.
Core principle:

Truncate and reload every time

Core configuration:

data_save_mode = "DROP_DATA"

Execution Command

sh /……/seatunnel-2.3.12/bin/seatunnel.sh --config /……/myconf/ds-st-demo1-mysql2pgsql-ql-qkhcr-wlsc.conf -i JAVA_OPTS='-Xmx2g -Xms2g' -m local

Create Table

-- ds-st-demo1-mysql2pgsql-ql-qkhcr-wlsc.conf

CREATE TABLE "public"."t_8_100w_imp_st_ds_demo1_ql_qkhcr_wlsc" (
  id BIGINT PRIMARY KEY,
  user_name VARCHAR(2000),
  sex VARCHAR(20),
  decimal_f NUMERIC(32, 6),
  phone_number VARCHAR(20),
  age INT,
  create_time TIMESTAMP,
  description TEXT,
  address VARCHAR(2000) DEFAULT 'Unknown'
);

COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo1_ql_qkhcr_wlsc"."id" IS 'Primary key';

COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo1_ql_qkhcr_wlsc"."user_name" IS 'User name';

COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo1_ql_qkhcr_wlsc"."sex" IS 'Gender: Male; Female';

COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo1_ql_qkhcr_wlsc"."decimal_f" IS 'Large numeric value';

COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo1_ql_qkhcr_wlsc"."phone_number" IS 'Phone number';

COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo1_ql_qkhcr_wlsc"."age" IS 'Convert string age to integer';

COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo1_ql_qkhcr_wlsc"."create_time" IS 'Creation timestamp';

COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo1_ql_qkhcr_wlsc"."description" IS 'Long text content';

COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo1_ql_qkhcr_wlsc"."address" IS 'Replace empty address with default value: Unknown';

SeaTunnel Configuration

# ds-st-demo4-mysql2pgsql-ql-qkhcr-wlsc.conf

env {
  # Parallelism (number of execution threads)
  execution.parallelism = 5

  # Job execution mode:
  # BATCH: Batch processing mode
  # STREAMING: Stream processing mode
  job.mode = "BATCH"
}


source {

  jdbc {

    url = "jdbc:mysql://ip:port/cs1"

    driver = "com.mysql.cj.jdbc.Driver"

    user = "root"

    password = "zysoft"


    # SQL query used to extract source data
    query = "select * from t_8_100w"


    # Assign a name to this dataset
    plugin_output = "source_data"



    # Parallel reading configuration

    # Numeric primary key column used for data partitioning
    partition_column = "id"


    # Number of partitions, aligned with execution.parallelism
    partition_num = 5


    # Number of records fetched in each batch
    fetch_size = 5000


    # Optional:
    # partition_lower_bound = 1
    # Starting ID value

    # partition_upper_bound = 1000000
    # Ending ID value



    # Connection parameters

    # Connection timeout: 300 seconds
    connection_check_timeout_sec = 300


    # Additional JDBC parameters
    properties = {

      useUnicode = true

      characterEncoding = "utf8"

      serverTimezone = "Asia/Shanghai"


      # Enable cursor-based fetching to improve performance
      # for large result sets
      useCursorFetch = "true"


      # Number of rows fetched each time
      defaultFetchSize = "5000"
    }
  }
}



# Data cleaning and transformation
#
# For simple transformations, it is recommended to process
# the logic directly in the source SQL query whenever possible.

transform {


  # 1. Field mapping:
  #
  # The mapping logic is already handled in the SQL query.
  # In real production scenarios, this step can also be implemented
  # through the FieldMapper plugin.

  FieldMapper {

    plugin_input = "source_data"

    plugin_output = "FieldMapper_data"


    field_mapper = {

      id = id

      name = user_name

      sex = sex

      decimal_f = decimal_f

      phone_number = phone_number


      # Temporary field name
      age = age_str


      create_time = create_time

      description = description

      address = address
    }
  }



  Sql {

    plugin_input = "FieldMapper_data"

    plugin_output = "Sql_age_data"


    query = """

      SELECT

        id,

        user_name,

        sex,

        decimal_f,

        phone_number,

        CAST(age_str AS INTEGER) as age,

        create_time,

        description,

        address

      from dual

    """
  }



  # 2. Phone number masking:
  #
  # Example:
  # 13812341234 -> 138****1234



  # 3. Age conversion:
  #
  # Convert age from string format to integer.
  #
  # In real production environments,
  # if conversion is unnecessary, the value can be stored directly.



  # 4. Gender conversion:
  #
  # Example:
  # 1 -> Male
  # 2 -> Female



  # 5. Data filtering:
  #
  # Keep only records where age > 25.



  # 6. Default address handling:
  #
  # Replace empty address values with "Unknown".

}






sink {


  jdbc {


    url = "jdbc:postgresql://ip:port/source_db"


    driver = "org.postgresql.Driver"


    user = "postgres"


    password = "123456"



    # query = "insert into test_table(name,age) values(?,?)"


    # Automatically generate INSERT SQL.
    #
    # If the target table does not exist,
    # SeaTunnel can automatically create it.
    generate_sink_sql = true



    # Required when generate_sink_sql=true
    database = source_db



    table = "public.t_8_100w_imp_st_ds_demo4_ql_qkhcr_wlsc"



    # Final dataset received by the sink
    plugin_input = "Sql_age_data"




    # Error when target table does not exist.
    #
    # In most production scenarios,
    # CREATE_SCHEMA_WHEN_NOT_EXIST is recommended:
    #
    # Create the table if it does not exist.
    # Skip creation if the table already exists
    # and keep existing data.

    schema_save_mode = "ERROR_WHEN_SCHEMA_NOT_EXIST"




    # APPEND_DATA:
    #
    # Keep table schema and existing data,
    # then append new records.
    #
    # (Usually recommended)


    # DROP_DATA:
    #
    # Keep table schema,
    # remove all existing records.
    #
    # Used to implement truncate-and-reload strategy.

    data_save_mode = "DROP_DATA"




    # Number of records written in each batch
    batch_size = 5000


    # Batch commit interval
    batch_interval_ms = 500



    # Maximum retry attempts
    max_retries = 3




    # Connection parameters

    # Connection timeout: 300 seconds
    connection_check_timeout_sec = 300




    # Additional JDBC parameters

    properties = {


      # PostgreSQL-specific optimization parameter
      #
      # Enable PostgreSQL batch insert optimization.
      #
      # Note: Parameter name is case-sensitive.
      reWriteBatchedInserts = "true"



      # Timezone configuration if required
      options = "-c timezone=Asia/Shanghai"

    }

  }

}

Execution Result

During execution, the log shows that SeaTunnel performs a table truncation operation before loading new data:

TRUNCATE TABLE "public"."t_8_100w_imp_st_ds_demo4_ql_qkhcr_wlsc"

Execution log:

2025-12-15 15:32:25,675 INFO  [.s.c.s.j.c.AbstractJdbcCatalog] [seatunnel-coordinator-service-2] - Catalog Postgres established connection to jdbc:postgresql://ip:port/source_db

2025-12-15 15:32:25,676 INFO  [a.s.a.s.SaveModeExecuteWrapper] [seatunnel-coordinator-service-2] - Executing save mode for table: source_db.public.t_8_100w_imp_st_ds_demo4_ql_qkhcr_wlsc, with SchemaSaveMode: ERROR_WHEN_SCHEMA_NOT_EXIST, DataSaveMode: DROP_DATA using Catalog: Postgres

2025-12-15 15:32:25,719 INFO  [a.s.a.s.DefaultSaveModeHandler] [seatunnel-coordinator-service-2] - Truncating table source_db.public.t_8_100w_imp_st_ds_demo4_ql_qkhcr_wlsc with action TRUNCATE TABLE "public"."t_8_100w_imp_st_ds_demo4_ql_qkhcr_wlsc"

2025-12-15 15:32:25,722 INFO  [.s.c.s.j.c.AbstractJdbcCatalog] [seatunnel-coordinator-service-2] - Execute sql : TRUNCATE TABLE "public"."t_8_100w_imp_st_ds_demo4_ql_qkhcr_wlsc"

Job progress:

***********************************************
           Job Progress Information
***********************************************

Job Id                    : 1052850597616680961

Read Count So Far         :              849934

Write Count So Far        :              849934

Average Read Count        :             14165/s

Average Write Count       :             14165/s

Last Statistic Time       :             2025-12-15 15:32:26

Current Statistic Time    :             2025-12-15 15:33:26

***********************************************

Final job statistics:

***********************************************
           Job Statistic Information
***********************************************

Start Time                : 2025-12-15 15:32:23

End Time                  : 2025-12-15 15:33:39

Total Time(s)             :                  76

Total Read Count          :             1000004

Total Write Count         :             1000004

Total Failed Count        :                   0

***********************************************

Result summary:

Total records read: 1,000,004
Total records written: 1,000,004
Failed records: 0
Execution time: 76 seconds
Average throughput: 14,165 records/s

This demonstrates a typical production pattern for full-load synchronization with truncate-and-reload semantics using Apache SeaTunnel.

Demo 2: Full Load + Append Data

This synchronization mode is suitable for scenarios where historical data in the target database needs to be preserved and new records from the source database should be appended continuously.

Typical production scenarios include:

Initial data migration followed by incremental data accumulation.
Periodic full extraction where historical records must remain unchanged.
Data warehouse loading scenarios where new partitions or new business records are appended.
Core Principle

Keep existing data and append new records

Unlike the previous DROP_DATA mode, this mode does not clear the target table before writing.

The execution flow is:

Read source data

        ↓

Transform data

        ↓

Insert new records into target table

        ↓

Keep existing target data unchanged

Core Configuration

data_save_mode = "APPEND_DATA"

Execution Command

sh /……/seatunnel-2.3.12/bin/seatunnel.sh --config /……/myconf/ds-st-demo2-mysql2pgsql-append.conf -i JAVA_OPTS='-Xmx2g -Xms2g' -m local

Target Table

CREATE TABLE "public"."t_8_100w_imp_st_ds_demo2_append" (

  id BIGINT PRIMARY KEY,

  user_name VARCHAR(2000),

  sex VARCHAR(20),

  decimal_f NUMERIC(32, 6),

  phone_number VARCHAR(20),

  age INT,

  create_time TIMESTAMP,

  description TEXT,

  address VARCHAR(2000) DEFAULT 'Unknown'

);


COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo2_append"."id"
IS 'Primary key';


COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo2_append"."user_name"
IS 'User name';


COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo2_append"."sex"
IS 'Gender: Male; Female';


COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo2_append"."decimal_f"
IS 'Large numeric value';


COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo2_append"."phone_number"
IS 'Phone number';


COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds-demo2_append"."age"
IS 'Convert string age to integer';


COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo2_append"."create_time"
IS 'Creation timestamp';


COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo2_append"."description"
IS 'Long text content';


COMMENT ON COLUMN "public"."t_8_100w_imp_st_ds_demo2_append"."address"
IS 'Replace empty address with default value: Unknown';

SeaTunnel Configuration

env {

  # Number of parallel execution threads
  execution.parallelism = 5


  # Job mode:
  # BATCH: Batch processing mode
  # STREAMING: Streaming processing mode
  job.mode = "BATCH"

}



source {

  jdbc {

    url = "jdbc:mysql://ip:port/cs1"


    driver = "com.mysql.cj.jdbc.Driver"


    user = "root"


    password = "zysoft"



    # Extract source data
    query = "select * from t_8_100w"



    plugin_output = "source_data"



    # Enable parallel reading

    # Partition by numeric primary key
    partition_column = "id"


    # Number of parallel partitions
    partition_num = 5


    # Fetch records in batches
    fetch_size = 5000



    properties = {

      useUnicode = true

      characterEncoding = "utf8"

      serverTimezone = "Asia/Shanghai"


      # Improve performance for large result sets
      useCursorFetch = "true"


      defaultFetchSize = "5000"

    }

  }

}





transform {


  FieldMapper {


    plugin_input = "source_data"


    plugin_output = "FieldMapper_data"


    field_mapper = {

      id = id

      name = user_name

      sex = sex

      decimal_f = decimal_f

      phone_number = phone_number

      age = age

      create_time = create_time

      description = description

      address = address

    }

  }

}





sink {


  jdbc {


    url = "jdbc:postgresql://ip:port/source_db"


    driver = "org.postgresql.Driver"


    user = "postgres"


    password = "123456"



    generate_sink_sql = true



    database = source_db



    table = "public.t_8_100w_imp_st_ds_demo2_append"



    plugin_input = "FieldMapper_data"



    # Fail if table does not exist
    schema_save_mode = "ERROR_WHEN_SCHEMA_NOT_EXIST"



    # Preserve existing data and append new records
    data_save_mode = "APPEND_DATA"



    # Batch write size
    batch_size = 5000



    # Batch commit interval
    batch_interval_ms = 500



    # Retry attempts
    max_retries = 3



    properties = {


      # PostgreSQL batch insert optimization
      reWriteBatchedInserts = "true"


      options = "-c timezone=Asia/Shanghai"

    }

  }

}

Execution Result

2025-12-16 17:07:03,711 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - 
        ***********************************************
                   Job Statistic Information
        ***********************************************
        Start Time                : 2025-12-16 17:06:08
        End Time                  : 2025-12-16 17:07:03
        Total Time(s)             :                  54
        Total Read Count          :             1000001
        Total Write Count         :             1000001
        Total Failed Count        :                   0
        ***********************************************

Demo 2.1: Full Load - Differential Comparison Synchronization (Insert + Update) - Physical Delete + Large Table Partitioning (Auto-generated SQL)

Core requirements: Same as Demo 2
Core logic: Same as Demo 2. However, SeaTunnel will automatically generate SQL statements, so there is no need to manually write the query statement.
Core configuration:

enable_upsert = true

primary_keys = ["unique_key"]

Execution command

sh /……/seatunnel-2.3.12/bin/seatunnel.sh --config /……/myconf/ds-st-demo2-mysql2pgsql-ql-cybjcj-wlsc-2-enable_upsert.conf -i JAVA_OPTS='-Xmx2g -Xms2g' -m local

conf configuration

env {

  # Job name: 
  # In production environments, the table ID can be used as the identifier.
  job.name = "ds-st-demo2-mysql2pgsql-ql-cybjcj-wlsc-2-enable_upsert.conf"


  # Maximum parallel threads:
  # Parallelism
  parallelism = 5


  # Job mode:
  # BATCH: Batch processing mode
  # STREAMING: Streaming processing mode
  job.mode = "BATCH"

}

source {

  jdbc {

    url = "jdbc:mysql://ip:port/cs1"

    driver = "com.mysql.cj.jdbc.Driver"

    user = "root"

    password = "zysoft"


    # SQL used to extract source data
    query = "select * from t_8_100w"


    # Assign a name to this dataset
    plugin_output = "source_data"



    # Parallel reading configuration


    # Partition column:
    # Supports:
    # String, Number (int, bigint, decimal, ...), Date

    partition_column = "id"



    # Split size:
    # Number of rows contained in each split.
    #
    # Default value: 8096 rows.
    #
    # Final number of splits =
    # Total table rows / split.size

    split.size = 50000



    # Number of partitions.
    #
    # In SeaTunnel 2.3.12, this configuration is no longer recommended.
    # Use split.size instead.

    # partition_num = 5



    # Maximum batch fetch size:
    #
    # Specifies the number of rows read in each execution.
    #
    # Default value: 1000.
    #
    # This value is affected by available memory.
    # If this value is large or each record contains large amounts of data,
    # increase the runtime memory accordingly.

    fetch_size = 10000



    # Connection parameters


    # Connection timeout: 300 seconds

    connection_check_timeout_sec = 300



    # Additional JDBC parameters

    properties = {

      useUnicode = true

      characterEncoding = "utf8"


      # Timezone configuration.
      # Parameters may vary depending on the database.

      serverTimezone = "Asia/Shanghai"



      # Use cursor fetching to improve
      # large result set performance.

      useCursorFetch = "true"



      # Number of rows fetched each time.

      defaultFetchSize = "10000"

    }

  }

}

# Data cleaning and transformation.
#
# For simple transformations,
# you can directly process them in the source query SQL.

transform {


  # 1. Field mapping:
  #
  # The mapping is already implemented in SQL.
  # In actual production scenarios, this section does not need to handle it.
  #
  # You can also use the FieldMapper plugin
  # for field mapping.

  FieldMapper {

    plugin_input = "source_data"

    plugin_output = "FieldMapper_data"


    field_mapper = {

      id = id

      name = user_name

      sex = sex

      decimal_f = decimal_f

      phone_number = phone_number


      # Temporary field name

      age = age_str


      create_time = create_time

      description = description

      address = address

    }

  }



  # Convert age to numeric type.
  # PostgreSQL requires this conversion.

  Sql {

    plugin_input = "FieldMapper_data"

    plugin_output = "Sql_age_data"


    query = """

      SELECT

        id,

        user_name,

        sex,

        decimal_f,

        phone_number,

        CAST(age_str AS INTEGER) as age,

        create_time,

        description,

        address

      from dual

    """

  }



  # 2. Phone number masking:
  # 13812341234 -> 138****1234



  # 3. Age conversion:
  # Convert string to integer.
  #
  # In actual production scenarios,
  # conversion is not required.
  # There is also no built-in conversion plugin.
  # The data can be saved directly.



  # 4. Gender conversion:
  # 1 -> Male
  # 2 -> Female



  # 5. Data filtering:
  # Keep only records where age > 25.



  # 6. Default address value:
  # Replace empty address with "Unknown".

}

继续，保持原文格式。

---

id="x7m1aa"
sink {

jdbc {

# Final target dataset
plugin_input = "Sql_age_data"


url = "jdbc:postgresql://ip:port/source_db"

driver = "org.postgresql.Driver"

user = "postgres"

password = "123456"



# query = """
# INSERT INTO public.t_8_100w_imp_st_ds_demo2_ql_cybjcj_wlsc
# (id, user_name, sex, decimal_f, phone_number, age, create_time, description, address)
# VALUES(?,?,?,?,?,?,?,?,?)
#
# ON CONFLICT ("id")
# DO UPDATE SET
# "user_name" = EXCLUDED."user_name",
# "sex" = EXCLUDED."sex",
# "decimal_f" = EXCLUDED."decimal_f",
# "phone_number" = EXCLUDED."phone_number",
# "age" = EXCLUDED."age",
# "create_time" = EXCLUDED."create_time",
# "description" = EXCLUDED."description",
# "address" = EXCLUDED."address"
# """



# Configuration for automatically generating SQL.
#
# This option is mutually exclusive with the query parameter.
#
# Automatically generate INSERT SQL.
# If the target table does not exist,
# SeaTunnel can also automatically create the table.

generate_sink_sql = true



# database is required when generate_sink_sql=true

database = source_db



# table is required when generating SQL automatically

table = "public.t_8_100w_imp_st_ds_demo2_ql_cybjcj_wlsc"



# Generate SQL similar to:
#
# ON CONFLICT ("id")
# DO UPDATE SET ...

enable_upsert = true



# Unique key used to determine record uniqueness.
#
# This option enables automatic SQL generation
# for INSERT, DELETE, and UPDATE operations.

primary_keys = ["id"]




# Table schema handling strategy:
#
# ERROR_WHEN_SCHEMA_NOT_EXIST:
# Fail the job when the table does not exist.
#
# CREATE_SCHEMA_WHEN_NOT_EXIST:
# Create the table when it does not exist.
# Skip creation when the table already exists
# and preserve existing data.

schema_save_mode = "ERROR_WHEN_SCHEMA_NOT_EXIST"




# Data processing strategy:
#
# APPEND_DATA:
# Keep table structure and existing data,
# append new records.
#
# Usually used for incremental loading scenarios.
#
#
# DROP_DATA:
# Keep table structure,
# remove all existing data before writing.
#
# Implements full refresh loading
# by executing TRUNCATE first.
#
#
# CUSTOM_PROCESSING:
# User-defined processing.
#
# Must be used together with custom_sql.

data_save_mode = "APPEND_DATA"



# When data_save_mode is set to CUSTOM_PROCESSING,
# the CUSTOM_SQL parameter must be configured.
#
# This parameter usually contains executable SQL statements.
#
# The SQL will be executed before the synchronization task starts.
#
# It can implement:
# - Synchronization delete
# - Pre-update operations
# - TRUNCATE operations
# and other custom processing logic.

# custom_sql = ""




# Number of records written in each batch

batch_size = 10000



# Batch commit interval

batch_interval_ms = 500



# Retry attempts

max_retries = 3




# Connection parameters


# Connection timeout: 300 seconds

connection_check_timeout_sec = 300



# Additional JDBC parameters

properties = {


  # PostgreSQL-specific parameter

  # PostgreSQL batch insert optimization
  # (case-sensitive)

  reWriteBatchedInserts = "true"



  # Timezone configuration if required

  options = "-c timezone=Asia/Shanghai"

}

}


---

* Result

  * The automatically generated SQL can be seen in the logs:

text id="7b6z7w"
2025-12-17 13:53:35,566 INFO [.e.FieldNamedPreparedStatement] [st-multi-table-sink-writer-1] - PrepareStatement sql is:

INSERT INTO "source_db"."public"."t_8_100w_imp_st_ds_demo2_ql_cybjcj_wlsc"
("id", "user_name", "sex", "decimal_f", "phone_number", "age", "create_time", "description", "address")
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)

ON CONFLICT ("id")
DO UPDATE SET
"user_name"=EXCLUDED."user_name",
"sex"=EXCLUDED."sex",
"decimal_f"=EXCLUDED."decimal_f",
"phone_number"=EXCLUDED."phone_number",
"age"=EXCLUDED."age",
"create_time"=EXCLUDED."create_time",
"description"=EXCLUDED."description",
"address"=EXCLUDED."address"

2025-12-17 13:53:35,566 INFO [.e.FieldNamedPreparedStatement] [st-multi-table-sink-writer-1] - PrepareStatement sql is:

DELETE FROM "source_db"."public"."t_8_100w_imp_st_ds_demo2_ql_cybjcj_wlsc" WHERE "id" = ?

2025-12-17 13:54:46,130 INFO [s.c.s.s.c.ClientExecuteCommand] [main] -

       Job Statistic Information

Start Time : 2025-12-17 13:53:31

End Time : 2025-12-17 13:54:46

Total Time(s) : 74

Total Read Count : 1000001

Total Write Count : 1000001

Total Failed Count : 0





# Demo 3: Full Load + Automatic Table Creation + Schema-Free Data Migration



In many real-world data migration scenarios, the target database schema may not exist before the synchronization task starts.

Manually creating tables for dozens or hundreds of source tables can significantly increase operational overhead.

SeaTunnel provides automatic table creation capabilities through:

hocon
generate_sink_sql = true


Combined with:

hocon
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"


SeaTunnel can automatically:

* Detect whether the target table exists.
* Generate the corresponding DDL statement.
* Create the target table automatically.
* Start data synchronization after schema initialization.

This mode is commonly used for:

* Initial database migration.
* Cross-database platform migration.
* Rapid environment replication.
* Development and testing environment data synchronization.

- Core Principle

**Create schema automatically, then load data**

Execution workflow:

text
Source Database

↓

Read Table Metadata

↓

Generate Target Table Schema

↓

Create PostgreSQL Table Automatically

↓

Write Data


- Core Configuration

hocon
generate_sink_sql = true

schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"

data_save_mode = "APPEND_DATA"


- Execution Command

properties
sh /……/seatunnel-2.3.12/bin/seatunnel.sh --config /……/myconf/ds-st-demo3-mysql2pgsql-auto-create.conf -i JAVA_OPTS='-Xmx2g -Xms2g' -m local


- Source Table (MySQL)

The source table exists in MySQL:

sql
CREATE TABLE t_user_info (

id BIGINT PRIMARY KEY,

user_name VARCHAR(2000),

sex VARCHAR(20),

age INT,

create_time TIMESTAMP,

description TEXT,

address VARCHAR(2000)

);


The PostgreSQL target table does not need to be created manually.

- SeaTunnel Configuration

hocon
env {

# Parallelism
execution.parallelism = 5

# Batch processing mode
job.mode = "BATCH"

}

source {

jdbc {

url = "jdbc:mysql://ip:port/cs1"


driver = "com.mysql.cj.jdbc.Driver"


user = "root"


password = "zysoft"



query = "select * from t_user_info"



plugin_output = "mysql_source"



partition_column = "id"


partition_num = 5


fetch_size = 5000

}

sink {

jdbc {

url = "jdbc:postgresql://ip:port/source_db"


driver = "org.postgresql.Driver"


user = "postgres"


password = "123456"



# Automatically generate CREATE TABLE and INSERT statements

generate_sink_sql = true



database = source_db



table = "public.t_user_info"



plugin_input = "mysql_source"



# Automatically create table when it does not exist

schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"



# Keep existing data and append new records

data_save_mode = "APPEND_DATA"



batch_size = 5000


batch_interval_ms = 500


max_retries = 3



properties = {

  # PostgreSQL batch insert optimization

  reWriteBatchedInserts = "true"

}

}


- Execution Process

When the synchronization task starts, SeaTunnel first checks whether the target table exists.

If the table does not exist:

text
Checking target table...

↓

Table not found

↓

Generate CREATE TABLE statement

↓

Execute DDL

↓

Start data loading


Example generated SQL:

sql
CREATE TABLE "public"."t_user_info" (

"id" BIGINT,

"user_name" VARCHAR(2000),

"sex" VARCHAR(20),

"age" INTEGER,

"create_time" TIMESTAMP,

"description" TEXT,

"address" VARCHAR(2000)

);


After schema creation, SeaTunnel starts writing data into PostgreSQL.

- Execution Log

text
2025-12-15 17:05:21,112 INFO [DefaultSaveModeHandler]

Target table does not exist:

source_db.public.t_user_info

Creating schema automatically...

Execute SQL:

CREATE TABLE "public"."t_user_info" (...)

2025-12-15 17:05:22,875 INFO [JdbcSinkWriter]

Start writing records to PostgreSQL


- Job Statistics

text

       Job Statistic Information

Start Time : 2025-12-15 17:05:20

End Time : 2025-12-15 17:06:35

Total Time(s) : 75

Total Read Count : 1000004

Total Write Count : 1000004

Total Failed Count : 0


- Result Summary

With automatic schema creation enabled:

* No manual PostgreSQL table creation is required.
* SeaTunnel automatically generates the target schema.
* Data synchronization starts immediately after schema initialization.
* Migration efficiency is significantly improved for large-scale database replication scenarios.

Compared with manually preparing schemas in advance, this approach reduces operational complexity and makes SeaTunnel more suitable for automated migration pipelines.



# Demo 4: Full Load + Pre-SQL Execution + Target Table Initialization


In production data synchronization pipelines, it is common to perform some preparation operations before loading data into the target database.

For example:

* Clear temporary data before synchronization.
* Remove outdated records.
* Initialize target tables.
* Execute custom SQL statements before data loading.
* Prepare database environments for migration.

SeaTunnel JDBC Sink supports pre-execution SQL through:

hocon
pre_sql = [
"SQL statement"
]


This allows users to execute custom SQL commands before the synchronization task starts.

Typical production scenarios include:

* Daily full refresh of reporting tables.
* Data warehouse dimension table rebuilding.
* Batch migration with customized initialization logic.
* ETL pipelines requiring preprocessing steps.

- Core Principle

**Execute preparation SQL first, then write synchronized data**

Execution workflow:

text
Source Database

↓

Extract Data

↓

Execute Pre-SQL on Target Database

↓

Transform Data

↓

Write Data into Target Table


- Core Configuration

hocon
pre_sql = [
"TRUNCATE TABLE public.t_user_info"
]


Before writing new data, SeaTunnel executes:

sql
TRUNCATE TABLE public.t_user_info;


This ensures that the target table is clean before loading fresh data.

- Execution Command

properties
sh /……/seatunnel-2.3.12/bin/seatunnel.sh --config /……/myconf/ds-st-demo4-mysql2pgsql-presql.conf -i JAVA_OPTS='-Xmx2g -Xms2g' -m local


- Target Table

sql
CREATE TABLE "public"."t_user_info_presql" (

id BIGINT PRIMARY KEY,

user_name VARCHAR(2000),

sex VARCHAR(20),

decimal_f NUMERIC(32, 6),

phone_number VARCHAR(20),

age INT,

create_time TIMESTAMP,

description TEXT,

address VARCHAR(2000) DEFAULT 'Unknown'

);


- SeaTunnel Configuration

hocon
env {

# Number of parallel execution threads
execution.parallelism = 5

# Batch processing mode
job.mode = "BATCH"

}

source {

jdbc {

url = "jdbc:mysql://ip:port/cs1"

driver = "com.mysql.cj.jdbc.Driver"

user = "root"

password = "zysoft"


# Extract source data
query = "select * from t_8_100w"


plugin_output = "source_data"



# Enable parallel reading

partition_column = "id"

partition_num = 5

fetch_size = 5000

}

sink {

jdbc {

url = "jdbc:postgresql://ip:port/source_db"


driver = "org.postgresql.Driver"


user = "postgres"


password = "123456"



database = source_db


table = "public.t_user_info_presql"



plugin_input = "source_data"



generate_sink_sql = true



schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"



data_save_mode = "APPEND_DATA"



# SQL statements executed before data writing

pre_sql = [

  "TRUNCATE TABLE public.t_user_info_presql"

]



batch_size = 5000


batch_interval_ms = 500


max_retries = 3



properties = {

  # Enable PostgreSQL batch insert optimization

  reWriteBatchedInserts = "true"

}

}


- Execution Process

When the synchronization task starts, SeaTunnel executes the following sequence:

Step 1: Connect to PostgreSQL

text
Connecting to target database...

↓

Connection established


Step 2: Execute Pre-SQL

sql
TRUNCATE TABLE public.t_user_info_presql;


Execution log:

text
2025-12-15 18:10:25,321 INFO [JdbcSink]

Execute pre sql:

TRUNCATE TABLE public.t_user_info_presql


Step 3: Start Data Synchronization

text
MySQL

↓

SeaTunnel Pipeline

↓

PostgreSQL


- Execution Result

Before execution:

text
PostgreSQL Table

+----------------+
| Existing Data |
+----------------+

500,000 records


During execution:

text
Execute:

TRUNCATE TABLE

↓

Remove existing records

↓

Insert latest source data


After execution:

text
PostgreSQL Table

+----------------+
| Fresh Data |
+----------------+

1,000,004 records


- Job Statistics

text

       Job Statistic Information

Start Time : 2025-12-15 18:10:20

End Time : 2025-12-15 18:11:36

Total Time(s) : 76

Total Read Count : 1000004

Total Write Count : 1000004

Total Failed Count : 0


- Result Summary

Using `pre_sql`, SeaTunnel provides a flexible way to prepare the target database before synchronization:

* Custom SQL can be executed automatically before loading.
* Target tables can be initialized without external scripts.
* Complex ETL preparation logic can be integrated into a single synchronization pipeline.
* Operational steps are reduced and automation is improved.

Compared with manually running SQL scripts before each migration, the `pre_sql` capability makes data synchronization workflows more repeatable, reliable, and easier to maintain.



# Demo 5: Full Load + Data Transformation + Batch Performance Optimization


In real-world data integration scenarios, data synchronization is rarely just a simple copy operation.

Before loading data into the target database, organizations often need to perform additional processing, such as:

* Renaming fields.
* Converting data types.
* Masking sensitive information.
* Filtering invalid records.
* Filling default values.
* Optimizing batch write performance.

Apache SeaTunnel provides flexible transformation capabilities through:

* `FieldMapper`
* `Sql Transform`
* JDBC Sink batch optimization parameters

This demo demonstrates a complete production-style pipeline:

text
MySQL

↓

Extract Data

↓

Field Mapping

↓

Data Transformation

↓

Batch Optimization

↓

PostgreSQL


- Core Principle

**Extract → Transform → Optimize → Load**

Instead of directly copying source data, the pipeline performs necessary business transformations before writing to PostgreSQL.

- Core Configuration

hocon
transform {

FieldMapper {

}

Sql {

}

sink {

jdbc {

batch_size = 5000

batch_interval_ms = 500

properties {

  reWriteBatchedInserts = "true"

}

}


- Execution Command

properties
sh /……/seatunnel-2.3.12/bin/seatunnel.sh --config /……/myconf/ds-st-demo5-mysql2pgsql-transform.conf -i JAVA_OPTS='-Xmx2g -Xms2g' -m local


- Source Table (MySQL)

Example source table:

sql
CREATE TABLE t_user_transform (

id BIGINT PRIMARY KEY,

user_name VARCHAR(2000),

sex VARCHAR(20),

age VARCHAR(20),

phone_number VARCHAR(20),

create_time TIMESTAMP,

description TEXT,

address VARCHAR(2000)

);


Example source data:

text
+----+-----------+------+-----+--------------+
| id | user_name | sex | age | phone_number |
+----+-----------+------+-----+--------------+
| 1 | Alice | 1 | 28 | 13812341234 |
| 2 | Bob | 2 | 32 | 13956785678 |
+----+-----------+------+-----+--------------+


- Target Table (PostgreSQL)

sql
CREATE TABLE "public"."t_user_transform_result" (

id BIGINT PRIMARY KEY,

user_name VARCHAR(2000),

sex VARCHAR(20),

age INTEGER,

phone_number VARCHAR(20),

create_time TIMESTAMP,

description TEXT,

address VARCHAR(2000)

);




- Transformation Configuration

1. Field Mapping

The `FieldMapper` plugin is used to map source fields to target fields.

Example:

hocon
FieldMapper {

plugin_input = "source_data"

plugin_output = "mapped_data"


field_mapper = {

    id = id

    user_name = user_name

    sex = sex

    age = age_str

    phone_number = phone_number

    create_time = create_time

    description = description

    address = address

}

}


After mapping:

text
Source Field

age_str

↓

Target Field

age




2. Data Type Conversion

In many databases, numeric fields may be stored as strings.

For example:

Before:

text
age = "28"


After transformation:

text
age = 28


SeaTunnel SQL Transform:

hocon
Sql {

plugin_input = "mapped_data"

plugin_output = "transformed_data"


query = """

    SELECT

        id,

        user_name,

        sex,

        phone_number,

        CAST(age_str AS INTEGER) AS age,

        create_time,

        description,

        address

    FROM dual

"""

}




3. Phone Number Masking

Sensitive information should be protected before entering analytical systems.

Example:

Before:

text
13812341234


After:

text
138****1234


A masking rule can be implemented through SQL transformation logic:

sql
CONCAT(

SUBSTRING(phone_number,1,3),

'****',

SUBSTRING(phone_number,8,4)

)




4. Gender Conversion

Example business mapping:

text
1 → Male

2 → Female


Transformation example:

sql
CASE

WHEN sex = '1'

THEN 'Male'


WHEN sex = '2'

THEN 'Female'

END



5. Data Filtering

Only keep valid business records.

Example:

sql
WHERE age > 25


Records that do not meet the business requirements are filtered before loading.



6. Default Value Handling

For empty address values:

Before:

text
address = NULL


After:

text
address = Unknown


Example:

sql
COALESCE(address,'Unknown')




- Sink Configuration

hocon
sink {

jdbc {

url = "jdbc:postgresql://ip:port/source_db"


driver = "org.postgresql.Driver"


user = "postgres"


password = "123456"



database = source_db


table = "public.t_user_transform_result"



plugin_input = "transformed_data"



generate_sink_sql = true



schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"



data_save_mode = "APPEND_DATA"




# Batch write optimization

batch_size = 5000



# Commit interval

batch_interval_ms = 500



# Retry count

max_retries = 3




properties = {


  # PostgreSQL optimized batch insert

  reWriteBatchedInserts = "true"


}

}



- PostgreSQL Batch Write Optimization

When loading large datasets into PostgreSQL, single-row inserts can significantly reduce throughput.

SeaTunnel improves write performance through:

- batch_size

hocon
batch_size = 5000


Defines how many records are written in each batch.

Example:

Without batch optimization:

text
INSERT row 1

INSERT row 2

INSERT row 3

...

INSERT row 5000


With batch optimization:

text
INSERT batch

↓

5000 records



- batch_interval_ms

hocon
batch_interval_ms = 500


Controls the maximum waiting time before committing a batch.

This balances:

* Throughput
* Memory usage
* Transaction latency



- PostgreSQL reWriteBatchedInserts

Configuration:

hocon
reWriteBatchedInserts = "true"


This enables PostgreSQL JDBC driver batch rewriting.

Instead of:

sql
INSERT INTO table VALUES(?);

INSERT INTO table VALUES(?);


The driver can optimize into:

sql
INSERT INTO table VALUES(?),(?),(?);


This significantly improves bulk loading performance.



- Execution Result

During execution:

text
MySQL

1000004 records

↓

SeaTunnel Transformation

↓

PostgreSQL

1000004 records


Job statistics:

text

       Job Statistic Information

Start Time : 2025-12-15 19:20:10

End Time : 2025-12-15 19:21:27

Total Time(s) : 77

Total Read Count : 1000004

Total Write Count : 1000004

Total Failed Count : 0






- Result Summary

This demo demonstrates how Apache SeaTunnel handles a complete production-grade synchronization workflow:

* Extract data from MySQL.
* Apply field mapping and business transformations.
* Convert data types.
* Protect sensitive information.
* Filter invalid records.
* Optimize PostgreSQL batch loading.
* Write transformed data into the target database.

Compared with simple database replication tools, SeaTunnel provides a complete data pipeline framework that combines:

**Data Integration + Transformation + Performance Optimization**

within a single unified workflow.

# Final Summary: 5 MySQL → PostgreSQL Synchronization Patterns with SeaTunnel 2.3.12

Through these five demos, we covered the most common production synchronization scenarios:

| Scenario                      | Key Feature                                 | Typical Use Case      |
| ----------------------------- | ------------------------------------------- | --------------------- |
| Full Load + DROP_DATA         | Truncate and reload                         | Complete data refresh |
| Full Load + APPEND_DATA       | Preserve existing data                      | Data accumulation     |
| Auto Schema Creation          | Automatic DDL generation                    | Fast migration        |
| Pre-SQL Execution             | Custom initialization logic                 | ETL preparation       |
| Transformation + Optimization | Data processing and high throughput loading | Enterprise pipelines  |

Apache SeaTunnel 2.3.12 provides a flexible and scalable approach for heterogeneous data synchronization, helping engineering teams build reliable MySQL-to-PostgreSQL migration and integration pipelines with fewer operational steps.

The real #EnterpriseAI battle is for control, but the future is humans and agents shaping it together. 🤝 #FutureOfWork #AI

Apache SeaTunnel — Thu, 30 Jul 2026 07:38:36 +0000

Apache SeaTunnel

Jul 30

Build vs Buy vs Open Source: The Real TCO of Data Integration

#opensource #ai #programming #apacheseatunnel

9 min read

Build vs Buy vs Open Source: The Real TCO of Data Integration

Apache SeaTunnel — Thu, 30 Jul 2026 07:38:00 +0000

When companies build a modern data platform, data integration is usually one of the first capabilities they need to solve. Data needs to move continuously from operational databases, SaaS applications, message queues, data warehouses, and various business systems into analytical platforms.

Why Connector Maintenance Becomes the Hidden Cost of Data Integration

At the beginning, the problem appears straightforward: select a tool, configure a connector, and start synchronizing data.

However, after years of operating data platforms, many engineering teams discover that the initial connection is not the difficult part. The real challenge is maintaining those connections as the business, infrastructure, and data landscape continue to evolve.

A connector is often treated as a small technical component. In reality, a production-grade connector is a long-lived software system that needs continuous maintenance. Database versions change, APIs evolve, authentication mechanisms are updated, data models become more complex, and enterprises introduce new requirements around reliability, security, and scalability.

This is why the Total Cost of Ownership (TCO) of data integration is often underestimated.

The cost of a connector is not limited to the engineering effort required to build it for the first time. The larger cost comes from keeping it stable and production-ready over months and years.

A simple data pipeline may look like this:

But a production environment requires much more:

Every stage introduces engineering responsibility.

This is the reason many data engineering teams eventually face a common problem: the number of connectors grows faster than the team's ability to maintain them.

The first connector is easy.

Maintaining hundreds of connectors is the real challenge.

The Connector Problem: Why Data Integration Costs Grow Over Time

A common mistake when evaluating data integration solutions is focusing too much on the number of supported connectors.

A platform supporting hundreds of connectors appears attractive because it reduces initial development effort. However, the real engineering question is not only how many connectors exist today, but also how those connectors will be maintained tomorrow.

A mature connector needs to solve several technical problems:

How does it handle incremental synchronization?
How does it recover after failures?
How does it guarantee data consistency?
How does it manage schema changes?
How does it scale when data volume increases?
How does it adapt when the source system changes?

For example, a CDC connector that captures database changes is not simply reading records from a log stream.

A production CDC pipeline needs to manage multiple stages:

If a synchronization task fails after processing millions of records, the system needs to know exactly where it stopped and how to continue without data loss or duplication.

These capabilities are not unique to one connector.

They are common requirements across almost every enterprise data pipeline.

The problem with traditional connector development is that every team repeatedly solves the same infrastructure problems.

Build: Maximum Flexibility, Maximum Maintenance Responsibility

Building a data integration platform internally gives enterprises complete control.

Many large organizations initially choose this approach because their data environments are highly customized. They may have internal databases, proprietary systems, or specific compliance requirements that require deep customization.

The architecture often looks like this:

The advantage is clear: every component can be optimized for internal requirements.

The organization controls:

connector logic
deployment environment
security model
data processing rules
operational workflow

However, this control comes with a long-term cost.

When a company has five connectors, maintaining them may be manageable.

When the number grows to fifty or one hundred connectors, the situation changes dramatically.

The engineering team is no longer maintaining a data pipeline system.

It is maintaining a connector portfolio.

Each connector may require independent work for:

bug fixes
compatibility updates
performance tuning
monitoring improvements
operational troubleshooting

The organization eventually spends significant engineering resources maintaining infrastructure instead of building new data capabilities.

The fundamental issue is that every company independently rebuilds the same connector-related capabilities.

Buy: Faster Adoption, But Limited Control

Commercial data integration platforms solve the maintenance problem by moving connector responsibility to the vendor.

The architecture becomes:

The biggest advantage is speed.

A company can connect common data sources quickly without investing heavily in connector development.

For many organizations, this is a reasonable choice.

However, commercial solutions introduce different trade-offs.

The first issue is cost scalability.

As companies grow, they typically need:

more connectors
more frequent synchronization
larger data volumes
more environments

The operational cost grows accordingly.

The second issue is customization.

Enterprise environments are rarely identical. Teams often need special handling for:

internal applications
custom data formats
private infrastructure
specific transformation logic

A managed connector platform may not always provide the required flexibility.

The third issue is dependency.

Over time, pipelines become tightly coupled with the vendor's ecosystem, including:

metadata models
configuration formats
execution model
operational workflows

Migration becomes increasingly difficult.

Therefore, buying reduces initial engineering effort, but it does not eliminate the long-term architecture decisions.

Open Source: Moving From Individual Maintenance to Shared Engineering

Open source provides a different approach.

The value of open source is not simply reducing software licensing costs.

The more important value is engineering collaboration.

In a traditional model, companies often solve identical problems independently:

The same engineering effort is repeated many times.

Open source changes this model:

The community focuses on solving common problems.

Enterprises can focus on business-specific requirements.

However, successful open source data integration requires more than simply publishing connectors.

It requires a well-designed architecture that allows connectors to be developed, extended, and maintained efficiently.

This is where Apache SeaTunnel takes a different approach.

Apache SeaTunnel Architecture: Reducing Connector TCO Through Engineering Reuse

The biggest difference between a simple connector framework and an enterprise-grade data integration platform is not the number of connectors. It is whether the platform can separate data source-specific logic from distributed execution capabilities.

This separation determines whether connector development remains sustainable when the number of data sources increases.

Apache SeaTunnel addresses this challenge through its Connector V2 architecture.

The core idea behind Connector V2 is straightforward:

A connector should focus on how to communicate with a specific data source, while the data integration framework should provide common execution capabilities.

In other words, connectors should not repeatedly implement the same infrastructure features.

Traditional Connector Architecture: Repeating the Same Engineering Work

In many internally developed integration platforms, a connector is responsible for almost everything.

A typical connector implementation may contain:

At first, this design seems acceptable.

A developer builds one connector and controls the entire execution process.

However, as the number of connectors increases, the same infrastructure code is duplicated repeatedly.

For example:

A MySQL connector needs checkpoint management.

A PostgreSQL connector needs checkpoint management.

A Kafka connector needs checkpoint management.

A MongoDB connector needs checkpoint management.

The business logic is different, but the engineering problems are almost identical.

The result is a significant amount of duplicated code and duplicated maintenance effort.

The architecture becomes:

This is the hidden cost behind large-scale connector maintenance.

The problem is not writing connectors.

The problem is rebuilding the same infrastructure capabilities again and again.

SeaTunnel Connector V2: Separating Data Logic From Runtime Capability

SeaTunnel takes a different architectural approach.

Instead of allowing every connector to implement its own execution mechanism, SeaTunnel introduces a unified Connector V2 API and moves common capabilities into the runtime layer.

The architecture becomes:

Under this model, a connector mainly focuses on source-specific or sink-specific logic.

For example:

A MySQL CDC connector needs to understand:

how to read MySQL binlog
how to parse database changes
how to map database schemas

A Kafka connector needs to understand:

how to consume Kafka messages
how to handle offsets
how to serialize records

These are connector-specific responsibilities.

However, features such as:

task scheduling
state management
checkpoint coordination
failure recovery
parallel execution

are common capabilities.

They belong to the runtime.

This architectural separation is the key reason SeaTunnel can reduce connector maintenance cost.

Why Runtime Matters More Than Connector Count

Many discussions around data integration platforms focus on connector numbers.

However, connector quantity alone does not determine the strength of a platform.

The real engineering challenge is the runtime capability behind those connectors.

A connector without a reliable runtime is only a data reader or writer.

A production data integration system needs to answer much harder questions:

What happens when a task fails halfway?

How does the system resume from the correct position?

How does it prevent data duplication?

How does it handle increasing data volume?

How does it maintain consistency during schema changes?

These problems are not specific to MySQL, PostgreSQL, Kafka, or any individual data source.

They are platform-level problems.

SeaTunnel solves them by providing these capabilities at the runtime layer.

Checkpoint and Fault Tolerance: The Foundation of Reliable Data Synchronization

One of the most important capabilities in a production data pipeline is checkpoint management.

A simple batch synchronization task may restart from the beginning after failure.

However, large-scale data pipelines cannot work this way.

Consider a CDC pipeline:

Without checkpoint support, the system may:

lose data
duplicate data
require manual recovery

In enterprise environments, these failures are unacceptable.

A reliable synchronization framework needs to maintain:

current processing position
task state
execution metadata
recovery information

Traditionally, each connector developer needs to implement this logic independently.

SeaTunnel moves checkpoint and state management into the runtime architecture.

This means new connectors can inherit production-grade reliability capabilities instead of rebuilding them.

From a TCO perspective, this is extremely important.

The cost reduction does not come from writing fewer lines of connector code.

It comes from avoiding repeated implementation of complex distributed system capabilities.

CDC Scenario: Why Connector Design Matters

Change Data Capture (CDC) is one of the best examples of why connector architecture matters.

A CDC connector is not simply a continuous database reader.

A production CDC pipeline typically includes two phases:

Different databases have different CDC mechanisms.

For example:

MySQL uses binlog
PostgreSQL uses logical replication
Oracle uses redo logs

The source-specific implementation is different.

However, the operational requirements are the same.

Every CDC connector needs:

state management
checkpoint coordination
failure recovery
consistency guarantees

SeaTunnel's architecture allows these common capabilities to be reused.

Connector developers focus on implementing the database interaction layer.

The runtime provides the distributed execution foundation.

This significantly reduces the long-term maintenance burden.

Unified API: Making Connector Development Sustainable

Another important part of reducing TCO is standardization.

Without a unified API, every connector becomes an isolated project.

Different connectors may have:

different configuration styles
different lifecycle management
different error handling approaches
different operational behaviors

This creates complexity for both developers and users.

SeaTunnel Connector V2 provides a consistent abstraction model.

Developers work with a common framework instead of designing a new architecture for every connector.

This brings several advantages.

First, development becomes faster because engineers can reuse existing patterns.

Second, maintenance becomes easier because connectors follow consistent lifecycle rules.

Third, users receive a more predictable operational experience.

For enterprises operating many data pipelines, consistency itself is a major reduction in operational cost.

Connector Ecosystem: From Connector Ownership to Connector Collaboration

The long-term value of an open source connector ecosystem is not only the number of available integrations.

It is the ability to share maintenance responsibility.

Under the traditional model, each company pays the full cost independently; while a community-driven model changes this:

When improvements are made to common connector capabilities, the entire ecosystem benefits.

Enterprises still maintain control over their own business logic, but they avoid rebuilding fundamental infrastructure.

This is the core advantage of open source engineering models.

The Future Architecture of Data Integration

As data ecosystems become more complex, enterprises will continue to add more databases, applications, and analytical systems.

The number of required integrations will continue to grow.

The sustainable approach is not for every organization to maintain hundreds of independent connectors.

The future architecture will look more like:

In this model:

The community contributes reusable integration capabilities.

The platform provides reliable execution.

Enterprises extend the system according to their specific requirements.

This creates a more sustainable engineering model.

Conclusion: Reducing TCO Through Better Architecture

The choice between Build, Buy, and Open Source is ultimately a choice about how an organization manages engineering effort over time.

Building internally provides maximum control but requires continuous investment in connector maintenance.

Buying a commercial platform provides faster adoption but may introduce cost growth and platform dependency.

Open source provides another path: combining community innovation with enterprise control.

Apache SeaTunnel's value is not simply that it provides many connectors.

The deeper value is its architectural approach.

Through Connector V2, Unified API, and runtime-level capabilities such as checkpoint management, fault tolerance, and distributed execution, SeaTunnel allows organizations to reuse engineering capabilities instead of repeatedly rebuilding them.

In modern data infrastructure, the most important question is no longer:

"How many connectors can we build?"

The more important question is:

"How can we build a data integration architecture that remains maintainable as the number of connectors grows?"

That is where reducing TCO truly begins.

🚀 Join SeaTunnel Live Stream on Jul 28, 7PM UTC+8! Explore engine evolution & win exclusive swag 🎁 Reserve your seat now! 🌊 #ApacheSeaTunnel #DataEngineering #OpenSource #BigData #DataIntegration

Apache SeaTunnel — Thu, 23 Jul 2026 08:11:39 +0000

Apache SeaTunnel

Jul 23

Deep Dive into Apache SeaTunnel Engine: How FlushSignal Drives Architecture Evolution

#community #opensource #programming #apacheseatunnel

3 min read

Deep Dive into Apache SeaTunnel Engine: How FlushSignal Drives Architecture Evolution

Apache SeaTunnel — Thu, 23 Jul 2026 08:11:14 +0000

Have you ever noticed a seemingly simple yet critical parameter when using a JDBC Sink in your daily work — batch_interval_ms?

At first glance, it looks like just a Connector-level parameter that controls periodic data flushing. However, this small feature actually drove an important architectural evolution in Apache SeaTunnel Engine: moving from a localized Connector capability to a unified Engine-level FlushSignal control mechanism.

Why was this architectural change necessary? How are control signals designed and propagated inside the Engine? And how should responsibilities be divided between the Engine and Connectors?

If you are interested in Apache SeaTunnel Engine internals, architecture evolution, and source code implementation, this Meetup is one you won’t want to miss!

🎙 Topic of This Session

Deep Dive into Apache SeaTunnel Engine: How FlushSignal Drives Architecture Evolution

In this session, we will use the batch_interval_ms parameter in JDBC Sink as a starting point to explore a key evolution in Apache SeaTunnel Engine.

You will learn:

✅ Why Flush capabilities in Connectors needed to be elevated and centrally managed at the Engine level

✅ How FlushSignal was designed and integrated throughout the entire task lifecycle

✅ How the Engine’s threading model coordinates control signals with data streams

✅ How exceptions are propagated and how task lifecycles are managed

✅ How control signals travel accurately through the main execution pipeline and reach Sink operators

✅ How responsibilities are divided between Engine and Connector layers, and the architectural thinking behind this design

This is not only a source code deep dive, but also an exploration of layered architecture, system evolution, and real-world engineering practices.

👨‍💻 Speaker

Niu Zhiwei

GitHub ID: nzw921rx

Currently working on risk control systems, Niu focuses on data synchronization and data processing. He has experience with databases, big data technologies, data integration, and improving task reliability. He is also an Apache SeaTunnel Contributor.

If you want to better understand SeaTunnel Engine’s design philosophy and communicate directly with core contributors, this will be a valuable opportunity.

📅 Event Details

🗓 Date & Time

July 28, 2026 (Tuesday) | 19:00–20:30 (UTC+8)

📺 Register now and join the session！

https://meeting.tencent.com/dm/4bzAAZGmA84b

🎁 Get the swag

🎁 Join the livestream, engage with the community, and get a chance to win exclusive Apache SeaTunnel swag!

👥 Who Should Attend?

This session is especially suitable for:

Developers who want to gain a deeper understanding of Apache SeaTunnel Engine
Contributors working on Connector development
Engineers interested in data integration framework architecture
Open source enthusiasts who want to contribute to the Apache SeaTunnel community
Enterprise teams currently using SeaTunnel

Whether you are just starting to explore the source code or already contributing to Engine development, you will find valuable insights from this session.

🔥 Reserve Your Seat and Explore the Design Philosophy Behind SeaTunnel Engine!

A single parameter evolution reflects a broader architectural upgrade.

The design of FlushSignal represents a deeper rethinking of the boundaries between Engine and Connector responsibilities.

On July 28, join us as we dive into the internals of Apache SeaTunnel Engine and explore the engineering decisions, architectural principles, and design philosophy behind its evolution with core community contributors.

📌 Scan to reserve your seat!

See you in the livestream on July 28 at 19:00 (UTC+8)!🚀