DEV Community: Apache SeaTunnel

How Basic Auth Works in Apache SeaTunnel Zeta Engine — Fix 401 Unauthorized API Errors

Apache SeaTunnel — Fri, 26 Jun 2026 09:57:49 +0000

Recently while reviewing the REST API authentication logic of Apache SeaTunnel’s Zeta Engine, I ran into a very common issue:
The Zeta Engine fully started and its REST service listened on the designated port normally, yet accessing endpoints like /overview, /running-jobs and /job-info kept returning:

HTTP/1.1 401 Unauthorized

If you encounter this error for the first time, you may easily assume the service failed to launch, you entered the wrong port number, or the API path is incorrect.

In reality, this error is mostly tied to the Basic Auth configuration of SeaTunnel Zeta Engine.
Once Basic Auth is enabled on Zeta Engine, clients can no longer send plain requests to REST APIs as before — valid authentication credentials must be attached in the request header.

Starting from this 401 error case, this article breaks down how Basic Auth operates inside SeaTunnel Zeta Engine, plus the correct client-side connection methods.

1. The Symptom: Receiving 401 When Calling Zeta REST APIs

Suppose we send a direct request to the Zeta Engine REST API:

curl http://localhost:8080/overview

If Basic Auth stays disabled, this call will return the engine’s overview information without issues.

But after enabling Basic Auth in the config, requests missing authentication headers will trigger this response:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="SeaTunnel Web UI"

This indicates the request successfully reached Zeta Engine, yet got intercepted by the authentication filter before hitting the actual REST Servlet.

To simplify the root cause chain:

Zeta Engine running normally
REST API endpoint fully functional
Client request missing Authorization header
Request intercepted by BasicAuthFilter → returns 401 error

Basic Auth itself is a straightforward mechanism. Its core function is appending an authentication string to the HTTP Request Header:

Authorization: Basic base64(username:password)

For example, with username admin and password admin, clients first combine the credentials into the string:

admin:admin

Encode this string with Base64, then inject the encoded value into the Authorization header.

2. Source Code Deep Dive: How BasicAuthFilter Intercepts Requests

All core Basic Auth logic of SeaTunnel Zeta Engine resides in the BasicAuthFilter class.
This class implements the standard Java Servlet Filter interface:

public class BasicAuthFilter implements Filter {
    private final HttpConfig httpConfig;

    public BasicAuthFilter(HttpConfig httpConfig) {
        this.httpConfig = httpConfig;
    }
}

The core trait of Servlet Filters: every request passes through the filter before reaching the target Servlet.
This means the Basic Auth validation logic is centralized in a universal filter, rather than hardcoded into individual API endpoints.

The core business logic lives in the doFilter method.
First, it checks whether Basic Auth has been toggled on:

if (!httpConfig.isEnableBasicAuth()) {
    chain.doFilter(request, response);
    return;
}

This snippet is critical.
If Basic Auth remains disabled, the request gets passed through unobstructed via:

chain.doFilter(request, response);

In short: REST APIs require zero authentication when enable-basic-auth is turned off.

When Basic Auth is enabled, the code proceeds to extract values from the HTTP request header:

HttpServletRequest httpRequest = (HttpServletRequest) request;
HttpServletResponse httpResponse = (HttpServletResponse) response;

String authHeader = httpRequest.getHeader("Authorization");

Next, it verifies the header exists and follows the standard "Basic " prefix format:

if (authHeader != null && authHeader.startsWith("Basic ")) {
    String base64Credentials = authHeader.substring("Basic ".length());
    String credentials =
            new String(Base64.decodeBase64(base64Credentials), StandardCharsets.UTF_8);

    final String[] values = credentials.split(":", 2);
}

This block executes four sequential steps:

Extract the full Authorization header from the request
Strip the leading "Basic " prefix
Base64-decode the remaining string content
Split the decoded string on the colon symbol to separate username and password

Take this header as an example:

Authorization: Basic YWRtaW46YWRtaW4=

Decoding the Base64 segment yields the raw credential pair:

admin:admin

The code then compares the parsed credentials against the values defined in the engine configuration:

if (username.equals(httpConfig.getBasicAuthUsername())
        && password.equals(httpConfig.getBasicAuthPassword())) {
    chain.doFilter(request, response);
    return;
}

Requests matching the configured username and password pass through the filter successfully.

If the header is missing, malformed, or contains mismatched credentials, the filter returns a 401 error response:

httpResponse.setHeader("WWW-Authenticate", "Basic realm=\"SeaTunnel Web UI\"");
httpResponse.sendError(HttpServletResponse.SC_UNAUTHORIZED, "Unauthorized");

The full execution flow of BasicAuthFilter can be summarized as:

Client request arrives at Zeta REST service
        ↓
Check if Basic Auth is enabled in configuration
        ↓
Disabled: Forward request directly to target API
        ↓
Enabled: Extract the Authorization header from HTTP request
        ↓
Parse and Base64-decode credentials to retrieve username + password
        ↓
Compare parsed credentials against configured username & password
        ↓
Credentials match: Forward request to target API
Credentials mismatch / invalid header: Return 401 Unauthorized

3. Configuration Breakdown: How enable-basic-auth, username and password Take Effect

Three core configuration items govern Basic Auth functionality:
By default, the configuration values are set as follows:

enable-basic-auth = false
basic-auth-username = admin
basic-auth-password = admin

A key detail to note:
Even though basic-auth-username and basic-auth-password default to admin, these two fields do nothing unless enable-basic-auth is activated.

The single switch controlling authentication enforcement is:

enable-basic-auth

When set to false, all Zeta Engine REST APIs operate without authentication requirements.
When set to true, every request routed through BasicAuthFilter must carry valid authentication credentials.

A complete sample configuration block:

seatunnel {
  engine {
    http {
      enable-http = true
      port = 8080

      enable-basic-auth = true
      basic-auth-username = "admin"
      basic-auth-password = "admin"
    }
  }
}

After enabling Basic Auth, plain unauthenticated requests will fail:

curl http://localhost:8080/overview

Response:

401 Unauthorized

Two valid approaches to send authenticated requests:

Short-form curl with built-in credential flag

curl -u admin:admin http://localhost:8080/overview

Explicitly inject the Authorization header

curl \
  -H "Authorization: Basic YWRtaW46YWRtaW4=" \
  http://localhost:8080/overview

The string YWRtaW46YWRtaW4= is the Base64 encoding of the credential pair admin:admin.

4. Client Implementation: Connect via the Authorization Header

Once you grasp the server-side authentication workflow, the client-side requirements become clear:
If Zeta Engine has Basic Auth enabled, every REST API request sent by the client must include this header:

Authorization: Basic base64(username:password)

For Java client implementations, you can leverage Spring’s built-in HttpHeaders#setBasicAuth to auto-generate the header without manual Base64 encoding:

HttpHeaders headers = new HttpHeaders();
headers.setBasicAuth("admin", "admin", StandardCharsets.UTF_8);

HttpEntity<Void> entity = new HttpEntity<>(headers);

ResponseEntity<Map> response = restTemplate.exchange(
        "http://localhost:8080/overview",
        HttpMethod.GET,
        entity,
        Map.class
);

For reusable code, wrap the authentication logic into a dedicated helper method:

private void applyBasicAuth(HttpHeaders headers, String username, String password) {
    if (username == null || username.trim().isEmpty()) {
        return;
    }

    if (password == null || password.trim().isEmpty()) {
        return;
    }

    headers.setBasicAuth(
            username.trim(),
            password,
            StandardCharsets.UTF_8
    );
}

Invoke this utility method before every REST API call to attach authentication universally:

HttpHeaders headers = new HttpHeaders();
headers.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));

applyBasicAuth(headers, username, password);

This method works seamlessly for both GET endpoints:

restTemplate.exchange(
        "http://localhost:8080/running-jobs",
        HttpMethod.GET,
        new HttpEntity<Void>(null, headers),
        List.class
);

And POST submission endpoints:

restTemplate.exchange(
        "http://localhost:8080/submit-job",
        HttpMethod.POST,
        new HttpEntity<>(configText, headers),
        Map.class
);

The core takeaway for client integration with Basic Auth-enabled Zeta Engine:

No changes to REST API endpoints themselves
No adjustments to API request paths
Only mandatory addition: the Authorization request header

Summary

The Basic Auth implementation inside SeaTunnel Zeta Engine is lightweight, yet it frequently triggers confusing 401 errors for first-time users.

Core takeaways condensed into six key points:

enable-basic-auth defaults to false; REST APIs require no authentication when disabled
Once Basic Auth is toggled on, Zeta Engine intercepts all requests via the BasicAuthFilter
Clients must attach the Authorization: Basic [encoded-string] header to every request
The encoded segment is the Base64 output of the username:password credential pair
Default values for basic-auth-username and basic-auth-password are both admin
SeaTunnel Web delivers a visual interface to simplify authentication setup and engine connection workflows

When you encounter a 401 Unauthorized response, avoid jumping to conclusions that the Zeta Engine failed to start or that your port/path values are incorrect.
Prioritize verifying these three points first:

Whether enable-basic-auth is enabled in your engine configuration
Whether your client request carries a valid Authorization header
Whether the username and password match the values defined in the engine config

With this authentication workflow fully understood, troubleshooting and integrating with SeaTunnel Zeta Engine’s REST APIs becomes far more straightforward.

Debugging a Distributed Job Stuck in CANCELING in Apache SeaTunnel

Apache SeaTunnel — Fri, 26 Jun 2026 09:41:39 +0000

Recently, I worked on an issue in Apache SeaTunnel where a job could sometimes stay in the CANCELING state forever after a user requested cancellation.

At first, I thought this would be a simple bug in the cancel logic. Maybe there was a deadlock, an infinite retry loop, or a missing state transition somewhere.

But after tracing the code more carefully, I realized the issue was more subtle. It was related to master-worker communication, master failover, job state recovery timing, and how one exception was handled during task status notification.

This post is a short troubleshooting note about what I found and why I added a separate Force Stop mechanism instead of directly rewriting the existing Cancel logic.

Background

SeaTunnel’s Zeta engine runs jobs across a cluster.

The master node manages job-level state, and worker nodes execute task groups. When a user cancels a job, the master needs to notify the worker that is running the task. After the task finishes or is canceled, the worker reports its final task state back to the master.

A simplified flow looks like this:

User requests cancel
        ↓
Master sends cancel request to worker
        ↓
Worker cancels or finishes the task
        ↓
Worker reports final task state to master
        ↓
Master updates the job state

This looks simple, but in a distributed system, each step can race with node failure, membership changes, or master recovery.

The Symptom

The symptom was that a job stayed in CANCELING indefinitely.

The user had already requested cancellation, but the job never moved to a final state such as CANCELED.

At first, I mainly focused on the master-to-worker cancel request path.

First Suspicion: The Cancel Request Path

On the master side, SeaTunnel sends a CancelTaskOperation to the worker node.

One important part of the logic is that it checks whether the current execution address still exists in the cluster membership before sending the cancel operation:

while (!taskFuture.isDone()
        && nodeEngine
                .getClusterService()
                .getMember(executionAddress = getCurrentExecutionAddress())
            != null) {
    try {
        nodeEngine
                .getOperationService()
                .createInvocationBuilder(
                        Constant.SEATUNNEL_SERVICE_NAME,
                        new CancelTaskOperation(taskGroupLocation),
                        executionAddress)
                .invoke()
                .get();
        return;
    } catch (Exception e) {
        Thread.sleep(2000);
    }
}

This looked suspicious to me.

If the worker node temporarily disappears from the cluster view, for example because of a heartbeat issue, the loop may exit without sending the cancel request at all.

So my first thought was:

Maybe the master believes it has handled cancellation, but the worker never received the cancel request.

This path is still worth paying attention to. However, I later realized that it was not enough to fully explain why the job could remain in CANCELING forever.

Even if the cancel request is missed, the task may eventually finish and report its final state to the master.

So I started looking at the opposite direction:

What happens when the worker reports its final task state back to the master?

The More Important Path: Worker-to-Master Notification

When a task reaches a final state, the worker calls notifyTaskStatusToMaster.

The method is designed to retry until the notification succeeds:

while (isRunning && !notifyStateSuccess) {
    InvocationFuture<Object> invoke =
            nodeEngine
                    .getOperationService()
                    .createInvocationBuilder(
                            SeaTunnelServer.SERVICE_NAME,
                            new NotifyTaskStatusOperation(
                                    taskGroupLocation, taskExecutionState),
                            nodeEngine.getMasterAddress())
                    .invoke();
    try {
        invoke.get();
        notifyStateSuccess = true;
    } catch (JobNotFoundException e) {
        logger.warning("send notify task status failed because can't find job", e);
        notifyStateSuccess = true;
    } catch (ExecutionException e) {
        if (e.getCause() instanceof JobNotFoundException) {
            logger.warning("send notify task status failed because can't find job", e);
            notifyStateSuccess = true;
        } else {
            Thread.sleep(sleepTime);
        }
    }
}

The retry logic itself looked reasonable at first.

But the JobNotFoundException handling was important.

If the worker receives JobNotFoundException, it sets notifyStateSuccess = true and stops retrying.

In many cases, this behavior makes sense. If the job no longer exists on the master, it may mean the job has already finished and was removed from the running job map.

But during master failover, this assumption can become dangerous.

Where JobNotFoundException Comes From

On the master side, the task state update checks runningJobMasterMap:

public void updateTaskExecutionState(TaskExecutionState taskExecutionState) {
    TaskGroupLocation taskGroupLocation = taskExecutionState.getTaskGroupLocation();
    JobMaster runningJobMaster = runningJobMasterMap.get(taskGroupLocation.getJobId());
    if (runningJobMaster == null) {
        throw new JobNotFoundException(
                String.format("Job %s not running", taskGroupLocation.getJobId()));
    }
    runningJobMaster.updateTaskExecutionState(taskExecutionState);
}

Normally, runningJobMaster == null means the job is not running anymore.

However, there is another possible timing window.

During master failover, the new master may not have fully restored runningJobMasterMap yet. If a worker sends its final task status during that window, the new master can fail to find the corresponding JobMaster and throw JobNotFoundException.

Then the worker treats this exception as success and stops retrying.

The problematic sequence can look like this:

A job is being canceled.
A master switch happens.
A worker finishes its task and sends the final task state.
The new master has not fully restored runningJobMasterMap yet.
The master throws JobNotFoundException.
The worker treats this as success and stops retrying.
The master never receives the final task state after recovery.
The job remains in CANCELING.

This was the key point for me.

The issue was not just that the cancel RPC might fail. The more important problem was that the worker’s final status notification could be lost during master recovery.

Why I Added Force Stop Instead of Rewriting Cancel

After finding this path, I considered whether the existing Cancel logic should be changed directly.

There were several possible directions:

change how JobNotFoundException is handled,
wait until runningJobMasterMap recovery is completed,
store more running job state in distributed storage,
or redesign the cancellation state transition.

But the issue was intermittent and timing-dependent. Also, the normal Cancel path is a sensitive part of the execution lifecycle. A direct change there could introduce new behavior changes or performance overhead.

So I chose a more practical approach first.

Instead of changing the meaning of normal Cancel, I added a separate Force Stop mechanism.

The idea was simple:

If graceful cancellation cannot make progress, operators need an explicit way to finalize the job state.

Cancel vs Force Stop

I tried to keep the difference between Cancel and Force Stop clear.

Cancel

Cancel is for graceful termination.

It asks the running task to stop and depends on the normal task lifecycle and worker notification path.

Force Stop

Force Stop is for operational recovery.

It should not depend on whether the remote worker can still respond correctly. The master finalizes the job state and cleans up based on that decision.

In short:

Cancel     = try to stop the job gracefully
Force Stop = finalize the job when Cancel cannot make progress

Force Stop is not meant to replace Cancel. It is a fallback path for stuck cases.

What I Learned

This issue taught me that a stuck job state is not always caused by the code that directly updates that state.

In this case, the cancel request path looked suspicious at first. But the more important problem was on the worker-to-master notification path.

JobNotFoundException looked like a reasonable terminal condition in normal cases, but during master failover it could also mean:

The new master has not recovered the job yet.

Those two meanings are very different.

That small difference can decide whether the worker should stop retrying or keep trying.

For me, this was a good reminder that in distributed systems, exception handling is part of the state machine. It is not just error handling.

Conclusion

The stuck CANCELING issue was not simply a failed cancel request.

A worker could finish its task and try to notify the master, but if this happened during master failover before the new master fully restored its running job state, the master could throw JobNotFoundException. Because the worker treated this exception as a successful terminal condition, it stopped retrying. As a result, the master could miss the final task state and keep the job in CANCELING.

Force Stop was added as a practical recovery mechanism for this kind of situation.

It does not replace normal Cancel. Instead, it gives operators a way to finalize a job when graceful cancellation cannot make progress.

The biggest lesson for me was simple:

In distributed systems, the hard part is not only sending a request. The hard part is deciding what the system should believe when the request races with failure, recovery, or a delayed state.

Tutorial: Incremental MySQL-to-Doris Synchronization with Apache SeaTunnel and Apache DolphinScheduler

Apache SeaTunnel — Thu, 18 Jun 2026 07:30:14 +0000

Data synchronization is one of the most common requirements in enterprise data platform development. As business volume continues to grow, full data synchronization can place increasing pressure on source databases while consuming substantial computing and storage resources. As a result, incremental synchronization has become the preferred approach in most production environments.

In this demo, we will combine Apache SeaTunnel and Apache DolphinScheduler to implement a typical offline incremental synchronization scenario. DolphinScheduler retrieves the synchronization checkpoint from the target system and passes it to SeaTunnel as a runtime parameter, enabling incremental data synchronization from MySQL to Apache Doris.

This article is based on an actual demonstration and provides a complete walkthrough of the environment setup, SeaTunnel configuration, and DolphinScheduler workflow configuration process.

For the full demo, please refer to:

https://youtu.be/ObUaVOuoDC8?si=xNTPvsMTo6ALhi5b

1. Environment Setup

This demonstration uses the following components:

Component	Version
Apache SeaTunnel	2.3.9
Apache DolphinScheduler	3.x
MySQL	8.4
Apache Doris	2.x

In this architecture:

MySQL serves as the source database.
Doris serves as the target database.
SeaTunnel is responsible for data synchronization.
DolphinScheduler handles workflow orchestration and scheduling.

2. Preparing Test Data

Before configuring the synchronization task, we first prepare sample business data.

In this demonstration, a database named shopping is used as the sample database, and an orders table is created.

The orders table contains an auto-incrementing primary key column:

order_id

This field will later be used as the incremental synchronization checkpoint.

To verify synchronization results, a batch of sample records is inserted into the table. Approximately 300 order records are generated using a script.

The following information is then inspected:

Current total number of orders
Current maximum order ID

These values will serve as references when configuring incremental synchronization logic later.

It is worth noting that order_id is used only for demonstration purposes. In real-world production scenarios, timestamp fields such as update_time or create_time are often used as incremental synchronization conditions.

3. Incremental Synchronization Design

Before configuring SeaTunnel, let's first understand the overall synchronization strategy.

The core idea is to use data that has already been synchronized into Doris to determine the current synchronization progress.

The workflow operates as follows:

Query the current maximum order ID in Doris.
Use this value as the synchronization checkpoint.
SeaTunnel reads records from MySQL whose order IDs are greater than this checkpoint.
Newly added records are written into Doris.
During the next execution, synchronization resumes from the latest checkpoint.

For example, if the current maximum order ID in Doris is:

SeaTunnel will execute the following condition:

WHERE order_id > 300

This ensures that only newly inserted records are processed during each run, preventing duplicate synchronization of existing data.

As emphasized during the demonstration, the incremental field does not necessarily have to be a primary key. Any field that can accurately identify newly added or modified data can be used.

4. Configuring the SeaTunnel Job

After defining the synchronization strategy, we can start configuring the SeaTunnel job.

Configure the JDBC Source

Since the source data resides in MySQL, the JDBC Source is used to read the data.

The core query is:

SELECT *
FROM orders
WHERE order_id > ${order_id}

The most important part is:

${order_id}

This value is not hardcoded. Instead, it will be dynamically supplied by DolphinScheduler.

When the workflow runs, SeaTunnel automatically replaces this variable with the actual synchronization checkpoint, enabling incremental extraction.

Configure Parallelism

The demonstration also configures task parallelism:

parallelism = 4

Increasing parallelism can significantly improve synchronization performance.

In production environments, the appropriate value should be determined based on available server resources and database workload.

Configure Partitioned Reads

To improve performance when reading large tables, partitioned reading is also introduced.

The partition column is:

order_id

Configuration:

partition_column = "order_id"

Combined with the partition_num parameter, the dataset is divided into multiple partitions that can be processed in parallel.

This approach can greatly improve synchronization efficiency for large-scale datasets.

Configure Fetch Size

Within the JDBC Connector, fetch_size can be used to control the number of records retrieved from the database per fetch operation.

Proper configuration of this parameter can reduce database round trips and improve overall read performance.

5. Configuring the Doris Sink

After completing the Source configuration, the next step is to configure the Doris Sink.

Automatic Table Creation

The demonstration first introduces:

create_schema

This parameter enables automatic creation of target tables.

By leveraging automatic table creation, users can significantly reduce the effort required to manually maintain Doris table schemas.

Configure Write Mode

Since this example uses incremental synchronization, append mode is selected:

save_mode = APPEND_DATA

APPEND_DATA is used because each synchronization run only processes newly added records and does not need to overwrite historical data.

Enable Two-Phase Commit

To ensure data consistency, the demonstration also introduces:

enable_2pc = true

Enabling this option activates the two-phase commit mechanism, providing more reliable data writes.

It also helps guarantee Exactly-Once semantics during data synchronization.

Performance Optimization Parameters

Several performance-related parameters are also discussed, including:

batch_size

and

buffer_size

These parameters primarily control batch write behavior and can significantly improve Doris ingestion performance.

6. Configuring the DolphinScheduler Runtime Environment

After completing the SeaTunnel configuration, the next step is to set up DolphinScheduler.

Create a Tenant

First, navigate to the Security Center.

Open the Tenant Management page and create a new tenant.

The demonstration specifically emphasizes that all tasks in DolphinScheduler are ultimately executed under a tenant identity. Therefore, tenant configuration is an essential step in preparing the runtime environment.

Create a User and Associate It with the Tenant

Next, navigate to the User Management page.

Create a user and associate it with the tenant created in the previous step.

Once configured, the user will have permission to execute tasks under the corresponding tenant.

Create an Environment

Next, open the Environment Management page.

Create a runtime environment for SeaTunnel.

Configure the following environment variable:

SEATUNNEL_HOME=/soft/seatunnel

This configuration tells DolphinScheduler where SeaTunnel is installed.

When a workflow executes a SeaTunnel task, DolphinScheduler uses this path to locate the corresponding execution scripts.

The demonstration highlights that this configuration is mandatory and should not be skipped.

7. Creating the Project and Workflow

After completing the environment setup, create a new project.

In this demonstration, a project named:

shopping

is created.

After entering the project, create a new workflow.

The workflow contains two core nodes:

SQL Task
SeaTunnel Task

The SQL Task is responsible for retrieving the synchronization checkpoint, while the SeaTunnel Task performs the actual data synchronization.

8. Configuring the SQL Task to Retrieve the Synchronization Checkpoint

This is the most critical step in the entire solution.

First, create an SQL Task and select the Doris datasource.

The purpose of this task is to determine how far synchronization has progressed by querying the latest synchronized order ID.

The SQL statement is:

SELECT IFNULL(MAX(order_id),0) AS order_id
FROM orders;

The demonstration specifically explains why the following statement should not be used directly:

SELECT MAX(order_id)
FROM orders;

The reason is that during the initial synchronization, the Doris table may still be empty.

In this case:

MAX(order_id)

returns:

NULL

If NULL is passed directly to the downstream SeaTunnel task, it may generate an invalid query condition.

Therefore:

IFNULL(MAX(order_id),0)

is used to convert NULL values into 0.

This ensures that the initial synchronization starts correctly from the very first record.

Configure an OUT Parameter

The query result must be passed to downstream tasks.

To accomplish this, create a custom parameter within the SQL Task.

Select the parameter type:

OUT

Set the parameter name to:

order_id

The SQL query result will then be stored as a workflow variable.

The SeaTunnel task can subsequently reference this variable directly.

9. Incremental Synchronization Workflow Logic

Once the SQL Task is configured, the entire incremental synchronization pipeline is established.

When the workflow runs:

The SQL Task queries the current maximum order_id in Doris.
The result is stored as a workflow variable.
SeaTunnel uses ${order_id} as the query condition.
Newly added records are extracted from MySQL.

Through this approach, offline incremental synchronization based on a business primary key can be implemented efficiently and reliably.

10. Conclusion

This example demonstrates how to implement offline incremental synchronization by combining Apache DolphinScheduler and Apache SeaTunnel.

SeaTunnel handles data extraction and loading, while DolphinScheduler manages synchronization checkpoint retrieval, parameter passing, workflow orchestration, and scheduling.

The key idea behind this solution is querying the maximum order_id from the target Doris table through an SQL Task and passing the result to SeaTunnel through an OUT parameter. SeaTunnel then uses the checkpoint to perform incremental extraction from MySQL.

For data warehouse construction, ODS synchronization, and recurring offline synchronization scenarios, this solution offers several advantages:

Simple implementation
Easy maintenance
Strong extensibility
Production-ready incremental synchronization capabilities

As a result, it provides a practical and highly valuable reference architecture for enterprise data platform development.

What It Takes to Become an Apache SeaTunnel Committer: Doyeon Kim's Story

Apache SeaTunnel — Thu, 18 Jun 2026 02:52:39 +0000

Do you remember the young contributor from South Korea whom we featured last year?

In our previous interview, From Newcomer to Power Contributor: South Korea’s Doyeon Kim Shines in Apache SeaTunnel in Just Six Months, we shared the story of a passionate newcomer who had already become one of the most active contributors in the Apache SeaTunnel community within just a few months of joining.

Fast forward two years, and we're excited to share some wonderful news: Doyeon Kim has officially been invited to become an Apache SeaTunnel Committer! 🎉👏

What's even more inspiring is that Doyeon is still a university student.

Through her exceptional curiosity, relentless commitment to learning, and consistently high-quality contributions to both the project and the community, she has earned the recognition and trust of contributors worldwide. Her growth over the past two years has been remarkable, and her promotion to Committer reflects the impact she has had on the Apache SeaTunnel ecosystem.

To celebrate this milestone, we sat down with Doyeon for another in-depth conversation.

How has her perspective evolved since our first interview? What challenges did she overcome along the way? And what lessons has she learned on her journey from student contributor to Apache SeaTunnel Committer?

Join us as we look back on her inspiring open-source journey and discover the experiences that helped shape the next chapter of her growth.

Personal Introduction

My name is Doyeon Kim, and my GitHub ID is dybyte.

I am currently a university student. My main technical interests are backend development, data integration, distributed systems, and Apache SeaTunnel.

I am especially interested in how data integration systems work internally, including connectors, checkpointing, failure recovery, and engine state
management.

Outside of development, I enjoy playing games, especially Overwatch.

Full Interview Transcript

How long have you been involved in open source? What attracts you to open source?

I have been involved in open source for about one year.

What attracted me to open source was the opportunity to work together with many people on the same project. I found it very interesting and enjoyable that contributors from different backgrounds can discuss ideas, review code, and improve the project together.

I also like that open source communities are active and open. Through open source, I can learn not only from the code itself, but also from discussions, reviews, and the way experienced contributors think about problems.

When did you start contributing to the Apache SeaTunnel community, and what motivated you to get involved?

I started contributing to Apache SeaTunnel in May 2025.

At first, I was mainly studying backend development and wanted to gain experience
by contributing to a real open source project. When I found SeaTunnel, I felt that the community was active, welcoming, and open to new contributors, so I decided to start contributing.

As I continued participating, I encountered many interesting problems related to connectors, engine behavior, checkpointing, and reliability. Through these experiences, my technical interests gradually expanded from backend development to data integration systems and distributed systems.

Now that you have been elected as a SeaTunnel Committer, could you summarize your contributions to the community, including both code and non-code contributions? Please describe specific solutions or initiatives as much as possible.

My contributions to Apache SeaTunnel include code contributions, larger ongoing proposals, pull request reviews, technical discussions, documentation improvements, and community activities.

On the code side, I have mainly contributed to the Zeta engine, connectors, Transform-V2, tests, and documentation.

In the Zeta engine area, I worked on improving job state handling, metrics handling, REST API stability, and failure-related edge cases. For example:

PR #9833: [Improve][Zeta] Improve job metrics handling with partitioning support https://github.com/apache/seatunnel/pull/
PR #9926: [Improve][Zeta] Filter tasks and pipelines by state https://github.com/apache/seatunnel/pull/
PR #10132: [Fix][Zeta] Fix unnecessary job state update https://github.com/apache/seatunnel/pull/
PR #10315: [Fix][Zeta] Fix memory leak when cancelling pending job https://github.com/apache/seatunnel/pull/
PR #10456: [Fix][Zeta] Fix NPE when querying pending job info https://github.com/apache/seatunnel/pull/

These changes helped me better understand SeaTunnel’s engine behavior,
job lifecycle, metrics handling, and failure recovery paths.

I have also contributed to connector and Transform-V2-related improvements. For example:

PR #10222: [Fix][Connector-V2] Use upload session for insert https://github.com/apache/seatunnel/pull/
PR #10263: [Fix][Transform-V2] Fix multiTable SQL transform
https://github.com/apache/seatunnel/pull/
PR #10319: [Fix][Connector-V2] Fix partitioning column selection logic
https://github.com/apache/seatunnel/pull/
PR #10361: [Feature][Transform-V2] Add FieldEncrypt transform for
encrypting selected fields
https://github.com/apache/seatunnel/pull/
PR #10603: [Feature][Transform-V2] Support AES_GCM algorithm in
FieldEncrypt
https://github.com/apache/seatunnel/pull/

These contributions involved fixing the connector or transform behavior, improving correctness, and adding new transform capabilities.

In addition to merged contributions, I have also been working on larger ongoing proposals and feature work. These PRs are still open, but I have already addressed the main review feedback, and they are currently waiting for further review or merge. I do not consider them completed features yet, but they represent areas where I have been investing deeper effort.

PR #10485: [Feature][Connector] Add BigQuery Sink Connector https://github.com/apache/seatunnel/pull/

This PR introduces a BigQuery Sink connector based on the BigQuery Storage Write API. Through this work, I learned a lot about connector design, checkpoint integration, write semantics, and failure recovery.

PR #10399: [Feature][Zeta] Add compaction support to IMAP external storage https://github.com/apache/seatunnel/pull/

This PR explores compaction support for IMap external storage. It helped me think more deeply about storage design, recovery behavior, and how to reduce long-term maintenance costs.

PR #10812: [Feature][Zeta] Decouple Hazelcast IMap via StateStore Abstraction https://github.com/apache/seatunnel/pull/

This PR proposes a StateStore abstraction layer in the Zeta engine. The goal is to reduce direct dependency on Hazelcast IMap and make engine state management easier to maintain and evolve. It represents one of the deeper areas I have been working on recently.

PR #10133: [Feature][Zeta] Report non-terminal job states https://github.com/apache/seatunnel/pull/

This PR is related to improving job state reporting and observability in the Zeta engine.

I have also helped improve test stability and CI reliability through several smaller fixes. Although these changes are not the main focus of my contributions, I believe they are still important for maintaining contributor productivity and community trust.

On the non-code side, I have reviewed pull requests from other contributors and participated in technical discussions. My reviews have covered areas such as connectors, Zeta engine behavior, configuration validation, tests, documentation, and maintainability.

During reviews, I try to focus not only on whether the code works, but also on whether the change is safe, understandable, and maintainable for the project in the long term.

I have also shared my experience through technical writing and SeaTunnel meetup participation. I hope these activities can help more people understand SeaTunnel and become interested in contributing to the community.

After being involved in the SeaTunnel project and community for quite some time, you likely have a deep understanding of both. In your opinion, what differentiates SeaTunnel from other competing products/projects? What are its strengths and weaknesses? What aspects of the SeaTunnel community motivate you to stay actively involved?

In my opinion, SeaTunnel is especially useful when users need distributed data integration and need to connect many different sources and sinks.

One of SeaTunnel’s strengths is that it provides many connectors while also supporting distributed execution. This makes it useful for ETL and ELT scenarios where users need to move, synchronize, or transform data across different systems.

Another interesting point is that SeaTunnel has its own engine, Zeta, while
also supporting engines such as Flink and Spark. This gives users more flexibility depending on their use case.

At the same time, I think SeaTunnel is more focused on data integration than on being a full-featured stream processing engine. For very complex stream processing logic, such as advanced aggregations, windows, or event-time processing, engines like Flink or Spark may still be more suitable depending on the scenario.

The value of SeaTunnel may be especially clear in scenarios where users need to integrate many systems or process data at a larger scale, so it is important to evaluate it based on each company’s requirements.

What motivates me to stay involved is that SeaTunnel still has many meaningful problems to solve. The project is active, the community is open, and I feel that individual contributors can still have a real impact on the project.

Have you ever performed re-development or custom enhancements to address SeaTunnel’s shortcomings? If so, have these improvements been contributed back to the community? Could you introduce the solution/design approach?

Since I am currently a student and not working in a company, I do not have experience customizing SeaTunnel for a company’s internal production use case.

However, through community contributions, I have worked on improvements that address limitations or potential reliability issues in SeaTunnel.

Has your company used SeaTunnel? If yes, what are the use cases or scenarios? If not, would you consider recommending it to your company? What would be your reasons for recommending it?

Since I am currently a university student, I do not have a company use case to share.

However, if I were in a situation where a company needed a data integration platform, I would consider recommending SeaTunnel depending on the requirements.

For example, if the company needs to connect many different data sources and sinks and needs distributed execution for ETL or ELT workloads, SeaTunnel could be a good option.

Of course, I would also consider the trade-offs carefully. The decision should depend on the company’s data volume, reliability requirements, operational cost, and whether SeaTunnel’s connector ecosystem matches the company’s needs.

What kind of support or growth opportunities do you hope continued participation in the SeaTunnel community can provide for your personal development?

Through continued participation in the SeaTunnel community, I hope to grow both technically and personally.

Technically, I want to deepen my understanding of data integration systems,
distributed systems, checkpointing, failure recovery, connector design, and
engine architecture.

I also hope to improve my communication skills. By discussing technical topics with community members, reviewing pull requests, and receiving feedback from experienced contributors, I have been able to learn a lot.

In addition, I want to improve my English communication skills. For example, participating in events such as ApacheCon, listening to technical talks, and giving presentations would be very valuable experiences for me.

Overall, the SeaTunnel community gives me opportunities to grow not only as a developer but also as a person who can communicate, collaborate, and contribute in an international open-source community.

What is your understanding of the Committer role within the community? What responsibilities and impact should a Committer have?

In my understanding, a Committer can contribute to the community in many ways, not only by writing code.

A Committer can review pull requests, help maintain code quality, guide contributors, and help new contributors become more familiar with the project.

I also think a Committer should regularly think about what actions can have a positive impact on the community. This includes giving constructive feedback, helping discussions move forward, and making decisions that are good for the
long-term health of the project.

So I believe the Committer role is not just a permission to merge code. It is also a responsibility to help the project and the community grow in a healthy direction.

Now that you have been elected as a Committer, do you have any thoughts you would like to share with the community, or suggestions for the future development of the project?

I am very grateful to be elected as an Apache SeaTunnel Committer.

While participating in the community, I have gained many valuable experiences. I am happy that my passion and contributions could be helpful to the community.

I would like to thank everyone who reviewed my pull requests, answered my questions, and discussed technical topics with me. Their feedback and support helped me continue contributing to the project.

There were some difficult moments, but I think all of those experiences became part of my growth.

At the moment, I do not have a specific proposal for the future direction of the
project. However, if I find areas that could be improved, I will continue to share my thoughts through GitHub issues, pull requests, and community discussions.

What are your personal plans for contributing to and promoting the further development of the community and project in the near future?

In the near future, I would like to continue helping contributors so that they
can contribute to SeaTunnel more easily.

One way I can do this is through pull request reviews. I want to continue reviewing PRs, giving feedback, and helping contributors understand the project better.

I am also interested in helping more people in the Korean open-source community learn about SeaTunnel and contribute to the project. I think this could help both new contributors and the SeaTunnel community.

In addition, I plan to continue sharing my experiences through platforms such as LinkedIn and Medium. By writing about SeaTunnel, my contribution experience, and technical problems I have worked on, I hope more people can become interested in the project.

Goodbye Data, Hello AI: My Biggest Takeaway from Snowflake Summit 2026

Apache SeaTunnel — Thu, 11 Jun 2026 10:39:40 +0000

By William Guo, CEO of WhaleOps & Snowflake Ambassador

I would like to thank Snowflake for inviting me to attend Snowflake Summit as a Snowflake Ambassador.

This Summit had a much greater impact on me than I had expected.

As many of you know, I have spent my career in the data industry. I started at Teradata, then moved to IBM. Later, I was responsible for big data initiatives at enterprises such as Lenovo, CICC, and Wanda Group. After that, I became a Member of the Apache Software Foundation, and today I am the CEO of WhaleOps Open Source. Because of this background, I have always paid close attention to developments across the data industry.

Before coming to the Summit, I originally thought Snowflake would launch a number of enterprise AI products or add AI-related capabilities on top of its existing data warehouse and data platform offerings.

For many years, people's understanding of Snowflake has been quite clear: it is a cloud data warehouse company and a representative of the Data Cloud era. Its core strengths revolve around data storage, compute, performance, security, governance, sharing, and elastic scalability.

However, after spending two days at the Summit, my impression changed completely.

The biggest takeaway I had from this year's Snowflake Summit was not that it had released some new data platform features. Rather, it is aggressively reconstructing its product positioning.

In my view, Snowflake is no longer satisfied with being defined as a Data Warehouse company. Nor does it simply want to become an AI Data Cloud. Instead, it aims to transform itself into an enterprise AI + Data platform, and perhaps even the foundation of the Agentic Enterprise, putting itself on a path that increasingly overlaps with companies like Anthropic.

If I had to summarize my personal impression of this Snowflake Summit in one sentence, it would be:

Goodbye Data, Hello AI.

Of course, "Goodbye Data" does not mean data is becoming less important.

On the contrary, data has become even more important.

What has changed is the way data platforms are expressed and understood.

In the past, when we talked about data platforms, we talked about how data should be stored, processed, shared, governed, and optimized for cost efficiency.

Today, Snowflake is talking about how AI can understand enterprise data, how Agents can use enterprise data, how business users can gain insights directly through natural language, and how enterprises can enable AI to execute tasks within secure and governed boundaries.

Snowflake Product VP Christian Kleinerman made a statement during the Platform Keynote that perfectly captures this shift:

Your AI-native enterprise starts here.

If this sentence had appeared at a typical AI conference, it might have sounded like a standard marketing slogan.

But in the context of Snowflake Summit, it carries a very different meaning.

Because Snowflake is not an AI-native company. Historically, it has been a data infrastructure company. When a company with that background begins reorganizing its entire product portfolio around AI, it signals that AI is no longer an add-on feature—it is becoming a force that reshapes the enterprise itself.

1 Snowflake's Transformation: From Data Warehouse to AI Platform

In the past, when I thought about Snowflake, the first things that came to mind were data warehousing, cloud-native architecture, elastic computing, the separation of storage and compute, data sharing, and unified governance.

What Snowflake solved were several long-standing problems in traditional data platforms: fragmented data, limited scalability, complex performance tuning, inconsistent governance, and high collaboration costs.

The central narrative of this year's Summit was clearly different.

Snowflake still talks about All Data, All Workloads, and All Users. It still talks about structured, semi-structured, and unstructured data. It still talks about Iceberg, OpenFlow, Streaming, Zero Copy, and Horizon Catalog.

However, these capabilities are no longer being positioned simply as components of a better data platform. Instead, they are being framed as the foundation for a new goal: enabling enterprise AI and Agents to operate on a unified data platform.

Christian Kleinerman also made another highly important statement during the Platform Keynote:

"We need a unified architecture, both AI and data."

This statement can almost be regarded as the strategic core of this year's Snowflake Summit. It is not saying, "We support AI too." Rather, it is saying that enterprises should not build a separate AI platform outside of their data platform.

Why?

Because if the AI platform and the data platform are separated, many of the same problems we experienced during the data era will reappear: new silos, new permission systems, new governance gaps, new cost black holes, and new security risks.

We spent more than a decade eliminating data silos. If we build AI on an entirely separate stack today, we are essentially creating AI silos all over again.

So Snowflake's answer is clear: AI and Data must be unified. Data, compute, semantics, governance, security, applications, and Agents should all form a closed loop within a single platform.

Viewed from this perspective, Snowflake's Summit slogan, Make AI Real for Business, is fundamentally about turning Data into the context, fuel, and execution foundation for AI.

In the past, data platforms were built for people. People wrote SQL, viewed dashboards, configured jobs, and performed analyses.

In the future, data platforms will increasingly be built for Agents. Agents will understand business questions, invoke data capabilities, generate analytical workflows, propose actions, and even participate directly in business processes.

This is what truly struck me at this Summit.

Snowflake is not simply adding an AI assistant on top of an existing Data Warehouse. It is using data to rebuild a new AI-native foundation for the Agentic Enterprise, which is also where OpenAI and Anthropic will ultimately compete.

That is why I believe Snowflake's transformation is far more aggressive than I originally imagined.

2 CoCo, CoWork, and Desktop: Snowflake Is "Paying Tribute" to Anthropic—And Revealing a Bigger Ambition

If the first layer of change is strategic positioning, then the second layer is the product portfolio itself.

At this year's Snowflake Summit, what impressed me most was not a traditional database feature or a performance improvement metric. Instead, it was the launch of an entire collection of AI Agent-centric products and components:

CoCo
CoWork
Desktop
Skill Catalog
VS Code Extension
Excel Add-in
MCP
ACP
Cloud Agents
Agent Teams
Automated Agents

When viewed together, they send a very clear signal:

Snowflake is reorganizing its product strategy in the same way an AI-native company would.

In fact, I would even say it is "paying tribute" to Anthropic.

Why do I say that?

Because AI-native companies such as Anthropic are no longer just building chatbots. They are building complete AI work systems, including Claude, Claude Code, Desktop, MCP, Artifacts, Skills, Computer Use, enterprise context, and security boundaries.

What they truly want to own is not merely a conversational interface, but the primary interface through which humans collaborate with software in the future.

The CoCo, CoWork, Desktop, Skill Catalog, and MCP/ACP announcements from Snowflake have remarkably strong parallels.

CoCo feels like Claude Code for the enterprise.

CoWork resembles an AI workspace for business users.

CoCo Desktop extends Snowflake's AI capabilities beyond the web console and into users' everyday work environments.

Skill Catalog packages Snowflake platform capabilities into discoverable, composable, and reusable skills that Agents can invoke.

So when I heard these announcements at the event, my first reaction was not:

"Snowflake has released a few more AI features."

Instead, it was:

Snowflake wants to repackage the data platform as a complete Enterprise AI Agent Operating System and enter the same strategic battleground occupied by OpenAI Enterprise and Anthropic Enterprise.

Snowflake officially announced that Cortex Code would no longer be called Cortex Code. It has been renamed to Snowflake CoCo:

"From here on, no more Cortex Code. It is officially Snowflake CoCo."

This statement is worth paying attention to.

The name Cortex Code still carried the feeling of being a coding assistant.

CoCo, on the other hand, feels much more like a standalone AI product.

Behind this rebranding is a larger ambition.

Snowflake does not want CoCo to be merely an assistant that helps users write SQL, generate code, or explain syntax. It wants CoCo to become the AI operating interface for the entire Snowflake platform.

Christian also mentioned during the keynote that over the past several months, CoCo has evolved beyond CLI and SnowSight experiences and expanded into MCP, ACP, SDKs, Agent Teams, Cloud Agents, automation capabilities, and Skill Catalog.

Among these, Skill Catalog is especially important. It enables users to share, discover, and reuse Skills. In essence, it is modularizing Snowflake platform capabilities and turning them into reusable tools for Agents.

This is extremely important.

Snowflake also explicitly announced upcoming Excel add-ins, VS Code extensions, and Marketplace partner integrations for CoCo.

During discussions at the event, many of us felt the Excel integration was particularly powerful because Excel remains the most familiar data workspace for business users.

VS Code, meanwhile, remains the most familiar workspace for developers.

Rather than forcing everyone into SnowSight, Snowflake is bringing CoCo directly into the environments where people already work.

This is also one of the most important principles behind AI-native products:

Do not force users to move into your interface. Bring your Agent into the user's workflow.

Therefore, the significance of CoCo is not that Snowflake now has its own Copilot.

The significance is that Snowflake is moving away from a traditional platform UI and toward an Agent Everywhere strategy.

Beyond CoCo, Snowflake also placed a major spotlight on CoWork at this Summit.

To be honest, when I first heard about CoWork, I was a bit puzzled. If Anthropic were launching CoWork, I could easily understand it, because Agents naturally require enterprise-grade collaboration. But from the perspective of a traditional data platform, CoWork did not seem like the kind of product Snowflake would be expected to release. CoCo helping data engineers write SQL, fix pipelines, and build applications makes perfect sense. OpenFlow, Streaming, Iceberg, and Horizon Catalog are also clear enhancements to the data platform. But what does CoWork have to do with a data warehouse?

After listening to the presentation, I gradually understood it. CoWork reveals Snowflake’s ambitions even more clearly. It is designed for business users, with the vision of enabling CEOs, sales teams, operations teams, marketers, and other business professionals to interact directly with enterprise data and gain insights as if they had their own personal Jarvis. Samsung shared a use case illustrating this idea: CoCo serves as the AI operating interface for data engineers and developers, while CoWork serves as the AI workspace for business users. Snowflake is not just trying to serve data teams; it wants to become part of the daily workflow of every business user across the enterprise.

At that point, I finally understood CoWork’s role. CoCo is reshaping back-end data engineering, while CoWork is reshaping front-end business decision-making. Together, they enable Snowflake to evolve from a data platform into an enterprise AI work platform. CoWork may seem far removed from the traditional Snowflake, but in reality, it is perhaps the closest thing to Snowflake’s future.

Building Agentic Enterprise Infrastructure—this is Snowflake’s true ambition. It also explains why I no longer see Snowflake as a traditional data company.

When traditional data companies launch products, they typically talk about performance improvements, cost reductions, additional connectors, or stronger governance capabilities.

Snowflake’s announcements this time felt much more like those of an AI company. It talked about Agents, Skills, Desktop, CoWork, natural language, business users, context, and security boundaries. In other words, Snowflake is repositioning itself from a Data Warehouse company into an Enterprise AI Platform company.

This should serve as a wake-up call for every data software company.

If even Snowflake has realized that the entry point to future data platforms will shift from SQL, BI, notebooks, and pipelines toward Agents, Skills, Context, and Workflows, then companies like ours that focus on ETL, DataOps, Data Ingestion, and Orchestration must also rethink what our products should look like.

This is not about a single product. It is a blueprint for how Snowflake is reorganizing its entire product portfolio for the AI era. CoCo, CoWork, Desktop, Skill Catalog, and MCP/ACP together reveal Snowflake’s new ambition: not just to manage data, but to become the entry point for enterprise AI.

3 AI Is Bringing Every Software Company Back to the Same Starting Line

The second major impression I took away from this Summit is that we are all part of the same ecosystem, and AI is bringing every software company back to the same starting line.

There was one moment that left a particularly deep impression on me.

Snowflake announced the Agentic Control Plane, or ACP. At that moment, I was genuinely shocked, because just last month we launched our own ACP product. Wait a minute—isn’t that a direct collision? If a giant like Snowflake is entering this space, am I finished!?

As I listened more carefully, I realized the two products are not exactly the same. Snowflake’s ACP focuses more on Snowflake-native data modeling, Text-to-SQL, semantic layers, and enabling Agents to understand and interact with Snowflake data. What we focus on is ETL, orchestration, pipelines, data synchronization, job scheduling, and the execution and governance of heterogeneous data systems. In fact, I quickly added the words “Data Engineering” in front of our product name before calling it an Agent Control Plane.

But the key point is not whether the two products are identical. The important thing is what this coincidence reveals: everyone is moving toward the same destination.

That destination is:

Future software systems must become systems that Agents can understand, invoke, orchestrate, and govern.

In the past, the differences between software companies came from many factors: brand recognition, customer base, sales channels, engineering scale, ecosystem strength, delivery capability, and product maturity. Large enterprises had their advantages, while startups faced their own challenges. But with the arrival of AI, something fascinating has happened: every software product now needs to be rebuilt for AI.

In the past, software interaction looked like this: people opened interfaces, clicked buttons, filled out forms, wrote SQL, reviewed logs, and handled exceptions.

In the future, software interaction may look like this: people define objectives, Agents understand context, invoke tools, generate plans, execute tasks, and return results. Humans increasingly take on the roles of validation, supervision, judgment, decision-making, and correction.

One of my strongest impressions from the Summit was this: when it comes to AI, every software company has been brought back to a new starting line.

Because in the AI era, every software product must be rebuilt from the ground up.

That is why Snowflake and we ended up launching similar categories of products at almost the same time. In the past, such a thing would have been difficult to imagine. Startups rarely released the same kind of products in parallel with large enterprises because the major players usually leveraged their vast resources to build everything first.

Today, however, this creates an enormous opportunity for startups.

In the past, competing directly with large enterprises on resources, branding, or customer scale was extremely difficult. But when AI begins to redefine software, large enterprises also carry historical baggage: legacy systems, legacy customers, legacy architectures, and legacy organizational processes. Startups, if they move quickly enough, can design products from day one with an Agent-native mindset.

That is what encouraged me most about attending Snowflake Summit. The kinds of products Snowflake announced this month are remarkably similar to some of the directions we ourselves explored last month. The scale, scenarios, and depth may differ, but it suggests that our understanding of where the industry is heading is remarkably aligned.

In the AI era, opportunities do not belong only to large enterprises. They also belong to entrepreneurs who can quickly recognize change and are willing to reinvent their products.

4 Looking Back at Ourselves Through Snowflake: How Do I Goodbye Data, Hello AI?

The biggest question Snowflake Summit left me with was not what Snowflake will become, but what we ourselves should become.

If Snowflake is already embracing Make AI Real for Business, then how should we, in turn, embrace Goodbye Data, Hello AI?

For years, we have worked in DataOps, ETL, Data Ingestion, Orchestration, and Pipelines. At their core, these disciplines are about managing the flow of data. We help customers move data from one system to another, schedule tasks based on dependencies, monitor failed jobs, stabilize data pipelines, and connect heterogeneous systems.

All of those things remain important. But once the AI era arrives, software itself is no longer the end goal.

In the past, we dealt with structured data, semi-structured data, files, logs, tables, fields, tasks, and workflows. In the future, we will also need to handle Knowledge, Context, Semantics, Business Rules, Lineage, Execution Memory, and Agent Actions.

When data is no longer just rows and columns inside tables, and no longer merely moving from a source to a destination, it becomes something far more important. Data becomes the context through which AI understands a business. It becomes the foundation upon which Agents take action. It becomes the fuel that powers enterprise automation.

Snowflake’s answer is to evolve from a Data Warehouse into an AI Data Platform.

Our answer is to evolve from a DataOps tool into a Data Engineering Harness for the AI era.

When people use Claude Code or Codex today, they are primarily working in Java or Python development environments. But the environment of a Data Engineer is fundamentally different. The business semantics are more complex, and the workflows are more complicated.

Snowflake’s CoCo is essentially a Data Warehouse Agent. But orchestration and data ingestion are not Snowflake’s core strengths. What data engineers truly need is a Data Engineering Harness that spans systems, databases, schedulers, and environments. In practice, this manifests as an Agentic Data Control Plane designed specifically for data engineers.

That may very well be the opportunity for WhaleOps.

There was a statement from Thomson Reuters at Snowflake Summit that left a deep impression on me:

“They can’t be wrong.”

The quote referred to professionals in legal, tax, and audit industries, where errors are simply unacceptable. The same principle applies to data engineering. That is why a Data Engineering Harness is inherently more complex than harnesses in many other domains.

Enterprise AI is not a toy.

The data tasks generated by Agents cannot merely look correct—they must actually be correct.

The analyses produced by Agents cannot simply sound convincing—they must be grounded in trustworthy data.

The data workflows executed by Agents cannot merely be automated—they must be governable, auditable, and reversible.

That is why I believe this represents our opportunity in the AI era—not simply to become smarter, but to become more trustworthy.

5 My Final Prediction: Snowflake Is Competing for the AI Entry Point, and If It Succeeds, Its Stock Could Rise Far Beyond 2×

Looking back on this Snowflake Summit, my biggest takeaway was not any single product announcement, but a much larger signal about the software industry:

AI is reshaping the entry points, forms, and value propositions of all software.

Snowflake is competing for the AI entry point. That is why it has set its sights on Anthropic as a competitor and is evolving from a Data Warehouse into an AI Data Platform.

In the future world of Data + AI, who will control the entry point—the owners of data, or the owners of AI?

My own view is that data is difficult to move, while AI platforms are relatively easy to switch.

As many people know, Snowflake’s stock price has roughly doubled over the past month. Personally, I believe that if Snowflake’s AI-entry-point strategy succeeds, its future upside will be far greater than the 2× gain we have seen over the past month.

Rethink what is possible,

The future of Snowflake AI looks incredibly promising.

(The views shared above are solely my personal opinions, do not represent any official position, and should not be considered investment advice.)

Apache SeaTunnel Monthly Engineering Digest (May 2026): 87 PRs, Stronger Connectors, Smarter Recovery

Apache SeaTunnel — Thu, 11 Jun 2026 10:22:25 +0000

Hello SeaTunnel community!

The Apache SeaTunnel May 2026 Monthly Report has finally arrived.

According to community statistics, a total of 87 pull requests were merged into the apache/seatunnel repository during May 2026. This month’s primary focus was on continuously improving Connector-V2, closing functional gaps, and making connectors truly production-ready. Significant efforts were also invested in the Zeta Engine, with enhancements across high availability, fault recovery, observability, and testing. At the same time, the community strengthened CI security and regression testing to ensure efficient and reliable development on the main branch.

This report includes:

A comprehensive review of all merged PRs, covering new features, performance improvements, bug fixes, and architectural enhancements (including the complete PR list).
In-depth analysis of key technical changes, their implementation details, and impact scope (with patch-level code snippets).
Reproducible methodologies for performance and stability validation (without fabricated benchmark results).
Insights into project evolution trends and future technical directions.
A complete list of all contributors who submitted merged PRs in May 2026, including GitHub usernames, contribution categories, and rankings.

1. Overall Snapshot

1.1 Four-Dimensional Statistics (87 PRs)

The data clearly shows that May was a month focused on making existing capabilities robust and production-ready. A large portion of the merged work addressed real-world operational challenges, including HA, recovery mechanisms, edge cases, resource and memory risks, and observability.

1.2 Module Distribution

seatunnel-connectors-v2: 32
seatunnel-engine: 21
seatunnel-connectors-v2/connector-cdc: 8
seatunnel-e2e: 6
docs: 3
other: 17

2. Feature Enhancements and Engineering Evolution

2.1 Connector-V2: HTTP Source Now Supports Binary Downloads (#10956)

This is one of the most user-visible connector enhancements introduced this month. HTTP Source has evolved from simply fetching JSON and text payloads to supporting full file and binary downloads.

The PR patch summary confirms the scope of the change (15 files changed, +758/-17):

Added format = "binary"
Added binary_chunk_size (default: 10 MB) for large-file chunking
Introduced a fixed output schema: (data: bytes, relativePath: string, partIndex: long)
Limited to BATCH mode
Added comprehensive unit tests, E2E tests, and documentation in both Chinese and English

Official example:

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Http {
    url = "http://example.com/files/report.pdf"
    method = "GET"
    format = "binary"
    binary_chunk_size = 10485760
    schema = {
      fields {
        data = bytes
        relativePath = string
        partIndex = long
      }
    }
  }
}

sink {
  LocalFile {
    path = "/tmp/download"
    file_format = "binary"
  }
}

Impact and upgrade recommendations:

Existing jobs remain unaffected as long as format=text continues to be used.
After upgrading, users should validate that downstream transforms and sinks properly handle bytes fields.
For large files, binary_chunk_size reduces peak memory usage from file-size scale to chunk-size scale, significantly lowering OOM risk. This can be validated through RSS and GC metrics.

2.2 Engine/Zeta: Progressive Validation for Dry-Run Mode (#10763)

[Feature][Zeta] Implement proper dry-run mode with progressive validation layer0

Engineering value:

Configuration errors are detected before job submission or startup rather than during runtime.
Provides a foundation for future progressive validation layers, including connector parameter validation, schema constraint verification, and permission/resource prechecks.

2.3 Ecosystem Expansion and Tooling Improvements

This month also saw progress in control-plane capabilities, observability, edge collection, and developer tooling:

#10878: STIP-24 Phase 1 EdgeSocket ingress (Edge Collection MVP)
#10491: Data lineage and performance tracing
#10184: Python SDK Client for SeaTunnel REST API
Multi-table and schema evolution support for RabbitMQ, Cassandra, SQL Server CDC, Postgres CDC, and more

3. Performance Optimization and Resource Risk Mitigation

3.1 Kudu: Dependency Upgrade Resolves Flink 1.15+ Compatibility Risks (#10974)

This PR primarily improves compatibility and runtime reliability, although it was categorized as a performance improvement.

Implementation details:

Upgraded kudu-client to reduce classpath conflicts with Flink 1.15+.
Removed explicit dependencies on Kudu’s shaded Guava libraries to avoid runtime issues caused by internal shaded API changes.

Suggested reproducible validation:

Run identical Kudu jobs on Flink 1.15+ and compare:

Job startup success rate
Startup latency percentiles (P50/P95)
Stability during 30-minute execution windows
Presence of slot allocation or release loops

3.2 Kafka: Reduce Default Cache Queue Size from 1024 to 2 to Prevent OOM (#10954)

The PR title explicitly states:

Reduce default reader_cache_queue_size from 1024 to 2 to prevent OOM

Quantitative interpretation:

Queue depth reduced by 512x (1024 → 2).
For workloads with large cached records (such as deserialized rows), this significantly reduces peak memory usage and GC pressure.

Recommended metrics:

Peak RSS memory
GC frequency
Young/Old GC duration
OOM occurrence rate
Throughput (rows/sec)

4. Bug Fixes and Architectural Improvements Defined May

4.1 Zeta: Master Failover Could Permanently Stall Jobs (#10836)

The PR title already highlights the affected scenarios:

affects BATCH / bounded source / job shutdown phase

Potential production impact:

Jobs that should terminate may remain indefinitely stuck.
Increased risk of resource leakage.
Potential duplicate scheduling.
Higher risk of inconsistent downstream results.

Recommended validation:

After simulating master failover or node failure, verify:

Jobs complete the shutdown phase correctly.
Final job states remain consistent.
State cleanup does not suffer from race conditions.

4.2 Security Fix: Path Traversal Vulnerability in Log REST API (#10628)

[Fix][Zeta] Fix path traversal vulnerability in log file REST API endpoints

Operational recommendations:

Apply access controls at gateway or ingress layers.
Restrict log API access to authorized users only.
Validate the fix using security scans and path traversal attempts such as ../.

4.3 Recovery and Race Condition Improvements (#10687 / #10842 / #10877)

Collectively, these fixes significantly strengthened Zeta’s recovery workflows, particularly around:

State cleanup after node failures
Restore process consistency
Reuse of recovery metadata
Elimination of race conditions

5. Deep Dive into Key Technical Changes

5.1 HTTP Binary Download Support (#10956): Design, Semantics, and Impact

Key characteristics:

format=binary treats the HTTP response body as raw bytes.
binary_chunk_size controls chunking for large files.
Fixed output schemas simplify downstream sink implementations.
Batch-only mode avoids ambiguity with streaming download semantics.

User value:

HTTP Source can now fetch both API data and files, reducing pipeline complexity.

Risks and recommendations:

After chunking, downstream sinks must correctly reconstruct files using the combination of relativePath and partIndex. The official binary file sink or sinks supporting byte-stream writes are strongly recommended.

5.2 SQLite Upsert Syntax Fix (#10880): Why `EXCLUDED` Matters

SQLite's UPSERT syntax requires conflicting updates to reference EXCLUDED.<column> rather than VALUES(<column>).

The PR updates the generated SQL as follows:

- `col`=VALUES(`col`)
+ `col`=EXCLUDED.`col`

Unit tests were added to ensure correctness and prevent regressions.

5.3 Milvus Exception Reference Fix (#10975): A Small Change with Large Operational Value

A single-line change delivers a significant troubleshooting improvement:

- loadStateResponse.getException()
+ queryResultsR.getException()

After the fix, error messages accurately reflect the actual failure source, reducing debugging effort and avoiding misleading diagnostics.

5.4 Zeta Flaky Test Fix (#10891): Eliminating Timing Dependencies

The patch replaces real job submissions and asynchronous waiting with:

Mock servers
Mock JobHistoryService
Constructed pendingJobDAGInfo objects

This removes non-deterministic timing dependencies and significantly improves CI stability.

6. Reproducible Framework for Performance and Stability Validation

Many of May’s PRs focused on compatibility improvements, stability enhancements, and safer default configurations rather than benchmark-oriented optimizations.

To ensure objective validation:

Default parameter changes: measure RSS, GC, OOM rate, and throughput.
Chunking strategies: measure peak memory, end-to-end latency, and output consistency.
Dependency upgrades: measure startup success rate, startup latency percentiles, and long-term stability.

Recommended validation workflow:

Fix dataset size and parallelism.
Fix hardware and network environments.
Define consistent metrics.
Compare only pre-change and post-change versions.
Preserve all logs and metric outputs.

Thanks to All Contributors

Apache SeaTunnel’s continuous growth and evolution would not be possible without every community member who contributes code, reviews, documentation, and ideas.

Thank you all for your dedication and generosity. Every line of code helps move the project forward and strengthens the ecosystem.

The following contributors made merged contributions during May 2026:

Rank	GitHub Username	Merged PRs	Major Contribution Categories
1	nzw921rx	18	Features ×2; Performance ×1; Bug Fixes ×6; Architecture ×9
2	zhangshenghang	12	Features ×0; Performance ×0; Bug Fixes ×8; Architecture ×4
3	DanielLeens	8	Features ×0; Performance ×0; Bug Fixes ×1; Architecture ×7
4	CosmosNi	8	Features ×1; Performance ×0; Bug Fixes ×6; Architecture ×1
5	davidzollo	5	Features ×1; Performance ×1; Bug Fixes ×1; Architecture ×2
6	JeremyXin	4	Features ×0; Performance ×0; Bug Fixes ×0; Architecture ×4
7	CloverDew	3	Features ×1; Performance ×0; Bug Fixes ×2; Architecture ×0
8	zhiliang-wu	2	Features ×1; Performance ×0; Bug Fixes ×0; Architecture ×1
9	yzeng1618	2	Features ×1; Performance ×0; Bug Fixes ×1; Architecture ×0
10	QuakeWang	2	Features ×0; Performance ×0; Bug Fixes ×2; Architecture ×0

(The ranking is based on the number of merged PRs during May. Contribution categories are aggregated using the four-quadrant classification adopted in this report.)

Save Your Seat: How HOPEGOO Built a Unified Multimodal Data Platform with Apache SeaTunnel

Apache SeaTunnel — Thu, 11 Jun 2026 08:41:17 +0000

As generative AI continues to evolve, enterprises are placing new demands on their data infrastructure. Beyond traditional structured data, multimodal data—including text, images, audio, and other content types—is growing rapidly. As data types become more diverse and data pipelines increasingly complex, building a unified, efficient, and easily governed data pipeline platform has become a key challenge for many organizations.

At the upcoming Apache SeaTunnel June Meetup, we are excited to welcome Xiaocheng Zhou, Data Development Engineer at HOPEGOO and Apache SeaTunnel Committer, who will present a session titled "Architecture and Practices for a Unified Multimodal Data Pipeline Platform."

Drawing from HOPEGOO’s real-world experience, this talk will explore how the company leveraged Apache SeaTunnel to consolidate data ingestion channels, build a unified batch-and-stream data platform, and gain valuable insights throughout the platform modernization journey.

Featured Speaker

Xiaocheng Zhou currently works at HOPEGOO, where he focuses on data platform development and operations. He is also an Apache SeaTunnel Committer and has been actively contributing to the SeaTunnel community and its ongoing evolution.

Background and Session Overview

As business scale continued to expand, HOPEGOO gradually accumulated multiple data synchronization systems, including:

Offline data synchronization services
Real-time lake ingestion pipelines
Legacy Sqoop-based synchronization jobs
An early-generation SeaTunnel platform

While these systems effectively supported business requirements at different stages of growth, the coexistence of multiple synchronization platforms eventually introduced several challenges as data volumes and use cases continued to grow:

Fragmented data ingestion entry points
Increased maintenance costs caused by multiple technology stacks
Difficulty establishing unified data governance
Reduced efficiency when onboarding new business scenarios
Challenges in standardizing platform capabilities across teams

At the same time, the rise of generative AI applications has significantly increased demand for multimodal data processing, creating new challenges for traditional data integration architectures.

Against this backdrop, HOPEGOO began exploring the construction of a unified data pipeline platform and ultimately selected Apache SeaTunnel as its core data integration engine, driving the consolidation and modernization of its data synchronization ecosystem.

Event Information

Topic: Architecture and Practices for a Unified Multimodal Data Pipeline Platform
Date & Time: June 23, 2026, 14:00–15:00 (UTC+8)
Live Streaming: https://meeting.tencent.com/dm/VABnzAOyh8Yx

Community Giveaways

As with our previous events, we have prepared exclusive Apache SeaTunnel community gifts for online attendees. Join the live session for a chance to win exciting prizes and community swag!

Reserve Your Spot Today

From managing multiple independent synchronization systems to building a unified data pipeline platform, and ultimately preparing data infrastructure for the multimodal AI era, HOPEGOO's journey offers valuable lessons for organizations facing similar challenges.

If you're interested in data integration platforms, unified batch-and-stream architectures, lakehouse implementations, or enterprise adoption stories of Apache SeaTunnel, we invite you to reserve your spot now!

https://meeting.tencent.com/dm/VABnzAOyh8Yx

Join this Meetup to connect with community contributors and industry practitioners!

Demo: Full Data Synchronization from MySQL CDC to PostgreSQL with Apache SeaTunnel

Apache SeaTunnel — Thu, 11 Jun 2026 08:24:06 +0000

This article provides a detailed walkthrough on using Apache SeaTunnel 2.3.9 to perform full data synchronization from MySQL CDC to PostgreSQL.

For the complete demonstration, please refer to the video:

Demo: Synchronizing Data from MySQL CDC to PostgreSQL with Apache SeaTunnel

Without further ado, let's dive into the MySQL-to-PostgreSQL synchronization scenario.

Version Requirements

MySQL → MySQL 8.3
PostgreSQL → PostgreSQL 13.2
Apache SeaTunnel → Apache SeaTunnel 2.3.9

All configuration files used in this article are available by replying with the keyword "Demo 01" on our WeChat official account.

Prerequisites

Verify Version Information

-- Check version information
SELECT VERSION();

Enable Replication

-- Check replication-related configurations
SHOW VARIABLES
WHERE variable_name IN (
'log_bin',
'binlog_format',
'binlog_row_image',
'gtid_mode',
'enforce_gtid_consistency'
);

MySQL CDC synchronization relies on reading MySQL binlog files. SeaTunnel cluster nodes act as replication replicas within the MySQL replication architecture.

Therefore, before configuring CDC synchronization, you must verify that:

Binary logging (binlog) is enabled.
Replication mode is enabled.

Note: For MySQL 8.0 and later versions, binlog is enabled by default. However, replication-related settings still need to be configured manually.

-- Enable replication (execute sequentially)
-- SET GLOBAL gtid_mode=OFF;
-- SET GLOBAL enforce_gtid_consistency=OFF;

SET GLOBAL gtid_mode=OFF_PERMISSIVE;
SET GLOBAL gtid_mode=ON_PERMISSIVE;
SET GLOBAL enforce_gtid_consistency=ON;
SET GLOBAL gtid_mode=ON;

Configure User Permissions

The CDC user must have replication privileges.

The two essential permissions are:

REPLICATION SLAVE
REPLICATION CLIENT

After granting permissions, refresh privileges.

CREATE USER 'test'@'%' IDENTIFIED BY 'password';

GRANT SELECT,
RELOAD,
SHOW DATABASES,
REPLICATION SLAVE,
REPLICATION CLIENT
ON *.*
TO 'test';

FLUSH PRIVILEGES;

SeaTunnel Cluster Configuration

Cluster Logging

Independent Log Files for Each Job

Configuration file:

log4j2.properties

By default, SeaTunnel outputs logs into a centralized log file.

However, in production environments, job management is typically performed on a per-job basis. Therefore, it is recommended to configure independent log files for each job.

This approach provides several advantages:

Easier monitoring
Faster troubleshooting
Better operational visibility
More efficient job-level management

Modify the configuration as follows:

############################ log output to file #############################
# rootLogger.appenderRef.file.ref = fileAppender

# Output logs into independent files for each job
rootLogger.appenderRef.file.ref = routingAppender

############################ log output to file #############################

Client Configuration

In production environments, SeaTunnel is typically installed under the /opt directory.

It is recommended to point SEATUNNEL_HOME to:

/opt/seatunnel

If multiple versions are installed or the installation path differs, it is recommended to create a symbolic link so that client and server environments remain consistent.

This helps avoid classpath-related issues and missing dependency errors.

SeaTunnel's submission scripts reference client-side environment variables, including classpaths and installation directories, using absolute paths.

If client and server configurations differ, job submission may fail.

# Create symbolic link
ln -s /opt/apache-seatunnel-2.3.9 /opt/seatunnel

# Configure environment variable
export SEATUNNEL_HOME=/opt/seatunnel

Environment Variable Configuration

For Linux servers, it is recommended to configure environment variables using the official approach by placing them under:

/etc/profile.d

Example:

echo 'export SEATUNNEL_HOME=/opt/seatunnel' >> /etc/profile.d/seatunnel.sh

echo 'export PATH=$SEATUNNEL_HOME/bin:$PATH' >> /etc/profile.d/seatunnel.sh

source /etc/profile.d/seatunnel.sh

Job Configuration

The following examples do not cover every available option. Instead, they focus on commonly used production configurations.

env {
  job.mode = "STREAMING"
  job.name = "DEMO"
  parallelism = 3

  checkpoint.interval = 30000
  checkpoint.timeout = 30000

  job.retry.times = 3
  job.retry.interval.seconds = 3
}

Let's start with the env section.

Because CDC synchronization is a streaming workload, the job mode must be configured as:

STREAMING

Next, configure the job name.

In production environments, it is recommended to use meaningful naming conventions based on database names or table names. This makes job identification and management much easier.

Parallelism

The example uses:

parallelism = 3

The optimal parallelism value depends on:

Cluster size
Available resources
Database performance characteristics

Checkpoints

The checkpoint interval is configured as:

30 seconds

If lower recovery latency is required, the interval can be reduced to 10 seconds or even lower.

Checkpoint timeout is also configured as:

30 seconds

If checkpoint creation exceeds this threshold, the job is considered failed.

Retry Policy

The example configures:

Retry attempts: 3
Retry interval: 3 seconds

These values can be adjusted according to production requirements.

MySQL CDC Source Configuration

The MySQL CDC source configuration is one of the most important parts of the entire synchronization task.

source {
  MySQL-CDC {
    base-url = "jdbc:mysql://192.168.8.101:3306/test?serverTimezone=Asia/Shanghai"
    username = "test"
    password = "123456"

    database-names = ["test"]

    # table-names = ["test.test_001","test.test_002"]

    table-pattern = "test\\.test_.*"

    table-names-config = [
      {
        "table":"test.test_002",
        "primaryKeys":["id"]
      }
    ]

    startup.mode = "initial"

    snapshot.split.size = "8096"
    snapshot.fetch.size = "1024"

    server-id = "6500-8500"

    connect.timeout.ms = 30000
    connect.max-retries = 3
    connection.pool.size = 20

    exactly_once = false

    schema-changes.enabled = true
  }
}

One important recommendation is to explicitly specify the timezone in the MySQL JDBC URL:

serverTimezone=Asia/Shanghai

This helps prevent timezone-related inconsistencies when extracting DATETIME and TIMESTAMP fields.

The username and password should belong to an account with replication permissions, capable of reading binlogs and accessing all tables involved in synchronization.

In most production environments, each synchronization task is typically configured for a specific database. Therefore, it is recommended to explicitly specify the database being synchronized:

database-names = ["test"]

Table Selection

SeaTunnel provides two approaches for selecting source tables:

Explicit Table List

table-names = [
  "test.test_001",
  "test.test_002"
]

Regular Expression Matching

table-pattern = "test\\.test_.*"

For large-scale initial synchronization scenarios involving many tables—or even an entire database—regular expressions are often the preferred option.

When using regular expressions, both the database name and table name must be included.

For example:

test\\.test_.*

The escaped dot (\\.) represents the literal separator between the database name and table name.

The pattern above matches all tables in the test database whose names begin with:

test_

This approach is especially useful for large-scale database synchronization projects.

Table-Level Custom Configuration

Additional table-specific settings can also be defined.

For example:

table-names-config = [
  {
    "table":"test.test_002",
    "primaryKeys":["id"]
  }
]

Suppose test_002 does not have a physical primary key.

In that case, a logical primary key can be specified manually to support synchronization and downstream processing.

Startup Mode

One of the most important options is:

startup.mode = "initial"

This is also the most common production configuration.

The initial mode performs:

Historical full synchronization
Continuous CDC synchronization

In other words:

Full Load + Incremental CDC

This approach is widely used when onboarding existing databases into real-time synchronization pipelines.

Snapshot Configuration

The following options control snapshot processing:

snapshot.split.size = "8096"
snapshot.fetch.size = "1024"

The default values work well in most cases.

For larger clusters or more powerful servers, these values can be adjusted to improve throughput.

Server ID Configuration

Another critical parameter is:

server-id = "6500-8500"

When SeaTunnel consumes MySQL binlogs, it behaves like a MySQL replica node.

MySQL replication requires every replica to have a unique server-id.

If not specified, a default value will be used.

However, the official recommendation is to configure a dedicated server-id range.

An important requirement is:

The server-id range must be larger than the configured parallelism; otherwise, synchronization tasks may fail during startup.

Connection Settings

The following parameters control connectivity:

connect.timeout.ms = 30000
connect.max-retries = 3
connection.pool.size = 20

For large datasets or slower networks, increasing timeout values may be beneficial.

Similarly, connection pool size can be increased for high-throughput synchronization workloads.

Exactly-Once Semantics

exactly_once = false

For most CDC analytics scenarios, strict transactional consistency is not required.

Therefore, disabling Exactly-Once semantics is often recommended because it can significantly improve synchronization performance.

If strong consistency guarantees are required by the business, this option can be enabled.

However, enabling Exactly-Once generally introduces additional overhead and may reduce throughput.

Schema Evolution

Another highly recommended feature is:

schema-changes.enabled = true

Schema Evolution allows SeaTunnel to automatically adapt to changes in source table structures.

For example:

Add Column
Drop Column
Rename Column
Modify Column

This reduces the need to manually modify synchronization jobs whenever schema changes occur.

However, it also introduces certain considerations.

For example, if a downstream application depends on a column name that is automatically renamed, related SQL statements may fail.

Therefore, users should balance automation and downstream compatibility according to their own requirements.

According to the official documentation, Schema Evolution currently supports:

add column
drop column
rename column
modify column

Not all DDL operations are supported.

For example:

CREATE TABLE
DROP TABLE

cannot currently be captured and propagated automatically.

Despite these limitations, Schema Evolution remains an extremely valuable capability and is highly recommended for most production environments.

PostgreSQL Sink Configuration

sink {
  jdbc {
    url = "jdbc:postgresql://192.168.8.101:5432/test"
    driver = "org.postgresql.Driver"

    user = "postgres"
    password = "123456"

    generate_sink_sql = true

    database = "test"

    table = "${database_name}.${table_name}"

    schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"

    data_save_mode = "APPEND_DATA"

    # enable_upsert = false
  }
}

The Sink configuration writes synchronized data into PostgreSQL.

In addition to the connection URL, JDBC driver, username, and password, one particularly useful feature is:

generate_sink_sql = true

When enabled, SeaTunnel automatically generates:

CREATE TABLE statements
INSERT statements
DELETE statements
UPDATE statements based on primary keys

This greatly simplifies synchronization configuration and eliminates the need for complex manual SQL development.

Understanding PostgreSQL Schema Hierarchy

When synchronizing data to PostgreSQL, it is important to understand the difference between MySQL and PostgreSQL object hierarchies.

MySQL contains:

Database → Table

PostgreSQL contains:

Database → Schema → Table

Therefore, if all synchronized tables should be stored in a specific PostgreSQL database, this must be configured appropriately.

It is also recommended that the PostgreSQL user have table creation permissions if automatic table creation is enabled.

Placeholder Support

SeaTunnel provides a powerful placeholder mechanism:

table = "${database_name}.${table_name}"

This feature is especially useful when synchronizing many tables.

Instead of manually defining every target table name, placeholders automatically generate target table mappings based on source metadata.

This significantly reduces maintenance effort in large-scale synchronization projects.

Save Modes

Schema Save Mode

schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"

This option is extremely useful for whole-database synchronization.

It automatically creates target schemas and tables when they do not already exist.

As a result, users can avoid many manual setup steps.

Data Save Mode

data_save_mode = "APPEND_DATA"

APPEND_DATA prevents existing synchronized data from being overwritten.

This makes it one of the safest and most commonly used modes in production environments.

Other save modes are also available and can be selected according to business requirements.

Upsert Configuration

enable_upsert

If you can guarantee that source-side data will never contain duplicate primary keys, disabling upsert operations may significantly improve synchronization performance.

However, if duplicate records are possible, it is recommended to keep Upsert enabled.

SeaTunnel can then perform primary-key-based updates automatically.

Refer to the official documentation for detailed parameter descriptions and supported behaviors.

Job Submission and Monitoring

After completing the configuration file, the synchronization job can be submitted using SeaTunnel's command-line tools.

Example:

./bin/start-seatunnel.sh --config /path/to/config.yaml --async

Parameter description:

config — Specifies the configuration file path.
async — Submits the job asynchronously. After submission, the command-line process exits immediately while the job continues running in the background.

After submission, the job can be monitored through the SeaTunnel UI.

Starting from version 2.3.9, SeaTunnel provides an intuitive web interface that allows users to view:

Job status
Execution logs
Throughput statistics
Data processing metrics

Synchronization Demonstration

In this demo, two tables were created:

test_001
test_002

After inserting sample records into MySQL, SeaTunnel successfully synchronized the data into PostgreSQL.

The demonstration also covers:

INSERT operations
DELETE operations
UPDATE operations
Schema changes

SeaTunnel successfully captures and synchronizes all these changes to PostgreSQL in real time.

Key Takeaways

Automatic Schema Synchronization

SeaTunnel supports automatic schema synchronization.

When the schema of a source MySQL table changes, the corresponding PostgreSQL table structure can be updated automatically as well.

Data Consistency

SeaTunnel ensures data consistency throughout the synchronization process.

All INSERT, DELETE, and UPDATE operations are accurately replicated to the target database, providing a reliable foundation for real-time analytics and data integration scenarios.

Meet Apache SeaTunnel's Newest Committer: 60+ PRs, Connector Innovation, and the Future of AI Data Integration

Apache SeaTunnel — Fri, 05 Jun 2026 09:51:05 +0000

Hey Community! Exciting news has arrived from the Apache SeaTunnel open-source community. Zeng Yi, a Big Data Engineer at China Telecom Cloud Technology, has been invited to join the ranks of Apache SeaTunnel Committers, bringing new energy and momentum to the project.

Within the Apache SeaTunnel community, Zeng Yi has already distinguished himself through outstanding technical expertise and strong engineering capabilities. His election as an Apache SeaTunnel Committer is a well-deserved recognition of his dedication to open source. He has actively contributed to a wide range of community initiatives, delivering high-quality code in areas such as Connector enhancements and engine adaptation. Beyond code contributions, he has also leveraged his extensive experience to improve documentation, participate in technical discussions, and help fellow developers solve problems, continuously driving the project's growth through practical action.

From his early days exploring the open-source world to becoming a key contributor to a top-tier Apache project, Zeng Yi's journey is filled with valuable experiences and unique insights. How did he find his direction in open source? What lessons can he share with aspiring contributors?

Let's dive into this in-depth community interview and hear his story firsthand!

Personal Profile

Interview Transcript

1. How long have you been involved in open source, and what attracts you to it?

If we take my first official PR submission to an Apache project as the starting point, I have been contributing to open source for about one year.

What attracts me most is the opportunity to collaborate with many outstanding developers in the community. During every PR review, reviewers provide feedback on naming conventions, edge cases, compatibility, test coverage, and many other aspects. These are valuable experiences that are not always available in such a concentrated form in day-to-day work.

Open source also allows individual contributions to create broader value. Fixing a problem within a company may only benefit a team or a specific project. However, when the same fix is merged into the community's main branch, it can benefit a much larger user base. That sense of impact is highly rewarding.

2. When did you start contributing to SeaTunnel, and what motivated you to get involved?

I started contributing to SeaTunnel in April 2025.

On April 23, I submitted my first PR (#9213), and on May 16, my first contribution was merged through PR #9305, officially making me a Contributor.

The motivation came directly from real-world challenges at work. At the time, we were building a data integration platform based on Apache SeaTunnel and Flink CDC, using Flink as the unified execution engine.

While supporting customer synchronization workloads, we encountered issues such as Oracle BLOB fields losing their original format after being read, and Doris Sink lacking flexible table name case handling. After implementing fixes internally, I realized these problems were quite common, so I organized the solutions into PRs and contributed them back to the community.

3. As a newly invited SeaTunnel Committer, could you summarize your contributions to the community, both technical and non-technical?

Since joining the Apache SeaTunnel community, I have continuously contributed in areas including Connector enhancements, engine adaptation, synchronization stability, documentation improvements, and community collaboration.

To date, I have had 60 PRs merged into the apache/seatunnel repository, covering modules such as connectors-v2, Zeta, Flink, CDC, Transform-v2, E2E, and Docs.

Code Contributions

File Connector Enhancements

I systematically improved large-file processing and continuous file discovery capabilities for HDFS.

To address the limitation of "one file equals one split," which restricted parallelism for large files, I implemented large-file split reading support for the HDFS File Source and introduced configurations such as enable_file_split and file_split_size.

For text, CSV, and JSON files, split processing respects line boundaries to prevent record corruption. For Parquet files, logical splitting is implemented based on RowGroups.

Continuous File Discovery

I added continuous discovery capabilities for FTP, SFTP, Local, and HDFS file sources.

This allows running jobs to continuously detect newly created or updated files, with support for the scan_interval configuration, making it suitable for periodic file drops and near real-time file synchronization scenarios.

Connector Enhancements and Bug Fixes

I contributed enhancements and fixes across Hive, JDBC, CDC, Iceberg, Doris, Kafka, and other connectors.

Examples include:

Hive Sink support for SchemaSaveMode and DataSaveMode
Automatic table creation
Schema management
Partition field support
Multiple storage format support
JDBC regex-based multi-table reading
PostgreSQL TIMESTAMP_TZ support
Iceberg time type fixes
Doris case sensitivity compatibility improvements
Kafka checkpoint offset recovery fixes
Flink Version Adaptation

I participated in SeaTunnel's support for Flink 1.20.1.

This involved introducing a dedicated translation layer for Flink 1.20, replacing fragile reflection-based implementations with Flink's official Sink2 API, and adding starters, build configurations, and E2E testing infrastructure.

Stability and Engineering Quality Improvements

I continuously worked on fixing unstable CI test cases, improving E2E test coverage, and addressing issues related to checkpoint recovery, transaction commits, duplicate XA XIDs, empty directory reading, CDC snapshot splits, Kafka offset restoration, and Zeta REST APIs.

Non-Code Contributions

Continuously improving SeaTunnel user and developer documentation, including connector documentation, parameter descriptions, usage examples, E2E configuration guides, onboarding materials, and architecture documents.
Actively participating in PR reviews and technical discussions.
Adjusting implementation approaches, adding tests, and improving documentation based on reviewer feedback, helping drive PRs from design to merge.

4. After contributing to SeaTunnel for some time, what do you see as its advantages and shortcomings compared to other solutions? What keeps you engaged in the community?

One of my strongest impressions is that SeaTunnel is highly aligned with real enterprise data integration needs.

Rather than focusing on a single data source, execution engine, or synchronization method, SeaTunnel aims to solve a broad range of heterogeneous data integration challenges.

Compared with competing solutions, one of SeaTunnel's most significant advantages is its rich Connector ecosystem.

It provides comprehensive coverage for commonly used systems such as JDBC, CDC, Hive, Doris, Kafka, Iceberg, HDFS, FTP/SFTP, and local file systems.

In enterprise environments, data integration rarely involves a single pipeline. Instead, organizations often require database-to-warehouse, database-to-lakehouse, file-to-Hive/Doris, CDC-to-Kafka, and many other combinations. SeaTunnel offers a relatively unified development and configuration experience across these scenarios.

Another major advantage is multi-engine support.

Different organizations have different technology stacks. Some workloads are better suited for Flink, while others may choose SeaTunnel Zeta. SeaTunnel does not lock users into a specific execution engine, which is highly valuable for enterprise adoption.

As for areas for improvement, I believe CDC capabilities can be further strengthened.

By learning from projects such as Flink CDC, SeaTunnel could continue improving schema change support across multiple data sources, consistency guarantees for different sinks, and recovery stability after failures.

Documentation and best practices can also be improved, especially around production deployment, troubleshooting, and performance tuning.

In addition, emerging areas such as AI data integration, unstructured data processing, and vector databases present exciting opportunities for future exploration.

What keeps me engaged is the community's high level of activity.

Issues, PRs, and discussions typically receive timely feedback, which makes contributions feel meaningful rather than isolated efforts.

I can clearly see the value of my contributions. Many problems originate from real business scenarios, and once resolved and contributed back, other users can immediately benefit.

Another important factor is the quality of community reviews.

Reviewers do not simply check whether code runs. They evaluate solution generality, edge cases, test completeness, documentation quality, and long-term maintainability.

Although this often requires multiple iterations, it significantly improves both the solution and my own engineering skills.

5. Have you developed custom solutions to address SeaTunnel's limitations? Have these been contributed back to the community?

Yes.

In fact, my initial involvement with SeaTunnel began because I encountered issues while using it.

At the time, we were building a data integration platform based on SeaTunnel and Flink CDC. During production synchronization tasks, we discovered problems related to Oracle BLOB field handling and Doris Sink table name case sensitivity.

While these might seem like minor details, they can directly affect synchronization results in production environments.

I first validated solutions internally and then submitted them to the community as PRs.

Later, much of my work focused on File Connectors.

For example, HDFS originally mapped one file to one split, which limited parallelism when processing very large files.

I introduced large-file split reading support with configurable split behavior and split sizes.

This required more than simply splitting files by byte offsets. Text, CSV, and JSON files must preserve record boundaries, while Parquet files are more naturally divided by RowGroups.

I also implemented continuous discovery for FTP, SFTP, Local, and HDFS file sources.

Many file synchronization scenarios involve periodic file arrivals rather than a fixed set of files prepared upfront, making continuous discovery essential.

Additionally, I participated in Flink 1.20.1 compatibility work because our product needed to standardize on a newer Flink version.

This included translation layers, starters, build configurations, and E2E testing to ensure SeaTunnel worked properly on Flink 1.20.1.

Most of these enhancements have already been contributed back to the community.

My philosophy is simple:

If a problem is fixed only internally, the team must maintain that patch indefinitely.

If the problem is common, contributing it back to the community creates greater value and allows the solution to benefit from community review and validation.

6. Does your company use SeaTunnel in production? What are the use cases? If not, would you recommend it, and why?

Yes, our company uses SeaTunnel in real production environments.

For customer-facing data integration scenarios, we have built a data integration platform based on open-source Apache SeaTunnel and Flink CDC, with Flink serving as the unified underlying execution engine.

Currently, we support a wide range of data sources, including various databases, data warehouses, data lakes, Kafka, HTTP, and many others.

Typical target systems include Hive, Doris, and Iceberg, which are used to support customer requirements such as data lake ingestion, data warehouse loading, and both real-time and batch synchronization.

From practical experience, SeaTunnel is well suited to serve as the foundation of a data integration platform.

On one hand, its broad Connector coverage allows it to adapt to different customer environments and heterogeneous data sources.

On the other hand, its configuration model and extensibility are relatively clear and straightforward, making it suitable for productization and enterprise-level packaging.

Of course, when delivering solutions to customers, we still perform additional adaptation and validation based on specific business scenarios, including complex data type compatibility, task stability, failure recovery, performance characteristics, and write capabilities for different target systems.

Overall, SeaTunnel provides significant value in heterogeneous data synchronization and lakehouse integration scenarios.

7. What kind of support do you hope the SeaTunnel community can provide for your personal growth?

I hope that through participating in the SeaTunnel community, I can continue improving my engineering capabilities, open-source collaboration skills, and technical perspective.

SeaTunnel involves many production-grade challenges, including data source integration, type conversion, task partitioning, fault tolerance and recovery, checkpointing, and multi-engine compatibility. Working on these areas greatly helps deepen my understanding of large-scale data systems.

At the same time, open-source collaboration has encouraged me to think beyond solving immediate business problems and to consider factors such as generality, compatibility, documentation quality, and test completeness.

Going forward, I would like to participate more actively in code reviews and community discussions, helping other Contributors improve their solutions and implementations.

In addition, if the community continues exploring areas such as AI, Agents, unstructured data, and vector databases, I would be very interested in participating and gaining hands-on experience in these emerging domains.

8. What is your understanding of the Committer role? What responsibilities should a Committer have within the community?

In my view, being a Committer is not simply about having merge permissions.

More importantly, a Committer is responsible for maintaining project quality and supporting the long-term growth of the community.

First, Committers should continue contributing.

Obtaining Committer status should not be the end of one's involvement. Instead, Committers should continue identifying problems, solving issues, and improving the areas where they have expertise.

Second, Committers should take code reviews seriously.

A good review is not merely about checking formatting or verifying that the code compiles successfully.

More importantly, reviewers should evaluate whether a solution is well designed, sufficiently general, capable of handling edge cases, compatible with existing functionality, and supported by adequate testing and documentation.

In many cases, a pull request becomes significantly clearer and more reliable after going through the review process.

Third, Committers should help new Contributors integrate into the community.

Many first-time contributors may not be familiar with project architecture, contribution guidelines, testing requirements, or communication processes.

Receiving timely and friendly feedback can greatly increase their confidence and encourage them to continue contributing.

For a community to grow sustainably, it cannot rely solely on a small number of contributors. It must continuously attract and nurture new participants.

Finally, Committers should also participate in strategic planning and project direction.

For example, determining which Connectors need priority improvements, which stability issues should be addressed first, and which documentation or testing gaps need attention all require collaboration between community members and real-world users.

9. How do you feel about becoming an Apache Software Foundation Committer? Do you have any message for the community or suggestions for the project's future development?

I am truly grateful to the community for recognizing my previous contributions and inviting me to become a SeaTunnel Committer.

For me, this is both an encouragement and a responsibility.

I originally became involved with SeaTunnel because of practical problems encountered in my daily work.

As I became more engaged, I realized that many business challenges are not isolated cases. When solutions are contributed back to the community, they can help many other users facing similar issues. That is one of the reasons I find open source so meaningful.

Throughout this journey, I am also deeply thankful to all the Reviewers and Contributors who have helped me along the way.

Many of my PRs went through multiple rounds of discussion and revision. While the process could sometimes be repetitive, the final solutions were always more complete and robust, and I learned a tremendous amount from those experiences.

Looking ahead, I hope SeaTunnel will continue strengthening the production-grade capabilities of its core Connectors, including CDC, JDBC, File, Hive, Doris, and Iceberg.

At the same time, I believe the project should continue investing in stability, observability, documentation, and best practices.

In addition, I plan to continue contributing to AI-related initiatives and practical implementations, including unstructured data processing, vector databases, LLM data pipelines, and Agent automation scenarios.

I hope to further explore how SeaTunnel's data integration capabilities can support these emerging technologies.

10. What are your plans for helping drive the project forward in the near future?

Over the coming period, I plan to continue contributing to Connector development and production stability improvements.

This includes enhancing commonly used Connectors, fixing issues, and improving E2E test coverage.

At the same time, I will place particular focus on AI-powered data integration.

Recently, the community has been discussing Knowledge Sync and Retrieval-Augmented Generation (RAG) capabilities.

The goal is to enable SeaTunnel to take on responsibilities related to enterprise knowledge synchronization and indexing, including:

Document discovery
Document parsing
Content chunking and segmentation
Embedding generation
Writing data into vector databases such as Milvus and Qdrant
Lifecycle management for document updates, deletions, and unchanged-content detection

Personally, I hope to participate in both the design and implementation of these capabilities.

By combining SeaTunnel's existing data integration strengths with AI and RAG scenarios, I believe we can unlock new possibilities for enterprise knowledge bases, unstructured data synchronization, and vector search data preparation workflows.

From Batch to Real-Time: A Practical Guide to MySQL CDC in SeaTunnel

Apache SeaTunnel — Fri, 05 Jun 2026 08:08:42 +0000

Real-Time Incremental Data Capture: Change Data Capture (CDC)

Official MySQL-CDC Documentation:
https://seatunnel.apache.org/docs/2.3.3/connector-v2/source/MySQL-CDC/
A single SeaTunnel CDC job can monitor multiple tables simultaneously and synchronize them in real time.
You must use MySQL JDBC Driver 8.0.33 or later.
- Download: https://downloads.mysql.com/archives/c-j/

1. How Many Databases Does SeaTunnel Support for CDC?

Documentation:

https://seatunnel.apache.org/docs/2.3.12/connector-v2/source

Official CDC Source Connectors Supported by SeaTunnel:

MongoDB CDC Source Connector
MySQL CDC Source Connector
OpenGauss CDC Source Connector
Oracle CDC Source Connector
PostgreSQL CDC Source Connector
SQL Server CDC Source Connector
TiDB CDC Source Connector

2. CDC Jobs Do Not Stop After SeaTunnel Starts

A standard CDC (Change Data Capture) job is designed to run continuously and indefinitely as a streaming service.

Core Difference

Offline / Batch Processing Jobs
- job.mode = "BATCH"
- Execute a full or incremental SQL query once.
- The job automatically finishes after processing all existing data.
CDC / Streaming Jobs
- job.mode = "STREAMING"
- After startup, the job will:

1. Optionally perform an **initial full snapshot** (when `scan.startup.mode = "initial"` is configured).
2. Suspend and continuously monitor the MySQL Binlog stream.
3. Immediately capture, process, and write any new database changes (INSERT, UPDATE, DELETE).
4. Continue monitoring indefinitely until the job is manually stopped or fails due to an error.

How to Stop a CDC Job

In a SeaTunnel terminal session, you can usually stop a CDC job gracefully using:

Ctrl + C

In production environments, jobs are typically stopped through schedulers, orchestration systems, or management platforms.

In simple terms, a CDC job behaves like a persistent subscription service that subscribes to database change logs and processes events in real time whenever they occur.

This is fundamentally different from a traditional batch job that runs once and exits.

💡 Configuration Recommendations and Common Pitfalls

About startup.mode

For first-time deployments, it is strongly recommended to use:

initial

This ensures:

A complete snapshot of current data is captured first.
Incremental synchronization seamlessly follows afterward.

If you configure:

latest

all historical data will be skipped and only changes occurring after startup will be captured.

About server-id

In production environments, always configure a unique and explicit value.

Avoid random values because conflicts can cause instability and unexpected behavior.

3. CDC Mode vs JDBC Mode

Comparison

Feature	MySQL CDC (Streaming)	JDBC Batch Processing
Job Mode	`STREAMING`	`BATCH`
Data Source	Database Binlog	SQL Query Result Set
Data Content	Change Event Stream (includes operation type, before/after images, metadata)	Static Data Snapshot
Core Configuration	`table-names`	`query` or `table + sql`
Synchronization Type	Real-time incremental (optionally full snapshot first)	One-time full or incremental batch
Job Lifecycle	Runs continuously until manually stopped	Automatically ends after reading all data
Typical Use Cases	Real-time warehouses, analytics, disaster recovery	T+1 reports, migrations, historical backfill
Source Database Load	Low continuous Binlog reading	Query-intensive batch execution
Data Consistency	Exactly-once guarantees	Depends on query conditions; duplicates or omissions may occur

Architecture Comparison

Under CDC mode, you cannot use a custom SQL query in the source connector like you would with JDBC.

For example:

SELECT * FROM table WHERE age > 20

is not supported in the CDC Source.

If filtering or transformation is required, it must be implemented in the Transform stage.

Why CDC Does Not Support Custom Queries

Different Data Sources

JDBC Batch

Reads the results of user-defined SQL statements.

MySQL CDC

Reads database Binlog events.
Internally uses Debezium.
Behaves like a MySQL replica that continuously receives row-level changes.

Different Data Structures

JDBC Batch

Returns ordinary row data.

MySQL CDC

Produces structured change events containing:

Changed row values
Operation types (op)
Metadata
Before image
After image

Examples:

+I  Insert
-U  Before Update
+U  After Update
-D  Delete

A traditional SQL query cannot generate this event structure.

4. How to Implement Filtering and Transformation in CDC Mode

Although filtering cannot be performed in the Source, several alternatives are available.

Method	Location	Description
Transform	SeaTunnel Transform Section	Recommended approach. Supports filtering, projection, renaming, and SQL processing.
Sink Processing	Sink Configuration	Some sinks support limited filtering or mapping capabilities.
Schema Evolution	Table Synchronization Layer	Sync selected columns to achieve indirect column filtering; requires advance schema changes and is typically used for long-term sync scenarios.

5. CDC Can Monitor Multiple Tables Simultaneously

When multiple tables need to be synchronized using a single CDC job, SeaTunnel provides a simple configuration model.

The key idea is:

One Source → Many Tables
Automatic Sink Routing

Configuring Multi-Table CDC

Simply list all tables inside:

table-names

Example:

source {
  MySQL-CDC {
    base-url = "jdbc:mysql://192.168.1.107:51382/cs1"
    username = "root"
    password = "zysoft"

    database-names = ["cs1"]

    table-names = [
      "cs1.t_8_100w",
      "cs1.order_table",
      "cs1.user_profile"
    ]

    startup.mode = "initial"
    server-id = 5400
    server-time-zone = "Asia/Shanghai"
  }
}

Sink Configuration: Automatic Routing

SeaTunnel JDBC Sink supports automatic table routing.

Core Configuration

sink {
  jdbc {

    url = "jdbc:mysql://192.168.1.107:51382/cs2"

    driver = "com.mysql.cj.jdbc.Driver"

    user = "root"

    password = "zysoft"

    generate_sink_sql = true

    database = "cs2"    # Target database

    # Key technique: dynamically map source table names using built-in variables
    table = "${table_name}"

    # table = "prefix_${table_name}"                  # Add a prefix if needed

    # table = "${database_name}_${table_name}_suffix" # Use database name and suffix

    schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"

    data_save_mode = "APPEND_DATA"

    batch_size = 5000

    # ... Other connection and performance tuning parameters remain unchanged

  }
}

Built-In Variables

SeaTunnel automatically provides:

${table_name}
${database_name}

During execution:

cs1.t_8_100w
   ->
cs2.t_8_100w

cs1.order_table
   ->
cs2.order_table

This creates a natural 1:1 table mapping.

Different Routing Strategies for Different Tables

If different tables require different write policies, configure multiple Sink definitions.

Example:

sink {

  # Sink 1: Dedicated to processing the t_8_100w table
  jdbc {

    # Explicitly define the target table
    table = "t_8_100w_target"

    # Route only data from the specified source table to this sink
    source_table_name = "t_8_100w"

    # ... Other configurations

  }
}

sink {

  # Sink 2: Dedicated to processing the order_table table
  # Different write strategies (such as data_save_mode) can be applied
  jdbc {

    table = "order_table_target"

    source_table_name = "order_table"

    # Example: Use a truncate-and-reload strategy for this table
    data_save_mode = "DROP_DATA"

    # ... Other configurations

  }
}

Reminding and Best Practices
1. Target Tables Ensure compatible target tables already exist, or enableschema_save_mode = “CREATE_SCHEMA_WHEN_NOT_EXIST
2. Performance Isolation All monitored tables share the same CDC stream. If one table generates massive changes, it may increase latency for others. For critical workloads, create separate CDC jobs.
3. Initial Snapshot When using startup.mode = "initial", all monitored tables are snapshotted during startup. Ensure sufficient database resources are available.

Summary:

For most multi-table CDC scenarios:

List multiple tables in table-names
Configure:

table = "${table_name}"

in the Sink

SeaTunnel will automatically route data to matching target tables.

If special processing is required for specific tables, use:

Transform SQL
Filters
Conditional routing

to customize behavior.

Routing and Filters If you want data to flow selectively into different sinks instead of being copied to every sink, you must configure a filter before each sink or use more advanced mechanisms such as side outputs (typically implemented through conditional expressions in the configuration file).

💡 Configuration Recommendations for Multi-Table CDC Scenarios

Based on the questions discussed earlier, a typical configuration for synchronizing multiple CDC tables into different target tables is shown below:

source {
  MySQL-CDC {
    table-names = ["cs1.t_order", "cs1.t_user", "cs1.t_log"]
    # ... other configurations
  }
}

sink {

  # Order table -> Order Archive Database
  jdbc {
    table = "t_order_archive"

    # Use filter to synchronize only t_order data
    filter {
      source_table_name == "t_order"
    }

    # ...
  }

  # User table -> User Analytics Database
  jdbc {
    table = "t_user_analysis"

    filter {
      source_table_name == "t_user"
    }

    data_save_mode = "OVERWRITE"   # Use overwrite strategy for this table

    # ...
  }

  # Log table -> Log Center (example writes to HDFS)
  hdfs {
    path = "/data/lake/log/${source_table_name}/dt=${now(date='yyyy-MM-dd')}"

    filter {
      source_table_name == "t_log"
    }

    # ...
  }
}

Steps to Implement MySQL CDC

1. Verify That MySQL Binlog Is Enabled

Use the following commands to check whether Binlog is enabled.

-- The simplest way to check
SHOW VARIABLES LIKE 'log_bin';

-- Shows current binlog file name and location
SHOW MASTER STATUS;

-- Verify binlog format
SHOW VARIABLES LIKE 'binlog_format';

-- Verify row image mode
SHOW VARIABLES LIKE 'binlog_row_image';

Enable Binlog If It Is Disabled

If Binlog is not enabled, modify the MySQL configuration file and restart MySQL.

[mysqld]

server-id = 123
# Configure a unique server ID

log_bin = /var/lib/mysql/mysql-bin
# Enable Binlog and specify storage path

binlog_format = ROW
# Must be configured as ROW mode

binlog_row_image = FULL
# Must be configured as FULL

expire_logs_days = 10
# Binlog retention period
# At least 2 days is recommended

2. Common MySQL CDC Parameters

Single-Table MySQL CDC Demo

Create Target Table

-- demo7-1-mysql-cdc2mysql-qxzh-st-107.conf

CREATE TABLE cs2.t_8_100w_imp_st_qxzh_cdc_demo7_1 (

  id BIGINT NOT NULL COMMENT 'Primary Key',

  user_name VARCHAR(2000) NULL COMMENT 'User Name',

  sex VARCHAR(20) NULL COMMENT 'Gender: Male/Female',

  decimal_f DECIMAL(32,6) NULL COMMENT 'Large Decimal Value',

  phone_number VARCHAR(20) COMMENT 'Phone Number',

  age INT NULL COMMENT 'Converted Age',

  create_time TIMESTAMP COMMENT 'Creation Time',

  description LONGTEXT NULL COMMENT 'Large Text',

  address VARCHAR(2000) NULL COMMENT 'Default Value for Empty Address: Unknown',

  PRIMARY KEY (id)

);

Execute the Job

# demo7-1-mysql-cdc2mysql-qxzh-st-107.conf

sh /data/tools/seatunnel/seatunnel-2.3.12/bin/seatunnel.sh \
--config /data/tools/seatunnel/myconf/demo7-1-mysql-cdc2mysql-qxzh-st-107.conf \
-m local

Complete Configuration File

# demo7-1-mysql-cdc2mysql-qxzh-st-107.conf

env {

  # Parallelism (number of threads)
  execution.parallelism = 5

  # Job mode:
  # BATCH = Batch Processing
  # STREAMING = Streaming Processing (required for CDC)
  job.mode = "STREAMING"
}

source {

  MySQL-CDC {

    base-url = "jdbc:mysql://ip:port/cs1"

    username = "root"

    password = "zysoft"

    # query is invalid in CDC mode

    # Source database
    database-names = ["cs1"]

    # Monitored tables
    # Table names must include database names
    table-names = ["cs1.t_8_100w"]

    # Startup mode:
    # initial = full snapshot + incremental changes
    # latest  = incremental changes only
    startup.mode = initial

    # Startup timestamp
    # Required when startup.mode=timestamp
    # startup.timestamp

    # Very important!
    # CDC client unique ID
    # Example: 5400 or range 5400-6408
    # Must not conflict with existing MySQL server IDs
    server-id = 5400

    # Stop mode
    # stop.mode

    # Required when stop.mode=timestamp
    # stop.timestamp

    # Database session time zone
    server-time-zone = "Asia/Shanghai"
  }
}

# CDC transformations must be implemented in Transform

transform {

  # 1. Field Mapping
  # Alternatively, use the FieldMapper plugin

  FieldMapper {

    field_mapper = {

      id = id

      name = user_name

      sex = sex

      decimal_f = decimal_f

      phone_number = phone_number

      age = age

      create_time = create_time

      description = description
    }
  }

  # 2. Phone number masking
  # 13812341234 -> 138****1234

  # 3. Age conversion
  # String -> Integer

  # 4. Gender conversion
  # 1 -> Male
  # 2 -> Female

  # 5. Data filtering
  # Keep only records where age > 25

  # 6. Address default value
  # Empty address -> "Unknown"
}

sink {

  jdbc {

    url = "jdbc:mysql://ip:port/cs2"

    driver = "com.mysql.cj.jdbc.Driver"

    user = "root"

    password = "zysoft"

    # Automatically generate insert SQL
    # Can also create tables automatically
    generate_sink_sql = true

    # Required when generate_sink_sql=true
    database = cs2

    table = "t_8_100w_imp_st_qxzh_cdc_demo7_1"

    # Fail if schema does not exist
    # Commonly used:
    # CREATE_SCHEMA_WHEN_NOT_EXIST
    schema_save_mode = "ERROR_WHEN_SCHEMA_NOT_EXIST"

    # APPEND_DATA
    # Keep existing data and append new records
    data_save_mode = "APPEND_DATA"

    # DROP_DATA
    # Clear table before loading
    # data_save_mode = "DROP_DATA"

    batch_size = 5000

    # Retry attempts
    max_retries = 3

    # Connection timeout
    connection_check_timeout_sec = 300

    properties = {

      useUnicode = true

      characterEncoding = "utf8"

      serverTimezone = "Asia/Shanghai"

      # Enable batch rewrite
      rewriteBatchedStatements = "true"

      # Enable compression
      useCompression = "true"

      # Disable server-side prepared statements
      useServerPrepStmts = "false"
    }
  }
}

Execution Result

After startup, the CDC job first performs the initial snapshot synchronization.

In this example:

Total records read: 1,000,009
Total records written: 1,000,009
Job status: RUNNING
Checkpoints continue to be generated successfully.

This indicates that:

The initial full snapshot has completed successfully.
Source and target data are consistent.
Checkpointing is functioning correctly.
The CDC job has entered continuous Binlog monitoring mode.
The streaming pipeline remains active and ready to capture new changes in real time.

Result Demonstration

Once the initial synchronization is complete:

INSERT operations are captured immediately.
UPDATE operations are synchronized in real time.
DELETE operations are propagated automatically.
The CDC job continues running until manually stopped.

At this point, the pipeline behaves like a long-running subscription service, continuously consuming MySQL Binlog events and delivering them to downstream systems in real time.

The Next Decade of Data Engineering: From Modern Data Stack to Data Engineering Harness

Apache SeaTunnel — Thu, 28 May 2026 09:44:46 +0000

Over the past decade, the core evolution of data engineering has been the deconstruction and reconstruction of traditional data warehouse architectures through the Modern Data Stack.

We separated data ingestion from databases, forming the Data Ingestion layer, using tools like FiveTran, Airbyte, and Apache SeaTunnel to solve ELT / CDC / Reverse ETL problems;
We separated compute from storage, forming cloud data warehouse and lakehouse systems such as Snowflake, Databricks, Iceberg, and Hive;
We separated orchestration from scripts, leading to orchestration systems like Apache Airflow and Apache DolphinScheduler;
SQL development, data modeling, lineage, data quality, BI, and AI analytics were further split into independent tools.

This architecture was undoubtedly progress. It moved data engineering away from the primitive era of “a bunch of scripts + Crontab” toward cloud-native infrastructure, elastic computing, engineering governance, and open ecosystems.

The greatest contribution of the Modern Data Stack was “decoupling,” and its biggest side effect was also “decoupling.”
Tools became more powerful, but data engineers were forced to switch between more systems than ever before: datasources in one place, synchronization configs in another, DAGs somewhere else, logs elsewhere, SQL stored in Git, and Snowflake / Iceberg / cloud warehouse execution results living in yet another environment.

As a result, many data engineers spend less time on data modeling, business understanding, metric definitions, architecture design, and cost optimization — and far more time configuring datasources, setting field mappings, dragging DAG nodes, modifying SQL, checking logs, and rerunning tasks. This is the hidden pain created by the Modern Data Stack: data engineers became trapped inside tools.

The emergence of engineering-focused AI systems like Codex and Claude Code is now changing the entire software engineering workflow. But how can data engineers truly achieve Vibe Coding? That is exactly the direction I’ve been exploring, and the core topic of this article.

I believe future data engineering will no longer revolve around “humans operating tools.” Instead, it will evolve into: Codex + Data Engineering Skills & Harness + Data Engineering SaaS + Cloud Data Warehouse Infrastructure.

In the past, the Modern Data Stack assumed that humans were the operational center: humans understood tools, clicked interfaces, connected workflows, and handled context switching. But in the AI and Agentic development era, data engineering should no longer mean “humans operating a pile of tools.” Instead, humans define objectives, Codex/Claude Code decompose and implement solutions automatically, the Data Engineering Skill & Harness layer provides engineering boundaries and translates them into cloud SaaS systems, Snowflake / Iceberg / cloud warehouses provide scalable compute, orchestration and synchronization engines ensure runtime stability, and humans become responsible for reviewing, governing, and making final decisions.

Once Codex and Claude Code deeply participate in data engineering, perhaps data engineers can finally be freed from the “Dirty Work” created by the Modern Data Stack, allowing data engineering to return from “tool operation” back to “engineering creation.”

I believe this organizational transformation is inevitable in the AI and Agent era.

1. The Problem with the Modern Data Stack: The Issue Is Not Weak Tools — It’s That Humans Spend Too Much Time Managing Complexity

Today’s data platforms are already extremely capable. Datasource management, batch synchronization, real-time CDC, SQL development, workflow orchestration, runtime logs, alerting, auditing, and lineage analysis are all widely available. But the more features platforms add, the more complex they become. Menus multiply, configurations grow deeper, and processes become longer.

Data engineers are no longer mastering tools — they are adapting themselves to tools. The once-popular Modern Data Stack essentially forced engineers to learn endless tools under the glamorous label of “Data Stack,” while in reality engineers became slaves to tools. Engineers should control tools, not endlessly relearn fragmented ecosystems.

Even a seemingly simple MySQL-to-Snowflake synchronization task may involve source schemas, target database/schema/warehouse/role settings, field type conversion, synchronization strategies, workflow dependencies, failure logs, downstream SQL, and reporting definitions. Even with the best visual tools, it still requires multiple drag-and-drop operations and configuration steps.

The real burden is not that any single technical challenge is difficult. The real burden is excessive context switching. Datasources live in one system, task configurations in another, scheduling elsewhere, logs elsewhere, SQL in Git or local files, and Snowflake execution results in cloud environments.

In the past, there was no better way, so humans had to do everything manually.

But once engineering AI systems like Codex and Claude Code emerged, many decisions became processable by large language models. Tiny repetitive actions became decomposable, callable, executable, and feedback-driven automatically. That made the emergence of the Data Engineering Harness possible.

A Data Engineering Harness is not simply another data platform. It is a data engineering capability framework designed specifically for AI systems and engineering agents.

It encapsulates datasource management, synchronization, CDC, SQL development, orchestration, log diagnostics, permission auditing, observability, cost governance, and human takeover mechanisms into engineering capabilities that Codex/Claude Code can invoke, humans can review, and enterprises can govern.

In other words, the Harness is not solving the question: “Can AI write SQL?” It is solving questions like:

After AI writes SQL, can it run safely?
After AI creates tasks, can they be audited and tracked?
After AI invokes Snowflake, can permissions and costs be controlled?
After AI generates workflows, can humans understand, confirm, and take over?

Therefore, the value of a Data Engineering Harness is not replacing data engineers, nor simply replacing data platforms. It upgrades data engineering from “humans manually operating tools” into “humans define goals, Codex executes tasks, platforms provide boundaries, and enterprises accumulate engineering know-how.”

2. Why Not Let Codex Directly Write Scripts?

Many people ask: if Codex can write SQL, Python, and invoke command lines, why do we still need a Data Engineering Harness? Why not simply let it connect directly to MySQL and Snowflake and generate scripts automatically?

This may work in personal experiments, but it fails in enterprise data engineering.

Enterprise data engineering is not simply “making a script run.” Production-grade systems require manageability, auditability, operations, collaboration, and governance. At minimum, enterprises must answer questions such as:

How do we restrict Codex/Claude Code behavior across development and production environments to avoid catastrophic actions?
How can runtime failures be interpreted and corrected automatically by AI?
How can other people, agents, or tools understand the generated engineering workflows?
Can failed tasks recover automatically through retries, checkpoint resume, or reruns?
Will table modifications affect downstream systems?
Can DAG dependencies be visualized?
Can synchronization, ETL, and Data Mapping processes be visually represented?
Who audits incidents when problems occur?

If AI generates temporary scripts every time, we simply replace “humans writing scripts” with “AI generating scripts.” Short-term productivity improves, but long-term technical debt explodes: inconsistent styles, unclear permissions, nonstandard logs, uncontrolled failures, and untraceable operations.

Eventually, data engineering falls back into the “Shell + Crontab era.”

That is why the future of enterprise AI data engineering is not about letting Codex run freely. It is about giving Codex clear engineering boundaries.

That is the true meaning of the Data Engineering Harness, and also the reason I designed the WhaleStudio Harness Suite. Harness does not restrict Codex or Claude Code — it makes them observable, manageable, and production-ready.

3. Data Engineering Harness Design Philosophy

Future Data Engineering Harness systems will no longer be traditional human-centered development platforms. They will become Harness & Skill suites designed specifically for Codex, Claude Code, and Agentic development.

Take WhaleStudio Harness Suite as an example. Previously:

Apache DolphinScheduler solved orchestration problems;
Apache SeaTunnel solved multi-datasource synchronization and CDC problems;
WhaleStudio integrated these capabilities into an all-in-one enterprise platform.

But in the era of large models and Codex/Claude Code, providing GUI interfaces for humans is no longer sufficient.

Future systems must simultaneously allow humans to review and take over, while enabling Codex/Claude Code to invoke, debug, and receive feedback through CLI interfaces and engineering contexts.

This means WhaleStudio must reorganize the core capabilities of DolphinScheduler and SeaTunnel — including orchestration, synchronization, CDC, SQL tasks, runtime execution, diagnostics, auditing, and observability — into an engineering capability layer that agents can invoke and debug, engineers can rapidly review, and enterprises can govern.

This is not about adding an “AI button” or chatbot onto old platforms. It is about redesigning software interaction models around agents as primary users.

From underlying engines to development feedback systems, every layer must become understandable, callable, observable, and controllable by AI systems.

Future data engineering platforms will not simply be feature collections. They will become containers for enterprise data engineering know-how.

Scheduling strategies, synchronization experience, SQL migration expertise, Snowflake/cloud warehouse cost optimization strategies, release workflows, and exception handling rules should all become part of Harness Memory and Skills. Codex/Claude Code should invoke not raw APIs, but proven enterprise engineering capabilities.

4. UI Will Not Disappear — It Will Become an Observability & Fine-Tuning Interface

Some people believe AI will make enterprise software UI irrelevant.

I disagree.

UI will not disappear, but its role will change. Previously, UI was the operational entry point: humans created datasources, configured tasks, dragged DAGs, scheduled workflows, and inspected logs.

In the future, many actions will be completed by Codex/Claude Code. But humans must still clearly understand what the agent created, which datasources were used, which Snowflake schemas were modified, which SQL changed, whether DAG dependencies are valid, why tasks failed, whether downstream systems are impacted, and whether human takeover is needed. Teams also need collaboration.

Nobody wants to read another person’s AI prompt history just to understand an engineering workflow. This creates demand for Observability + Fine-Tuning Interfaces.

Future UI systems will no longer focus on step-by-step manual operations. Instead, they will help humans review, fine-tune, and build trust in AI-generated engineering workflows.

UI should visualize execution plans, SQL diffs, DAG dependencies, runtime states, failure logs, and cost risks.

In short:

CLI is for Codex execution.
GUI is for human review.
Harness connects both worlds.

The best future UI may not even be static pages. It may dynamically generate review interfaces around specific engineering actions: SQL migration diffs, synchronization confirmation, DAG risk analysis, cost estimation, and deployment approvals.

UI becomes the trust layer between humans and AI-generated engineering systems.

5. Future Data Engineers: From Tool Operators to Engineering Commanders

Data engineers will not disappear. But they will diverge into two categories.

One group will remain tool operators: configuring platforms, editing SQL, checking logs, and manually dragging DAGs. These skills still matter, but they will increasingly be automated by agents.

The other group will move upward: understanding business goals, designing data models, governing cloud warehouse costs, understanding orchestration/synchronization/CDC relationships, and encoding team experience into Harness systems.

Future elite data engineers may not be the people who know the most tools. They will be the people who best organize engineering capabilities.

They will know:

what can be automated;
what requires human confirmation;
what should become Harness rules;
and what should remain human judgment.

In the past, data engineers revolved around tools. In the future, tools, Codex/Claude Code, and cloud capabilities will revolve around engineering goals.

Conclusion: The Future of Data Engineering Is Not Humanless — Humans Finally Move to a Higher Level

In the future, engineers who only know how to manually operate Modern Data Stack tools may become obsolete, just like developers who only know how to manually write Java code.

But engineers who understand business, data engineering, cloud warehouses, AI workflows, and Harness systems will become increasingly valuable.

And this is not some distant vision.

In one of my experimental demos, I already completed an entire MySQL-to-Snowflake ETL pipeline with automated SQL orchestration creation in just 10 minutes using Codex and WhaleStudio Harness.

Through CLI-based capabilities, the system automatically identified datasources, created synchronization tasks, generated visual DAGs, executed workflows, inspected logs, converted SQL into Snowflake-compatible pipelines, debugged runtime failures, and corrected issues automatically.

Through this demo, you can experience how future data engineers may work.

The next decade of data engineering will not be about adding more tools. It will be about AI deeply integrating into tools, understanding goals, respecting boundaries, and operating under human review. And that is what Data Engineering Harness truly means.

Building Metadata Capabilities in Apache SeaTunnel: A Committer’s Journey

Apache SeaTunnel — Thu, 28 May 2026 09:34:59 +0000

Recently, Apache SeaTunnel welcomed several talented and highly motivated new Committers, and Wang Xuepeng is one of them.

As a long-time contributor, Wang Xuepeng’s promotion to Committer was no coincidence. Over the years, he has quietly contributed a tremendous amount to the community, and everyone has witnessed his dedication. From first stepping into the open-source world to becoming a Committer of an Apache top-level project, he has accumulated plenty of stories and valuable insights along the way.

What inspired his journey? What experiences and lessons does he want to share with the community? Let’s take a closer look at this exclusive interview with him!

Personal Introduction

Interview Transcript

How long have you been involved in open source? What attracts you to open source?

I started getting involved in open source in 2023. What attracts me most is the sense of achievement when the code I write can actually be used within the industry.

When did you start contributing to SeaTunnel? What was the trigger?

I joined WhaleOps in 2023, which was also when I first started engaging with open source.

Now that you’ve been elected as a SeaTunnel Committer, could you summarize your contributions to the community, including both code and non-code contributions?

Most of my major feature PRs have focused on building SeaTunnel’s metadata capabilities.

When running SeaTunnel jobs and writing job configurations, users often need to manually enter datasource connection information. For file-based tasks, users also need to manually define field mappings. To address these issues, I designed an SPI interface called MetadataProvider.

The interface mainly exposes two methods:

Map<String, Object> datasourceMap(String connectorIdentifier, String metaDataDatasourceId);
Optional<TableSchema> tableSchema(String metaDataTableId);

Previously, some users in the community mentioned that datasource usernames and passwords were stored in Nacos with read-only access permissions. In scenarios like this, users can implement a custom metadata center to better protect sensitive connection information.

Community Contribution Summary

PR Link	Description
https://github.com/apache/seatunnel/pull/5663	Added `save_mode` functionality to SeaTunnel
https://github.com/apache/seatunnel/pull/10402	Integrated Gravitino with SeaTunnel
https://github.com/apache/seatunnel/pull/10586	Designed the metadata SPI interface for SeaTunnel
https://github.com/apache/seatunnel/pull/10657	Enhanced the metadata SPI interface for SeaTunnel
https://github.com/apache/seatunnel/pull/10838	Added dynamic metadata functionality based on the metadata SPI interface

After contributing to SeaTunnel for so long, you must have developed a deep understanding of both the project and the community. Compared with competing products, what do you think are SeaTunnel’s strengths and weaknesses? What keeps you actively involved in the community?

One major advantage of SeaTunnel is the flexibility of its engine choices. Teams already familiar with Flink or Spark can adopt it with a very low learning curve. For lightweight data synchronization scenarios, the Zeta engine is an even better choice.

As for weaknesses, I think the web platform still has a lot of room for improvement.

What attracts me most to the SeaTunnel community is the opportunity to discuss implementation solutions with talented contributors from different technical fields. It helps me improve my own skills while broadening my perspective.

Have you ever done any secondary development based on SeaTunnel’s shortcomings? Have you contributed those improvements back to the community? Could you briefly introduce your solution?

Yes, I’ve done secondary development for SeaTunnel. Most of the time, when I encounter bugs during usage, I first fix them in our company repository and then submit the same fixes back to the open-source community.

What kind of support do you hope the SeaTunnel community can provide for your personal growth in the future?

As long as people actively participate in community discussions — whether it’s creating issues, submitting PRs, or reviewing PRs — they will definitely improve their technical abilities.

What does the Committer role mean to you? What responsibilities should a Committer take within the community?

I believe a Committer should first ensure code quality. Secondly, Committers should help guide the community in a positive direction, such as mentoring newcomers on how to submit PRs.

Now that you’ve become a Committer, what would you like to say to the community? Do you have any suggestions for the project’s future development?

First of all, I’m very happy to become a Committer. It means becoming part of the Apache Foundation, which is truly a valuable identity and experience. I also want to thank all the community members who guided and helped me along the way.

What are your future plans in the community to further promote the project’s development?

I will continue contributing in the metadata area, and in the future, I plan to expand further into data lineage capabilities.

DEV Community: Apache SeaTunnel

How Basic Auth Works in Apache SeaTunnel Zeta Engine — Fix 401 Unauthorized API Errors

1. The Symptom: Receiving 401 When Calling Zeta REST APIs

2. Source Code Deep Dive: How BasicAuthFilter Intercepts Requests

3. Configuration Breakdown: How enable-basic-auth, username and password Take Effect

4. Client Implementation: Connect via the Authorization Header

Summary

Debugging a Distributed Job Stuck in CANCELING in Apache SeaTunnel

Background

The Symptom

First Suspicion: The Cancel Request Path

The More Important Path: Worker-to-Master Notification

Where JobNotFoundException Comes From

Why I Added Force Stop Instead of Rewriting Cancel

Cancel vs Force Stop

Cancel

Force Stop

What I Learned

Conclusion

Tutorial: Incremental MySQL-to-Doris Synchronization with Apache SeaTunnel and Apache DolphinScheduler

1. Environment Setup

2. Preparing Test Data

3. Incremental Synchronization Design

4. Configuring the SeaTunnel Job

Configure the JDBC Source

Configure Parallelism

Configure Partitioned Reads

Configure Fetch Size

5. Configuring the Doris Sink

Automatic Table Creation

Configure Write Mode

Enable Two-Phase Commit

Performance Optimization Parameters

6. Configuring the DolphinScheduler Runtime Environment

Create a Tenant

Create a User and Associate It with the Tenant

Create an Environment

7. Creating the Project and Workflow

8. Configuring the SQL Task to Retrieve the Synchronization Checkpoint

Configure an OUT Parameter

9. Incremental Synchronization Workflow Logic

10. Conclusion

What It Takes to Become an Apache SeaTunnel Committer: Doyeon Kim's Story

Personal Introduction

Full Interview Transcript

Goodbye Data, Hello AI: My Biggest Takeaway from Snowflake Summit 2026

1 Snowflake's Transformation: From Data Warehouse to AI Platform

2 CoCo, CoWork, and Desktop: Snowflake Is "Paying Tribute" to Anthropic—And Revealing a Bigger Ambition

3 AI Is Bringing Every Software Company Back to the Same Starting Line

4 Looking Back at Ourselves Through Snowflake: How Do I Goodbye Data, Hello AI?

5 My Final Prediction: Snowflake Is Competing for the AI Entry Point, and If It Succeeds, Its Stock Could Rise Far Beyond 2×

Apache SeaTunnel Monthly Engineering Digest (May 2026): 87 PRs, Stronger Connectors, Smarter Recovery

1. Overall Snapshot

1.1 Four-Dimensional Statistics (87 PRs)

1.2 Module Distribution

2. Feature Enhancements and Engineering Evolution

2.1 Connector-V2: HTTP Source Now Supports Binary Downloads (#10956)

2.2 Engine/Zeta: Progressive Validation for Dry-Run Mode (#10763)

2.3 Ecosystem Expansion and Tooling Improvements

3. Performance Optimization and Resource Risk Mitigation

3.1 Kudu: Dependency Upgrade Resolves Flink 1.15+ Compatibility Risks (#10974)

3.2 Kafka: Reduce Default Cache Queue Size from 1024 to 2 to Prevent OOM (#10954)

4. Bug Fixes and Architectural Improvements Defined May

4.1 Zeta: Master Failover Could Permanently Stall Jobs (#10836)

4.2 Security Fix: Path Traversal Vulnerability in Log REST API (#10628)

4.3 Recovery and Race Condition Improvements (#10687 / #10842 / #10877)

5. Deep Dive into Key Technical Changes

5.1 HTTP Binary Download Support (#10956): Design, Semantics, and Impact

5.2 SQLite Upsert Syntax Fix (#10880): Why EXCLUDED Matters

5.3 Milvus Exception Reference Fix (#10975): A Small Change with Large Operational Value

5.4 Zeta Flaky Test Fix (#10891): Eliminating Timing Dependencies

6. Reproducible Framework for Performance and Stability Validation

Thanks to All Contributors

Save Your Seat: How HOPEGOO Built a Unified Multimodal Data Platform with Apache SeaTunnel

Featured Speaker

Background and Session Overview

Event Information

Community Giveaways

Reserve Your Spot Today

Demo: Full Data Synchronization from MySQL CDC to PostgreSQL with Apache SeaTunnel

5.2 SQLite Upsert Syntax Fix (#10880): Why `EXCLUDED` Matters