DEV Community: Apache SeaTunnel

Why I Reworked a BigQuery Sink from Pending Streams to Buffered Streams

Apache SeaTunnel — Tue, 22 Sep 2026 10:00:24 +0000

While working on the BigQuery Sink Connector for Apache SeaTunnel, I initially thought the main challenge would be straightforward: write rows into BigQuery correctly.

But as the implementation evolved, I realized the real problem was deeper than just calling the BigQuery API. The hard part was aligning BigQuery’s external write visibility model with SeaTunnel’s checkpoint and restore lifecycle.

This post summarizes the design issue I encountered while using the BigQuery Storage Write API, why the initial pending-stream design was not strong enough for checkpoint recovery, and why I eventually redesigned the batch write path around buffered streams.

Initial Design: Pending Streams for Batch Writes

BigQuery Storage Write API provides several write stream types. At first, Pending Stream looked like a natural fit for batch writes.

The flow is roughly:

Create PENDING stream
AppendRows
FinalizeWriteStream
BatchCommitWriteStreams

Data written to a pending stream is not visible to readers until BatchCommitWriteStreams is called. Since BigQuery can commit multiple pending streams atomically, this looked like a good match for a checkpoint-based sink commit protocol.

The initial implementation looked roughly like this:

@Override
public Optional<BigQueryCommitInfo> prepareCommit() {
    flush();
    streamWriter.finalizeStream();
    return Optional.of(new BigQueryCommitInfo(streamWriter.getStreamName()));
}
@Override
public List<Void> snapshotState(long checkpointId) {
    this.streamWriter.close();
    this.streamWriter = BigQueryBatchWriter.of(client, config);
    return Collections.emptyList();
}

At first glance, this looked similar to a two-phase commit protocol.

prepareCommit()  -> finalize stream
commit()         -> BatchCommitWriteStreams

However, during review, a subtle but important problem surfaced.

The Problem: Ownership Gap After a Failed Checkpoint

In SeaTunnel’s sink lifecycle, prepareCommit() is called before the checkpoint is fully completed. The issue is that prepareCommit() already performs an external side effect by calling FinalizeWriteStream.

Consider this scenario:

1. Checkpoint N starts.
2. prepareCommit() finalizes stream A.
3. commitInfo(stream A) is created.
4. The job fails before checkpoint N completes.
5. The job restores from the last 
completed checkpoint N-1.

After restore, restoreWriter(states)receives only the state of the last completed checkpoint. Failed checkpoint state is discarded.

This means stream A, which was finalized during the failed checkpoint N, is not part of the restored state.

That is the core issue.

FinalizeWriteStream has already been called,
but the checkpoint does not contain a durable decision
about whether this finalized stream should be committed,
abandoned, or cleaned up.

At first, I thought that since pending-stream data is invisible before BatchCommitWriteStreams, abandoning the stream might be acceptable. However, that explanation was not strong enough. To claim clear checkpoint recovery semantics, the connector needs a more explicit story about external side effects.

Why restoreWriter() Should Not Commit Failed-Checkpoint Streams

One seemingly simple solution is to discover and commit the finalized pending stream during restore.

But this is unsafe.

Remember that restore starts from the last completed checkpoint, N-1. Therefore, records that were processed during checkpoint N may be replayed after restore.

If restoreWriter() discovers stream A and commits it, the following can happen:

records in stream A become visible in BigQuery
+
the same records are replayed after restore from checkpoint N-1
+
the replayed records are written again and committed later

This could produce duplicate rows if the connector commits a stream that belongs to a failed checkpoint.

So the important point is:

restoreWriter() receives only successful checkpoint state.
A stream finalized by a failed checkpoint is not part of that state.
If we try to commit such an external stream during restore,
we may publish records that the engine is about to replay.

Therefore, streams finalized by failed checkpoints should not be blindly committed during restore.

The issue is not that Pending Streams always cause duplicates; the issue is that the connector has no durable decision for finalized-but-uncommitted streams created by failed checkpoints.

Redesign: Switching to Buffered Streams

To avoid this ownership gap, I redesigned the batch write path to use Buffered Streams instead of Pending Streams.

Buffered Streams do not commit the whole stream through a finalize-and-batch-commit protocol. Instead, they make data visible up to a specific stream offset.

The flow looks roughly like this:

Create BUFFERED stream
AppendRows(offset=N)
FlushRows(offset=M)

Data appended to a buffered stream is not immediately visible. It becomes visible only after FlushRows advances visibility up to a specific offset.

This model maps much better to checkpoint-based recovery.

A checkpoint represents the processing position of the engine. A buffered stream offset represents the external write position in BigQuery.

The new design looks like this:

writer state:
  streamName
  nextOffset
  checkpointId
prepareCommit(checkpointId):
  flush()
  return BigQueryCommitInfo(streamName, flushOffset = nextOffset - 1)
snapshotState(checkpointId):
  return BigQuerySinkState(streamName, nextOffset, checkpointId)
commit(commitInfo):
  FlushRows(streamName, flushOffset)
restoreWriter(states):
  select the latest completed checkpoint state
  restore writer with streamName + nextOffset

Now the external write position is represented by:

streamName + nextOffset

This position is stored in checkpoint state. After restore, the writer can resume from the last completed checkpoint’s external write position.

Understanding State and CommitInfo

One important lesson was that writer state and commit info are related, but they mean different things.

Write on Medium
The writer state represents where the writer should resume appending after restore.

BigQuerySinkState {
    String streamName;
    long nextOffset;
    long checkpointId;
}

The commit info represents what should become visible in BigQuery after the checkpoint completes.

BigQueryCommitInfo {
    String streamName;
    long flushOffset;
}

For example:

writer state:
  streamName = S
  nextOffset = 100
commit info:
  streamName = S
  flushOffset = 99

This means:

The writer has appended rows up to offset 99.
If the checkpoint completes, BigQuery should flush visibility up to offset 99.
If the job restores from this checkpoint, the writer should resume from offset 100.

This separation made the recovery model much clearer.

Managing Offsets Correctly

The most important detail in the buffered-stream design is offset management.

The offset is not a record identifier. It is not a primary key. It is also not a checkpoint id.

The offset is the append position inside a specific BigQuery write stream.

If there are multiple parallel writers, each writer should own its own stream, and each stream should manage its offset independently.

For example:

writer-0:
  stream S0
  nextOffset = 100
writer-1:
  stream S1
  nextOffset = 250

Even if the checkpoint id is 10, the BigQuery append offset should not be 10. The checkpoint id is only metadata. The BigQuery offset must represent the append position inside that stream.

So the model is:

checkpointId:
  used to identify and select checkpoint state
nextOffset:
  used as the next append position in the BigQuery stream
flushOffset:
  nextOffset - 1, used by FlushRows

This distinction is important, especially when the sink runs with parallelism.

Committer Uses FlushRows Instead of BatchCommitWriteStreams

With Pending Streams, the committer used BatchCommitWriteStreams.

With Buffered Streams, the committer uses FlushRows.

FlushRowsRequest request =
        FlushRowsRequest.newBuilder()
                .setWriteStream(info.getStreamName())
                .setOffset(Int64Value.of(info.getFlushOffset()))
                .build();
FlushRowsResponse response = client.flushRows(request);

This makes rows visible up to the requested offset.

So after a checkpoint completes, the committer advances BigQuery visibility up to the checkpoint’s flushOffset.

Keeping CDC Separate

This redesign only applies to the batch write path.

I did not change the CDC path to use Buffered Streams. BigQuery CDC ingestion has its own semantics around _CHANGE_TYPE, _CHANGE_SEQUENCE_NUMBER, and primary keys. It should not be forced into the same checkpoint-offset model as batch writes.

The resulting structure is:

batch mode:
  Buffered Stream
  streamName + nextOffset in checkpoint state
  FlushRows on commit
cdc / streaming mode:
  existing streaming path
  BigQuery CDC semantics

This keeps the batch recovery model explicit without changing CDC behavior unnecessarily.

Testing Challenges

This change depends on real BigQuery Storage Write API behavior, especially Buffered Streams, explicit offsets, and FlushRows. A local emulator was not sufficient to validate these semantics reliably.

I tested the updated batch path against a real BigQuery environment.

The verified scenarios included:

- creating a Buffered Stream
- appending rows with explicit offsets
- running a batch write with checkpoint enabled
- verifying that rows become visible in BigQuery after FlushRows

A fully deterministic failure-recovery E2E test is much harder. It would require controlling checkpoint barrier timing, injecting failure at a precise point, restoring the job, and verifying BigQuery visibility and duplicates.

For automated tests, a more practical approach is to cover the recoverable metadata path:

- BigQuerySinkState serialization
- selecting the latest state by checkpointId
- advancing nextOffset only after append success
- creating FlushRows commit info

This does not replace a full failure-recovery E2E test, but it validates the most important internal recovery metadata.

Lessons Learned

The biggest lesson I learned is that connector exactly-once semantics are not just about calling a commit API.

A connector must answer questions like:

1. When does data become visible in the external system?
2. What happens to external side effects if a checkpoint fails?
3. What external write position should the writer restore from?
4. What does writer state mean?
5. What does commitInfo mean?
6. How should offsets behave when append fails and retries happen?

The initial Pending Stream design looked natural from BigQuery’s batch commit perspective. But when combined with SeaTunnel’s checkpoint and restore lifecycle, it created an ownership gap for finalized streams after failed checkpoints.

The Buffered Stream design is more complex, but it gives the connector a clearer recovery model by storing the external write position as streamName + offset.

Closing Thoughts

This work was not just about adding a BigQuery Sink Connector.

It was about aligning an external system’s write visibility model with a stream processing engine’s checkpoint lifecycle.

The first design was not perfect. Review revealed an important failure scenario. But by understanding the issue, revisiting BigQuery stream types, and redesigning the batch path around Buffered Streams, the connector’s recovery semantics became much clearer.

Open source review can be painful, but it often forces us to think beyond whether the code works in the happy path. It pushes us to reason about boundaries, failure windows, and system guarantees.

For me, this BigQuery Sink work was exactly that kind of experience.

How Apache SeaTunnel Handles Kafka Topic Partition Expansion Without a Job Restart

Apache SeaTunnel — Fri, 18 Sep 2026 09:16:22 +0000

1. Scenario

When using an Apache SeaTunnel streaming job to consume data from a Kafka topic, what happens if the topic is expanded from 2 partitions to 4?

By default, the new partitions are not consumed, and none of the data written to them will be processed unless the SeaTunnel job is restarted.

The reason is that SeaTunnel’s Kafka Source scans the available partitions when the job starts. Without additional configuration, it does not automatically detect partitions added later.

Restarting the job can force SeaTunnel to scan the topic again, but this approach introduces several issues:

The entire job is paused during the restart. If checkpoints are used for recovery, there is also a risk of missing data depending on the recovery and offset state.
When start_mode = EARLIEST, restarting may cause the newly assigned partitions to be consumed from the beginning, which can replay historical data and result in duplicate records.

SeaTunnel provides the partition-discovery.interval-millis parameter specifically for this scenario. It periodically scans Kafka for newly added partitions and automatically adds them to the consumer, so the streaming job does not need to be restarted.

2. Hands-On Guide

2.1 Prerequisites

A Kafka cluster
Apache SeaTunnel 2.3.12

2.2 Create a Topic with Two Partitions

Create a Kafka topic named ksource with two partitions:

bin/kafka-topics.sh --create --topic ksource \
  --bootstrap-server ip:9092 \
  --partitions 2 --replication-factor 1

2.3 Configure the SeaTunnel Streaming Job

Create a SeaTunnel configuration file at job/k.conf.

There are three key settings to pay attention to:

Set job.mode to STREAMING.
Enable checkpointing explicitly with checkpoint.interval.
Set partition-discovery.interval-millis to a positive value.

env {
  parallelism = 2
  job.mode = "STREAMING"
  checkpoint.interval = 5000
}

source {
  Kafka {
    bootstrap.servers = "ip:9092"
    topic = "ksource"
    consumer.group = "seatunnel_k_group"
    partition-discovery.interval-millis = 5000
    start_mode = "EARLIEST"
    format = "json"
    schema = {
      fields {
        id = INT
        cusname = STRING
        amount = DOUBLE
      }
    }
  }
}

sink {
  Console {}
}

The key settings are:

partition-discovery.interval-millis = 5000: SeaTunnel scans Kafka for new partitions every 5 seconds.
start_mode = EARLIEST: When a new partition is discovered, SeaTunnel starts consuming from the earliest available offset.
Schema field types such as INT, STRING, and DOUBLE are basic types and can be specified directly.

2.4 Start the Job and Verify Normal Consumption

Start the SeaTunnel job in local mode:

bin/seatunnel.sh --config job/k.conf -m local

Open another terminal and produce three messages to ksource. Because the topic currently has only two partitions and no key is specified, the messages will be distributed across partitions 0 and 1:

bin/kafka-console-producer.sh --broker-list ip:9092 --topic ksource

Enter the following messages, one per line:

{"id":1,"cusname":"Zhang San","amount":100.50}
{"id":2,"cusname":"Li Si","amount":200.00}
{"id":3,"cusname":"Wang Wu","amount":300.00}

The corresponding records should appear in the SeaTunnel job console, confirming that the source is consuming data normally.

2.5 Expand the Topic from 2 Partitions to 4

Now increase the number of partitions from 2 to 4:

bin/kafka-topics.sh --alter --topic ksource \
  --bootstrap-server ip:9092 \
  --partitions 4

Verify the result:

bin/kafka-topics.sh --describe --topic ksource \
  --bootstrap-server ip:9092

The topic should now contain four partitions: 0, 1, 2, and 3.

2.6 Produce More Data and Check the New Partitions

Continue producing six messages to ksource. Because no key is specified, Kafka's default partitioning behavior distributes the records across all four partitions. Some of them should therefore be written to the newly added partitions 2 and 3.

bin/kafka-console-producer.sh --broker-list ip:9092 --topic ksource

Enter:

{"id":4,"cusname":"Zhao Liu","amount":400.00}
{"id":5,"cusname":"Sun Qi","amount":500.00}
{"id":6,"cusname":"Zhou Ba","amount":600.00}
{"id":7,"cusname":"Wu Jiu","amount":700.00}
{"id":8,"cusname":"Zheng Shi","amount":800.00}
{"id":9,"cusname":"Qian Shiyi","amount":900.00}

The six records should gradually appear in the SeaTunnel console, including those written to the newly added partitions 2 and 3.

The job continues running without a restart, and data from the new partitions is consumed automatically.

2.7 Comparison: What Happens Without Dynamic Partition Discovery?

Now remove partition-discovery.interval-millis from the configuration and repeat Steps 2.4 through 2.6.

The results are:

Messages produced before the topic expansion: consumed normally.
Messages produced after the expansion and written to new partitions 2 and 3: not consumed.
Restarting the SeaTunnel job: the new partitions are detected and their data can then be consumed.

The difference between the two configurations is summarized below:

3. Important Considerations

3.1 The Starting Offset of New Partitions Depends on `start_mode`

New partitions do not have existing checkpoint offsets. The starting position therefore depends on the configured start_mode.

With EARLIEST, SeaTunnel consumes the new partition from its earliest available offset. If the partition already contains historical data, that data will be consumed as well.

With LATEST, SeaTunnel only consumes new records written after the partition is discovered.

3.2 Avoid Setting the Discovery Interval Too Low

Each discovery scan sends metadata requests to the Kafka cluster. A shorter interval allows SeaTunnel to detect partition changes more quickly, but it also increases the request frequency and therefore the load on Kafka.

A 5-second interval is a reasonable starting point. If partition changes are infrequent, you can consider increasing the interval to 30–60 seconds.

3.3 Kafka Partitions Can Only Be Increased, Not Decreased

Kafka supports increasing the number of partitions for a topic, but does not support reducing the partition count. Reducing partitions could result in data loss.

3.4 Partition Expansion Triggers Consumer Group Rebalancing

When the number of partitions changes, Kafka rebalances the consumer group and redistributes partitions among consumers.

A brief fluctuation in consumption during the rebalance is expected behavior.

3.5 Existing Partitions Are Not Affected

Dynamic partition discovery only handles newly added partitions.

Existing partitions continue consuming from their checkpointed offsets, so enabling partition discovery does not cause SeaTunnel to reprocess data from existing partitions.

3.6 Types with Precision or Generic Parameters Must Be Quoted

Types containing commas or angle brackets must be enclosed in double quotes, for example:

"decimal(10,2)"
"array<int>"
"map<string, int>"

Otherwise, the HOCON parser will report a syntax error.

4. Summary

The partition-discovery.interval-millis parameter solves a specific problem: automatically detecting newly added Kafka partitions while a SeaTunnel streaming job is already running.

Set the parameter to a positive interval, and SeaTunnel periodically checks Kafka for new partitions and automatically starts consuming from them without requiring a job restart.

There are three key points to remember:

Streaming mode + checkpointing + a positive partition discovery interval.

The starting offset for a newly discovered partition depends on start_mode, while the discovery interval should be chosen based on the balance between detection latency and Kafka cluster load.

With this configuration in place, expanding a Kafka topic no longer requires manually restarting the SeaTunnel job just to make the new partitions visible to the consumer.

🚀 Move Redis data without writing migration scripts! Explore how Apache SeaTunnel handles String, Hash, Set, and ZSet with practical Redis-to-Redis configs. #ApacheSeaTunnel #Redis #DataMigration #DataEngineering

Apache SeaTunnel — Fri, 11 Sep 2026 07:28:22 +0000

Apache SeaTunnel

Sep 11

Redis Data Migration Without Scripts: How Apache SeaTunnel Handles String, Hash, Set, and ZSet

#programming #apacheseatunnel #redis #datascience

5 min read

Apache SeaTunnel had a busy August: 195 PRs merged, 13 new connectors added, and major upgrades to Zeta observability, CDC, Checkpoint recovery, and CI. #ApacheSeaTunnel #DataEngineering #OpenSource

Apache SeaTunnel — Fri, 11 Sep 2026 07:27:38 +0000

Apache SeaTunnel

Sep 11

195 Merged PRs, 13 New Connectors: What’s New in Apache SeaTunnel This August?

#apacheseatunnel #programming #datascience #bigdata

10 min read

195 Merged PRs, 13 New Connectors: What’s New in Apache SeaTunnel This August?

Apache SeaTunnel — Fri, 11 Sep 2026 07:19:40 +0000

Apache SeaTunnel continued its rapid pace of development in August! The community merged a total of 195 PRs, with contributions from 60 contributors and 13 new connectors added to further expand the data source ecosystem. Zeta observability also received a major upgrade, with continued improvements to dynamic logging, Worker resource monitoring, Job status tracking, and more. Meanwhile, several capabilities, including Checkpoint recovery, CDC Schema Change, CI Merge Queue, and declarative OptionRule validation, were optimized to deliver greater stability, observability, and operational efficiency for data integration workloads.

Let’s take a closer look at the updates that stood out in Apache SeaTunnel this August!

1. Merged PRs This Month

1.1 Overall Statistics

Metric	Value	Description
Successful Merge Commits	195	Includes automatic merges by Merge Queue and manual merges, consistent with independent verification via `git log --oneline`
Estimated Merged PRs	180+	Some PRs contain multiple commits
Average Merges per Day	7.0	28 merge days in the month; 195 / 28 = 6.96 ≈ 7.0
Daily Peak Merges	20	Monthly peak reached on August 23
Net Code Changes	+114,181	+132,340 additions / -18,159 deletions; reporting convention retained. Independent recalculation: 134,165 / 18,143 / net +116,022
Unique Files Changed	1,687	—

1.2 PR Classification

What the numbers tell us: Bug fixes accounted for the largest share of PRs at 35.7%, reflecting the community’s continued focus on stability. New feature development contributed the largest volume of code changes, with +76,352 net lines added, accounting for approximately 66.9% of the total. Much of this came from 13 new connectors and enhancements to core APIs. The 37 documentation PRs (18.9%) also marked a new monthly high for the year.

2. Top 25 Contributors

3. Code Change Analysis

3.1 Overall Code Volume

Metric	Value	Notes
Total lines added	+132,340	Independently recalculated with `git log --numstat \
Total lines deleted	-18,159	Same as above
Net lines added	+114,181	Feature: +76,352; other categories: +37,829
File change references	2,187	Includes duplicates; files modified by multiple commits are counted multiple times
Unique files changed	1,687	Independently verified with {% raw %}`sort -u \
Average changes per commit	+675/-93 lines, 11.2 files	The average was pushed up by large PRs such as #11413 HugeGraph

3.2 Distribution by Technical Area

Based on actual net lines added by category
{% raw %}

┌──────────────────────────────────────────────────────────────┐
│ New Features [Feature]   ████████████████████████████████ 66.9% │ +76,352 net
│ Bug Fixes [Fix]/[Bug]    ████████████████                  14.8% │ +16,859 net
│ Improvements [Improve]   ████████                           6.6% │  +7,478 net
│ Documentation [Doc]/[Docs] ██████████                       9.5% │ +10,846 net
│ Tests/CI                  ██                                0.4% │    +472 net
│ Other                     ██                                1.9% │  +2,174 net
└──────────────────────────────────────────────────────────────┘

3.3 Major Refactoring and Architectural Changes

Change	Files Affected	Change Volume	Scope	Risk Level	Verified Contributors
seatunnel-shade module refactoring (#9993)	116	+1,042/-5,890	Third-party dependency shading and uber JAR build process across all Shade packaging	⚠️ Medium-High	hawk9821
Declarative OptionRule migration across 10+ connectors	~150 (estimated)	~+3,500/-1,500 (estimated)	Configuration validation for Pulsar/MongoDB/HDFS/S3/Iceberg/HBase/Cassandra/ActiveMQ connectors	✅ Low (backward-compatible and non-breaking; validation is stricter, so previously accepted boundary configurations may now fail at startup instead of failing silently)	Multi-contributor effort (itzrohan007/amanbbdniit.mishra/Linz1248/nikk4645/zhang-arvin/claire040217, etc.)
Zeta REST API v2 expansion (#11984/#11909/#11982)	57 (estimated across 3 PRs)	+3,102/-62 (estimated)	`/api/v2/jobs`, `/api/v2/nodes`, `/api/v2/logs`, `/api/v2/nodes/{id}/resources`, and other endpoints	⚠️ Medium (response bodies add new fields such as `stateTransitions`; clients using strict JSON Schema may need `ignoreUnknown=true`)	SEZ9 + goutamadwant
Jackson Relocate added and later reverted (#11673 → #11851)	~3 (related changes)	Corresponding additions/removals	Jackson version conflicts between `hadoop-aws` and the uber JAR (#11673 original commit: dev.loustler; #11851 revert: hawk9821)	✅ Stable after revert; current state is consistent with the reverted version	#11673 + #11851 (#11522 shenghang’s openGauss anti-hijacking fix was related)
Markdown RAG metadata alignment (#10990/#11740)	41 (2 commits combined)	+2,286/-72	File Source + Knowledge Sync end-to-end field contract	✅ Low (new fields are optional)	yzeng1618

4. Top 7 Updates Users Will Notice

✨ TOP 1: 13 New Connectors Expand the Data Source Ecosystem

Estimated total change volume: approximately +40,000 lines / 500+ files

Connector	Type	Contributor	Primary Use Case	Estimated Maturity
BosFile (Baidu Intelligent Cloud BOS)	Source + Sink	programmerloverun	Multi-cloud object storage coverage, aligned with S3/Azure/OSS	✅ Full E2E-level support, consistent with other File connectors
Azure Cosmos DB	Source	ilovezeri333 (first-time contributor)	Azure NoSQL migration into data lakes	✅
NebulaGraph	Vertex Sink	goutamadwant	Writing vertices to graph databases; Edge support is planned	⚠️ Beta (Vertex only)
SNMP (v2c/v3)	Source	goutamadwant	Collecting network device monitoring data	✅
PostHog	Source	goutamadwant	Extracting user behavior data from product analytics platforms	✅
Google Pub/Sub (Source + Sink)	Source + Sink	goutamadwant	GCP messaging events into data lakes and write-back workflows	✅
Azure Queue Storage	Sink	goutamadwant	Cloud-native asynchronous message delivery to Azure	✅
Couchbase	Sink	srijan-singh	Distributed document database integration	⚠️ Beta
DB2 CDC	Source (CDC)	davidzollo365	Incremental synchronization from IBM DB2	✅ Incremental support
PythonSource	Source	escheduler	User-defined Python-script-based data sources	✅
NATS JetStream	Sink	rucciva (first-time contributor)	Cloud-native messaging and cloud-edge collaboration scenarios	✅
HugeGraph (Source + Sink refactoring)	Source + Sink	liu7777jx (first-time contributor)	Multi-mapping support for the Baidu open-source graph database	✅ Largest single PR this month at 15,358 lines
BigQuery enhancements (Multi-Table + UniverseDomain)	Sink enhancement	merlin.launay (first-time contributor)	Writing multiple source tables to a GCP data warehouse	✅ See TOP 3

What this means for users: Cloud storage coverage now spans five major storage ecosystems: S3, Azure, OSS, BOS, and HDFS. For example, in an overseas advertising scenario, the Cosmos DB → BosFile combination can directly support compliant cross-cloud synchronization from Azure public cloud environments to domestic BOS storage. PythonSource also expands the flexibility available for custom data source scenarios.

✨ TOP 2: Major Observability Upgrades for Zeta

REST v2 + Dynamic Log Levels + Worker Resource APIs

New and enhanced endpoints: approximately 10+ REST endpoints, all covered by independent E2E tests.

Key improvements include:

Dynamic log levels: PUT /api/v2/logs/level accepts a JSON body such as logger=xxx, level=DEBUG, with the new level taking effect within 30 seconds. Production troubleshooting no longer requires a cluster restart. This behavior is deterministic based on the implementation.
Fine-grained Worker resources: GET /api/v2/nodes/{id}/resources provides real-time monitoring of CPU, heap, and thread-pool utilization.
Job state transition tracking: The response now includes a stateTransitions array, making it possible to trace the source and trigger time of each failover or manual restart.
Metrics isolation fix (#11492): In multi-Worker deployments, metrics were previously written to a global Registry, which could cause Prometheus/Grafana dashboards to display values under the wrong Worker. This has been fixed, with expected dashboard accuracy improving from approximately 70% to 95%+ (estimate).
Schema-first multi-table sink metrics alignment (#12001): Metrics from individual tables in multi-table synchronization no longer interfere with one another, which is expected to significantly improve the accuracy of per-tenant table-level metrics.

Operational impact: Combining online DEBUG-level adjustment with state-transition tracing is estimated to reduce the average time required to diagnose typical production issues by approximately 30%–60%. This is a conservative estimate based on the workflow change from “restart first, then inspect logs” to “inspect signals online and adjust logging levels,” rather than a benchmark result.

✨ TOP 3: BigQuery Sink Adds Multi-Table and UniverseDomain Support

Key improvements:

A single SeaTunnel job can now map N source tables to N corresponding BigQuery tables, eliminating the need to split the workload into N independent SeaTunnel jobs.
The bigquery-universe-domain configuration supports different Google API domains, including the standard Google domain (googleapis.com), EU domains, and private/custom domains.
Combined with Flink/Spark CDC upstream pipelines, this can provide a practical configuration for a multi-database, multi-table MySQL → BigQuery pipeline with end-to-end latency measured in minutes.

Value for users: For typical overseas retail and advertising workloads, moving from N jobs to a single multi-table job can reduce resource consumption by an estimated 30%–40% under comparable throughput. This has not been validated against a standardized benchmark and is provided for reference only. Operational complexity is reduced accordingly.

✨ TOP 4: LATEST_COMPLETED Checkpoint Recovery + Prometheus Checkpoint Flush Alignment

LATEST_COMPLETED recovery strategy (#11421): A new restore.strategy = LATEST_COMPLETED option avoids the risk of recovering from a Checkpoint that was still in progress when a failure occurred, reducing the possibility of reading incomplete state.
Prometheus Checkpoint Flush (#11827): This addresses occasional loss of data points from the final few seconds when Spark/Flink timer-driven flushes are not aligned with the Checkpoint Barrier. An enhancement in #11778 further moved the PrometheusWriter to an engine-level FlushSignal, binding the flush timing to the Checkpoint barrier.

Value: For high-consistency workloads such as financial payments and order reconciliation, the risk of data loss during Checkpoint recovery moves from an occasional issue reported by the community toward theoretical zero data loss, significantly increasing confidence in recovery consistency. This is a qualitative statement rather than a benchmark result.

✨ TOP 5: CDC COMMENT Schema Change Support + Stronger Recovery Stability

Enhancement	Problem Solved
COMMENT Schema Change event support	Changes such as `ALTER TABLE ... MODIFY COLUMN ... COMMENT 'xxx'` in MySQL/PostgreSQL/Oracle can now be correctly propagated through the CDC pipeline to downstream Iceberg/Paimon/Hive metadata, addressing delays in synchronizing column comments with downstream BI data dictionaries
Event routing fix #11837	COMMENT events were previously routed incorrectly into the DML pipeline, causing parsing errors; this has been fixed
MySQL CDC long-running E2E	Added a 12-hour fault injection → recovery → data consistency regression test case (E2E #11946, with 192 new lines)
MongoDB snapshot failure stack preservation	Previously, the exception stack could be lost; it can now be used directly to identify the root cause without rerunning the job
SQL Server CDC Resume LSN	Fixed a 0–3 record offset issue in the resume position after restart (#11410)
Oracle startup by SCN #11171	Added an option to specify the starting SCN for incremental recovery

Enterprise CDC impact: Schema evolution now covers the “last mile” for column comments, allowing metadata changes to propagate correctly through the CDC pipeline. For long-running CDC workloads, the estimated recovery success rate after 12-hour fault-injection tests improved from approximately 85%+ to over 95%. This is a conservative estimate based on CI pass-rate improvements.

✨ TOP 6: CI Merge Queue + E2E Shard Rebalancing + Hung Job Protection

Key mechanisms:

Merge Queue: Changes are first placed into a temporary queue, where the full CI suite, including high-risk E2E tests, is executed. Once all checks pass, changes are merged linearly into dev. This avoids the historical problem of the dev branch remaining red for days.
E2E shard rebalancing: The test suite was rebalanced from 7 shards to 12, reducing the longest shard from more than 90 minutes to an estimated 25–40 minutes.
Automatic timeout for hung jobs (#11718): Added protection for hung CI jobs, including the 66-minute boundary observed with Paimon S3 hung jobs.

Value: Conservatively estimated, the time contributors spend waiting for CI results is reduced by approximately 25%–35% on average. The Merge Queue design also provides a theoretical baseline stability of 95%+ for the dev branch by preventing prolonged red states.

✨ TOP 7: Declarative OptionRule Validation Expands Across 10 Connector Migration PRs

PR list: 10 independent migrations + 1 additional Iceberg alignment enhancement:

Pulsar (#11985 Linz1248) — First-time contribution
MongoDB fetch-size (#11886 amanbbdniit.mishra) — First-time contribution
HDFS + S3 sink (#11881 itzrohan007) — First-time contribution
HBase timestamp (#11803 goutamadwant)
Iceberg (#11921 claire040217) — First-time contribution + Iceberg alignment enhancement (#11675 claire040217)
Cassandra (#11964 Aryadeepta) — First-time contribution
ActiveMQ (#12004 nikk4645) — First-time contribution
RabbitMQ source (#11795)
IoTDBv2 SQL dialect (#11839)
Email (#11817)

Item	Before: Traditional Procedural Validation	After: Declarative OptionRule
Validation code location	Distributed across multiple Factory/constructor classes, typically around 100–200 lines	Centralized in OptionRule definitions, typically around 30–80 lines
Error messages	Mixed Chinese/English, NPEs, or raw stack traces	Standardized `OptionMessage` JSON: `{field, required, range...}`
SQL vs. API consistency	Implemented separately in two places	One shared OptionRule implementation
Development cost for new contributors (estimated)	Requires understanding and modifying multiple locations, with a higher risk of mistakes	Follow the template and fill in the required rules; estimated effort reduction of ≈30%–40%

What users will notice: Configuration errors can now produce clear validation messages instead of occasionally surfacing as difficult-to-understand NPE stack traces. For example, partitionDiscoveryIntervalMillis=-1 in Pulsar now returns an explicit validation error indicating that the field value is -1 and must be greater than 0. This is a deterministic improvement in configuration validation.

Appendix: Quick Reference for Key Changes

Commit Hash	PR #	Type	Summary
`96048a7b8`	#11167	Feature-New	Azure Cosmos DB Source (ilovezeri333, first-time contributor)
`a9f69848a`	#10780	Feature-New	DB2 CDC Source (davidzollo365)
`99e53aad8`	#11952	Feature-New	BosFile Source & Sink (programmerloverun)
`9f229ce3e`	#11865	Feature-New	NebulaGraph Vertex Sink (goutamadwant)
`3534417f6`	#11968	Feature-New	SNMP Source (goutamadwant)
`e352beba2`	#11998	Feature-New	PostHog Source (goutamadwant)
`c5b58851b`	#11989	Feature-New	Google Pub/Sub Source (goutamadwant)
`e97555a62`	#11877	Feature-New	Google Pub/Sub Sink (goutamadwant)
`8725e1e13`	#11939	Feature-New	Azure Queue Storage Sink (goutamadwant)
`ead62107e`	#11198	Feature-New	Couchbase Sink (srijan-singh)
`4ebb87b8f`	#11337	Feature-New	PythonSource connector (escheduler)
`a6beccf8b`	#11460	Feature-New	NATS JetStream Sink (rucciva, first-time contributor)
`543f2a5a6`	#11413	Feature-New	HugeGraph Source + Sink multi-mapping (liu7777jx, first-time contributor; largest PR this month, +14,135/-1,223)
`f0046a002`	#11649	Feature-Enhancement	BigQuery Multi-Table + UniverseDomain (merlin.launay, first-time contributor)
`9ec9c4fe8`	#11984	Feature-Zeta	Zeta dynamic log level REST v2 (SEZ9)
`0071e8fa6`	#11909	Feature-Zeta	Zeta Worker Resource REST API (goutamadwant)
`7c73866c7`	#11957	Feature-Zeta	Zeta Job Detail real-time metrics charts (lindaluo83)
`f9b1b330f`	#11963	Feature-CI	CI Merge Queue introduced (zniu70696)
`c44f5b338`	#11982	Improve-Zeta	Zeta Job `stateTransitions` exposure (SEZ9)
`2b51d3614`	#11421	Feature-Zeta	Zeta `LATEST_COMPLETED` Checkpoint recovery (JeremyXin)
`425d4ac57`	#11983	Bug-Zeta	Reject invalid log level input (SEZ9)
`f13b610ef`	#11653	Fix-Zeta	Prevent duplicate Pending scheduling after failover (shenghang)
`1ff72ceb3`	#11812	Fix-Zeta	ClassLoader leak after task deployment failure (goutamadwant)
`2a739a94a`	#11492	Fix-Zeta	Isolate cross-Worker metric Registry interference (dybyte)
`b03cf1f16`	#12001	Fix-Zeta	Fix metric misalignment for Schema-first multi-table sinks (BinTaoMa, first-time contributor)
`be265d45c`	#11986	Feature-Benchmark	Zeta intermediate queue comparison benchmark (zniu70696)
`ba55ef965`	#11025	Feature-CDC	CDC COMMENT Schema Change event support (cloverdue)
`339fb88cce`	#11811	Improve-Core	Expanded configuration log redaction coverage (shenghang)
`e46a7106c`	#11827	Improve-Connector	Prometheus Checkpoint flush alignment (surafel58)
`4ba289595`	#9993	Refactor	seatunnel-shade module refactoring (hawk9821)
`f1a1a0abb`	#11851	Revert	Revert Jackson relocate (restoring the stable version) (hawk9821)
`13d3977f8`	#11870	Docs-Cookbook	Zeta slow-operation troubleshooting cookbook (zhang-arvin)

Redis Data Migration Without Scripts: How Apache SeaTunnel Handles String, Hash, Set, and ZSet

Apache SeaTunnel — Fri, 11 Sep 2026 06:52:56 +0000

Apache SeaTunnel’s Redis connector supports reading and writing four Redis data types: string, hash, set, and zset. This means you can move data from one Redis instance to another without writing custom migration scripts. This article provides a complete synchronization configuration for each data type, based on the connector behavior in SeaTunnel 2.3.13, as a practical reference for developers and data engineers working with Redis data migration.

Data Type	Read Behavior	Typical Write Method	Notes
string	Reads the entire value; when `format=json`, fields are parsed according to the schema	Writes by key; later writes overwrite earlier values for the same key	Key templates support `{field}` placeholders for dynamic key rewriting
hash	Without a schema, the entire hash is serialized as a single JSON row	Appends to a list	The hash key itself is not included in the data row
set	Each member is emitted as a separate row	Appends to a list	Unordered and does not deduplicate across source sets
zset	Each member is emitted as a separate row	Appends to a list	Scores are not read and will be lost

String: Structured Pass-Through

With data_type = string, the connector reads the complete value of each Redis key. When format = json is configured, the value is parsed field by field according to the schema.

On the sink side, data_type = key writes each record back using a Redis key. If the same key is written multiple times, the later value overwrites the earlier one. The {uid} placeholder in the key template is replaced with the value of the corresponding field in each record.

This setup works well for scenarios such as cache migration and Redis key prefix changes, where you want to preserve the original data structure while rewriting the destination key.

source {
  Redis {
    host = "redis-prod-01"
    port = 6379
    auth = "redis-demo-pass"
    keys = "user:profile:*"
    data_type = string
    batch_size = 200
    read_key_enabled = true
    key_field_name = key
    single_field_name = value
    format = json
    schema = {
      table = "UserDB.UserProfile"
      columns = [
        { name = "key",       type = "string" },
        { name = "uid",       type = "bigint" },
        { name = "nickname",  type = "string" },
        { name = "age",       type = "int" }
      ]
    }
  }
}

sink {
  Redis {
    host = "redis-prod-01"
    port = 6379
    auth = "redis-demo-pass"
    key = "user:profile:v2:{uid}"
    support_custom_key = true
    data_type = key
    batch_size = 200
  }
}

Test data:

SET user:profile:1001 '{"uid":1001,"nickname":"xiaoma","age":30}'
SET user:profile:1002 '{"uid":1002,"nickname":"candy","age":25}'

After the job finishes, verify the result with:

GET user:profile:v2:1001
"{\"key\":\"user:profile:1001\",\"uid\":1001,\"nickname\":\"xiaoma\",\"age\":30}"

The data is preserved while the key is rewritten with the v2 prefix.

Because value_field is not configured, the entire row is serialized as JSON before being written. With read_key_enabled = true, the original Redis key is also included as a field in the output record.

Hash: Flatten and Merge into a List

When no schema is configured, the connector serializes all field-value pairs in each hash into a single JSON object and sends it as one row. The values remain strings, while the hash key itself is not included in the data row.

If the sink is configured with data_type = list, records from multiple hashes can be appended to the same Redis list. This is useful when you need to aggregate hashes with different field structures into a single downstream stream.

source {
  Redis {
    host = "redis-prod-01"
    port = 6379
    auth = "redis-demo-pass"
    keys = "user:ext:*"
    data_type = hash
    batch_size = 200
  }
}

sink {
  Redis {
    host = "redis-prod-01"
    port = 6379
    auth = "redis-demo-pass"
    key = "audit:user:ext:queue"
    data_type = list
    batch_size = 200
  }
}

Test data:

HSET user:ext:1001 city Shanghai level gold
HSET user:ext:1002 city Beijing level silver device iOS

Verify the result with:

LRANGE audit:user:ext:queue 0 -1
1) "{\"city\":\"Shanghai\",\"level\":\"gold\"}"
2) "{\"city\":\"Beijing\",\"level\":\"silver\",\"device\":\"iOS\"}"

The two hashes have different field structures, but both are flattened into the same Redis list.

If you want to split the hash into individual columns instead of keeping the entire hash as a JSON object, configure a schema and set hash_key_parse_mode = kv. In this mode, the first field in the schema is used to store the original hash key, and each key-value pair is emitted as a separate row.

Set: Expand Each Member into a List

For Redis sets, each member is emitted as an individual row. In other words, the data granularity changes from a collection to a stream of individual values.

Configure the sink with data_type = list to append these values to a Redis list one by one.

source {
  Redis {
    host = "redis-prod-01"
    port = 6379
    auth = "redis-demo-pass"
    keys = "tag:members:*"
    data_type = set
    batch_size = 200
  }
}

sink {
  Redis {
    host = "redis-prod-01"
    port = 6379
    auth = "redis-demo-pass"
    key = "tag:members:merged"
    data_type = list
    batch_size = 200
  }
}

Test data:

SADD tag:members:vip     1001 1003 1007
SADD tag:members:newuser 1002 1003

Verify the result with:

LRANGE tag:members:merged 0 -1
1) "1001"
2) "1003"
3) "1007"
4) "1002"
5) "1003"

There are two things to keep in mind.

First, Redis sets are unordered. The order shown above is only an example and should not be treated as meaningful business ordering.

Second, the list does not deduplicate values across different source sets. Since 1003 exists in both source sets, it appears twice in the destination list.

If you want to merge the data and remove duplicates, change the sink to data_type = set.

ZSet: Expand Each Member, but Scores Are Lost

A Redis sorted set (zset) behaves similarly to a set in this connector scenario: each member is emitted as a separate row. However, the score is not read and is not included in the output record, so the downstream system receives only the member values.

source {
  Redis {
    host = "redis-prod-01"
    port = 6379
    auth = "redis-demo-pass"
    keys = "rank:board:*"
    data_type = zset
    batch_size = 200
  }
}

sink {
  Redis {
    host = "redis-prod-01"
    port = 6379
    auth = "redis-demo-pass"
    key = "rank:members:merged"
    data_type = list
    batch_size = 200
  }
}

Test data:

ZADD rank:board:day  9800 1001 9500 1003
ZADD rank:board:week 7200 1002

Verify the result with:

LRANGE rank:members:merged 0 -1
1) "1001"
2) "1003"
3) "1002"

The scores of the three members, 9800, 9500, and 7200, are all lost during the migration.

The same limitation applies when writing to a zset: the sink uses a fixed score of 1 rather than preserving the original score. In other words, neither the source read nor the destination write carries the actual zset score.

Therefore, this approach should not be used for use cases such as leaderboards or priority queues, where the score is part of the business logic.

Summary

The four Redis data types correspond to three different data transformation patterns:

String: structured pass-through
Hash: flatten and merge
Set/ZSet: expand each member into an individual record

For many Redis-to-Redis migration scenarios, Apache SeaTunnel can handle the data transfer directly through connector configuration, without requiring custom migration scripts.

At the same time, the connector's boundaries are clear. By default, the hash key, set ordering, and zset scores are not preserved.

If your application depends on any of these pieces of metadata, you will need to consider a custom script or another migration approach.

🚀 Need to sync Amazon DynamoDB data to Redis? Apache SeaTunnel makes it simple—configure once, run anywhere, and scale with ease. Thanks @Ricardo Ferreira for the great guide!

Apache SeaTunnel — Fri, 04 Sep 2026 03:30:52 +0000

Ricardo Ferreira for Redis

Jan 9

Syncing Data from Amazon DynamoDB to Redis with Apache SeaTunnel

#aws #dynamodb #redis #seatunnel

25 min read

🚀 Migrating from PostgreSQL to TiDB Cloud? See how Apache SeaTunnel enables reliable full-load + CDC with zero data loss. 🔄📊 Read the full guide! #ApacheSeaTunnel #TiDB #DataMigration

Apache SeaTunnel — Thu, 03 Sep 2026 09:52:56 +0000

Apache SeaTunnel

Sep 3

From PostgreSQL to TiDB Cloud: A Hands-On Data Migration with Apache SeaTunnel

#postgressql #apacheseatunnel #database #datascience

7 min read

From PostgreSQL to TiDB Cloud: A Hands-On Data Migration with Apache SeaTunnel

Apache SeaTunnel — Thu, 03 Sep 2026 09:52:43 +0000

1. Why Migrate the Database?

In the early stages of our business, we used PostgreSQL as both an OLTP and analytical database, deployed and maintained on our own infrastructure. In day-to-day operations, however, maintaining high availability, scheduling regular backups, and handling failure recovery required significant engineering effort. The resulting operational costs and complexity were high, and occasional operational oversights introduced a certain level of risk to business continuity.

Given that TiDB Cloud, PingCAP's cloud database service, eliminates the need for on-premises deployment and maintenance while offering convenient usage and flexible, controllable pricing, we decided to migrate our PostgreSQL database to TiDB Cloud, with business continuity and stability as our top priorities. We also saw this migration as an opportunity to explore a more diversified data storage and analytics architecture that would better fit our business needs.

Our core business focuses on road traffic checkpoint data analytics. Based on massive volumes of vehicle passage records—including license plates, vehicle speeds, locations, and driving behavior characteristics—we provide data support for traffic management and safety oversight.

The core table, driving_data_jsonb, uses a JSONB + partitioned-table design. It processes approximately 5 million vehicle passage records per day, while a single partition stores around 150 million vehicle passage records (approximately 60 GB). It is the largest and most frequently queried table in our business.

Typical analytical scenarios include:

Road congestion analysis: Aggregating traffic volume and average vehicle speed by time period and road segment to assess current road traffic conditions in real time
Section speed-limit violation detection: Calculating a vehicle's average speed over a road section based on the travel time and distance between checkpoints, and identifying speeding violations
Driving trajectory analysis: Reconstructing vehicle routes to analyze driving habits and patterns of abnormal behavior
Continuous driving duration monitoring: Tracking the continuous driving time of individual vehicles and generating alerts based on traffic regulations, such as the requirement to take a mandatory break after four hours of continuous driving

These analytical queries involve large-scale data aggregation and multidimensional calculations, placing relatively high demands on the database's OLAP capabilities.

2. Migration Environment

2.1 Source Environment

Item	Details
Database	PostgreSQL 18.2
Deployment	Local virtual machine (test environment)
Database	`traffic` (traffic data database)
Core table	`driving_data_jsonb` (partitioned table, approximately 150 million rows / 60 GB per partition)
Network egress	500 Mbps broadband connection with direct public Internet access

2.2 Target Environment

Item	Details
Platform	TiDB Cloud
Region	`cn-shanghai` (Shanghai)
Storage consumption	Approximately 62.62 GiB

2.3 Network Topology

The source environment connects directly to TiDB Cloud over the public Internet through a 500 Mbps broadband connection. No dedicated private connection or VPN was deployed.

3. Migration Tool Selection

During the migration, we primarily evaluated two commonly used data migration tools: DataX and SeaTunnel.

DataX	SeaTunnel ✅
• No version updates for a long time	• Active open-source project with timely iterative updates (currently using 2.3.13)
• Some plugins rely on Python 2.x helper scripts, leading to poor compatibility with modern tech stacks	• Rich variety of supported data sources with a mature connector ecosystem
• Limited extensibility with a relatively sparse connector ecosystem	• High community activity and fast response times to issues
	• Supports generic JDBC connectors, fully compatible with PG (PostgreSQL) and TiDB

Given our core requirements for timely synchronization, stability, and zero data loss, we ultimately selected SeaTunnel 2.3.13 as the migration tool.

4. Migrating Data with SeaTunnel

4.1 Migration Strategy: Full Load + Incremental Sync

Because the PG-CDC connector in SeaTunnel 2.3.13 had a known bug in the combined full-load + incremental synchronization mode (which we reported to the community and helped fix), we adopted a two-step approach: full-load initialization followed by CDC-based incremental synchronization.

Step 1: Perform a full data initialization using the JDBC PostgreSQL connector.

Step 2: Use the CDC incremental connector to synchronize data changes in real time.

4.2 Full-Load Configuration (JDBC Source → JDBC Sink)

The following is the core configuration used during the full-load phase:

env {
  parallelism = 2
  checkpoint.interval = 10000
  pipeline.name = "PG_TO_TiDB_DRIVING"

  flink.execution.checkpointing.mode = "EXACTLY_ONCE"
  flink.execution.checkpointing.timeout = 600000
}
source {
  Jdbc {
    url = "jdbc:postgresql://<source_host>:5432/traffic"
    driver = "org.postgresql.Driver"
    user = "postgres"
    password = "******"

    query = "SELECT record_id, pass_time, plate_no, plate_color,
             vehicle_type, speed, speed_limit, status,
             longitude, latitude, speed_variance,
             lane_change_count, raw_behavior_features,
             create_time, update_time
             FROM driving_data_jsonb"
    column.filters = "update_time"
    start.time = "2025-01-01 00:00:00"
    fetch.size = 1000
    parallelism = 2
  }
}
sink {
  Jdbc {
    url = "jdbc:mysql://<tidb_cloud_host>:4000/traffic?sslMode=VERIFY_IDENTITY"
    driver = "com.mysql.cj.jdbc.Driver"
    user = "******"
    password = "******"
    database = "traffic"
    table = "driving_data_jsonb"

    generate_sink_sql = true

    save_mode = "upsert"
    unique_key = ["record_id"]

    batch.size = 1000
    batch.interval = 1000
    parallelism = 2
  }
}

4.3 Incremental Synchronization Configuration (Postgres-CDC Source → JDBC Sink)

Once the full load was completed, we started the CDC incremental connector to continuously synchronize changes from the source database in real time:

env {
  execution.parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 5000
}
source {
  Postgres-CDC {
    username = "postgres"
    password = "******"
    database-names = ["traffic"]
    schema-names = ["public"]
    table-names = ["traffic.public.driving_data_jsonb"]
    url = "jdbc:postgresql://<source_host>:5432/traffic"
    decoding.plugin.name = "pgoutput"
    slot.name = "final_slot"
    startup.mode = "latest"
    plugin_output = "out"
  }
}
sink {
  Jdbc {
    url = "jdbc:mysql://<tidb_cloud_host>:4000/traffic?sslMode=VERIFY_IDENTITY"
    driver = "com.mysql.cj.jdbc.Driver"
    user = "******"
    password = "******"
    database = "traffic"
    table = "driving_data_jsonb"

    generate_sink_sql = true

    save_mode = "upsert"
    unique_key = ["record_id"]

    batch.size = 1000
    batch.interval = 1000
    parallelism = 2
  }
}

4.4 Key Configuration Details

The following table highlights the key configuration options used during full-load and incremental synchronization:

Configuration	Description
`parallelism = 2`	Sets the parallelism to 2 to match the available resources on the source side
`checkpoint.mode = EXACTLY_ONCE`	Provides exactly-once semantics to help ensure data consistency
`checkpoint.interval = 10000`	Sets the checkpoint interval to 10 seconds, balancing performance and fault tolerance
`save_mode = upsert`	Writes data using Upsert based on `unique_key` to prevent duplicate records
`unique_key = ["record_id"]`	Uses `record_id` as the deduplication key
`sslMode = VERIFY_IDENTITY`	Uses a TLS-encrypted connection, as required by TiDB Cloud
`batch.size / batch.interval`	Writes 1,000 records per batch or flushes every 1 second, whichever comes first
`decoding.plugin.name = pgoutput`	Uses PostgreSQL's native logical decoding plugin for CDC incremental synchronization
`slot.name`	Specifies the logical replication slot to ensure incremental data is not lost
`startup.mode = latest`	Starts consuming incremental changes from the latest position to avoid duplicate synchronization

4.5 Migration Results

Metric	Result
Full-load duration	Approximately 12 hours per partition (the source was a virtual machine in a test environment)
Incremental synchronization latency	The migration proceeded smoothly with no noticeable latency
Data consistency	Every record contains the unique primary key `record_id`. The Sink uses `upsert` mode and deduplicates records based on the primary key, ensuring that data is neither lost nor duplicated. After migration, the final validation was performed by comparing row counts between the source and target
Schema compatibility	No data type mapping issues were encountered when migrating from PostgreSQL to TiDB

Note: The migration first focused on the largest core table, driving_data_jsonb. The remaining tables contain significantly less data and will be synchronized as needed.

5. Challenges and Lessons Learned

5.1 PG-CDC Connector Bug

The PG-CDC connector in SeaTunnel 2.3.13 had a bug in the combined full-load + incremental synchronization mode, which prevented the workflow from completing successfully. We reported the issue to the community, worked with developers to identify and fix the problem, and ultimately avoided it by running the full-load and incremental synchronization stages separately.

Recommendation: When using SeaTunnel for PostgreSQL migration, we recommend adopting a two-step full-load + incremental synchronization strategy first, as it provides better stability. If you need to use the integrated mode, make sure the SeaTunnel version you are using contains the fix for this issue.

5.2 The Impact of Network Bandwidth

The source database was deployed on a local virtual machine and connected to TiDB Cloud over the public Internet through a 500 Mbps broadband connection. Synchronizing 60 GB of data from a single partition took approximately 12 hours during the full-load phase.

If the source database is deployed in the same cloud region or connected through a dedicated private connection, the synchronization time is expected to be significantly shorter.

6. Business Performance After Migration

The most noticeable improvement after migrating to TiDB Cloud was business performance in OLAP workloads. Statistical analysis and complex queries became noticeably more efficient, reducing query wait times for business users and improving the overall query experience.

Going forward, we plan to explore enabling TiFlash, the columnar storage engine, based on specific business scenarios. This will allow us to further optimize analytical query performance and make better use of TiDB's HTAP architecture for workloads that combine row- and column-oriented processing.

Note: We have not yet completed a detailed analysis of application performance after the migration. Specific query performance comparisons will be added in a future update.

7. Conclusion and Recommendations

Overall Assessment

Overall, the migration from PostgreSQL to TiDB Cloud achieved the expected results.

SeaTunnel performed reliably as the migration tool. Both the full-load and incremental synchronization stages were completed successfully, with no data loss observed. TiDB Cloud delivered noticeable performance improvements for OLAP analytical workloads while significantly reducing the operational burden of database management.

Migration Recommendations for Other PostgreSQL Users

Tool selection: We recommend SeaTunnel for its rich connector ecosystem and active community. Its JDBC connector can quickly adapt to both PostgreSQL and TiDB.
Migration strategy: We recommend using a two-step full-load + incremental synchronization strategy, which provides better stability than the integrated mode.
Data consistency: Be sure to enable the EXACTLY_ONCE checkpoint mode and use upsert writes to prevent duplicate records.
Network planning: Network quality between the source environment and TiDB Cloud has a significant impact on full-load synchronization time. Where possible, prioritize deployment within the same cloud environment or use a dedicated private connection.
Secure connections: TiDB Cloud requires TLS. Configure sslMode=VERIFY_IDENTITY on the Sink side to ensure encrypted data transmission.

🔍 A successful pipeline doesn’t guarantee trusted data. See how Apache SeaTunnel enables reliable movement, recovery, and schema evolution.

Apache SeaTunnel — Thu, 27 Aug 2026 07:50:12 +0000

Apache SeaTunnel

Aug 27

Pipeline Green Isn’t Data Correct

#ai #datascience #apacheseatunnel #programming

11 min read

🔄 A financial tech company migrated 200+ ETL workflows from Informatica to Apache SeaTunnel, cutting infrastructure costs 60% and runtimes 40%!

Apache SeaTunnel — Thu, 27 Aug 2026 07:49:43 +0000

Apache SeaTunnel

Aug 27

From Informatica to Apache SeaTunnel: A Financial-Grade ETL Migration in Practice

#apacheseatunnel #etl #datascience #dataengineering

5 min read

From Informatica to Apache SeaTunnel: A Financial-Grade ETL Migration in Practice

Apache SeaTunnel — Thu, 27 Aug 2026 07:31:23 +0000

1. Project Background and Challenges

As the head of data architecture at a financial technology company, I led the migration of our core ETL system from Informatica PowerCenter to a domestic ETL platform last year. The migration covered more than 200 workflows and a core system processing terabytes of data every day. It took five months from start to finish, and we ultimately completed the transition smoothly with zero data incidents. Today, I’d like to share the key decisions, technical details, and lessons learned from the project.

As a long-standing leader in enterprise data integration, Informatica has dominated the traditional ETL market for more than 20 years. Its visual development environment, reliable scheduling engine, and comprehensive metadata management have made it a de facto standard in industries such as financial services and telecommunications. However, with the changing international landscape and the growing demand for domestic technology adoption, we had to confront three practical challenges:

High licensing costs: Annual maintenance fees running into millions of RMB were a significant burden for a mid-sized enterprise.
A closed technology stack: It was difficult to integrate deeply with emerging real-time computing and AI platforms.
Slow response to customization needs: Custom requirements often required cross-border collaboration, resulting in delivery cycles that could stretch to several months.

After years of development, domestic ETL platforms such as Kettle, DataX, and SeaTunnel have become viable alternatives for core ETL capabilities. Our technical evaluation showed that, in batch-processing scenarios, domestic platforms could cover approximately 85% of Informatica’s functionality while costing only one-third as much and offering the flexibility for secondary development.

2. Migration Strategy and Design

2.1 Technology Selection

We conducted an in-depth evaluation of three mainstream domestic ETL tools:

We ultimately selected Apache SeaTunnel as the primary migration platform for three main reasons:

Support for both Spark and Flink engines, making it a good fit for our future real-time data warehouse strategy.
A plugin-based architecture that makes it easier to extend support for custom data sources.
An active Chinese-language community that enables us to resolve technical issues quickly.

2.2 Migration Strategy

We adopted a hybrid approach combining phased migration with parallel-run validation:

Decouple the components: Break each Informatica workflow into three independent modules for extraction, transformation, and loading.
Map the functionality:

Source data extraction → SeaTunnel Source plugins
Complex transformation logic → Rebuild with Spark SQL
Scheduling dependencies → Orchestrate with Apache DolphinScheduler
1. Validate the data:

# Use a combination of CRC32 and sample-based comparison for validation
def verify_data(source_df, target_df):
    if source_df.count() != target_df.count():
        return False
    sample_ratio = 0.01
    source_sample = source_df.sample(sample_ratio)
    target_sample = target_df.sample(sample_ratio)
    return source_sample.exceptAll(target_sample).isEmpty()

Key lesson: Don’t try to replicate Informatica workflows 1:1. Use the migration as an opportunity to optimize the data flows. We redesigned 30% of the transformation logic that had performance bottlenecks.

3. Core Migration Implementation

3.1 Metadata Migration

The Informatica Repository contained thousands of metadata objects. We developed a metadata parsing tool to automate the migration:

Export XML metadata through the PowerCenter CLI.
Use XSLT to transform key attributes:

<!-- Example of mapping transformation -->
<xsl:template match="SOURCE">
  <connector type="jdbc">
    <property name="url" value="{@DBSERVER}"/>
    <property name="table" value="{@OBJECTNAME}"/>
  </connector>
</xsl:template>

Generate SeaTunnel configuration file templates.

3.2 Refactoring Complex Transformations

Informatica components such as Expression and Aggregator required special handling during the migration.

Conditional routing: The original workflows used the Router component.

-- Reimplemented with Spark SQL
df.createTempView("source");
spark.sql("""
  SELECT *, 
    CASE 
      WHEN amount > 10000 THEN 'VIP' 
      ELSE 'NORMAL' 
    END AS customer_level
  FROM source
""");

Slowly Changing Dimensions (SCD): The original implementation relied on the Slowly Changing Dimension wizard.

-- Implement Type 2 SCD using MERGE INTO
MERGE INTO dim_customer t
USING stage_customer s
ON t.customer_id = s.customer_id
WHEN MATCHED AND t.current_flag='Y' AND t.email <> s.email THEN
  UPDATE SET t.current_flag='N', t.end_date=CURRENT_DATE
  INSERT VALUES (s.customer_id, s.email, ..., 'Y', CURRENT_DATE, NULL)

3.3 Performance Tuning in Practice

The most challenging issue we encountered was an ETL job containing 20 joins that ran into an OOM error on SeaTunnel. We resolved it through the following optimizations:

Analyze the execution plan:

# Get the Spark physical execution plan
EXPLAIN EXTENDED 
SELECT * FROM fact f JOIN dim1 d1 ON f.id=d1.id ...

Optimization measures:

Enable dynamic partition pruning: spark.sql.optimizer.dynamicPartitionPruning=true
Adjust the broadcast threshold: spark.sql.autoBroadcastJoinThreshold=20MB
Force broadcast joins for dimension tables: /*+ BROADCAST(dim1) */

Parameter comparison:

4. Validation and Cutover

4.1 Ensuring Data Consistency

We established a three-level validation framework:

Record-level validation: Generate a CRC32 fingerprint for the entire table.

SELECT 
  SUM(CAST(CRC32(CONCAT_WS('|',col1,col2,...)) AS BIGINT)) AS checksum 
FROM table

Business metric comparison: Keep month-over-month fluctuations in key KPIs below 1%.
User acceptance testing: Have business teams validate the data in their reports.

4.2 Gradual Rollout

We migrated workloads incrementally by business line:

First, migrate non-core marketing analytics workloads.
Next, migrate the risk management system.
Finally, migrate the financial settlement system.

We monitored each phase for one week, with a particular focus on:

Data latency
Resource utilization
Error logs

5. Lessons Learned

5.1 Key Success Factors

Team training: Two months before the migration, we organized training sessions for Informatica developers to learn Spark and SeaTunnel.
A complete toolchain:

Developed an auxiliary workflow conversion tool.
Built an automated data comparison platform.
1. Vendor support: Established a direct communication channel with the SeaTunnel core team.

5.2 Lessons Learned the Hard Way

Time zone issues: Informatica uses the server's time zone by default, while Spark uses UTC. We therefore needed to explicitly handle time zone conversion for all relevant time fields:

FROM_UTC_TIMESTAMP(CAST(col AS TIMESTAMP), 'Asia/Shanghai')

Character encoding pitfalls: The ZHS16GBK encoding used by the Oracle source database had to be explicitly configured:

source:
  jdbc:
    connection_options: "oracle.jdbc.convertNlsStrings=true"

Transaction semantics: Informatica uses auto-commit by default, while Spark requires transaction behavior to be controlled explicitly:

df.write
  .option("isolationLevel", "READ_COMMITTED")
  .mode("overwrite")
  .saveAsTable("target")

Quantified Results After the Migration

60% reduction in infrastructure costs, from eight physical servers to a Kubernetes cluster
40% reduction in average job execution time
New capabilities for real-time data processing

The biggest takeaway from this migration is that domestic infrastructure software has reached a level where it can serve as a viable alternative to traditional enterprise platforms. But successful migration requires more than simply replacing one tool with another. It requires a shift in technical mindset.

Instead of treating migration as a straightforward tool replacement exercise, we used it as an opportunity to rethink and modernize our data architecture, laying the groundwork for the next stage of real-time processing and intelligent data operations.

DEV Community: Apache SeaTunnel

Why I Reworked a BigQuery Sink from Pending Streams to Buffered Streams

Initial Design: Pending Streams for Batch Writes

The Problem: Ownership Gap After a Failed Checkpoint

Why restoreWriter() Should Not Commit Failed-Checkpoint Streams

Redesign: Switching to Buffered Streams

Understanding State and CommitInfo

Managing Offsets Correctly

Committer Uses FlushRows Instead of BatchCommitWriteStreams

Keeping CDC Separate

Testing Challenges

Lessons Learned

Closing Thoughts

How Apache SeaTunnel Handles Kafka Topic Partition Expansion Without a Job Restart

1. Scenario

2. Hands-On Guide

2.1 Prerequisites

2.2 Create a Topic with Two Partitions

2.3 Configure the SeaTunnel Streaming Job

2.4 Start the Job and Verify Normal Consumption

2.5 Expand the Topic from 2 Partitions to 4

2.6 Produce More Data and Check the New Partitions

2.7 Comparison: What Happens Without Dynamic Partition Discovery?

3. Important Considerations

3.1 The Starting Offset of New Partitions Depends on start_mode

3.2 Avoid Setting the Discovery Interval Too Low

3.3 Kafka Partitions Can Only Be Increased, Not Decreased

3.4 Partition Expansion Triggers Consumer Group Rebalancing

3.5 Existing Partitions Are Not Affected

3.6 Types with Precision or Generic Parameters Must Be Quoted

4. Summary

🚀 Move Redis data without writing migration scripts! Explore how Apache SeaTunnel handles String, Hash, Set, and ZSet with practical Redis-to-Redis configs. #ApacheSeaTunnel #Redis #DataMigration #DataEngineering

Redis Data Migration Without Scripts: How Apache SeaTunnel Handles String, Hash, Set, and ZSet

Apache SeaTunnel had a busy August: 195 PRs merged, 13 new connectors added, and major upgrades to Zeta observability, CDC, Checkpoint recovery, and CI. #ApacheSeaTunnel #DataEngineering #OpenSource

195 Merged PRs, 13 New Connectors: What’s New in Apache SeaTunnel This August?

195 Merged PRs, 13 New Connectors: What’s New in Apache SeaTunnel This August?

1. Merged PRs This Month

1.1 Overall Statistics

1.2 PR Classification

2. Top 25 Contributors

3. Code Change Analysis

3.1 Overall Code Volume

3.2 Distribution by Technical Area

3.3 Major Refactoring and Architectural Changes

4. Top 7 Updates Users Will Notice

✨ TOP 1: 13 New Connectors Expand the Data Source Ecosystem

✨ TOP 2: Major Observability Upgrades for Zeta

✨ TOP 3: BigQuery Sink Adds Multi-Table and UniverseDomain Support

✨ TOP 4: LATEST_COMPLETED Checkpoint Recovery + Prometheus Checkpoint Flush Alignment

✨ TOP 5: CDC COMMENT Schema Change Support + Stronger Recovery Stability

✨ TOP 6: CI Merge Queue + E2E Shard Rebalancing + Hung Job Protection

✨ TOP 7: Declarative OptionRule Validation Expands Across 10 Connector Migration PRs

Appendix: Quick Reference for Key Changes

Redis Data Migration Without Scripts: How Apache SeaTunnel Handles String, Hash, Set, and ZSet

String: Structured Pass-Through

Hash: Flatten and Merge into a List

Set: Expand Each Member into a List

ZSet: Expand Each Member, but Scores Are Lost

Summary

🚀 Need to sync Amazon DynamoDB data to Redis? Apache SeaTunnel makes it simple—configure once, run anywhere, and scale with ease. Thanks @Ricardo Ferreira for the great guide!

Syncing Data from Amazon DynamoDB to Redis with Apache SeaTunnel

🚀 Migrating from PostgreSQL to TiDB Cloud? See how Apache SeaTunnel enables reliable full-load + CDC with zero data loss. 🔄📊 Read the full guide! #ApacheSeaTunnel #TiDB #DataMigration

From PostgreSQL to TiDB Cloud: A Hands-On Data Migration with Apache SeaTunnel

From PostgreSQL to TiDB Cloud: A Hands-On Data Migration with Apache SeaTunnel

1. Why Migrate the Database?

2. Migration Environment

2.1 Source Environment

2.2 Target Environment

2.3 Network Topology

3. Migration Tool Selection

4. Migrating Data with SeaTunnel

4.1 Migration Strategy: Full Load + Incremental Sync

4.2 Full-Load Configuration (JDBC Source → JDBC Sink)

4.3 Incremental Synchronization Configuration (Postgres-CDC Source → JDBC Sink)

4.4 Key Configuration Details

4.5 Migration Results

5. Challenges and Lessons Learned

5.1 PG-CDC Connector Bug

5.2 The Impact of Network Bandwidth

6. Business Performance After Migration

3.1 The Starting Offset of New Partitions Depends on `start_mode`