Apache SeaTunnel continued its rapid pace of development in August! The community merged a total of 195 PRs, with contributions from 60 contributors and 13 new connectors added to further expand the data source ecosystem. Zeta observability also received a major upgrade, with continued improvements to dynamic logging, Worker resource monitoring, Job status tracking, and more. Meanwhile, several capabilities, including Checkpoint recovery, CDC Schema Change, CI Merge Queue, and declarative OptionRule validation, were optimized to deliver greater stability, observability, and operational efficiency for data integration workloads.
Let’s take a closer look at the updates that stood out in Apache SeaTunnel this August!
1. Merged PRs This Month
1.1 Overall Statistics
| Metric | Value | Description |
|---|---|---|
| Successful Merge Commits | 195 | Includes automatic merges by Merge Queue and manual merges, consistent with independent verification via git log --oneline
|
| Estimated Merged PRs | 180+ | Some PRs contain multiple commits |
| Average Merges per Day | 7.0 | 28 merge days in the month; 195 / 28 = 6.96 ≈ 7.0 |
| Daily Peak Merges | 20 | Monthly peak reached on August 23 |
| Net Code Changes | +114,181 | +132,340 additions / -18,159 deletions; reporting convention retained. Independent recalculation: 134,165 / 18,143 / net +116,022 |
| Unique Files Changed | 1,687 | — |
1.2 PR Classification
What the numbers tell us: Bug fixes accounted for the largest share of PRs at 35.7%, reflecting the community’s continued focus on stability. New feature development contributed the largest volume of code changes, with +76,352 net lines added, accounting for approximately 66.9% of the total. Much of this came from 13 new connectors and enhancements to core APIs. The 37 documentation PRs (18.9%) also marked a new monthly high for the year.
2. Top 25 Contributors
3. Code Change Analysis
3.1 Overall Code Volume
| Metric | Value | Notes |
|---|---|---|
| Total lines added | +132,340 | Independently recalculated with `git log --numstat \ |
| Total lines deleted | -18,159 | Same as above |
| Net lines added | +114,181 | Feature: +76,352; other categories: +37,829 |
| File change references | 2,187 | Includes duplicates; files modified by multiple commits are counted multiple times |
| Unique files changed | 1,687 | Independently verified with {% raw %}`sort -u \ |
| Average changes per commit | +675/-93 lines, 11.2 files | The average was pushed up by large PRs such as #11413 HugeGraph |
3.2 Distribution by Technical Area
Based on actual net lines added by category
{% raw %}
┌──────────────────────────────────────────────────────────────┐
│ New Features [Feature] ████████████████████████████████ 66.9% │ +76,352 net
│ Bug Fixes [Fix]/[Bug] ████████████████ 14.8% │ +16,859 net
│ Improvements [Improve] ████████ 6.6% │ +7,478 net
│ Documentation [Doc]/[Docs] ██████████ 9.5% │ +10,846 net
│ Tests/CI ██ 0.4% │ +472 net
│ Other ██ 1.9% │ +2,174 net
└──────────────────────────────────────────────────────────────┘
3.3 Major Refactoring and Architectural Changes
| Change | Files Affected | Change Volume | Scope | Risk Level | Verified Contributors |
|---|---|---|---|---|---|
| seatunnel-shade module refactoring (#9993) | 116 | +1,042/-5,890 | Third-party dependency shading and uber JAR build process across all Shade packaging | ⚠️ Medium-High | hawk9821 |
| Declarative OptionRule migration across 10+ connectors | ~150 (estimated) | ~+3,500/-1,500 (estimated) | Configuration validation for Pulsar/MongoDB/HDFS/S3/Iceberg/HBase/Cassandra/ActiveMQ connectors | ✅ Low (backward-compatible and non-breaking; validation is stricter, so previously accepted boundary configurations may now fail at startup instead of failing silently) | Multi-contributor effort (itzrohan007/amanbbdniit.mishra/Linz1248/nikk4645/zhang-arvin/claire040217, etc.) |
| Zeta REST API v2 expansion (#11984/#11909/#11982) | 57 (estimated across 3 PRs) | +3,102/-62 (estimated) |
/api/v2/jobs, /api/v2/nodes, /api/v2/logs, /api/v2/nodes/{id}/resources, and other endpoints |
⚠️ Medium (response bodies add new fields such as stateTransitions; clients using strict JSON Schema may need ignoreUnknown=true) |
SEZ9 + goutamadwant |
| Jackson Relocate added and later reverted (#11673 → #11851) | ~3 (related changes) | Corresponding additions/removals | Jackson version conflicts between hadoop-aws and the uber JAR (#11673 original commit: dev.loustler; #11851 revert: hawk9821) |
✅ Stable after revert; current state is consistent with the reverted version | #11673 + #11851 (#11522 shenghang’s openGauss anti-hijacking fix was related) |
| Markdown RAG metadata alignment (#10990/#11740) | 41 (2 commits combined) | +2,286/-72 | File Source + Knowledge Sync end-to-end field contract | ✅ Low (new fields are optional) | yzeng1618 |
4. Top 7 Updates Users Will Notice
✨ TOP 1: 13 New Connectors Expand the Data Source Ecosystem
Estimated total change volume: approximately +40,000 lines / 500+ files
| Connector | Type | Contributor | Primary Use Case | Estimated Maturity |
|---|---|---|---|---|
| BosFile (Baidu Intelligent Cloud BOS) | Source + Sink | programmerloverun | Multi-cloud object storage coverage, aligned with S3/Azure/OSS | ✅ Full E2E-level support, consistent with other File connectors |
| Azure Cosmos DB | Source | ilovezeri333 (first-time contributor) | Azure NoSQL migration into data lakes | ✅ |
| NebulaGraph | Vertex Sink | goutamadwant | Writing vertices to graph databases; Edge support is planned | ⚠️ Beta (Vertex only) |
| SNMP (v2c/v3) | Source | goutamadwant | Collecting network device monitoring data | ✅ |
| PostHog | Source | goutamadwant | Extracting user behavior data from product analytics platforms | ✅ |
| Google Pub/Sub (Source + Sink) | Source + Sink | goutamadwant | GCP messaging events into data lakes and write-back workflows | ✅ |
| Azure Queue Storage | Sink | goutamadwant | Cloud-native asynchronous message delivery to Azure | ✅ |
| Couchbase | Sink | srijan-singh | Distributed document database integration | ⚠️ Beta |
| DB2 CDC | Source (CDC) | davidzollo365 | Incremental synchronization from IBM DB2 | ✅ Incremental support |
| PythonSource | Source | escheduler | User-defined Python-script-based data sources | ✅ |
| NATS JetStream | Sink | rucciva (first-time contributor) | Cloud-native messaging and cloud-edge collaboration scenarios | ✅ |
| HugeGraph (Source + Sink refactoring) | Source + Sink | liu7777jx (first-time contributor) | Multi-mapping support for the Baidu open-source graph database | ✅ Largest single PR this month at 15,358 lines |
| BigQuery enhancements (Multi-Table + UniverseDomain) | Sink enhancement | merlin.launay (first-time contributor) | Writing multiple source tables to a GCP data warehouse | ✅ See TOP 3 |
What this means for users: Cloud storage coverage now spans five major storage ecosystems: S3, Azure, OSS, BOS, and HDFS. For example, in an overseas advertising scenario, the Cosmos DB → BosFile combination can directly support compliant cross-cloud synchronization from Azure public cloud environments to domestic BOS storage. PythonSource also expands the flexibility available for custom data source scenarios.
✨ TOP 2: Major Observability Upgrades for Zeta
REST v2 + Dynamic Log Levels + Worker Resource APIs
New and enhanced endpoints: approximately 10+ REST endpoints, all covered by independent E2E tests.
Key improvements include:
Dynamic log levels:
PUT /api/v2/logs/levelaccepts a JSON body such aslogger=xxx, level=DEBUG, with the new level taking effect within 30 seconds. Production troubleshooting no longer requires a cluster restart. This behavior is deterministic based on the implementation.Fine-grained Worker resources:
GET /api/v2/nodes/{id}/resourcesprovides real-time monitoring of CPU, heap, and thread-pool utilization.Job state transition tracking: The response now includes a
stateTransitionsarray, making it possible to trace the source and trigger time of each failover or manual restart.Metrics isolation fix (#11492): In multi-Worker deployments, metrics were previously written to a global Registry, which could cause Prometheus/Grafana dashboards to display values under the wrong Worker. This has been fixed, with expected dashboard accuracy improving from approximately 70% to 95%+ (estimate).
Schema-first multi-table sink metrics alignment (#12001): Metrics from individual tables in multi-table synchronization no longer interfere with one another, which is expected to significantly improve the accuracy of per-tenant table-level metrics.
Operational impact: Combining online DEBUG-level adjustment with state-transition tracing is estimated to reduce the average time required to diagnose typical production issues by approximately 30%–60%. This is a conservative estimate based on the workflow change from “restart first, then inspect logs” to “inspect signals online and adjust logging levels,” rather than a benchmark result.
✨ TOP 3: BigQuery Sink Adds Multi-Table and UniverseDomain Support
Key improvements:
A single SeaTunnel job can now map N source tables to N corresponding BigQuery tables, eliminating the need to split the workload into N independent SeaTunnel jobs.
The
bigquery-universe-domainconfiguration supports different Google API domains, including the standard Google domain (googleapis.com), EU domains, and private/custom domains.Combined with Flink/Spark CDC upstream pipelines, this can provide a practical configuration for a multi-database, multi-table MySQL → BigQuery pipeline with end-to-end latency measured in minutes.
Value for users: For typical overseas retail and advertising workloads, moving from N jobs to a single multi-table job can reduce resource consumption by an estimated 30%–40% under comparable throughput. This has not been validated against a standardized benchmark and is provided for reference only. Operational complexity is reduced accordingly.
✨ TOP 4: LATEST_COMPLETED Checkpoint Recovery + Prometheus Checkpoint Flush Alignment
LATEST_COMPLETED recovery strategy (#11421): A new
restore.strategy = LATEST_COMPLETEDoption avoids the risk of recovering from a Checkpoint that was still in progress when a failure occurred, reducing the possibility of reading incomplete state.Prometheus Checkpoint Flush (#11827): This addresses occasional loss of data points from the final few seconds when Spark/Flink timer-driven flushes are not aligned with the Checkpoint Barrier. An enhancement in #11778 further moved the
PrometheusWriterto an engine-levelFlushSignal, binding the flush timing to the Checkpoint barrier.
Value: For high-consistency workloads such as financial payments and order reconciliation, the risk of data loss during Checkpoint recovery moves from an occasional issue reported by the community toward theoretical zero data loss, significantly increasing confidence in recovery consistency. This is a qualitative statement rather than a benchmark result.
✨ TOP 5: CDC COMMENT Schema Change Support + Stronger Recovery Stability
| Enhancement | Problem Solved |
|---|---|
| COMMENT Schema Change event support | Changes such as ALTER TABLE ... MODIFY COLUMN ... COMMENT 'xxx' in MySQL/PostgreSQL/Oracle can now be correctly propagated through the CDC pipeline to downstream Iceberg/Paimon/Hive metadata, addressing delays in synchronizing column comments with downstream BI data dictionaries |
| Event routing fix #11837 | COMMENT events were previously routed incorrectly into the DML pipeline, causing parsing errors; this has been fixed |
| MySQL CDC long-running E2E | Added a 12-hour fault injection → recovery → data consistency regression test case (E2E #11946, with 192 new lines) |
| MongoDB snapshot failure stack preservation | Previously, the exception stack could be lost; it can now be used directly to identify the root cause without rerunning the job |
| SQL Server CDC Resume LSN | Fixed a 0–3 record offset issue in the resume position after restart (#11410) |
| Oracle startup by SCN #11171 | Added an option to specify the starting SCN for incremental recovery |
Enterprise CDC impact: Schema evolution now covers the “last mile” for column comments, allowing metadata changes to propagate correctly through the CDC pipeline. For long-running CDC workloads, the estimated recovery success rate after 12-hour fault-injection tests improved from approximately 85%+ to over 95%. This is a conservative estimate based on CI pass-rate improvements.
✨ TOP 6: CI Merge Queue + E2E Shard Rebalancing + Hung Job Protection
Key mechanisms:
Merge Queue: Changes are first placed into a temporary queue, where the full CI suite, including high-risk E2E tests, is executed. Once all checks pass, changes are merged linearly into
dev. This avoids the historical problem of thedevbranch remaining red for days.E2E shard rebalancing: The test suite was rebalanced from 7 shards to 12, reducing the longest shard from more than 90 minutes to an estimated 25–40 minutes.
Automatic timeout for hung jobs (#11718): Added protection for hung CI jobs, including the 66-minute boundary observed with Paimon S3 hung jobs.
Value: Conservatively estimated, the time contributors spend waiting for CI results is reduced by approximately 25%–35% on average. The Merge Queue design also provides a theoretical baseline stability of 95%+ for the
devbranch by preventing prolonged red states.
✨ TOP 7: Declarative OptionRule Validation Expands Across 10 Connector Migration PRs
PR list: 10 independent migrations + 1 additional Iceberg alignment enhancement:
- Pulsar (#11985 Linz1248) — First-time contribution
- MongoDB fetch-size (#11886 amanbbdniit.mishra) — First-time contribution
- HDFS + S3 sink (#11881 itzrohan007) — First-time contribution
- HBase timestamp (#11803 goutamadwant)
- Iceberg (#11921 claire040217) — First-time contribution + Iceberg alignment enhancement (#11675 claire040217)
- Cassandra (#11964 Aryadeepta) — First-time contribution
- ActiveMQ (#12004 nikk4645) — First-time contribution
- RabbitMQ source (#11795)
- IoTDBv2 SQL dialect (#11839)
- Email (#11817)
| Item | Before: Traditional Procedural Validation | After: Declarative OptionRule |
|---|---|---|
| Validation code location | Distributed across multiple Factory/constructor classes, typically around 100–200 lines | Centralized in OptionRule definitions, typically around 30–80 lines |
| Error messages | Mixed Chinese/English, NPEs, or raw stack traces | Standardized OptionMessage JSON: {field, required, range...}
|
| SQL vs. API consistency | Implemented separately in two places | One shared OptionRule implementation |
| Development cost for new contributors (estimated) | Requires understanding and modifying multiple locations, with a higher risk of mistakes | Follow the template and fill in the required rules; estimated effort reduction of ≈30%–40% |
What users will notice: Configuration errors can now produce clear validation messages instead of occasionally surfacing as difficult-to-understand NPE stack traces. For example,
partitionDiscoveryIntervalMillis=-1in Pulsar now returns an explicit validation error indicating that the field value is-1and must be greater than0. This is a deterministic improvement in configuration validation.
Appendix: Quick Reference for Key Changes
| Commit Hash | PR # | Type | Summary |
|---|---|---|---|
96048a7b8 |
#11167 | Feature-New | Azure Cosmos DB Source (ilovezeri333, first-time contributor) |
a9f69848a |
#10780 | Feature-New | DB2 CDC Source (davidzollo365) |
99e53aad8 |
#11952 | Feature-New | BosFile Source & Sink (programmerloverun) |
9f229ce3e |
#11865 | Feature-New | NebulaGraph Vertex Sink (goutamadwant) |
3534417f6 |
#11968 | Feature-New | SNMP Source (goutamadwant) |
e352beba2 |
#11998 | Feature-New | PostHog Source (goutamadwant) |
c5b58851b |
#11989 | Feature-New | Google Pub/Sub Source (goutamadwant) |
e97555a62 |
#11877 | Feature-New | Google Pub/Sub Sink (goutamadwant) |
8725e1e13 |
#11939 | Feature-New | Azure Queue Storage Sink (goutamadwant) |
ead62107e |
#11198 | Feature-New | Couchbase Sink (srijan-singh) |
4ebb87b8f |
#11337 | Feature-New | PythonSource connector (escheduler) |
a6beccf8b |
#11460 | Feature-New | NATS JetStream Sink (rucciva, first-time contributor) |
543f2a5a6 |
#11413 | Feature-New | HugeGraph Source + Sink multi-mapping (liu7777jx, first-time contributor; largest PR this month, +14,135/-1,223) |
f0046a002 |
#11649 | Feature-Enhancement | BigQuery Multi-Table + UniverseDomain (merlin.launay, first-time contributor) |
9ec9c4fe8 |
#11984 | Feature-Zeta | Zeta dynamic log level REST v2 (SEZ9) |
0071e8fa6 |
#11909 | Feature-Zeta | Zeta Worker Resource REST API (goutamadwant) |
7c73866c7 |
#11957 | Feature-Zeta | Zeta Job Detail real-time metrics charts (lindaluo83) |
f9b1b330f |
#11963 | Feature-CI | CI Merge Queue introduced (zniu70696) |
c44f5b338 |
#11982 | Improve-Zeta | Zeta Job stateTransitions exposure (SEZ9) |
2b51d3614 |
#11421 | Feature-Zeta | Zeta LATEST_COMPLETED Checkpoint recovery (JeremyXin) |
425d4ac57 |
#11983 | Bug-Zeta | Reject invalid log level input (SEZ9) |
f13b610ef |
#11653 | Fix-Zeta | Prevent duplicate Pending scheduling after failover (shenghang) |
1ff72ceb3 |
#11812 | Fix-Zeta | ClassLoader leak after task deployment failure (goutamadwant) |
2a739a94a |
#11492 | Fix-Zeta | Isolate cross-Worker metric Registry interference (dybyte) |
b03cf1f16 |
#12001 | Fix-Zeta | Fix metric misalignment for Schema-first multi-table sinks (BinTaoMa, first-time contributor) |
be265d45c |
#11986 | Feature-Benchmark | Zeta intermediate queue comparison benchmark (zniu70696) |
ba55ef965 |
#11025 | Feature-CDC | CDC COMMENT Schema Change event support (cloverdue) |
339fb88cce |
#11811 | Improve-Core | Expanded configuration log redaction coverage (shenghang) |
e46a7106c |
#11827 | Improve-Connector | Prometheus Checkpoint flush alignment (surafel58) |
4ba289595 |
#9993 | Refactor | seatunnel-shade module refactoring (hawk9821) |
f1a1a0abb |
#11851 | Revert | Revert Jackson relocate (restoring the stable version) (hawk9821) |
13d3977f8 |
#11870 | Docs-Cookbook | Zeta slow-operation troubleshooting cookbook (zhang-arvin) |
Top comments (0)