How a Single JDBC Parameter Drove Engine Evolution in Apache SeaTunnel Zeta

Lead-in: In a data integration system, a simple configuration parameter often masks intricate underlying engineering design. In this Apache SeaTunnel Meetup recap, the speaker takes the batch_interval_ms parameter of the JDBC Sink as an entry point to deep-dive into the batch-processing mechanisms during data writing—scaling up the perspective from the Connector layer to the overarching architecture of the SeaTunnel Zeta Engine.

Meetup Video Playback: https://youtu.be/L2QZefyJP88?si=WRZmqn_SRE4Ba_lC

About the Speaker

Zhiwei Niu: Apache SeaTunnel Contributor (GitHub ID: nzw921rx). Currently focused on risk control, specializing in data synchronization and processing. Possesses deep expertise in databases and big data technology, with extensive hands-on experience in data integration and task stability.

In a data synchronization system, many issues initially appear isolated to a specific Connector. However, as you dig deeper, you often discover that they actually touch upon the core runtime mechanisms of the entire data processing engine.

This sharing stems from a seemingly simple parameter in JDBC Sink: batch_interval_ms.

The parameter’s goal is straightforward: when data in the buffer sits longer than a specified threshold, a flush should be triggered even if the batch size hasn't been met—thereby minimizing data synchronization latency.

Yet during practical implementation, the challenge quickly escalated beyond the bounds of the JDBC Connector itself.

The core issue was not how JDBC executes SQL, but rather a fundamental runtime challenge every data synchronization engine must solve:

• Who manages scheduled background tasks?
• On which thread should the flush be executed?
• How are exceptions propagated to the main Task upon a flush failure?
• How can lifecycle operations (such as Checkpoints, close, and cancel) avoid concurrency conflicts?

Going a step further: when the Source temporarily yields no incoming data, can the system still trigger a flush on a schedule?

Thus, what started as a JDBC Sink parameter, batch_interval_ms, gradually evolved into a design overhaul at the SeaTunnel Zeta Engine level, ultimately giving birth to the Engine-Level FlushSignal design in STIP-23.

01 The Root Cause: Addressing Data Latency

In batch writing scenarios for the JDBC Sink, two conditions typically trigger a flush: batch_size and batch_interval_ms.

The logic behind batch_size is intuitive. As data flows into the Sink, the system buffers the records and checks whether the buffered count has hit the set threshold. Once met, a batch write is executed.

For instance:

if (buffer.size >= batchSize) {
    flush();
}

This logic fits naturally inside writeRecord(), because the buffer size only increments when new data arrives.

However, the semantics of batch_interval_ms are entirely different.

It does not mean "when the next record arrives, check by the way if the elapsed time since the last flush exceeds the limit." Instead, it demands a true time-based trigger mechanism: even if no new data arrives, as long as the configured interval has elapsed since the last flush, a flush operation must be triggered.

In high-throughput scenarios, the difference between these two semantics might be negligible because records arrive continuously, driving regular execution of writeRecord() and time checks.

However, in real-world production environments, many workloads do not run at a continuous high throughput.

Consider Change Data Capture (CDC) streams, small table synchronizations, off-peak business periods, or temporary lulls in a specific Source partition. Unwritten data might be idling in the buffer, but because no new records arrive, the system never enters the next writeRecord() call.

Consequently, the core promise of batch_interval_ms—triggering flushes purely based on time—fails to hold up.

This is precisely why the issue could not be patched inside the JDBC Sink alone and called for a fundamental rethink from the Engine's execution mechanism.

02 Starting from a Single Parameter

Approach 1: Spawning Background Threads inside the Connector

When first tackling this problem, the natural instinct was to introduce a scheduled task directly inside the JDBC Connector.

For example, using a ScheduledExecutorService to spin up a background thread:

scheduledExecutor.scheduleAtFixedRate(() -> {
    flush();
}, batchIntervalMs, batchIntervalMs, TimeUnit.MILLISECONDS);

This approach superficially meets the requirement. Even when the Source produces no data, the background thread fires on schedule and calls flush().

However, once plugged into the SeaTunnel Task execution model, serious issues emerge.

In the standard data pipeline, writeRecord() and flush() run on the Task's main execution path. By spawning a background thread, flush() becomes a separate, uncoordinated path managed entirely by the Connector.

Thread execution splits as follows:

Executed on Task Thread:

writeRecord(record)

Executed on Connector Background Thread:

flush()

This means two threads can concurrently access the same buffer.

At the same time, a flush might execute concurrently with Checkpoints, schema evolution, or close routines. Furthermore, exceptions occurring on the background thread cannot naturally propagate to the main Task thread, and stopping this timer thread gracefully during task failure, cancellation, or shutdown becomes a major challenge.

If every Sink Connector were to implement scheduled flushes this way, each would end up reinventing the wheel to handle thread lifecycles and timer synchronization.

Thus, the fundamental flaw became clear: to support a single timing parameter, the Connector was forced to assume responsibilities that belong strictly to the runtime engine.

This was a clear boundary violation.

Approach 2: Checking Elapsed Time within writeRecord

Another line of thought was to avoid background threads altogether and simply check the timestamp inside writeRecord():

public void writeRecord(Row row) {
    buffer.add(row);

    if (buffer.size() >= batchSize) {
        flush();
        return;
    }

    if (System.currentTimeMillis() - lastFlushTime >= batchIntervalMs) {
        flush();
    }
}

This avoids extra threads. Flushing and writing remain strictly on the single-threaded execution path, exceptions propagate cleanly to the Task, and lifecycle management remains simple.

Yet, it misses the primary objective.

If no new data arrives, writeRecord() is never invoked.

Hence, what this actually implements is not "flush every 5 seconds," but rather "when the next record arrives, if more than 5 seconds have elapsed, execute a flush."

While this subtle difference is imperceptible under steady data flows, it completely falls apart in low-throughput, intermittent CDC, small table sync, or idle Source scenarios—leaving buffered data stranded indefinitely.

In short, neither approach fulfills the true semantics of batch_interval_ms.

Spawning background threads inside the Connector guarantees time-based execution but introduces severe concurrency, exception propagation, and lifecycle headaches. On the flip side, evaluating time within writeRecord() preserves the single-threaded execution model but fails when Sources go idle.

The core issue was never JDBC implementation—it was abstraction hierarchy.

batch_interval_ms fundamentally requires a core Engine capability:

Even when no incoming data exists, the Engine must generate a control event based on time. This event does not manipulate the Sink directly; instead, it flows down the standard data pipeline, allowing the Sink to execute the flush on its own consumer thread.

In short:

Engine Timer → FlushSignal → Sink flushAction

This is precisely the core problem STIP-23 was built to solve.

03 Zeta Task Execution Model: Integrating FlushSignal into the Engine

Before exploring how the Engine generates FlushSignals, we must first understand how tasks run inside the SeaTunnel Zeta Engine. FlushSignal is not an isolated trick; it must integrate seamlessly into Zeta's existing data processing framework and traverse the data pipeline managed by the Engine.

Without a firm grasp of task scheduling, data flow, and control event processing, it is hard to see why FlushSignal was designed as an Engine-Level Signal rather than a Connector-managed utility.

Task Execution Model: Scheduling and Dispatching

In the Zeta Engine, once a data synchronization job is submitted, it is not executed directly by a single thread. Instead, the Engine orchestrates, splits, and dispatches the job into distinct Tasks across the cluster.

The Engine oversees the complete job lifecycle—including job submission, resource coordination, Task instantiation, and runtime status monitoring. Based on the logical execution plan, the Engine maps different data processing stages to corresponding Tasks, which perform the actual work.

From an execution perspective, a Task is not an isolated bit of business logic, but a managed execution unit operating under the Engine. It wraps input, transformation, and output logic, passing data according to runtime rules defined by the Engine.

This architecture ensures Connectors do not need to worry about task scheduling or thread management. A Connector simply provides data processing logic, while task lifecycles, state management, and threading models are handled uniformly by the Engine.

This serves as the foundation for FlushSignal.

If a Connector spawns its own Timer to call flush() directly, it bypasses the Task execution model entirely, spawning an unmonitored side-channel that the Engine cannot manage.

Data Pipeline: How Records Flow

Beyond task execution, we must look at how regular data records traverse the Zeta Engine.

SeaTunnel follows a classic data flow topology:

Source → Transform → Sink

The Source extracts data from external systems and translates it into internal, standardized SeaTunnel records.

Next, records enter the Transform phase, where field mappings, filtering, and transformations are applied according to user configurations.

Finally, transformed records pass downstream to the Sink, which writes them to the target storage system.

Crucially, records are not directly handed off from Source to Sink via direct method calls; they flow through data channels managed by the Engine.

This implies that every stage in the data flow operates under unified Engine control, including memory buffer passing, thread models, and task status tracking.

For FlushSignal, this architectural trait is vital.

If the Engine already boasts a battle-tested data transmission channel, the most logical design for new control events is to reuse this existing pipeline rather than building a side-channel.

In other words, FlushSignals should not bypass the engine via Timer threads calling Sinks directly. They should enter the Engine's data channel just like ordinary records.

Control Events: How SchemaChange Synchronizes

Alongside standard data records, the SeaTunnel Engine manages a second vital class of data: control events.

SchemaChange represents a classic example.

In CDC or dynamic schema synchronization scenarios, the structure of source data can evolve dynamically (e.g., adding columns, altering data types).

These alterations are not standard data records; they are structural metadata events that must be synchronized downstream.

Consequently, SchemaChange events travel directly along the data pipeline.

They are not processed in isolation at the Source, nor do they bypass Transform or Sink modules. Instead, they flow downstream as special control events within the stream, recognized and passed along at each stage.

Transforms typically pass structural change events through without needing to interpret their business logic.

Ultimately, the Sink acts on the SchemaChange event, applying DDL updates to the target system.

This proves that the SeaTunnel Engine data stream carries both operational Data Records and runtime Control Events.

The design of FlushSignal leverages this exact pattern. If SchemaChange can travel seamlessly through the data pipeline as a control event, FlushSignal can do the same.

Checkpoint: The Barrier Processing Pipeline

Beside SchemaChange, Checkpoint Barriers represent another critical control event within the Zeta Engine.

In Exactly-Once processing scenarios, Checkpoints ensure state consistency and data delivery guarantees.

During a Checkpoint, the Engine generates a Barrier and injects it into the data channel.

The Barrier flows downstream alongside data records, establishing a consistent state boundary across task operators.

Upon receiving a Barrier, a Task executes corresponding routines—such as persisting operator states and coordinating Checkpoint completion—ensuring the overall pipeline can safely resume from a known checkpoint state in case of failure.

This underlines a fundamental design rule:

Control events do not need to bypass the data pipeline; they can natively leverage existing Engine channels.

Barriers are not dispatched via sideband thread calls; they propagate through the Task's data stream.

This directly aligns with the philosophy behind FlushSignal.

FlushSignal is not an out-of-band call, but a brand-new control event type. Like SchemaChange and Checkpoint Barriers, it is generated by the Engine and propagated via existing data pipelines.

By examining task scheduling, record flow, SchemaChange propagation, and Checkpoint Barrier management, we see that the SeaTunnel Zeta Engine already possesses a robust data and event processing framework.

Therefore, when JDBC Sink ran into the batch_interval_ms limitation, the right solution was not adding local timer threads to JDBC, but introducing a new control event into the Engine's native execution model.

This explains why STIP-23 ultimately opted for an Engine-Level FlushSignal. Rather than introducing a specialized patch for JDBC, it introduces a standardized runtime signal for time-driven flushes built atop SeaTunnel's existing Task and event architecture.

04 Engine-Level Flush Abstraction

During JDBC Sink development, the underlying challenge exposed by batch_interval_ms was not how to execute a flush, but how to elevate flushes to an Engine-level scheduled capability.

Traditional Connector designs treat flushes as localized buffer commits, evaluated inside writeRecord() as volume thresholds are met. However, time-driven triggers differ fundamentally from data-driven triggers. While batch_size relies on continuous incoming data, batch_interval_ms requires a flush to fire once an interval elapses—even if zero new records enter the pipeline.

Hence, flushing can no longer live as a localized Connector method call. It must be promoted to an Engine-managed runtime event.

The core philosophy of STIP-23 is to abstract Flush operations into Engine-level FlushSignals. Connectors no longer manage local Timers, nor do background threads invoke Sink flush routines directly. Instead, the Engine generates Signals based on configured time intervals, propagating them along SeaTunnel's existing data channels so Sink operators can handle them according to their own semantics.

Under this model, FlushSignal behaves identically to existing SeaTunnel control events. The data pipeline handles standard Data Records alongside Checkpoint Barriers and SchemaChange events. FlushSignal enters the record stream as a first-class control event, giving the Engine unified lifecycle and propagation control.

From Flush Method to Signal Event

Translating Flush operations from direct method calls into Signal events marks the turning point of this architecture.

Traditionally, flushes happened entirely inside Sink operators. For example, in the JDBC Sink, when internal buffers hit capacity, executeBatch() flushed data to the database. This pattern worked flawlessly for volume-based triggers since incoming records continually updated buffer states.

However, once timing triggers enter the picture, the model breaks down. batch_interval_ms implies: "After a specified duration, attempt a flush even if no new records arrive." If this logic remains trapped inside writeRecord(), the execution degrades to "check if timed out whenever the next record happens to arrive," which misses the mark for scheduled flushes.

To illustrate how different event types flow through SeaTunnel's data channel, we use specific identifiers:

• R denotes DataRecord (standard operational data);
• Ck denotes Checkpoint Barrier (marking checkpoint state boundaries);
• Sc denotes SchemaChange (representing metadata evolution events);
• F denotes FlushSignal (representing time-driven flush intent).

In a standard data stream, records and barriers propagate sequentially:

R → R → R → Ck → R → R → Ck

When SchemaChange occurs, it injects directly into the stream:

R → R → Sc → R → R → Ck

SchemaChange operates as a inline control event without disturbing the overarching delivery model.

FlushSignal follows this exact paradigm:

R → F → Sc → R → R → Ck → R

Rather than bypassing the data path to notify the Sink directly, it steps into the stream as a native event type.

Thus, FlushSignal does not introduce a ad-hoc invocation path; it elevates flushing into a runtime signal that the Engine can observe, route, and schedule.

How the Engine Triggers Signals

The creation of FlushSignals is handled exclusively by the Engine, completely freeing Connectors from managing background threads.

The complete Engine trigger flow proceeds as follows:

The process begins with job configuration.

When setting:

sink.flush.interval

to a non-zero value, the FlushSignal capability activates. Setting it to 0 disables the feature without affecting legacy job configurations.

During task startup, SourceFlowLifeCycle registers the corresponding Timer. Subsequently, the timerFlushWorker inside TaskExecutionService executes recurring tasks based on fixed-delay scheduling.

Here, the Timer's responsibility is strictly constrained: it only fires periodic notifications and triggers onTimerTick(). The Timer thread itself never executes Sink flushes or manipulates SinkWriter objects directly.

When a Timer tick occurs, the Engine calls:

collector.sendFlushSignal(jobId, taskId)

injecting a FlushSignal into the Source output.

Notably, injecting a FlushSignal relies on the exact same checkpointLock used by Checkpoints.

This design prevents race conditions between FlushSignals and Checkpoint Barriers, as both act as state-altering control events within the data processing path.

By generating FlushSignals at the Source according to standard execution models, the Engine avoids forcing direct executions on Sinks via unmanaged background threads.

Signal Propagation across the Pipeline

Once injected, the FlushSignal travels downstream via SeaTunnel's native Record channel.

The signal propagation chain proceeds as follows:

The signal may transit through internal Engine components like Queue / Disruptor buffers along the way.

The end-to-end responsibilities are divided cleanly:

• Engine Timer generates the Signal;
• Source injects the Signal into the stream;
• Transform transparently passes the Signal through;
• SinkFlowLifeCycle detects the Signal and executes the flush routine.

At the Source stage, FlushSignal is broadcast downstream via:

sendRecordToNext()

reaching all downstream consumers through:

output.received(record)

In short, FlushSignal relies on the exact same transmission mechanics as standard Data Records.

When passing through Transforms, no transformation or business logic is applied.

Upon recognizing a Signal, the Transform calls collector.collect(record) directly, skipping transform(). This guarantees FlushSignals bypass record-level processing while preserving their sequence in the stream.

Finally, the FlushSignal arrives at SinkFlowLifeCycle.

The Sink identifies the item as a Signal rather than a Data Record, routing it to:

processSignal()

which executes the registered flush action.

This highlights a core architectural principle:

The signal propagates down shared record channels, but only the Sink interprets and acts upon its semantic meaning.

How Signals Handle Backpressure

Once inside the data channel, FlushSignals must handle backpressure gracefully when buffer queues reach capacity.

SeaTunnel applies distinct queuing policies depending on event types:

For standard DataRecords:

put() / ringBuffer.next()

When queues fill up, records block and wait for available capacity to guarantee zero data loss.

For Checkpoint Barriers:

Barriers similarly execute:

put() / ringBuffer.next()

blocking until space frees up to ensure state integrity across Checkpoints.

However, FlushSignal uses a non-blocking strategy:

offer() / tryPublishEvent()

If the queue is full:

Immediately returns false

and:

Drops the current Signal

In other words, FlushSignal never blocks the core data pipeline when queues are saturated.

This design makes complete sense because FlushSignal carries no business payloads—it signals a transient flush intent.

If internal queues are already saturated, the system is actively processing a heavy volume of data. Under high load, prioritizing raw business records and checkpoint integrity over timed flushes is the correct trade-off.

Furthermore:

prepareClose:
Signal bypassed directly

and:

Queue Full:
FlushSignalQueueFailureTotal +1

These explicit safeguards ensure FlushSignals never block job shutdown, error recovery, or core data throughput under stress.

This keeps FlushSignal lightweight and highly resilient without risking stream stability.

How the Sink Executes Flushes

When a FlushSignal reaches SinkFlowLifeCycle, the Sink alone decides how to execute the operation.

Throughout this entire process, the Engine remains agnostic about JDBC-specific SQL flushing logic or database commands.

The Engine's job ends once the FlushSignal is delivered to the Sink.

Upon receiving the Signal, SinkFlowLifeCycle enters:

processSignal()

and routes execution based on signal classification.

For standard Records:

Record → sinkWriter.write()

continuing normal data ingestion.

For FlushSignals:

FlushSignal → flushAction

executing the specific flush behavior registered by the Connector.

This enforces a clean separation of concerns between Engine and Connector.

The Engine supplies FlushSignal infrastructure without dictating what concrete actions a Sink must take upon arrival.

Because flush semantics vary widely across storage systems:

• For JDBC Sinks, a flush means executing executeBatch() SQL statements.
• For other Sinks, it might trigger bulk HTTP uploads, stream loads, or memory buffer commits.

Hence, each Connector explicitly opts in by registering its own flushAction.

This makes FlushSignal an extensible Engine primitive rather than a hardcoded trick bound to a single Connector.

Ultimately, flushing evolved from an internal JDBC Sink parameter into a unified runtime control event across the SeaTunnel Engine. The Engine governs signal generation and propagation, while Connectors govern local execution—collaborating through clean interfaces.

This represents a key architectural step forward in STIP-23. It solves far more than batch_interval_ms—it equips the SeaTunnel Zeta Engine with a robust, generalized runtime control framework.

05 Exactly-Once Guarantees

Elevating scheduled flushing into an Engine-level FlushSignal solved the timing issue, but introduced a crucial challenge: how to preserve SeaTunnel's strict Exactly-Once delivery semantics when FlushSignals trigger mid-stream.

For Sinks leveraging XA transactions, a Flush does not equal a Transaction Commit. A FlushSignal can trigger localized batch updates or prepare transaction boundaries, but it must never bypass Checkpoints to commit transactions independently. Under SeaTunnel's Exactly-Once model, transaction states are tightly bound to Checkpoint State. Only transactions wrapped inside a persisted Checkpoint qualify for final commits.

Therefore, Engine-Level FlushSignals must adhere to a strict invariant: FlushSignals may advance internal transaction states, but they can never commit transactions independently of Checkpoints.

Constraints of Exactly-Once

Before FlushSignal was introduced, Sink transaction commits were managed entirely by Checkpoints. When a Checkpoint Barrier arrived, the system persisted transaction states, executing final commits only during the notifyCheckpointComplete phase.

If a FlushSignal were to trigger an XA COMMIT directly, this state model would break down.

The pattern on the left side of the diagram is dangerous because transactions are committed before the Checkpoint State is recorded.

If an XID executes XA COMMIT before its Checkpoint successfully persists, SeaTunnel's state manager has no record of that transaction being committed upon recovery.

If the Source fails shortly after and restarts from the previous Checkpoint, it will re-consume and re-send those exact records—causing duplicate writes in the target database.

Thus, neither Timers nor FlushSignals can ever commit transactions directly.

The correct approach is shown on the right side of the diagram. Here, the FlushSignal simply advances the active transaction into a PREPARE state, leaving the final COMMIT waiting for Checkpoint completion.

In short: PREPARE can occur early, but COMMIT must wait for Checkpoint.

This allows FlushSignal to adjust the cadence of transaction staging without compromising Exactly-Once boundaries.

Following this principle, the current JDBC XA Writer implementation does not register timer-based flushes, keeping transaction commits firmly anchored to the Checkpoint lifecycle.

Splitting XA Transactions

With direct commits off the table, the next question becomes: how does FlushSignal influence XA transaction demarcation?

The target design model functions as follows:

• R denotes DataRecord;
• F1, F2 denote FlushSignals;
• CK denotes Checkpoint Barrier.

When a FlushSignal arrives, it does not finish the Checkpoint lifecycle. Instead, it helps the Sink split large transaction boundaries into manageable chunks.

Transactions are chunked as follows:

txn-1: R R R F1
txn-2: R R F2
txn-3: R R CK

When F1 arrives, the Sink executes:

executeBatch() → XA PREPARE → beginTx()

completing data staging for that specific segment.

Subsequent records flow into a fresh transaction scope until the next FlushSignal arrives.

When F2 arrives, it triggers another PREPARE sequence for that segment.

Crucially, these prepared transactions remain uncommitted in the database:

pendingCommitInfo and pendingStates are staged in memory, then aggregated and persisted during the next Checkpoint.

FlushSignal simply helps the Sink partition monolithic transactions into multiple prepared XA transactions, while leaving final execution rights to Checkpoints.

When the Checkpoint Barrier finally arrives:

prepare txn-3
snapshotState()
merge pending
    ↓
Checkpoint State: [txn-1, txn-2, txn-3]

The Checkpoint State records all prepared transactions (txn-1, txn-2, txn-3).

Finally, during the execution of:

notifyCheckpointComplete

the engine calls:

XA COMMIT ALL

committing all prepared segments in one atomic step.

This design allows FlushSignal and Exactly-Once semantics to coexist seamlessly. FlushSignal provides flexible transaction boundaries, while Checkpoint maintains absolute control over final commits.

Failover and Recovery Rules

In distributed execution environments, tasks can fail at any moment. Therefore, robust recovery rules are required to resolve transaction states during failovers.

The golden rule remains: Recovery depends strictly on whether an XID was persisted in a completed Checkpoint State.

FlushSignal itself plays no role during recovery; it merely influences when transactions enter the prepared state.

Depending on when a failure occurs, recovery falls into three distinct scenarios:

Scenario 1: Failure occurs before Checkpoint completion.

Though txn-A and txn-B were prepared via F1 and F2, their metadata never made it into a persisted Checkpoint State.

Recovery sequence:

State contains no txn-A / txn-B
    ↓
XA RECOVER
    ↓
ROLLBACK
    ↓
Source replayed from CK-N

Uncommitted prepared transactions are safely rolled back, and the Source replays stream data from the last valid Checkpoint.

Scenario 2: Failure occurs after Checkpoint State is saved, but before COMMIT.

Here, XIDs were successfully persisted inside the Checkpoint State.

Recovery sequence:

State contains txn-A / txn-B / txn-C
    ↓
restoreCommit
    ↓
COMMIT

The system reads transaction metadata from the Checkpoint State and commits them to the target storage.

Scenario 3: Failure occurs after Checkpoint commits complete.

Recovery sequence:

Transactions already committed
    ↓
XA RECOVER returns empty
    ↓
no-op

No orphan transactions remain; the system resumes standard execution.

Across all three failure modes, FlushSignal never compromises SeaTunnel's core Exactly-Once guarantees. It simply introduces finer control over transaction staging within the overarching Checkpoint framework.

FlushSignal handles staging, Checkpoint State handles persistence, and notifyCheckpointComplete handles execution.

This clean separation ensures Engine-Level flushing operates safely within XA transactions while establishing a extensible pattern for future runtime events.

06 System Boundaries and Architecture Roles

The journey from fixing JDBC's batch_interval_ms to introducing an Engine-Level FlushSignal forced a fundamental rethink of the architectural boundaries between SeaTunnel Engine and its Connectors.

Initially, looking at the problem purely through the lens of the JDBC Connector led to localized fixes.

Because the JDBC Sink needed timed flushes, the immediate thought was to spawn a local Timer inside the JDBC plugin.

However, deeper analysis revealed that Timer management, thread coordination, exception propagation, and task lifecycle management are not JDBC-specific problems at all. They are fundamental runtime capabilities that belong to the core execution engine.

If every Connector implemented its own Timer, code duplication would explode across the codebase.

Each Connector would need to reinvent how to launch, stop, and safeguard background threads against race conditions with Checkpoints, job cancellations, and teardowns.

Eventually, a simple configuration parameter would force Connectors to maintain increasingly complex runtime logic.

That is a classic anti-pattern.

Through the design of FlushSignal, SeaTunnel re-established clear domain boundaries between Engine and Connector.

Connectors focus strictly on external system semantics.

For instance, the JDBC Sink knows how to construct batch SQL statements, execute transactions, handle prepare/commit/rollback semantics, and perform flushes when requested.

These operational details belong exclusively to the Connector.

The Engine manages generic, system-wide runtime mechanics.

This includes Timer lifecycles, control event generation, stream signal routing, and ensuring Signals execute safely on the correct Task thread.

These capabilities belong to the platform engine, not individual plugins.

Therefore, instead of having JDBC manage local Timers, the cleaner architecture dictates:

1. Connector registers a flushAction.
2. Engine generates and routes the FlushSignal downstream to the Sink.
3. Sink executes the flush action safely on its dedicated processing thread.

This cleanly decouples timing mechanics from target storage behaviors.

In STIP-23, the Engine exposes this configuration capability cleanly:

sink.flush.interval = 5000

This setting defines scheduled behavior at the Engine level.

When enabled, the Engine periodically emits FlushSignals. Setting it to 0 disables signal emission, keeping legacy jobs operating as normal.

Yet, the Engine never invokes external flush actions directly.

The Engine remains intentionally ignorant of whether a JDBC Sink flushes via executeBatch(), or whether another Sink relies on bulk HTTP posts, stream loads, or memory buffer commits.

Connectors opt in by registering an action via the Sink Context:

context.registerFlushAction(() -> {
    flush();
});

Registration is entirely at the Connector's discretion.

The overarching win here is that Timers never invoke Sink operations directly out-of-band.

If a Timer thread executed out-of-band flushes directly:

Timer Thread → SinkWriter.flush()

the system would fall back into the original multi-threading trap, reintroducing:

• Race conditions between flushes and normal writes;
• Uncaught exceptions bypassing Task error channels;
• Fragile thread lifecycle management during teardowns.

Instead, STIP-23 restricts the Timer to emitting FlushSignals. Once injected into standard record channels, the Signal flows downstream to SinkFlowLifeCycle, executing safely within the main consumer thread.

The end-to-end execution flow moves predictably:

Timer → FlushSignal → Data Path → SinkFlowLifeCycle → flushAction.run()

Flushing is successfully brought back into SeaTunnel's unified execution model.

07 Key Takeaways: What FlushSignal Teaches Us About Engine Evolution

The journey from a JDBC batch_interval_ms parameter to an Engine-Level FlushSignal started as a local patch, but ultimately revealed how modern data engines must evolve through proper abstraction. The longevity of an engine feature depends not on how many lines of code are written, but on whether domain boundaries are respected, existing architectures are reused, and extensible hooks are provided.

1. Define Semantics First: Clarify Capability Boundaries

Designing FlushSignal required establishing strict semantic boundaries for what a flush means. A FlushSignal does not guarantee a successful commit, nor does it imply data is immediately visible externally—it simply grants the Sink an opportunity to perform a flush.

Different storage engines define "success" in fundamentally different ways. At-Least-Once processing accepts replaying data upon failure, whereas Exactly-Once processing relies strictly on Checkpoints and transaction boundaries to eliminate duplicates. Thus, FlushSignal can only trigger execution opportunities; it can never replace a Connector's internal consistency mechanisms.

This explains why the Engine must never force all Sinks to execute flushes blindly. Across different Connectors, a flush might mean executing batch SQL, staging transactions, or triggering custom API calls. If the Engine ignored these nuances, it would risk corrupting transaction states and breaking Exactly-Once guarantees.

A reusable capability must clearly state what it delivers—and what it intentionally leaves to lower layers.

2. Evolve atop Existing Foundations: Reuse the Core Execution Model

FlushSignal avoids introducing ad-hoc sideband execution paths. Instead, it introduces a new event type directly into the existing Task pipeline.

Had the Timer invoked the Sink directly:

Timer Thread → SinkWriter.flush()

it might have seemed easier to code, but it would have introduced thread concurrency issues—causing race conditions with active writes, Checkpoints, and task shutdowns while masking background exceptions from the main Task thread.

Instead, FlushSignal leverages SeaTunnel's battle-tested data channels. The Engine Timer emits a Signal, injecting it at the Source to flow naturally along the Source → Transform → Sink path.

Along this chain, the Source injects the Signal, Transforms pass it through untouched, and SinkFlowLifeCycle detects it to run the local flush action. FlushSignals share the exact thread and queue models as Data Records, eliminating the need for unmanaged sideband threads.

This reflects a fundamental rule of engine design: new features should seamlessly build upon core abstractions rather than stacking special-case hacks.

3. Extension Points Dictate Evolution Costs: From Local Patches to Framework Abstractions

The value of FlushSignal extends far beyond resolving JDBC flush timeouts; it sets a precedent for how Engine and Connector responsibilities should be divided.

The Engine manages generic infrastructure—Timer scheduling, Signal generation, lifecycle coordination, and stream routing. Meanwhile, Connectors handle target-specific semantics—executing actual flushes, opting into timing features, and guaranteeing local transaction integrity.

By offering clean flushAction registration via Context APIs or SPIs, Connectors can freely opt into scheduled capabilities without the Engine needing to know lower-level implementation details.

This modular architecture delivers three distinct advantages: default behaviors preserve legacy Connector stability, new capabilities honor existing transactional semantics, and future runtime control requirements can leverage this exact same event-driven abstraction.

Looking back at where this started, batch_interval_ms was never just a missing parameter inside a JDBC Connector—it exposed a missing runtime control abstraction within the Engine itself. Moving from localized thread hacks to Engine-Level Signals represents a classic transition from quick-fix engineering to clean architectural abstraction.

A mature data engine does not attempt to hardcode every edge case up front. Instead, it establishes robust abstractions and extension points, allowing new capabilities to integrate cleanly, safely, and at minimal cost.