600,000 Records/Sec with Sub-100ms Latency: Inside SeaTunnel Zeta’s Rigorous Performance Test

Written by Niu Zhiwei

Figure 1: JMH Summary in a single run, including complete Pipelines and SeaTunnelRow hot path operations

Figure 2: Throughput, Latency, Growth Ratio, and Valid Samples across Five Pipelines under Identical Load

1. Interpreting the Test Results

This round of benchmarking was executed on Java 8, with a 4 GiB heap, 4 visible JVM processors, and a pipeline parallelism of 4. Each job run processed 1,000,000 records, targeting a planned input rate of 600,000 records/sec, with a payload size of 256 characters per record.

The results yield three primary insights:

1. Stable and Sustainable Processing: Across all five scenario pipelines, Sink throughput consistently held between 588,000 and 591,000 records/sec. The P99 latency registered between 100 and 101 ms, while the Growth Ratio hovered tightly between 0.99 and 1.00. Additionally, all 75 out of 75 samples were marked valid. These metrics confirm that under the tested workload, the system suffered no continuous backlog accumulation.
2. Feature Overhead Within Margin of Error: The maximum discrepancy in Pipeline JMH Scores across scenarios was roughly 0.87%, whereas the margins of error for this run spanned 0.88% to 1.38%. Statistically, this indicates no observable feature-induced overhead under this configuration—though it stops short of declaring any feature "zero-overhead."
3. Fixed Test Load vs. Capacity Limit: The 600,000 records/sec figure reflects a pre-set, fixed workload rather than the absolute performance ceiling. Evaluating maximum capacity would require systematically scaling up input rates while closely tracking throughput, P99 latency, and latency growth trends.

In summary, these data points demonstrate that the benchmark path is fully connected and sustainable under current loads. They provide a sound baseline for standardizing performance comparisons across various engine capabilities, rather than delivering an absolute performance verdict stripped of runtime context.

2. Why Micro-benchmarks and Complete Pipelines Co-exist

SeaTunnel employs a two-tier benchmarking strategy, with each level engineered to answer distinct operational questions.

SeaTunnelRow Micro-benchmarks: Pinpointing Hot Paths

The SeaTunnelRowBenchmark isolates basic operations along the critical execution paths of Sources, Transforms, and Sinks. This includes Row creation, field reads, cloning, projections, Options handling, and size calculations.

Operation	Score (ops/ms)	CV	Primary Scope & Meaning
copyPlainRow	22,328.74	1.01%	Full clone of a standard Row
copyProjectedPlainRow	59,769.89	1.57%	Projected clone (4 out of 8 fields)
copyProjectedRowWithOptions	49,282.32	0.59%	Projected clone with Options processing
copyRowWithOptions	20,789.92	0.22%	Full clone of a Row containing Options
copyRowWithTracePayload	20,499.68	0.27%	Full clone of a Row with Trace Payload
copyThenMutateCopiedOptions	7,305.63	1.01%	Clone followed by Options mutation
createRowAndGetBytesSize	5,837.52	0.68%	Row instantiation paired with size calculation
createRowWithSetField	7,668.90	0.59%	Field-by-field Row construction
getBytesSizeCached	254,677.57	1.13%	Reading pre-cached byte sizes
readFields	28,347.47	0.47%	Reading and consuming all fields

Higher ops/ms values indicate superior performance (e.g., copyPlainRow at 22,328.74 ops/ms corresponds to roughly 22.33 million operations per second).

These micro-metrics are primarily designed for cross-version comparisons of identical methods or for benchmarking functionally similar operations. Crucially, a high score in getBytesSizeCached relative to createRowAndGetBytesSize cannot be translated linearly to overall pipeline speedups: the former merely fetches cached state, whereas the latter encompasses both object instantiation and initial byte calculation.

Comparing structurally similar cloning operations reveals that full cloning with Options exhibits a ~6.9% drop in throughput compared to plain cloning, while adding a Trace Payload decreases throughput by ~8.2%. These granular variances offer useful optimization directives, though their real-world impact must always be validated through end-to-end Pipeline benchmarks.

Zeta Pipeline Benchmarks: End-to-End System Validation

The full Pipeline benchmark provisions live single-node Zeta Masters and Workers. Bounded jobs are executed through actual Client invocation, configuration parsing, task submission, and scheduling paths.

Cluster initialization occurs during the Trial Setup phase and is explicitly excluded from active timing intervals. Every benchmark invocation submits a bounded job carrying 1,000,000 records. The timed measurement window encompasses job configuration generation, submission, scheduling, Source generation, optional Transforms, Sink execution, and completion wait times.

Compared to isolated unit calls, this architecture mirrors production execution while eliminating external environment jitter via in-memory Sources and Blackhole Sinks—making it the ideal setup for assessing engine-level code updates.

3. Theoretical Foundations of the Benchmark Design

Benchmark credibility rests on two pillars: statistical reliability of the collected metrics, and accurate alignment between measured indicators and actual performance bottlenecks. Achieving the former requires handling JVM variability, isolating independent samples, and controlling experimental noise; achieving the latter demands clear boundaries between throughput and latency measurements.

Our benchmarking framework builds upon three foundational studies in systems evaluation:

Research Study	Core Problem Addressed	Key Findings & Recommendations
Statistically Rigorous Java Performance Evaluation (OOPSLA 2007)	Java execution varies per run. How do we extract statistically sound conclusions?	Distinguish startup from steady-state performance; treat independent JVM runs as critical experimental units; report both point estimates and uncertainty metrics.
Rigorous Benchmarking in Reasonable Time (ISMM 2013)	Builds, JVM instances, and iterations introduce variance. How should finite benchmark budgets be allocated?	Identify variance across execution layers; run calibration experiments to direct repetition budgets toward the primary sources of uncertainty.
Benchmarking Distributed Stream Data Processing Systems (ICDE 2018)	Where should throughput and latency timing begin in stream processing systems?	Employ open-loop workload generation and event-time tracking; evaluate sustainable throughput alongside backlog accumulation and tail latency.

Together, these studies form a continuous methodology: verifying sample validity, spending execution budgets efficiently, and establishing correct measurement boundaries.

JVM Performance Evaluation: From Raw Repetition to Statistically Sound Sampling

Java runtime performance is non-deterministic. Even with fixed codebases, parameters, and physical hardware, variables like JIT compilation timing, Garbage Collection (GC) pauses, thread scheduling, heap memory layout, and OS interruptions introduce performance drift across runs.

Relying strictly on the "best run out of N" distorts true operational metrics. Selecting the peak result answers only "how fast the system ran under optimal conditions." As N scales, the probability of sampling an abnormally lucky run increases, systematically skewing conclusions in favor of higher-variance implementations.

Furthermore, evaluation frameworks must differentiate between startup performance and steady-state capability. Startup metrics capture JVM spin-up, class loading, initialization overhead, and cold execution; steady-state metrics isolate sustained processing power post-initialization and JIT compilation. Conflating these two regimes compromises data integrity.

Because iterations within a single JVM instance share JIT compilation profiles, compiled code artifacts, heap state, and GC history, they cannot be treated as statistically independent trials. Independent JVM invocations (represented as Forks in JMH) constitute indispensable units of measurement. Reliable reporting must present point estimates accompanied by confidence intervals, rather than isolated averages or peak figures.

Allocating Experimental Budgets: Identifying Variance Layers

Increasing benchmark iterations blindly does not guarantee a proportional rise in data precision. Performance experiments operate across nested structural layers:

Build
 └─ JVM Execution / Fork
       └─ Measurement Iteration
            └─ Pipeline Invocation

Variance can emerge at every layer: rebuilding code alters binary layouts; launching new JVMs resets JIT and heap configurations; while iterations within the same JVM are susceptible to localized runtime jitter.

If primary variance stems from JVM Forks, increasing iteration counts within a single Fork from 5 to 50 will not mitigate inter-Fork discrepancies; budget is better spent allocating additional independent Forks. Conversely, if code compilation introduces non-trivial drift, repetitions must be elevated to the Build layer.

We begin by conducting calibration experiments: capturing raw samples across all layers, estimating variance contributions alongside execution costs for Builds, Forks, and Iterations, and directing experimental budgets toward dominant noise sources. Consequently, static Fork and Iteration counts serve merely as baselines rather than unchangeable configurations.

Equally critical is differentiating between random errors and systematic bias. Additional repetitions improve random error estimations, but cannot correct for structural biases like host background load or execution ordering. If a baseline runs on an idle machine while a candidate runs under system load, higher sample counts merely yield a more precise measurement of a flawed setup.

Stream Processing Metrics: Shifting from Stability to Accuracy

While the first two studies establish statistical rigor, the third addresses metric validity: whether measured parameters truly reflect stream processing capabilities.

Traditional closed-loop workload generators issue new batches only after previous ones finish processing. If the processing engine slows down, the generator throttles its issuance rate accordingly, obscuring queuing delays occurring prior to engine ingestion. This creates a misleading pattern where an overloaded system reports deceptively low internal processing latencies due to Coordinated Omission.

To prevent this, input rates must operate independently of downstream processing speed, and latency must be measured against an event's original scheduled creation time. Otherwise, queueing prior to Source ingestion is ignored, yielding metrics that capture processing speed post-ingestion while omitting end-to-end event delay.

This underscores the distinction between sustained throughput and transient peak processing rates. Sustained throughput represents the maximum input rate an engine can process indefinitely without accumulating persistent backlogs or inflating event-time latency. Throughput, data integrity, tail latency, and latency growth trends must be analyzed holistically; evaluating any single metric in isolation risks incorrect performance assessments.

Synthesis of the Three Methodologies

These three principles form a rigorous benchmark evaluation workflow: establish whether the scope targets startup, steady-state, sub-routine hot paths, or end-to-end architectures; identify independent sampling units and variance sources; and verify that throughput and latency boundaries accurately reflect production constraints.

Statistical analysis cannot salvage flawed boundary definitions, nor can valid boundaries compensate for uncalibrated sample variance. Only when both domains align can benchmark results evolve from isolated execution figures into actionable engineering evidence.

4. Operationalizing Theory in Zeta Pipeline Benchmarks

Translating these theoretical principles into the Zeta Pipeline benchmarking suite relies on three implementation choices: preserving independent JVM samples, allocating iteration budgets to primary variance layers, and employing open-loop generation to measure true queuing delays.

Plan-Time Scheduling with BenchmarkSource

The system's Source schedules planned generation times using absolute temporal offsets:

scheduledTime = startTime
              + wholeSeconds × 1000
              + remainder × 1000 / rate

Generation schedules remain decoupled from downstream processing delays. Upon reaching the Sink, event latency is computed as:

event-time latency = Sink Receipt Time - Planned Generation Time

Any processing bottlenecks within the engine immediately register in the P50, P95, and P99 percentiles, preventing Source-side throttling from masking true internal delays.

Isolating Engine Overhead via Deterministic Components

Test components are engineered to minimize stochastic noise:

• Source: Generates a fixed volume of memory records with static payload sizes.
• Transform: Executes deterministic hashing ops and outputs verifiable checksums.
• Sink: Bypasses external IO; tracks row counts, throughput, latency distributions, and checksums.
• Pipeline Uniformity: Every pipeline scenario retains identical record counts, parallelism settings, and resource allocations, altering only the target functional capabilities.

The five benchmark scenarios cover baseline data flows, Transforms, real-time busyness metrics, StainTrace tracking, and composite feature sets. This guarantees uniform workloads while isolating incremental engine overhead.

Data Validation as a Prerequisite for Sample Validity

High throughput paired with unverified or dropped data yields invalid metrics. Every completed job undergoes structural validation:

• processed_rows must match expected_rows exactly.
• Baseline Source -> Sink pipelines must output a checksum of 0.
• Transform-enabled pipelines must produce non-zero checksums.
• Latency percentiles must fit within tracked bounds without hitting overflow buckets.
• P99 latencies and growth ratios must satisfy strict safety thresholds.

A sample is included in final reporting only when data completeness, functional execution, and latency metrics pass validation.

Fork Configurations, Warmups, and Machine-Readable Outputs

The default JMH setup runs 3 independent Forks, each including 3 warmup iterations and 5 measurement iterations. Warmup results are excluded from aggregated reporting; only measurement iterations populate final datasets.

Because a single measurement iteration executes multiple full Pipeline invocations, a reported 75/75 metric represents 75 validated, end-to-end job runs rather than 75 individual data rows or 15 JMH iterations.

Alongside Markdown summaries, the suite exports raw JMH JSON, per-run Pipeline JSON, normalized metrics, and environmental metadata. Version control commits, JDK builds, JVM arguments, container images, CPU topologies, kernel revisions, host memory, and load parameters are logged to guarantee run-to-run comparability.

The 3 Forks × 5 Measurement Iterations setup provides an initial baseline budget. As continuous integration collects historical performance trends across dedicated hardware, these repetition rates, execution rules, and regression thresholds will be adjusted dynamically.

5. Deciphering Dual-Layer Metrics

Differentiating JMH Scores from Pipeline Throughput

The Pipeline benchmark registers 1,000,000 logical operations per invocation. JMH converts total job execution time into an ops/s score:

JMH Score ≈ 1,000,000 / Total Job Duration

This total duration includes job setup, configuration parsing, submission, task scheduling, Source generation, Transform processing, Sink consumption, and teardown synchronization.

Conversely, Pipeline JSON records a narrower measurement window:

Pipeline Throughput
    = processed_rows / (Final Sink Receipt Time - Initial Sink Receipt Time)

This isolates the interval during which the Sink actively receives data, excluding setup and teardown overheads. Consequently, reported JMH Scores range between 454,000 and 458,000 ops/s, while direct Pipeline Throughput tracks between 588,000 and 591,000 records/sec. This variance reflects distinct measurement boundaries rather than metric divergence.

Metric Definitions: Score, Error, CV, and Units

Field	Description	Analytical Application
Score	Estimated throughput measured by JMH	Higher values denote greater processing throughput.
Error	Half-width of the confidence interval expressed as a percentage of the Score	Defines the score range: Score × (1 ± Error%).
CV	Coefficient of Variation (Standard Deviation divided by Mean)	Lower values reflect higher sample stability within the run.
Unit	Measurement units	Reported as ops/s for Pipelines and ops/ms for Row micro-benchmarks.

Error and CV reflect internal variance within a single test run; they do not account for cross-node hardware variations, CPU scheduling noise, or system load fluctuations. Small score variations (~1%) between commits should not be interpreted as performance regressions without cross-run validation.

Latency Percentiles, Growth Ratios, and Valid Samples

• P50: Median latency; establishes the baseline execution delay.
• P95 / P99: Tail latency metrics; exposes queueing delays, GC pauses, and thread contention.
• Max: Peak recorded latency; captures extreme outliers, evaluated alongside percentiles.
• Growth: Ratio comparing late-stage P99 latency against early-stage P99 latency to identify backlog accumulation.
• Valid: Count of fully validated, non-overflowing measurement samples completed during testing.

The Growth Ratio is calculated using the following formula:

Growth = (Late-Stage P99 + 1) / (Early-Stage P99 + 1)

Throughput tracks completed work volume, Latency quantifies record wait time, and the Growth Ratio tracks whether queueing delays are escalating over time. Evaluating all three together is essential for accurate performance diagnostics.

6. Zeta Engine Performance Analysis

Allocated 4 visible JVM processors and a 4 GiB heap, SeaTunnel Zeta instantiated single-node Master and Worker runtimes, executing end-to-end Source -> Transform -> Sink pipelines through standard submission and scheduling paths. The engine maintained Sink throughput between 588,000 and 591,000 records/sec, capped P99 latency around 100 ms, and successfully validated all 75 out of 75 measurement runs.

Growth Ratios across all five pipeline configurations remained close to 1.0, proving that the engine handled a steady 600,000 records/sec input stream without accumulating internal latency backlogs. Incorporating Transforms, real-time busyness metrics, and StainTrace features introduced no throughput degradation beyond standard error bounds. While this does not imply "zero computational cost," it confirms these features introduce no measurable performance bottlenecks along core data paths.

These results reflect performance under controlled test parameters rather than absolute system limits, omitting external connector IO, multi-node network transport, and Checkpoint persistence costs. Nevertheless, within these test bounds, SeaTunnel Zeta demonstrated high throughput, stable tail latencies, and efficient functional overhead management—establishing itself as an engine for unified batch and stream synchronization across databases, message queues, data warehouses, and data lakes.

7. Key Takeaways and Engineering Summary

Designing an effective benchmarking suite requires establishing clear evaluation goals before writing test routines. Applying concepts like startup vs. steady-state performance, sample independence, variance layer allocation, open-loop testing, and sustained throughput prevents generating precise yet uninformative metrics.

Once testing goals are defined, key execution variables must be held constant: timing boundaries, workload profiles, and hardware resource limits. Implementing strict output validation guarantees that throughput and latency figures are evaluated only on correct, fully processed datasets.

Finally, benchmark runs should separate warmup phases from measurement windows, retain independent iterations, and track throughput alongside tail latency, Growth Ratios, and variance metrics. Single runs on shared build runners highlight performance signals, but dedicated test environments are required to collect historical distributions and establish reliable performance regression thresholds.

The value of benchmarking lies not in generating a single optimized metric, but in building a repeatable, verifiable testing framework that tracks performance shifts and catches regressions over time.