TL;DR — Most streaming pipelines report latency numbers that measure the wrong thing. The gap between event time and processing time is not a minor implementation detail — it is the fundamental reason dashboards look healthy while business logic silently produces stale or wrong results. Understanding watermarks, late data, and window semantics is the only way to build pipelines that are honest about what they know and when they know it.
Every streaming pipeline has a latency number on its dashboard. That number is almost always a lie — not because engineers are careless, but because most pipelines measure the wrong clock. They report how quickly data moves through the system, not how accurately the system reflects the world that generated the data. Those are completely different things, and conflating them is one of the most persistent sources of silent correctness failures in production data infrastructure.
Two Clocks, One Pipeline
Every event in a stream carries two timestamps. The first is event time: when something actually happened in the real world — a user clicked a button, a sensor recorded a reading, a transaction cleared. The second is processing time: when the event arrived at a particular stage of your pipeline.
In a perfect world these would be identical. In practice, they diverge constantly. Mobile clients buffer events while offline and flush them minutes or hours later. IoT devices have clock skew. Upstream services retry failed deliveries. Kafka partitions rebalance and cause brief consumer lag spikes. A single network hiccup can push a batch of events into the next processing window even though they all happened before the window boundary.
When your monitoring says "p99 latency: 800ms," it almost certainly means the 99th percentile of processing time latency — the delay from broker ingestion to some downstream sink. It says nothing about how old the events themselves were when they arrived. A pipeline can have sub-second processing latency while routinely operating on data that is five minutes stale.
Why Windows Make the Problem Concrete
Windowed aggregations are where event-time confusion graduates from theoretical concern to production incident. Consider a tumbling five-minute window that counts transactions per merchant. If you assign windows by processing time, a transaction that happened at 12:04:58 but arrived at 12:05:03 lands in the wrong window. Your 12:00–12:05 count is understated; your 12:05–12:10 count is overstated. The error is small per event, but it compounds across millions of events and it is systematically biased in the direction of recency — the most recent window always looks artificially inflated because late arrivals from earlier windows keep leaking into it.
Assign windows by event time instead, and you solve the misclassification problem — but you immediately face a new one: when do you close the window? You cannot wait forever for late events. You need a principled way to say "I am confident enough that all events for this window have arrived; I will emit the result now."
That mechanism is the watermark.
Watermarks Are a Bet, Not a Guarantee
A watermark is a statement of the form: "I believe all events with timestamps earlier than T have now been observed." It is not a guarantee. It is a heuristic — a bet your pipeline makes about the maximum out-of-orderness of your data stream.
Systems like Apache Flink let you define watermark strategies explicitly. A bounded-out-of-orderness watermark says: "Assume events arrive at most N milliseconds late." Set N too low and you close windows prematurely, discarding late data as it arrives after the watermark has already advanced past its window. Set N too high and you hold state in memory longer, increase end-to-end latency, and delay downstream consumers from seeing results.
This is not a parameter you can tune once and forget. The right value for N depends on your data source, your network topology, your client SDK behavior, and your tolerance for late-data errors — and all of those things change over time. A mobile app update that adds aggressive offline buffering can silently invalidate a watermark strategy that worked fine for a year.
The honest engineering position is: watermarks encode assumptions, and those assumptions should be monitored like any other SLO. Track the distribution of event-time lag at ingestion. Alert when the 99th percentile of lag exceeds your watermark bound. Otherwise you will not know that your pipeline has started silently dropping late events until a business analyst notices that last Tuesday's numbers look wrong in hindsight.
The Late Data Problem Has No Clean Solution
Once you accept that watermarks are bets, you have to decide what to do when the bet is wrong — when an event arrives after its window's watermark has already advanced. You have three options, none of them free:
Drop the event. Simple, but you are silently losing data. Acceptable only if you have quantified the business impact and it is genuinely negligible.
Emit a corrected result. Reopen the window, recompute, and send a retraction followed by an updated aggregate. This is what Flink's allowed lateness mechanism supports. Downstream consumers must be built to handle retractions — most are not.
Route late events to a side output. Preserve them for reconciliation, audit, or replay. This is operationally honest but requires a separate reconciliation process that most teams never actually build.
The worst choice is the most common one: drop late events silently and report a latency number that makes the pipeline look healthy. This is how you end up with fraud detection models trained on subtly incomplete feature windows, or revenue dashboards that quietly undercount during periods of high mobile traffic.
Idle Partitions and Watermark Stalls
There is a failure mode that bites almost every team the first time they run a multi-partition Flink or Spark Structured Streaming job on uneven traffic: watermark stalls caused by idle partitions.
In most streaming frameworks, the global watermark advances to the minimum watermark across all partitions. If one Kafka partition receives no events for several minutes — because traffic to that shard is genuinely zero, or because a producer is down — the global watermark freezes. Windows stop closing. Downstream results stop emitting. The pipeline appears to be processing data normally but is actually stuck, holding state and waiting for a partition that may not produce events for a long time.
Flink addresses this with idle source detection: after a configurable timeout with no events, a partition is marked idle and excluded from the watermark minimum calculation. But this is opt-in configuration that many teams never enable, and the timeout value requires the same careful tuning as the watermark bound itself. A timeout that is too short causes watermarks to advance past legitimate gaps in sparse data streams.
What Honest Latency Measurement Looks Like
A pipeline that takes event-time semantics seriously instruments at least three things:
Event-time lag at ingestion: the difference between wall-clock time and the event timestamp when the event is first consumed. This tells you how out-of-order your upstream data actually is.
Watermark lag: how far behind the current watermark is relative to wall-clock time. A growing watermark lag means your pipeline is falling behind or stalling on idle partitions.
Late event rate per window: what fraction of events for a given window arrive after that window's watermark has closed. This is the direct measure of how often your watermark bet is wrong.
None of these metrics are exotic. They are all derivable from timestamps that are already present in your events and your framework's internal state. The reason most pipelines do not track them is that setting up processing-time latency monitoring is easy and the numbers look good, while setting up event-time monitoring requires understanding the semantics first.
The Correctness Conversation Your Team Is Not Having
The deeper issue is cultural. Streaming pipeline correctness is treated as an infrastructure concern — something the platform team handles — rather than a domain-level contract between the pipeline and its consumers. Downstream teams build models and dashboards on top of aggregations without knowing whether those aggregations are event-time correct, what the watermark strategy is, or what happens to late data.
The fix is to make the contract explicit. Document, for every windowed aggregation your pipeline produces: the window type and size, the watermark strategy and its assumed maximum lateness, the late-data policy, and the expected late-event rate under normal conditions. Treat a change to any of these as a breaking schema change. Version it accordingly.
That discipline is more valuable than any framework feature. Watermarks, allowed lateness, and side outputs are tools. The judgment about what your data source actually does and what your downstream consumers can actually tolerate — that is the engineering work that determines whether your pipeline is telling the truth.
Top comments (0)