TL;DR — A watermark in a Kafka and Flink pipeline decides how much of the world a real-time feature has seen when the model reads it. Training on a batch replay sees a more complete world than production did, which creates a directional, systematic skew. Treat lateness policy as a versioned model parameter, log what serving actually saw, and measure lateness per source.
Most teams tune the learning rate, the context window, and the retrieval top-k. Almost nobody tunes the watermark. Yet in a real-time ML system built on Kafka and Flink, the watermark decides how much of reality a feature has seen at the moment the model reads it. It shapes your predictions as much as any model setting does, and it never shows up in an experiment tracker.
My thesis: the lateness policy of your stream processor is a hyperparameter of your model, and it should be versioned, logged, and evaluated like one.
What a watermark actually promises
Event time is when something happened. Processing time is when your system found out. In a clean world they are the same. In the real world they diverge because of mobile clients that go offline, retries, partner feeds delivered in batches, cross-region replication, and consumer lag after a deploy.
A watermark is the stream processor's heuristic claim: "I believe I have seen everything with an event time earlier than T." It is a guess, usually a bounded out-of-orderness assumption such as "events arrive at most N seconds late." When the watermark passes the end of a window, the window fires. Events that show up afterward are dropped, routed to a side output, or trigger a corrected result, depending on how you configured allowed lateness.
Take a feature like "number of card transactions in the last ten minutes." The value the model sees at request time is not the true count. It is the count of events that arrived before the watermark closed the relevant window, plus whatever you do with stragglers. That is a different quantity, and it depends on a number somebody picked during initial development.
The skew is directional, not random
Training/serving skew is usually described as a bug: two code paths computing the same thing differently. Watermark skew is subtler because both paths can be correct and still disagree.
Here is the typical setup. Serving reads features computed by the live streaming job. Training data comes from a batch job that replays the full event log, or from a warehouse table built after the fact. By the time the batch job runs, every late event has landed. The training feature reflects the complete history. The serving feature reflects the history as of the watermark.
So the training set is systematically cleaner and more complete than what production ever serves. The model learns, for example, that a count of seven means something specific. In production, the same underlying situation produces a count of five, because two events were still in flight. The model is not seeing noise. It is seeing a consistent undercount that training told it never happens.
It gets worse because late data is rarely random. Events arrive late for reasons that correlate with the label:
Offline or poorly connected clients often differ in behavior from well-connected ones.
Retry storms happen during incidents, which are exactly when anomalies cluster.
Batched partner feeds carry a distinct population with its own base rate.
Fraudulent activity may be deliberately shaped to look like normal, delayed traffic.
If the late tail is informative, then dropping it online and including it offline produces a skew aligned with the signal you care about. Aggregate metrics will not flag this, since the features look healthy and the pipeline is green. The model just underperforms in a way that looks like drift.
Two failure modes that live in Kafka partitions
Watermark behavior across partitions adds operational traps. A stream processor typically advances its watermark based on the slowest input. That has two consequences.
An idle partition stalls everything. If one Kafka partition stops receiving data, because of a low-traffic key range or an upstream outage, the combined watermark can stop advancing. Windows stop firing. Your features go stale while the job reports healthy throughput. Serving keeps happily reading the last computed values.
The fix for that creates the opposite problem. Configuring an idle timeout lets the watermark move on without the quiet partition. That unblocks the pipeline, but when the partition wakes up, its events are now late relative to a watermark that has moved past them. You traded staleness for silent data loss, and the loss concentrates in whichever keys live on the partition that went quiet.
Neither behavior is a bug. Both are defaults interacting with traffic shape. That is why they belong in your model's risk register rather than only in the platform team's runbook.
Treat lateness as a model parameter
The goal is not to eliminate lateness. Waiting longer for completeness costs latency, and a fraud decision that arrives after the transaction clears is worth little. The goal is to make the tradeoff explicit and make training match it. Several practices follow from that.
Log what serving saw. The most reliable training data for a real-time model is the feature vector actually served at prediction time, joined to the label later. It encodes your real watermark behavior, including incident days and idle-partition stalls. Reconstructing features after the fact is the path that produces the skew.
If you must replay, replay under the same policy. Run the same streaming logic over recorded events with the same lateness configuration, deterministically in event time. A batch SQL reimplementation with no lateness cutoff is a different feature, even if it shares a name.
Version the lateness config with the model. Allowed lateness, out-of-orderness bounds, and idle timeouts should be pinned alongside the model artifact. Changing the watermark delay for a latency win is a model change. Ship it with an evaluation, not a quiet config edit.
Measure lateness per source. Track the distribution of event-time-to-arrival gaps for each upstream, not one global p99. A tail that looks fine overall can hide one feed that is consistently minutes behind.
Count what you drop. Route late events to a side output and monitor the volume, ideally sliced by the features and segments your model cares about. A rising dropped-event rate is an early warning that precedes any accuracy metric.
Expose freshness to the model or the policy around it. Passing a "data age" or completeness signal as a feature lets the model discount a count it knows is partial. At minimum, serving logic can fall back to a safer default when freshness exceeds a threshold.
The cost curve belongs to the business, not the platform
Who decides how long to wait? Platform teams tend to pick a watermark delay that keeps latency dashboards green. Data scientists assume features are complete because the offline evaluation said so. Nobody owns the gap between them.
The right owner is whoever can price the tradeoff. For some use cases, such as recommendations or ranking, a slightly undercounted feature costs little, and low latency wins. For others, such as fraud or risk, the late tail may carry the most valuable signal, and a longer wait or a two-stage design is justified: a fast provisional score now, a corrected score when the window closes. Streaming engines support emitting early results and refinements, and ML systems should use that capability deliberately rather than treating a window as a single-shot answer.
The broader point is that "real-time" is not a property you get from Kafka and Flink. It is a negotiated position between freshness and completeness, and that position leaks directly into model behavior. If you can recite your model's learning rate but not its allowed lateness, you are tuning the wrong knob and ignoring the one that production is tuning for you.
Put the watermark in the experiment config. Put the served features in the training set. Then the skew has nowhere to hide.
Top comments (0)