Gowtham Potureddi

Posted on Aug 16

Windowing in Stream Processing: Tumbling, Hopping, Session & Global Windows

#python #sql #interview #dataengineering

A stream never ends, and that single fact breaks every aggregation you learned on batch data. You cannot run SUM, COUNT, or AVG over an infinite sequence, because the answer would never be ready — there is always one more event coming. windowing in stream processing is the mechanism that fixes this: it slices an unbounded stream into bounded chunks — windows — so that "revenue in the last 5 minutes", "unique users this hour", or "events per session" become questions with finite, emittable answers. Choosing the right window shape, and knowing what happens to events that arrive out of order or late, is the difference between a dashboard that is roughly right and a pipeline that is provably correct.

This guide walks the four window families every stream processor supports — tumbling, hopping/sliding, session, and global — and the two concepts that make them trustworthy: the distinction between event time vs processing time, and the watermark that tells the engine when a window is safe to close. Every section pairs the theory with real code across Apache Flink, Spark Structured Streaming, Kafka Streams, and Apache Beam/Dataflow, then a worked interview scenario with the answer, a step-by-step trace, and a concept-by-concept breakdown of why it is correct. By the end you will be able to read a requirement — "hourly aggregates, events up to 10 minutes late, per user session" — and map it straight to a window type, a watermark, and an allowed lateness setting without guessing.

When you want hands-on reps alongside the reading, drill the streaming practice library →, rehearse rolling aggregates on the sliding-window practice library →, and sharpen time-based queries with the time-series practice library →.

On this page

Event time vs processing time, watermarks & why windows exist
Tumbling (fixed) windows — non-overlapping, per-window aggregates
Hopping / sliding windows — size, slide & overlap
Session windows — inactivity gap, dynamic per-key windows
Global windows, triggers & side outputs for late data
Cheat sheet — windowing recipes across engines
Frequently asked questions
Practice on PipeCode

1. Event time vs processing time, watermarks & why windows exist

Windows turn an infinite stream into finite, emittable answers — and the clock you pick decides whether they are correct

The one-sentence framing that changes how you reason about streaming: a window is a rule for collecting events into a bounded group so an aggregate can be computed and emitted, and the single most important choice is which clock the window reads — the time the event actually happened (event time) or the time your system saw it (processing time) — because on a real network those two clocks disagree, and the disagreement is exactly where streaming bugs live. Every window type in this article is just a different rule for drawing those group boundaries; watermarks and allowed lateness are the machinery that decides when a group is complete.

Why windows exist at all. Batch systems have a natural boundary — the file, the table, the day's partition — so GROUP BY terminates. A stream has no end.

Unbounded input. A Kafka topic or a Pub/Sub subscription produces events forever; an aggregate over "all events" would never finish and never emit.
Windows impose a boundary. By grouping events into finite windows (by time, by count, or by activity gaps), the engine can compute a result for each window and release it downstream.
The result is incremental. Instead of one answer at the end of time, you get a stream of answers — one per window — which is exactly what dashboards, alerts, and feature pipelines consume.

The three clocks — know all three cold. Interviewers probe this because picking the wrong clock silently corrupts results.

Event time. The timestamp embedded in the event, recording when it actually occurred (a click at 10:00:03 on the user's device). Windowing on event time gives correct, reproducible results regardless of network delay — replaying the same data always yields the same windows.
Processing time. The wall-clock time of the machine when the operator processes the event. Simple and low-latency, but non-deterministic: a network hiccup shifts events into the wrong window, and a replay produces different results.
Ingestion time. A middle ground — the timestamp assigned when the event enters the streaming system. More stable than processing time, less accurate than true event time.

Watermarks — the engine's estimate of completeness. A watermark is a special marker that flows with the stream and asserts "I believe I have now seen all events with event time ≤ T."

When the watermark passes a window's end, the engine considers that window complete and fires its result.
Watermarks are generated from the data (for example, max-seen-timestamp minus a bounded out-of-orderness of a few seconds), so they trade a little latency for a lot of correctness.
A watermark is a heuristic, not a guarantee — which is why late data (events arriving after the watermark has passed their window) still needs an explicit policy.

Allowed lateness and triggers — what to do about stragglers. The completeness-vs-latency dial.

Allowed lateness keeps a window's state around for an extra grace period after the watermark passes, so genuinely late events can still update the result before the window is finally garbage-collected.
Triggers decide when a window emits: on the watermark (default), early (before the watermark, for low-latency previews), or late (re-firing when late data arrives).
Side outputs / dropped-late metrics catch events that arrive after even the allowed-lateness grace has expired, so nothing is silently lost.

The three trade-offs every window forces. No windowing choice is free; each one moves you along three axes, and interviewers want to hear you name the trade-off, not just the API.

Latency vs completeness. Fire early and you emit fast but risk missing late events; wait for the watermark plus allowed lateness and you are complete but slower. The watermark delay and lateness grace are literally the two dials on this axis.
State cost vs smoothness. Overlapping windows (sliding) and never-closing windows (global) hold more state and emit more often; disjoint windows (tumbling) are cheapest. size/slide and the retention policy set the price.
Correctness vs simplicity. Event time with watermarks is correct and reproducible but adds timestamp/watermark plumbing; processing time is trivial but non-deterministic. Pick the clock the requirement's guarantee demands, not the one that is easiest to wire.

The four window families at a glance. Keep this orientation table in your head; every later section is one row of it in depth.

Window	Parameters	Overlap?	Boundary driven by	Canonical use
Tumbling (fixed)	size	none	fixed clock intervals	per-minute / hourly roll-ups, billing
Hopping / sliding	size + slide	yes (size/slide)	fixed clock intervals	moving averages, rolling counts
Session	inactivity gap	none (but merges)	the data (activity gaps)	sessionization, per-visit metrics
Global	trigger (+ evictor)	one window	triggers only	every-N-events, CEP, count-based emit

Worked example — the same stream aggregated on event time vs processing time

Detailed explanation. The clearest way to feel the difference between the two clocks is to run the same events through both and watch them land in different windows. Consider four events for one sensor, each with an event_ts (when it happened) and an arrival_ts (when the operator saw it); one event was delayed on the network. Event-time windowing places each event by event_ts, so the delayed event still lands in its true minute; processing-time windowing places it by arrival_ts, so the delay smears it into the wrong minute.

Event-time bucketing — the reading is grouped by when it was generated, so results are correct and replay-stable.
Processing-time bucketing — the reading is grouped by when it arrived, so a delay moves it to a later window and the two windows are both wrong.
The tell — if a requirement says "correct hourly totals even with delays", it is an event-time requirement.

Question. Four temperature readings arrive; one is delayed. Which 1-minute window does each land in under event time vs processing time?

Input.

event_id	event_ts	arrival_ts	note
e1	10:00:10	10:00:11	on time
e2	10:00:50	10:00:52	on time
e3	10:00:58	10:01:30	delayed 32s
e4	10:01:05	10:01:06	on time

Code.

# Apache Flink (PyFlink) — same stream, two clocks
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.window import TumblingEventTimeWindows, TumblingProcessingTimeWindows
from pyflink.common import Time, WatermarkStrategy, Duration

env = StreamExecutionEnvironment.get_execution_environment()

# EVENT TIME: assign timestamps from the event, bounded out-of-orderness 5s
event_time = (
    stream
    .assign_timestamps_and_watermarks(
        WatermarkStrategy
          .for_bounded_out_of_orderness(Duration.of_seconds(5))
          .with_timestamp_assigner(lambda ev, ts: ev.event_ts_ms))
    .key_by(lambda ev: ev.sensor_id)
    .window(TumblingEventTimeWindows.of(Time.minutes(1)))   # bucket by event_ts
    .reduce(lambda a, b: a.merge(b))
)

# PROCESSING TIME: bucket by wall-clock arrival — no timestamps/watermarks
processing_time = (
    stream
    .key_by(lambda ev: ev.sensor_id)
    .window(TumblingProcessingTimeWindows.of(Time.minutes(1)))  # bucket by arrival
    .reduce(lambda a, b: a.merge(b))
)

Step-by-step trace.

Event time reads event_ts: e1, e2, e3 (10:00:58) all fall in the [10:00, 10:01) window; e4 (10:01:05) falls in [10:01, 10:02).
The watermark (max-seen minus 5s) only crosses 10:01:00 after e3's true time is accounted for, so the delayed e3 still lands in its correct minute.
Processing time reads arrival_ts: e1, e2 land in [10:00, 10:01); but e3 arrives at 10:01:30, so it lands in [10:01, 10:02) — the wrong minute.
e4 lands in [10:01, 10:02) under both clocks; only e3 diverges.

Output:

Window	Event-time members	Processing-time members
[10:00, 10:01)	e1, e2, e3	e1, e2
[10:01, 10:02)	e4	e3, e4

Why this works — concept by concept:

Event-time bucketing — grouping by the embedded event_ts is what makes the delayed e3 land in its true 10:00 window; the result is correct and identical on every replay.
Watermark grace — bounding out-of-orderness by 5 seconds lets Flink wait a beat before closing the 10:00 window, so a slightly-late event is still counted rather than lost.
Processing-time drift — reading the wall clock is simple but non-deterministic; the 32-second network delay alone moved e3 into the wrong minute and made both windows wrong.
Cost — event time costs a little latency (you hold windows open until the watermark advances) but buys correctness and reproducibility; processing time is lowest-latency but only acceptable when approximate, replay-unstable results are fine.

The late-data drill — reading a completeness guarantee before you pick a window

Detailed explanation. The single most transferable streaming skill is extracting the completeness guarantee from a requirement before you choose a window or a watermark, because the guarantee dictates every downstream setting. Each requirement hides its answer in one of a few families — the window shape (fixed / overlapping / gap-based / trigger-driven), the clock (event vs processing), the lateness bound (how late can data be and still count), and the late-data policy (update, drop, or side-output). Name the family and the configuration writes itself.

Shape words — "every 5 minutes, non-overlapping" → tumbling; "rolling / moving / every minute over the last hour" → sliding/hopping; "per session / per visit / burst of activity" → session; "no natural boundary, emit every N events" → global + trigger.
Clock words — "correct despite delays", "when it actually happened", "reproducible" → event time; "as-fast-as-possible", "approximate is fine" → processing time.
Lateness words — "up to 10 minutes late", "stragglers must still count" → allowed lateness = 10 min; "must be exact eventually" → keep state + re-fire.
Late-policy words — "correct the result" → accumulating + late trigger; "quarantine bad/late records" → side output; "drop and count" → dropped-late metric.

Question. For each requirement clause, name the window shape, the clock, and the lateness policy.

Input.

Requirement clause	Window shape	Clock	Late policy
"per-minute counts, correct despite delays"	tumbling	event time	allowed lateness
"rolling 1-hour active users every 5 min"	sliding	event time	allowed lateness
"group a user's clicks into visits"	session	event time	gap + merge
"emit a partial total every 1000 events"	global	either	trigger-driven

Code.

# The 4-step drill, applied to every streaming requirement:
1. SHAPE  : fixed | overlapping | gap-based | trigger-driven   -> window type
2. CLOCK  : correct-despite-delay? -> event time; else processing time
3. LATENESS: "up to X late" -> allowedLateness = X (event time only)
4. POLICY : correct | quarantine | drop-and-count -> trigger / side output

Step-by-step trace.

Take clause 2: "rolling 1-hour active users every 5 min" → SHAPE = overlapping (size 1h, slide 5m) = sliding/hopping.
"active users" over a delayed clickstream implies correctness → CLOCK = event time.
No explicit lateness bound is stated, so choose a sensible default (a few minutes) → LATENESS = allowed lateness.
The metric should be corrected as late clicks arrive → POLICY = accumulating with a late trigger.

Output:

Clause	Named shape	Clock	Policy
per-minute counts	tumbling	event time	allowed lateness
rolling active users	sliding (1h/5m)	event time	accumulate + late trigger
clicks into visits	session (gap)	event time	gap merge
every 1000 events	global	either	count trigger

Rule of thumb. If you cannot name the window shape and the clock in one phrase each, re-read the requirement — you are not ready to write the window assigner yet.

2. Tumbling (fixed) windows — non-overlapping, per-window aggregates

A tumbling window slices the stream into equal, back-to-back buckets so every event belongs to exactly one window

The invariant to burn in: a tumbling (fixed) window has a single size parameter, tiles the timeline into contiguous non-overlapping intervals of that size, and assigns every event to exactly one window — so the aggregates partition the data perfectly with no double-counting. If a requirement says "per-minute", "hourly", "daily totals", or "every N minutes, non-overlapping", it is a tumbling window and nothing else.

What defines a tumbling window.

One parameter: size. A 5-minute tumbling window produces [10:00, 10:05), [10:05, 10:10), [10:10, 10:15) — abutting, equal, gapless.
Exactly-once membership. Each event lands in precisely one window (the one its timestamp falls into), so SUM across windows equals SUM over the whole stream — no overlap, no gaps.
One aggregate per window. When the watermark passes a window's end, the engine emits that window's result (count, sum, average, distinct-count) and moves on.
The default choice. Tumbling is the simplest and most common window; reach for it whenever you need periodic, non-overlapping roll-ups.

How the four engines spell it. The concept is identical; the assigner names differ.

Flink — TumblingEventTimeWindows.of(Time.minutes(5)) (or TumblingProcessingTimeWindows).
Spark Structured Streaming — window(col("event_time"), "5 minutes") with a single duration and no slide argument.
Kafka Streams — TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)) (a hopping window whose advance equals its size is a tumbling window).
Apache Beam / Dataflow — FixedWindows.of(Duration.standardMinutes(5)).

Setting the watermark delay — not too eager, not too patient. The bounded out-of-orderness you pass is the single knob that trades latency for how many events land on time.

Zero delay fires each window the instant its end passes, but any out-of-order event is immediately late — rarely correct on a real network.
Too large a delay buys completeness at the cost of latency: the window sits open long after most data has arrived, delaying every downstream result.
Right-size it to the realistic out-of-orderness (often a few seconds), and let allowedLateness — not the watermark — absorb the rarer, longer stragglers.

Window alignment — the detail that surprises people. Tumbling windows are aligned to the epoch, not to the first event.

A 1-hour tumbling window produces [10:00, 11:00), [11:00, 12:00) — aligned to the top of the hour — regardless of when your job started or when the first event arrived.
This alignment is why two independent jobs computing hourly totals agree on the same boundaries, and why a replay reproduces identical windows.
Some engines let you supply an offset (for example, to align daily windows to a business day in a specific timezone) — reach for it only when a non-epoch boundary is a real requirement.

Where tumbling windows are the right tool.

Periodic reporting — per-minute request counts, hourly revenue, daily active users.
Billing / metering — non-overlapping intervals are essential so no unit of usage is counted twice.
Alerting on rates — "more than N errors in a 1-minute window" is a tumbling threshold.
Feature engineering — non-overlapping per-interval features (events-per-minute) feed models without leaking the same event into adjacent features.

Common trap answers to pre-empt.

Using a sliding window when the requirement is non-overlapping — over-counts and wastes compute; if buckets must not overlap, it is tumbling.
Windowing on processing time for a "correct totals" requirement — a delay smears counts across buckets; use event time.
Forgetting a watermark / allowed lateness — without them, late events either never count or the window never closes.

Flink 5-minute tumbling count — a worked teaching example

Detailed explanation. Consider a clickstream where you need the number of page views per URL every 5 minutes, computed on event time so a slightly-delayed click still counts in its real bucket. This is the textbook tumbling-count pattern: assign timestamps and a watermark, key by URL, apply a tumbling event-time window, and aggregate.

Question. Produce a per-URL page-view count for each non-overlapping 5-minute event-time window, tolerating up to 1 minute of lateness.

Input.

Field	Value
Source	Kafka topic `clicks`
Key	`url`
Window	5-minute tumbling, event time
Lateness	up to 1 minute

Code.

// Apache Flink (Java) — 5-minute tumbling count per URL on event time
DataStream<Click> clicks = env
    .fromSource(kafkaSource, WatermarkStrategy
        .<Click>forBoundedOutOfOrderness(Duration.ofSeconds(5))
        .withTimestampAssigner((c, ts) -> c.eventTimeMillis()),
        "clicks");

clicks
    .keyBy(c -> c.url())
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))  // fixed, non-overlapping
    .allowedLateness(Time.minutes(1))                       // stragglers still count
    .aggregate(new CountAggregate())                        // COUNT(*) per window
    .addSink(bigQuerySink);

Step-by-step trace.

Each click is stamped with its eventTimeMillis; the watermark trails the max seen timestamp by 5 seconds to absorb minor out-of-orderness.
keyBy(url) partitions the stream so each URL is counted independently.
TumblingEventTimeWindows.of(5 min) assigns every click to exactly one 5-minute bucket by its event time — no overlap.
When the watermark crosses a window's end, CountAggregate emits that window's count; allowedLateness(1 min) holds the window state one extra minute so a late click re-fires an updated count.

Output:

url	window_start	views
/home	10:00	1240
/home	10:05	1310
/pricing	10:00	88

Rule of thumb. For periodic, non-overlapping roll-ups, reach for a tumbling event-time window with a small watermark delay and a modest allowed-lateness grace — it is the simplest correct choice.

Spark Structured Streaming tumbling window with watermark — a worked teaching example

Detailed explanation. Spark Structured Streaming expresses windows declaratively in SQL/DataFrame form: you call window(event_time, "5 minutes") as a grouping key, and you must pair it with withWatermark so Spark knows how long to keep window state for late data before dropping it. The watermark threshold you pass is also, implicitly, your allowed-lateness bound — state older than the watermark is evicted and later events for it are dropped.

withWatermark("event_time", "10 minutes") — accept events up to 10 minutes late; drop later ones and free their state.
window(col("event_time"), "5 minutes") — a single duration = tumbling; group by it like any column.
Output mode update/append — append emits a window only once it is finalised (after the watermark passes); update emits running results.
State cleanup — the watermark is what lets Spark bound state so the streaming query does not grow forever.

Question. Compute a 5-minute tumbling sum of amount per store_id, accepting events up to 10 minutes late.

Input.

Column	Type
`event_time`	TIMESTAMP
`store_id`	STRING
`amount`	DOUBLE

Code.

# Spark Structured Streaming (PySpark) — 5-min tumbling sum with a 10-min watermark
from pyspark.sql.functions import window, col, sum as _sum

agg = (
    events
    .withWatermark("event_time", "10 minutes")          # allowed lateness = 10 min
    .groupBy(
        window(col("event_time"), "5 minutes"),          # tumbling: size only, no slide
        col("store_id"))
    .agg(_sum("amount").alias("revenue"))
)

(agg.writeStream
    .outputMode("append")            # emit each window once finalised past the watermark
    .format("console")
    .option("truncate", False)
    .start())

Step-by-step trace.

withWatermark("event_time", "10 minutes") tells Spark the latest an event may be while still updating its window; state for windows older than watermark is evicted.
window(event_time, "5 minutes") with a single duration tiles the timeline into non-overlapping 5-minute buckets and acts as a grouping key.
groupBy(window, store_id).sum(amount) computes revenue per store per bucket.
outputMode("append") waits until a window's end passes the watermark, then emits it exactly once — a finalised, non-overlapping revenue row.

Output:

window	store_id	revenue
[10:00, 10:05)	s-1	4210.50
[10:00, 10:05)	s-2	980.00
[10:05, 10:10)	s-1	4675.25

Rule of thumb. In Structured Streaming, a single duration in window(...) means tumbling, and withWatermark is mandatory — the watermark both admits late data and bounds state so the query does not leak memory.

Interview scenario on per-minute revenue with a late-data guarantee

You must publish per-minute revenue totals per store from a payments stream. Payments can arrive up to 3 minutes late because of mobile retries, and the published number for a minute must be corrected if late payments land. Totals must never double-count and must be reproducible on replay.

Solution Using tumbling event-time windows + watermark + allowed lateness

Answer choices (as an interviewer would present them).

A. Processing-time tumbling windows of 1 minute, emit immediately, no lateness handling.
B. Event-time tumbling 1-minute windows with a watermark and 3-minute allowed lateness, accumulating so late payments correct the total.
C. A sliding 1-minute window advancing every 10 seconds.
D. A global window that emits the running total every 1000 payments.

Code.

Elimination:
A  processing-time, no lateness -> wrong bucket on delay, no correction   [reject: correctness]
C  sliding window               -> overlapping, double-counts revenue      [reject: over-count]
D  global + count trigger       -> not per-minute; no minute boundaries    [reject: wrong shape]
B  event-time tumbling + wm + allowedLateness -> per-minute, correct, fixable [ACCEPT]

Step-by-step trace.

Constraint keywords: "per-minute" + "never double-count" → non-overlapping = tumbling; "reproducible on replay" + "correct despite delay" → event time; "up to 3 minutes late" + "must be corrected" → watermark with 3-minute allowed lateness, accumulating.
A uses processing time, so a delayed payment lands in the wrong minute and is never corrected — eliminate on correctness.
C is a sliding window: its overlapping buckets count the same payment in multiple windows, so the per-minute totals double-count — eliminate.
D never produces minute boundaries at all; a count trigger is orthogonal to "per-minute" — wrong shape, eliminate.
B tiles the stream into non-overlapping event-time minutes, uses the watermark to emit each minute, and holds state 3 extra minutes so a late payment re-fires a corrected total.

Output:

store_id	minute	revenue	note
s-1	10:00	512.00	on-time emission
s-1	10:00	547.50	corrected after late payment
s-1	10:01	488.25	on-time emission

Why this works — concept by concept:

Tumbling non-overlap — because each payment belongs to exactly one minute, the totals partition revenue perfectly and cannot double-count, which is the hard requirement.
Event-time clock — bucketing by the payment's own timestamp makes the result reproducible on replay and puts delayed payments in their true minute.
Watermark + allowed lateness — the watermark emits the minute promptly while 3-minute allowed lateness (accumulating) holds state long enough for a late payment to re-fire a corrected total.
Cost — you pay for holding each window's state for size + allowed-lateness (≈4 minutes) and a modest re-fire on late data; bounded and cheap versus a sliding window's N-fold state or a global window's unbounded growth.

Streaming
Topic — streaming
Tumbling-window aggregation problems

Practice →

Analytics Topic — time-series Per-interval time-series roll-up problems

Practice →

3. Hopping / sliding windows — size, slide & overlap

A sliding window has two knobs — size and slide — so windows overlap and every event feeds several of them, giving smooth moving aggregates

The invariant: a hopping/sliding window is defined by two parameters — a size (how much history each window covers) and a slide/advance (how often a new window starts) — and whenever the slide is smaller than the size, consecutive windows overlap, so a single event belongs to size / slide windows at once, producing overlapping moving aggregates rather than disjoint buckets. When a requirement says "rolling", "moving average", "over the last hour, updated every 5 minutes", it is a sliding window.

Size vs slide — the whole idea.

Size sets coverage: a 1-hour size means each window summarises the last hour of data.
Slide/advance sets cadence: a 5-minute slide starts a fresh window every 5 minutes, so you get a fresh "last hour" result every 5 minutes.
Overlap is the consequence: with size 1h and slide 5m, each event falls into 60/5 = 12 overlapping windows, so aggregates move smoothly instead of jumping at hard boundaries.
Tumbling is the special case where slide equals size (overlap = 0).

Hopping vs continuous sliding — terminology across engines. The distinction interviewers like to probe:

Hopping window — discrete windows that "hop" forward by a fixed slide (Kafka Streams, Beam call this hopping; a 10-min window advancing every 1 min).
Sliding window (Flink/Spark sense) — the same discrete-slide idea; Flink's SlidingEventTimeWindows takes size + slide.
Continuous sliding (some engines / Flink's SlidingWindows over CEP) — conceptually re-evaluated on every event; more expensive, used for pattern matching.
Practical takeaway — in Flink, Spark, Kafka Streams, and Beam, "sliding/hopping" means size + slide, and the overlap multiplies state and output volume by size/slide.

The state-cost consequence of overlap — do the arithmetic. Overlap is the sliding window's superpower and its bill.

Fan-out. Each event is buffered in size/slide open windows simultaneously; a 1h window sliding every 1 minute holds each event in 60 windows.
Output amplification. You emit size/slide times more result rows than a tumbling equivalent — the downstream sink and storage must absorb it.
Mitigations. Use incremental/associative aggregations (sum, count, HyperLogLog distinct) so each window keeps a tiny accumulator instead of the raw events, and keep the slide no finer than the dashboard actually refreshes.

Where sliding windows are the right tool.

Moving averages / rolling sums — a 15-minute moving average of latency updated every minute.
Rolling unique counts — "active users in the last hour" refreshed every 5 minutes for a live dashboard.
Smoothing spiky metrics — overlapping windows damp the sawtooth you get from hard tumbling boundaries.
Anomaly / threshold detection — a rolling error rate that updates frequently catches spikes sooner than a once-per-interval tumbling roll-up.

Common trap answers.

Using a sliding window for billing/metering — overlap double-counts usage; those must be tumbling.
Choosing a tiny slide relative to size — size/slide is your fan-out; a 1h/1s window puts each event in 3600 windows and can overwhelm state and output.
Ignoring that output volume multiplies — every event now emits into many windows; downstream and storage must handle the amplification.

Kafka Streams hopping window — a worked teaching example

Detailed explanation. Kafka Streams models a hopping window with TimeWindows.ofSizeAndGrace(size).advanceBy(slide): advanceBy is the slide. A 10-minute window advancing every 1 minute yields a fresh "last 10 minutes" count every minute, and each record contributes to 10 overlapping windows. The grace period is Kafka Streams' allowed lateness — records later than grace are dropped.

Question. Count events per key over a 10-minute window that advances every 1 minute, tolerating 2 minutes of lateness.

Input.

Field	Value
Source	topic `events` (keyed)
Size	10 minutes
Advance (slide)	1 minute
Grace (lateness)	2 minutes

Code.

// Kafka Streams (Java) — 10-min hopping window, 1-min advance, 2-min grace
KStream<String, Event> events = builder.stream("events");

events
    .groupByKey()
    .windowedBy(
        TimeWindows.ofSizeAndGrace(Duration.ofMinutes(10), Duration.ofMinutes(2))
                   .advanceBy(Duration.ofMinutes(1)))   // slide = 1 min -> overlap
    .count(Materialized.as("hopping-counts"))
    .toStream()
    .to("event-counts");

Step-by-step trace.

TimeWindows.ofSizeAndGrace(10 min, 2 min) sets a 10-minute window with a 2-minute grace (allowed lateness).
.advanceBy(1 min) makes a new window open every minute, so at any instant 10 windows are open and each event is counted in all 10.
groupByKey().count() maintains a running count per key per window in the hopping-counts state store.
As event time advances, each window closes when the stream time passes its end + grace, and its final count is emitted; records later than grace are dropped.

Output:

key	window_start	window_end	count
u-9	10:00	10:10	42
u-9	10:01	10:11	45
u-9	10:02	10:12	47

Rule of thumb. In Kafka Streams, advanceBy is the slide and size/advance is how many windows each record feeds — keep that ratio small enough that state and output volume stay affordable.

Spark sliding window moving average — a worked teaching example

Detailed explanation. Spark Structured Streaming turns a sliding window on with a second duration argument: window(event_time, "10 minutes", "1 minute") means size 10m, slide 1m. That produces overlapping windows and, with an average aggregation, a moving average updated every minute. As always, withWatermark bounds state and admits late data.

Two durations = sliding. The third argument (slide) distinguishes it from a tumbling window's single duration.
Overlap fan-out — each event contributes to size/slide windows (here 10).
Watermark — still mandatory to bound state and set allowed lateness.
Aggregation — avg, sum, approx_count_distinct all work per window.

Question. Compute a 10-minute moving average of latency_ms per service, refreshed every 1 minute, accepting 5 minutes of lateness.

Input.

Column	Type
`event_time`	TIMESTAMP
`service`	STRING
`latency_ms`	DOUBLE

Code.

# Spark Structured Streaming (PySpark) — 10-min size, 1-min slide moving average
from pyspark.sql.functions import window, col, avg

moving_avg = (
    metrics
    .withWatermark("event_time", "5 minutes")            # allowed lateness = 5 min
    .groupBy(
        window(col("event_time"), "10 minutes", "1 minute"),  # size=10m, slide=1m
        col("service"))
    .agg(avg("latency_ms").alias("avg_latency_ms"))
)

(moving_avg.writeStream
    .outputMode("append")
    .format("console")
    .start())

Step-by-step trace.

withWatermark(..., "5 minutes") admits events up to 5 minutes late and bounds the state Spark keeps for overlapping windows.
window(event_time, "10 minutes", "1 minute") opens a new 10-minute window every minute, so each metric feeds 10 overlapping windows.
avg(latency_ms) per (window, service) yields a 10-minute average that refreshes every minute — a moving average.
append emits each window once its end passes the watermark, giving a smooth per-minute stream of 10-minute averages.

Output:

window	service	avg_latency_ms
[10:00, 10:10)	api	182.4
[10:01, 10:11)	api	179.1
[10:02, 10:12)	api	176.8

Rule of thumb. A second duration in Spark's window(...) is the slide and turns a tumbling window into a sliding one — use it for moving averages and rolling counts, and mind that state scales with size/slide.

Interview scenario on a rolling active-user count

A live dashboard needs the count of distinct active users over the last 1 hour, refreshed every 5 minutes, from a global clickstream. Clicks can be a couple of minutes out of order. The dashboard should update smoothly, not jump once an hour, and must be reproducible.

Solution Using event-time sliding windows (size 1h, slide 5m) with a watermark

Answer choices.

A. Tumbling 1-hour windows — one result per hour.
B. Sliding window, size 1 hour, slide 5 minutes, event time, approx_count_distinct, with a watermark.
C. Processing-time sliding window, size 1 hour, slide 5 minutes.
D. Global window emitting distinct count every 5 minutes with no size bound.

Code.

Elimination:
A  tumbling 1h        -> updates once/hour, jumps hard, not "every 5 min"   [reject: cadence]
C  processing time    -> out-of-order clicks land wrong, non-reproducible   [reject: correctness]
D  global, no size    -> counts all-time users, not "last hour"             [reject: wrong window]
B  event-time sliding 1h/5m + watermark -> rolling, smooth, correct         [ACCEPT]

Step-by-step trace.

Constraint keywords: "last 1 hour" → window size 1h; "refreshed every 5 minutes" + "smooth" → slide 5m (overlapping); "reproducible" + "out of order" → event time + watermark.
A is tumbling, so it only produces a result once an hour and jumps at the boundary — fails "every 5 minutes / smooth" — eliminate.
C uses processing time, so out-of-order clicks land in the wrong windows and replays differ — eliminate on correctness.
D drops the 1-hour size, so it counts all-time active users, not the last hour — wrong window semantics — eliminate.
B opens a fresh 1-hour window every 5 minutes on event time, uses a watermark for out-of-orderness, and approx_count_distinct keeps the distinct-user state affordable.

Output:

window	active_users
[09:00, 10:00)	51,204
[09:05, 10:05)	51,880
[09:10, 10:10)	52,133

Why this works — concept by concept:

Size vs slide — size 1h delivers the "last hour" coverage while slide 5m delivers the "every 5 minutes" cadence; the two knobs map directly onto the two clauses in the requirement.
Overlap for smoothness — because consecutive windows overlap by 55 minutes, the active-user line moves gradually instead of resetting on the hour, which is exactly what a live dashboard wants.
Event time + watermark — bucketing by click time makes the metric correct and replay-stable despite a couple minutes of out-of-orderness.
Cost — each click feeds 12 windows, so state and output are ~12× a tumbling equivalent; using approx_count_distinct (HyperLogLog) keeps the distinct-count state bounded rather than storing every user id per window.

Sliding window
Topic — sliding-window
Sliding-window and rolling-aggregate problems

Practice →

Streaming Topic — streaming Moving-average and rolling-count streaming problems

Practice →

4. Session windows — inactivity gap, dynamic per-key windows

A session window has no fixed size — it opens on activity and closes after a gap of silence, so its width is decided by the data itself, per key

The invariant: a session window is defined by a single gap parameter (the maximum inactivity allowed within one session); it groups a key's events into a window that keeps extending as long as new events arrive within the gap, and closes only when no event has arrived for longer than the gap — so window boundaries and widths are data-driven and unique per key. When a requirement says "per visit", "per session", "group activity into bursts separated by idle time", it is a session window.

What defines a session window.

One parameter: the gap. A 30-minute gap means events less than 30 minutes apart belong to the same session; a gap longer than 30 minutes starts a new one.
Dynamic width. Sessions are as long as the activity that feeds them — a 2-minute burst and a 3-hour marathon are both single sessions.
Per-key. Each key (user, device, IP) gets its own independent sessions.
Merging. When a late or intervening event bridges two previously-separate sessions (the gap between them shrinks below the threshold), the engine merges them into one — a distinguishing behaviour session windows must support.

How the four engines spell it.

Flink — EventTimeSessionWindows.withGap(Time.minutes(30)) (or ProcessingTimeSessionWindows); dynamic-gap variants exist for per-event gaps.
Spark Structured Streaming — session_window(col("event_time"), "30 minutes") as a grouping key, with a watermark; supports dynamic gaps via an expression.
Kafka Streams — SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30)).
Apache Beam / Dataflow — Sessions.withGapDuration(Duration.standardMinutes(30)).

Tuning the gap — the one decision that defines your sessions. The gap is the whole model, so choose it from behaviour, not habit.

Too small and a single visit fragments into many short sessions (a user who pauses to read gets cut off), inflating session counts and deflating durations.
Too large and unrelated visits merge into one, hiding the natural rhythm and overstating engagement.
Pick it from data — a common heuristic is the 95th-percentile inter-event gap within known-single visits; web analytics conventionally lands near 30 minutes, but IoT or gaming may want seconds or hours.
Merging is why the gap works — because an out-of-order event can bridge two sessions, the engine must merge windows whose combined gap falls below the threshold; this is what keeps the count correct under out-of-orderness.

Where session windows are the right tool.

Web/product analytics — group a user's clicks into visits to compute session length, pages-per-session, bounce.
IoT / device bursts — a device that wakes, transmits a burst, and sleeps forms a natural session.
Fraud / behaviour — a burst of actions from one account between idle periods.
Support / conversation grouping — messages within a gap form one conversation, a longer silence starts the next.

Common trap answers.

Forcing a fixed window onto session data — a 30-minute tumbling window arbitrarily splits a visit that straddles the boundary; sessions follow the activity instead.
Ignoring merging — an out-of-order event can bridge two sessions; an implementation that cannot merge produces wrong session counts.
No watermark on session windows — you still need one to decide when a gap has truly elapsed on event time and when to close/emit.

Flink session windows — a worked teaching example

Detailed explanation. Consider a clickstream where you want, per user, the number of clicks and the duration of each browsing session, where a session ends after 30 minutes of inactivity. Flink's EventTimeSessionWindows.withGap does exactly this: events for a user within 30 minutes coalesce into one window, and a gap longer than 30 minutes starts a new one; out-of-order events that bridge two sessions trigger a merge.

Question. Compute clicks-per-session and session duration per user with a 30-minute inactivity gap.

Input.

Field	Value
Source	Kafka topic `clicks` (keyed by `user_id`)
Gap	30 minutes inactivity
Clock	event time
Metric	count + (max_ts − min_ts) per session

Code.

// Apache Flink (Java) — per-user session windows with a 30-min inactivity gap
clicks
    .keyBy(c -> c.userId())
    .window(EventTimeSessionWindows.withGap(Time.minutes(30)))  // gap-based, dynamic width
    .allowedLateness(Time.minutes(5))
    .process(new ProcessWindowFunction<Click, SessionStat, String, TimeWindow>() {
        @Override
        public void process(String user, Context ctx,
                            Iterable<Click> clicks, Collector<SessionStat> out) {
            long start = ctx.window().getStart();
            long end   = ctx.window().getEnd();
            long n = 0;
            for (Click c : clicks) n++;
            out.collect(new SessionStat(user, start, end, n, end - start));
        }
    })
    .addSink(bigQuerySink);

Step-by-step trace.

keyBy(userId) isolates each user's stream so sessions are per-user.
EventTimeSessionWindows.withGap(30 min) opens a session on the first click and keeps extending it as long as the next click is within 30 minutes.
If a click arrives more than 30 minutes after the previous one, the current session closes (fires) and a new one opens.
An out-of-order click that lands between two sessions — bridging the gap to under 30 minutes — causes Flink to merge them into one window before the ProcessWindowFunction computes the stats.

Output:

user_id	session_start	session_end	clicks	duration_min
u-3	10:02	10:19	14	17
u-3	11:40	11:43	3	3

Rule of thumb. Reach for a session window whenever "activity separated by idle time" defines the grouping; the gap is your only knob, and the engine handles the merging that makes it correct.

Beam / Dataflow session windows — a worked teaching example

Detailed explanation. Apache Beam (and Google Cloud Dataflow) expresses sessions with Sessions.withGapDuration(...), applied via Window.into(...). Beam's model cleanly separates the window (Sessions) from the trigger and allowed lateness, so you can, for example, fire on the watermark and re-fire on late data with accumulating mode — the same late-correction pattern as fixed windows, but with data-driven boundaries.

Sessions.withGapDuration(gap) — the gap-based assigner.
triggering(AfterWatermark.pastEndOfWindow()) — emit when the session's gap has provably elapsed.
withAllowedLateness(Duration) — keep session state for late events past the watermark.
accumulatingFiredPanes() — late events re-fire an updated session result.

Question. Group per-user events into sessions with a 30-minute gap, emit on the watermark, and allow 10 minutes of lateness with accumulation.

Input.

Field	Value
Key	`user_id`
Gap	30 minutes
Trigger	on watermark, re-fire late
Allowed lateness	10 minutes, accumulating

Code.

// Apache Beam (Java) — session windows, watermark trigger, 10-min allowed lateness
PCollection<KV<String, Event>> sessions = events
    .apply(Window.<KV<String, Event>>into(
              Sessions.withGapDuration(Duration.standardMinutes(30)))
        .triggering(AfterWatermark.pastEndOfWindow()
            .withLateFirings(AfterProcessingTime.pastFirstElementInPane()))
        .withAllowedLateness(Duration.standardMinutes(10))
        .accumulatingFiredPanes());

PCollection<KV<String, Long>> perSession = sessions
    .apply(Count.perKey());   // events per user per session

Step-by-step trace.

Window.into(Sessions.withGapDuration(30 min)) assigns each user's events into gap-based sessions; nearby events merge, distant ones split.
AfterWatermark.pastEndOfWindow() fires each session once the watermark shows the 30-minute gap has truly elapsed.
withLateFirings(...) plus withAllowedLateness(10 min) and accumulatingFiredPanes() mean a late event within 10 minutes re-fires an updated, accumulated session result.
Count.perKey() produces the event count per user per session.

Output:

user_id	pane	events	note
u-8	on-time	9	watermark firing
u-8	late	11	corrected after late events

Rule of thumb. Beam separates window / trigger / lateness cleanly, so you can attach the same watermark-plus-allowed-lateness late-correction policy to session windows that you use for fixed windows.

Interview scenario on web-analytics sessionization

You must sessionize a global clickstream into user visits with a 30-minute inactivity gap, computing pages-per-session and session duration. Clicks arrive slightly out of order, an out-of-order click may bridge two visits, and results feed both a real-time dashboard and a warehouse. Minimise wasted state.

Solution Using event-time session windows with gap + merge + watermark

Answer choices.

A. Tumbling 30-minute windows keyed by user.
B. Event-time session windows with a 30-minute gap, a watermark, and merging, keyed by user.
C. Sliding 30-minute windows advancing every minute, keyed by user.
D. A single global window per user, emitting on every click.

Code.

Elimination:
A  tumbling 30m   -> arbitrary cuts split a visit at the boundary          [reject: wrong shape]
C  sliding 30m/1m -> overlapping windows, no session semantics, huge state [reject: over-count]
D  global per user-> never closes a visit; unbounded state                 [reject: no boundary]
B  event-time session (gap=30m) + watermark + merge -> true visits         [ACCEPT]

Step-by-step trace.

Constraint keywords: "visits with a 30-minute inactivity gap" → session window, gap 30m; "out-of-order click may bridge two visits" → merging required; "slightly out of order" → event time + watermark.
A is tumbling: a visit straddling 10:29→10:31 is split across two 30-minute buckets, corrupting session length — eliminate.
C is sliding: overlapping fixed windows have no notion of inactivity and multiply state; they cannot express a variable-width visit — eliminate.
D never detects the end of a visit and grows state without bound — eliminate.
B opens a session per user, extends it while clicks are within 30 minutes, merges when an out-of-order click bridges two, and uses the watermark to close and emit — with allowed lateness kept small to minimise state.

Output:

user_id	session_start	session_end	pages	duration_min
u-42	10:05	10:41	12	36
u-42	13:10	13:14	2	4

Why this works — concept by concept:

Gap-based boundaries — a session window's single gap parameter matches the requirement exactly; the visit is as long as the activity, never arbitrarily cut by a fixed boundary.
Session merging — supporting merge is what makes an out-of-order click that bridges two visits produce one correct session instead of two fragments.
Event time + watermark — bucketing by click time and closing on the watermark makes both the dashboard and the warehouse see identical, reproducible sessions.
Cost — session state lives only for gap + allowed lateness after the last event, so keeping allowed lateness small bounds the per-user state and minimises waste, unlike sliding or global alternatives.

Streaming
Topic — streaming
Sessionization and event-grouping problems

Practice →

Analytics Topic — real-time-analytics Real-time sessionization and analytics problems

Practice →

5. Global windows, triggers & side outputs for late data

A global window has no natural end — it puts all of a key's data in one window and hands the "when to emit" decision entirely to triggers

The invariant: a global window assigns every event for a key to one single, never-ending window, so time boundaries do not fire anything — you must attach an explicit trigger (on count, on processing time, on a custom condition) to decide when the window emits, and usually an evictor to control what state is retained; it is the escape hatch for aggregations that are not naturally time-boxed. Global windows are also where you meet the two late-data tools every streaming engineer must know: allowed lateness and side outputs.

What defines a global window.

One window, forever. All events for a key go into the same window; there is no automatic firing.
Triggers do everything. Without a trigger a global window never emits — you attach CountTrigger, ProcessingTimeTrigger, or a custom trigger to decide when.
Evictors manage state. Because the window never closes, an evictor (e.g. keep the last N elements or the last T time) prevents unbounded state growth.
Use it when time is not the boundary. "Emit every 1000 events", "fire when a special sentinel arrives", or CEP-style pattern conditions are global-window territory.

Triggers and evictors — the control surface.

CountTrigger.of(N) — fire when the window has accumulated N elements.
ProcessingTimeTrigger / ContinuousEventTimeTrigger — fire on a wall-clock or event-time cadence.
Purging vs non-purging — PurgingTrigger clears state after firing (disjoint batches); a plain trigger keeps accumulating.
Evictors — CountEvictor, TimeEvictor bound retained elements so the never-ending window does not leak memory.

Late data: allowed lateness and side outputs. Even with time windows, some events arrive after the watermark and after allowed lateness — this is where side outputs matter.

Allowed lateness — a grace period after the watermark during which late events still update the window; after it expires, state is dropped.
Side output (OutputTag) — Flink routes events later than allowed lateness to a separate stream so they can be logged, reprocessed, or reconciled instead of silently dropped.
Dropped-late metrics — engines expose counters (e.g. numLateRecordsDropped) so you can alarm on excessive lateness.
Interview signal — being able to say "late data past allowed lateness goes to a side output for reconciliation, and I alarm on the dropped-late metric" demonstrates production maturity.

Early, on-time, and late firings — the trigger vocabulary. Even on time windows, Beam-style triggers let one window emit multiple panes, and naming them cleanly is an interview signal.

Early firing — emit a speculative pane before the watermark for a low-latency preview (e.g. every 10 seconds), knowing it may be revised.
On-time firing — the default pane when the watermark passes the window end; the "trust me" result.
Late firing — a re-fire triggered by a within-allowed-lateness event that corrects the on-time pane.
Accumulating vs discarding — accumulating panes each contain the full running result (later panes supersede earlier ones); discarding panes contain only the delta since the last firing (downstream must add them up).

Common trap answers.

A global window with no trigger — it will never emit; a trigger is mandatory.
A global window with no evictor for an unbounded aggregation — state grows forever; bound it.
Silently dropping late data — production pipelines route it to a side output and monitor the drop count rather than losing it.
Assuming a global window is exactly-once by itself — correctness still depends on the trigger (purging vs accumulating) and idempotent sinks, not the window type.

Flink global window with a count trigger — a worked teaching example

Detailed explanation. Suppose you must emit a running aggregate every 1000 events per key, regardless of time — a classic non-time-boxed requirement. A global window plus CountTrigger.of(1000) does this; wrapping it in a PurgingTrigger makes each fired batch disjoint (state clears after each 1000), and a count evictor can cap retained elements.

Question. Emit a per-key sum of amount every 1000 events, with each batch independent.

Input.

Field	Value
Key	`device_id`
Window	global (no time boundary)
Trigger	every 1000 events, purging
Metric	SUM(amount) per batch

Code.

// Apache Flink (Java) — global window firing every 1000 events, purging state each batch
events
    .keyBy(e -> e.deviceId())
    .window(GlobalWindows.create())                          // one never-ending window
    .trigger(PurgingTrigger.of(CountTrigger.of(1000)))       // fire + clear every 1000
    .aggregate(new SumAggregate())                           // SUM(amount) per 1000-batch
    .addSink(sink);

Step-by-step trace.

GlobalWindows.create() puts all of a device's events into one window with no time-based firing.
CountTrigger.of(1000) fires the window once it has accumulated 1000 events for that key.
Wrapping it in PurgingTrigger clears the window state right after firing, so the next 1000 events form a fresh, disjoint batch (no double-counting across batches).
SumAggregate emits the sum of amount for each 1000-event batch per device.

Output:

device_id	batch	events	sum_amount
d-1	1	1000	24,510.00
d-1	2	1000	25,003.50

Rule of thumb. For count-based or condition-based emission, use a global window with an explicit trigger; add a PurgingTrigger or evictor so state does not accumulate forever.

Side outputs for late events — a worked teaching example

Detailed explanation. The production-grade late-data pattern in Flink is: window on event time with a watermark, set allowedLateness, and attach a side output so events later than the allowed-lateness grace are routed to a dedicated stream rather than dropped. You then reconcile the side stream (reprocess into the warehouse, alert, or store for audit) and monitor the dropped-late metric.

OutputTag<T> — names the side channel for late events.
.sideOutputLateData(tag) — on the windowed stream, diverts too-late events there.
result.getSideOutput(tag) — access the late stream to reconcile it.
Why not drop — silently discarding late data corrupts totals with no audit trail; a side output preserves correctness and observability.

Question. Compute 1-minute event-time counts with 2 minutes of allowed lateness, and capture events later than that in a side output for reconciliation.

Input.

Field	Value
Window	1-minute tumbling, event time
Allowed lateness	2 minutes
Too-late events	routed to side output
Downstream	main → warehouse; side → reconcile

Code.

// Apache Flink (Java) — tumbling count with allowed lateness + side output for late data
final OutputTag<Event> lateTag = new OutputTag<Event>("late-events") {};

SingleOutputStreamOperator<Count> counts = events
    .keyBy(e -> e.key())
    .window(TumblingEventTimeWindows.of(Time.minutes(1)))
    .allowedLateness(Time.minutes(2))        // late-but-within-grace still update the window
    .sideOutputLateData(lateTag)             // later than grace -> side output, not dropped
    .aggregate(new CountAggregate());

counts.addSink(warehouseSink);               // on-time + corrected results
DataStream<Event> tooLate = counts.getSideOutput(lateTag);
tooLate.addSink(reconciliationSink);         // audit / reprocess later data

Step-by-step trace.

Events are windowed into 1-minute event-time buckets; the watermark decides when each minute first fires.
allowedLateness(2 min) keeps each window's state 2 extra minutes, so events within that grace re-fire a corrected count into the main stream.
sideOutputLateData(lateTag) diverts events that arrive later than the 2-minute grace to the late-events side stream instead of dropping them.
getSideOutput(lateTag) exposes that late stream so it can be reconciled (reprocessed or audited), and the engine's dropped-late counter stays near zero because nothing is silently lost.

Output:

stream	key	window	value
main	k-7	10:00	320 (corrected)
side (late)	k-7	10:00	4 events for reconciliation

Rule of thumb. In production, pair allowed lateness with a side output: within-grace late data corrects the window, past-grace late data goes to a side stream you reconcile and monitor — never a silent drop.

Processing-time triggers for early, speculative emission — a worked teaching example

Detailed explanation. A global (or long event-time) window paired only with an on-watermark trigger emits once, at the end — which is useless for a live dashboard that needs a number every few seconds. The fix is a processing-time trigger that fires speculative, partial results on a wall-clock cadence while the window stays open, then a final on-watermark firing that supersedes them. This is the "early result now, correct result later" pattern, and interviewers probe whether you know that early firings are updates, not additional rows.

Processing-time trigger — fire every N seconds of wall-clock time regardless of event completeness.
Speculative result — a partial aggregate the UI treats as provisional.
Accumulating mode — each firing emits the running total, so the latest firing replaces the previous value for that key/window.
Final firing — the on-watermark (or window-close) firing is the authoritative result.

Question. A live "orders in the last hour" counter must refresh every 5 seconds but still converge to the exact hourly count. Configure the trigger.

Input.

Requirement	Mechanism
Refresh cadence	processing-time trigger, 5 s
Correct final value	on-watermark final firing
UI semantics	accumulating (replace, not append)
Window	1-hour event-time window

Code.

// Flink: early speculative firings every 5s, final firing on watermark
stream
  .keyBy(o -> o.shopId)
  .window(TumblingEventTimeWindows.of(Time.hours(1)))
  .trigger(ContinuousProcessingTimeTrigger.of(Time.seconds(5)))  // early fires
  .allowedLateness(Time.minutes(10))
  .aggregate(new CountAgg());
// UI keys on (shopId, windowStart) and OVERWRITES the value on each firing.

Step-by-step trace.

The hour window opens; every 5 seconds the processing-time trigger fires a partial count for the elapsed portion of the hour.
The dashboard keys each firing by (shopId, windowStart) and overwrites the displayed number — so the counter climbs smoothly, not duplicated.
When the watermark passes the window end, a final firing emits the authoritative hourly count that supersedes all speculative values.
Late events within allowed lateness trigger one more corrected firing.

Output:

Firing @	shopId	count	status
t+5s	s-1	118	speculative
t+3600s	s-1	4,207	final

Rule of thumb. Use processing-time triggers for "show me something now" liveness and an on-watermark final firing for correctness; make the sink treat firings as overwrites, never inserts.

Interview scenario on exactly-once late-data reconciliation

A billing pipeline aggregates per-minute usage on event time. Most events are on time, some are up to 5 minutes late, and a small fraction arrive hours late after a client comes back online. Totals must be exact eventually, nothing may be silently dropped, and re-processing must not double-count. Design the windowing and late-data handling.

Solution Using event-time windows + allowed lateness + side output reconciliation

Answer choices.

A. 1-minute event-time windows, no allowed lateness; drop anything past the watermark.
B. 1-minute event-time windows with 5-minute allowed lateness (accumulating) for the common case, a side output for hours-late events, and idempotent reconciliation of the side stream.
C. Processing-time 1-minute windows, emit immediately.
D. One global window per key that never emits until the stream ends.

Code.

Elimination:
A  no lateness, drop late  -> silent loss, totals never exact              [reject: correctness]
C  processing time         -> wrong minute on delay, not reproducible      [reject: correctness]
D  global, emit at end     -> a stream has no end; never emits             [reject: no boundary]
B  event-time + allowedLateness + side output + idempotent reconcile [ACCEPT]

Step-by-step trace.

Constraint keywords: "per-minute usage" → tumbling 1-minute; "exact eventually" + "nothing silently dropped" → allowed lateness + side output; "re-processing must not double-count" → idempotent (accumulating / keyed upsert) writes.
A drops late data past the watermark, so hours-late events are lost and totals are never exact — eliminate on correctness.
C uses processing time, so any delay corrupts the minute and replays differ — eliminate.
D never emits on an unbounded stream — eliminate.
B handles the common 5-minute lateness by keeping window state (accumulating) so those events correct the minute, routes the rare hours-late events to a side output, and reconciles the side stream with an idempotent upsert keyed by (minute, key) so reprocessing overwrites rather than adds.

Output:

path	key	minute	usage	note
main (on time)	acct-9	10:00	512	first emission
main (≤5m late)	acct-9	10:00	547	corrected within grace
side → reconcile	acct-9	10:00	553	hours-late, idempotent upsert

Why this works — concept by concept:

Event-time windows — bucketing usage by its own timestamp makes every minute reproducible and puts delayed events in their true minute, the base requirement for exact billing.
Allowed lateness (accumulating) — a 5-minute grace with accumulation lets the common late events re-fire a corrected minute without a separate pipeline.
Side output for the long tail — hours-late events exceed any reasonable grace, so routing them to a side stream preserves them for reconciliation instead of dropping them — the "nothing silently dropped" guarantee.
Idempotent reconciliation — writing corrections as an upsert keyed by (minute, key) means reprocessing the side stream overwrites the total rather than adding to it, delivering "exact eventually" without double-counting.
Cost — you pay bounded state for the 5-minute grace on every window plus a small, rare side-stream reconciliation; far cheaper than holding all windows open indefinitely to catch the long tail inline.

Analytics
Topic — real-time-analytics
Late-data and reconciliation problems

Practice →

SQL
Topic — window-functions
Window-function and running-aggregate SQL problems

Practice →

Cheat sheet — windowing recipes across engines

Window-type → when-to-use lookup (memorise this table).

Requirement keyword	Window type	Key parameters
"per-minute / hourly / daily, non-overlapping"	Tumbling (fixed)	size
"rolling / moving average / every 5 min over last hour"	Hopping / sliding	size + slide
"per visit / per session / activity between idle gaps"	Session	inactivity gap
"every N events / no natural time boundary / CEP condition"	Global	trigger (+ evictor)
"correct despite delays / reproducible"	any + event time	watermark
"as fast as possible / approximate is fine"	any + processing time	none
"events up to X late must still count"	any + event time	allowedLateness = X
"nothing may be silently dropped"	any	side output for late data

Flink / Spark / Kafka Streams / Beam API cross-reference.

Window	Flink	Spark Structured Streaming	Kafka Streams	Beam / Dataflow
Tumbling	`TumblingEventTimeWindows.of(size)`	`window(t, "5 minutes")`	`TimeWindows.ofSizeWithNoGrace(size)`	`FixedWindows.of(size)`
Sliding/hopping	`SlidingEventTimeWindows.of(size, slide)`	`window(t, "10 minutes", "1 minute")`	`TimeWindows.ofSize(size).advanceBy(slide)`	`SlidingWindows.of(size).every(slide)`
Session	`EventTimeSessionWindows.withGap(gap)`	`session_window(t, "30 minutes")`	`SessionWindows.ofInactivityGap(gap)`	`Sessions.withGapDuration(gap)`
Global	`GlobalWindows.create()` + trigger	(custom via flatMapGroupsWithState)	(custom processor)	`GlobalWindows()` + trigger

Watermark + allowed-lateness checklist.

Assign event-time timestamps and a watermark with a small bounded out-of-orderness (seconds), not zero.
Set allowedLateness to the realistic straggler bound (minutes), not hours — hours-late data belongs in a side output.
Choose accumulating mode when late data should correct a result; discarding mode when panes are independent.
Route past-grace late data to a side output and alarm on the dropped-late-records metric.
In Spark, withWatermark is mandatory for windowed aggregations — it both admits late data and bounds state.

Event-time vs processing-time decision line. Need correctness, reproducibility, or a "despite delays" guarantee → event time + watermark. Need lowest latency and approximate is acceptable → processing time. Ingestion time is the stable-but-approximate middle ground.

Interview-signal one-liners. "Tumbling for non-overlapping roll-ups; sliding for moving aggregates; session for gap-based visits; global for count/condition emission." · "The watermark is the engine's estimate of completeness; allowed lateness is the grace after it." · "Late-past-grace goes to a side output, not the floor."

Frequently asked questions

What is windowing in stream processing and why is it needed?

Windowing in stream processing is the practice of grouping the events of an unbounded stream into bounded chunks — windows — so that aggregations like counts, sums, and averages become finite and emittable. Without windows, an aggregate over an infinite stream would never complete, because there is always another event coming. The window is the boundary that lets the engine compute a result "for this minute / this session / this batch" and release it downstream continuously.

Tumbling vs sliding vs session windows — how do I choose?

Match the window to the shape of the requirement. Choose a tumbling window for periodic, non-overlapping roll-ups (per-minute, hourly, daily) where each event must count exactly once — this is the default for billing and reporting. Choose a sliding/hopping window (size + slide) for rolling or moving aggregates that update more often than they cover, like a 1-hour active-user count refreshed every 5 minutes. Choose a session window (an inactivity gap) when the grouping is activity-driven, such as sessionizing a user's clicks into visits.

What is a watermark and how does allowed lateness relate to it?

A watermark is a marker flowing through the stream that represents the engine's estimate that "all events with event time up to T have probably arrived", and when it passes a window's end the window fires. Because a watermark is only a heuristic, some events still arrive after it — allowed lateness is the extra grace period during which those late events can still update the window before its state is discarded. Together they tune the trade-off between low latency (fire early) and completeness (wait for stragglers).

Event time vs processing time — which should I window on?

Window on event time whenever results must be correct, reproducible, or robust to delays — event time buckets each record by when it actually happened, so a network hiccup or a replay never changes the answer. Reach for processing time only when you need the lowest possible latency and an approximate, replay-unstable result is acceptable, since it buckets by wall-clock arrival and a delay silently moves records into the wrong window. Most correctness-sensitive pipelines — billing, metrics, sessionization — use event time with a watermark.

What happens to late data that arrives after the window closes?

It depends on your configuration. Within the allowed lateness grace period, a late event still updates its window (and, in accumulating mode, re-fires a corrected result). After that grace expires, the default is to drop the event and increment a dropped-late-records metric — but production pipelines instead route past-grace events to a side output so they can be logged, reconciled, or reprocessed idempotently rather than silently lost. Choosing the grace length and the side-output policy is a core part of correct windowing in stream processing.

What is a global window and when would I use it?

A global window assigns every event for a key to one single, never-ending window, so it never fires on its own — you attach an explicit trigger (fire every N events, on a processing-time cadence, or on a custom condition) to control emission, and usually an evictor to bound state. Use it when the aggregation boundary is not time-based: "emit a running total every 1000 events", complex-event-processing pattern conditions, or firing when a sentinel event arrives. It is the flexible escape hatch when tumbling, sliding, and session windows do not fit.

Practice on PipeCode

Turn windowing theory into streaming reflexes

Guides explain tumbling, sliding, session, and global windows. PipeCode drills build the instinct interviewers test — reading a requirement, naming the window shape and the clock, and defending the watermark and allowed-lateness settings under a timer. Pipecode.ai is Leetcode for Data Engineering — scenario-first practice on streaming, sliding-window, and time-series problems tuned to the trade-offs real stream-processing systems force you to make.

Practice streaming problems →
Practice sliding-window problems →