Batch or stream?
That's the wrong first question. The better one is: does your data ever end?
If you've worked anywhere near big data, you've sat through this debate. Someone wants Kafka and Flink. Someone else says a nightly Spark job is fine. But what it is all about?
Stream processing reacts to each event the moment it happens.
Batch waits for a time window or a volume threshold, then processes everything in one go.
At first glance it looks like a per-feature choice. Pick one, move on.
but it isn't.
Bounded data has an end. Yesterday's song plays, last month's orders. Because it ends, a batch job can give you a complete answer.
Unbounded data never ends. Plays are still coming in while you read this. A stream can't wait for "all the data", so the best it can give you is a fresh answer.
Fresh, complete, cheap. You rarely get all three.
So the real work is understanding your data and how events move through your system. Then for each feature you ask one thing: how fast does this event lose its value?
Take a music app:
A bot spamming plays to fake streams? Worth catching in seconds. Stream.
Paying artists for last month's plays? Nobody needs that in real time, they need it correct. Batch.
The trending chart? Honestly, both. A stream for what's hot right now, and a batch job that fixes the counts once late data lands.
Most real systems end up looking like that. A few features on each side, and a couple that need both.
I turned this into a small course: Spark and the shuffle, Kafka offsets, Flink watermarks, exactly-once, backpressure, replaying a week of bad results, and the four questions I use to route any feature to batch, stream, or both.
Top comments (0)