There is a question most data teams never actually ask before standing up streaming infrastructure: who is consuming this data, and how fast do they genuinely need it?
Skip that question, and you end up paying $4,200 a month for sub-second latency that your analysts are querying once an hour over their morning coffee.
This is not a rare edge case. It is the default outcome when infrastructure decisions are made under social pressure instead of engineering constraints.
The Latency Nobody Is Consuming
Here is a useful exercise. Pull up your last 30 days of query logs against whatever "real-time" data source your team maintains. Bucket the queries by the time granularity they actually operate at. How many are filtering by the last 60 seconds? How many are filtering by the last hour, the last day, or some arbitrary date range?
For most teams, the distribution looks something like this: a handful of operational dashboards refreshing every few minutes, and everything else running at hourly or daily granularity. The streaming pipeline is working extremely hard to keep data fresh to the second. The business is checking it at 9am.
The technical term for this mismatch is waste. The political term for it is "modern infrastructure."
The Career Risk That Doesn't Show Up on the AWS Bill
The actual driver of unnecessary streaming adoption is not technical enthusiasm. It is asymmetric career risk.
If you push for Kafka and the bill is high, that cost gets absorbed into the infrastructure budget. Nobody puts "championed expensive streaming pipeline for a use case that didn't need it" on a post-mortem. If you argue against streaming and recommend a micro-batch job on a cron schedule, you have now made yourself the person who said no to real-time. That label follows you into the next planning meeting.
Saying "we don't need real-time" carries a higher personal cost than the infrastructure bill itself. So engineers default to yes, and the Kafka cluster gets provisioned, and six months later it is a load-bearing piece of the stack that nobody wants to touch.
This is not a criticism of the engineers making those calls. It is a systems observation. The incentive structure of most engineering orgs actively punishes the conservative, correct answer.
The Question That Actually Clarifies the Decision
Before any conversation about tooling, there is one question worth spending real time on:
What is the actual decision latency our business requires?
Not "what would be cool to have." Not "what does the architecture diagram look like if we go real-time." What is the real human or automated decision that consumes this data, and how stale can that data be before the decision quality degrades?
If the answer is milliseconds, you probably need streaming. If the answer is "our ops team reviews churn alerts every morning," you need a batch job and a good scheduler.
A rough framework:
Decision latency < 1 minute → Streaming is justified
Decision latency 1-60 minutes → Micro-batch, evaluate carefully
Decision latency > 1 hour → Batch. Full stop.
This is not a sophisticated model. It is deliberately blunt, because most teams benefit more from a forcing function than from a nuanced framework that can be argued around indefinitely.
What "Good Enough" Actually Costs
The team that was running Kafka at $4,200 a month for an analytics workload had a micro-batch alternative available the whole time. A lightweight setup using scheduled incremental loads, a modest compute instance, and a simple orchestration layer would have handled the same data volumes with the same outcomes for roughly $800 a month.
The delta was not buying better latency. The delta was buying the ability to say "we run on Kafka" in architecture reviews.
That is a $3,400 monthly expenditure on perceived credibility. Some organizations can absorb that without blinking. Plenty cannot, and the teams inside those organizations often do not even realize the cost is optional.
The Boring Infrastructure Defense
Batch jobs and micro-batch pipelines are not legacy technology. They are appropriate technology for the majority of analytics and reporting workloads that exist in production today. A well-written scheduled job that runs every 15 minutes, handles failures cleanly, and sends alerts when something goes wrong is genuinely harder to build than it sounds, and it solves the actual problem most teams have.
The engineering challenge worth taking seriously is not "how do we stream this." It is "how do we build something reliable, observable, and cheap enough that we can maintain it without heroics."
That is a less exciting sentence to write in a design doc. It also tends to produce systems that are still running correctly two years later, which is the metric that actually matters.
The takeaway is simple: audit what your downstream consumers actually do with the data before you architect for the data's freshness. The pipeline should serve the decision, not the other way around.
Top comments (0)