You inherit a new codebase. There's a Kafka cluster humming away in the infrastructure. You open the consumer group dashboard and see... fourteen messages in the last hour. The brokers are healthy. The Schema Registry is running. ZooKeeper is doing whatever ZooKeeper does. And somewhere, a $400/month managed cluster is processing roughly what a cron job and a database table could handle before your morning coffee gets cold.
This is not a rare story.
Kafka Solves One Specific Problem
Kafka is genuinely excellent at one thing: reliably moving ordered event streams between multiple independent systems at massive scale, with durability, replay capability, and back-pressure handling built in. When you have hundreds of thousands of events per second, multiple downstream consumers each doing different things with the same stream, and hard requirements around ordering and replay, Kafka earns every bit of its operational weight.
The problem is that description matches maybe 5% of the workloads it actually gets deployed on.
Teams reach for Kafka because it sounds serious. It signals that you're building something scalable. It shows up in every architecture diagram for companies like Netflix and LinkedIn. What those diagrams don't show is the decade of scale those companies hit before Kafka became the right answer.
What Running Kafka Below Scale Actually Costs You
Here's a condensed version of a real situation: a team spends three months configuring brokers, getting Schema Registry working, migrating off ZooKeeper, writing producers and consumers, debugging partition assignment, and setting up monitoring. The workload they're handling? Around 1.4 messages per minute.
That's not an exaggeration. It's the kind of thing that happens when "we might need to scale this" becomes "we should build for scale right now."
The costs aren't just financial. They're:
- Cognitive load. Every new developer has to learn Kafka concepts before they can touch the pipeline.
- Failure surface. More moving parts means more places for things to go wrong at 2am.
- Deployment complexity. A simple bug fix now requires coordination across broker configs, schema versions, and consumer group offsets.
- Opportunity cost. That's engineer time that didn't go into the actual product.
A PostgreSQL table with a processed_at column and a background worker would have shipped in a day and failed less.
The Signals That Actually Matter
So when does Kafka make sense? There's a short checklist worth running through before you commit:
Multiple independent consumers on the same stream. If you have one consumer reading events and doing one thing with them, a queue or even a database table is simpler and faster to operate. Kafka's fan-out model shines when three or four completely separate services need the same event stream independently.
Replay requirements. If you need to reprocess historical events, whether for debugging, rebuilding a derived dataset, or onboarding a new consumer to historical data, Kafka's log retention is genuinely useful. If you don't need replay, you're paying for a feature you'll never use.
Scale that justifies the overhead. Below around 10,000 messages per day, the overhead is almost never worth it. A managed queue service, a simple worker pulling from a database, or even a well-structured cron job will be cheaper, faster to operate, and easier to debug.
Event-driven workflows across team boundaries. When multiple teams own different services that need to react to the same events without tight coupling, Kafka's decoupling model becomes genuinely valuable. For a single team running a single service, it's indirection for its own sake.
What to Use Instead
If your volume is low and your consumers are few, here's a rough hierarchy of alternatives:
< 1,000 messages/day → Cron job + database table
1K - 10K messages/day → Managed queue (SQS, Cloud Tasks, etc.)
10K - 1M messages/day → Evaluate based on consumer count and replay needs
> 1M messages/day → Kafka or a purpose-built streaming platform
This isn't a hard rule, but it gives you a starting point that isn't "default to the most complex option."
A managed queue handles retries, dead-letter queues, and visibility timeouts out of the box. A database-backed worker pattern is debuggable with a single SQL query. These aren't compromises, they're appropriate tools for the actual problem.
The Actual Takeaway
Architecture decisions compound. Choosing Kafka when you don't need it doesn't just slow down today's work, it shapes how every new engineer onboards, how you debug incidents, and how much infrastructure you're paying to maintain six months from now.
The right question isn't "could this eventually need Kafka?" It's "does this need Kafka right now, given what it actually does today?" Most of the time, the honest answer is no, and the team that ships a boring, debuggable background worker will outrun the team still configuring broker replication factors for a feature that processes fourteen events an hour.
Top comments (0)