The Kafka Conversation Every Small Team Has (And What Comes After It)
Eight engineers. Someone says "we should add Kafka." Someone else asks why. The answer is some version of "so our services can talk to each other without being coupled." It's not a wrong answer. It's just not a reason to run Kafka.
This conversation happens on almost every team that hits a certain scale. And the instinct is understandable, Kafka is battle-tested, it's what Netflix and LinkedIn built on, and the docs make it look approachable. But approachable is doing a lot of work in that sentence.
What You're Actually Signing Up For
Kafka isn't just a message queue. It's a distributed log system built to handle millions of events per second across many consumers, with durable replay, partitioning, and replication baked in. That's genuinely powerful, and genuinely expensive to operate when you don't need all of it.
For a team of eight, you're probably not worried about millions of events per second. You're worried about keeping a handful of services loosely coupled, maybe getting some async processing in, and not waking up at 2am because a consumer group is stuck.
Kafka solves those problems, but it also introduces a new class of problems: broker management, partition rebalancing, offset tracking, schema registry decisions, connector configs. You're trading one kind of complexity for another.
Lightweight Patterns That Actually Work
Before reaching for Kafka, it's worth being honest about what your system needs event-driven architecture to do.
Postgres LISTEN/NOTIFY is underrated for low-to-medium volume internal events. If you're already running Postgres (you probably are), you can publish events from a transaction and have a consumer pick them up, all without a separate broker. It's not durable the way Kafka is, and it won't survive a consumer restart gracefully, but for many use cases that's fine.
Outbox pattern over a simple queue (SQS, RabbitMQ, even Redis Streams) solves the most common async problem, reliably publishing events from a service after a DB write, without a distributed log. You write to an outbox table in the same transaction, a poller picks it up and sends it to a queue, consumers process it. Simple, debuggable, and you can run it with tools you already understand.
Redis Streams hit a sweet spot for teams that want durability and consumer groups without Kafka's operational weight. You get replay, multiple consumers, and persistence, at a fraction of the infrastructure cost.
The Real Question Behind the Architecture Decision
The thing that often gets skipped in these conversations is: what does "event-driven" actually mean for your specific services, right now?
If your services need to react to state changes in near-real-time, you need some form of event propagation. But the mechanism matters a lot less than the guarantees you actually need. Do you need replay? For how long? Do you need multiple independent consumers? Do you care about ordering at the partition level?
Most small teams, if they answer those questions honestly, find they need something much simpler than Kafka. A well-structured outbox with a reliable queue handles a huge chunk of real-world event-driven use cases.
When Kafka Actually Makes Sense
Kafka is genuinely the right answer when the volume is high, when you have multiple independent consumers that need to replay history at different offsets, or when you're building something where the stream itself, the ordered, durable log, is the core data structure.
Financial data pipelines are a classic example. Market data feeds, trade execution events, audit logs that regulatory teams need to replay arbitrarily far back, these are cases where Kafka's properties aren't overhead, they're the point. When events have real monetary or compliance consequences, the durability and replay guarantees justify the operational cost.
But if you're not there yet, borrowing the architecture before you need its properties just adds weight.
A Pattern Worth Stealing
Start with the simplest thing that gives you loose coupling: an outbox table, a poller, and a queue your consumers already understand. Write it so it could be replaced later. When you actually hit the limits of that, when you genuinely need replay at scale, or you have ten consumers that need to process the same event stream independently, migrate to Kafka knowing exactly why you're doing it.
That's a better story than "someone said we should add Kafka."
Top comments (0)