The Kafka Tax Is Real, Here's What Small Teams Actually Use Instead
Kafka is a genuinely impressive piece of infrastructure. It handles millions of events per second, survives broker failures, and keeps financial exchanges humming. It also requires a dedicated team to operate it well, a ZooKeeper (or KRaft) cluster to manage, careful partition tuning, and a monitoring setup that can itself become a full-time job.
For small teams moving fast, that overhead can quietly eat the velocity that event-driven architecture is supposed to give you in the first place.
What the "Kafka tax" actually looks like
It's not just the ops burden. It's the cognitive tax. Before you write a single line of business logic, you're making decisions about:
- Number of partitions (and you can only go up, never down)
- Replication factor and minimum in-sync replicas
- Retention policies and disk sizing
- Consumer group lag monitoring
- Schema registry, or the chaos of not having one
For a team of two or three engineers shipping a product, that's a significant upfront investment before you've proven the event-driven model even makes sense for your use case.
The lightweight patterns worth knowing
Redis Streams is often the first stop. It gives you a consumer group model similar to Kafka's, persistence, and backpressure handling, all on infrastructure most teams already run. Throughput caps out well below Kafka, but for most internal event pipelines, you're nowhere near those limits anyway.
Postgres LISTEN/NOTIFY is underrated for low-volume event flows. If you're already on Postgres, you can emit events from a trigger or application layer and have consumers react in near-real-time. It's not durable in the same way Kafka is, but for workflows where eventual delivery via polling is acceptable, it removes a whole infrastructure dependency.
NATS JetStream sits in interesting middle ground. Lightweight to operate, fast, and offers at-least-once delivery with persistence. It's a good fit if you want something closer to Kafka's guarantees without the cluster management overhead.
SQLite with a polling loop sounds laughable until you realize how many internal event pipelines at early-stage companies are basically this, and it works fine until it doesn't.
Where lightweight patterns break down
These alternatives all share a common ceiling: they're not designed for high-cardinality, high-frequency data where you need strict ordering across partitions, complex replay, or consumer fan-out at scale.
Financial market data is a good stress test. Tick-by-tick price feeds, order book updates, and trade confirmations arrive at rates and with ordering requirements that quickly expose the limits of Redis Streams or a LISTEN/NOTIFY setup. The moment you need to reconstruct a precise sequence of events across multiple symbols in parallel, the simpler tools start showing their seams.
That's not a failure of the lightweight approach, it's just a different problem class. The mistake is treating Kafka as a default when you're in the simple zone, and treating the lightweight tools as "good enough" when you've silently crossed into the complex zone.
How to actually decide
A rough heuristic: if your event volume fits in a single Postgres table comfortably and your consumers can tolerate polling latency measured in seconds, start with Postgres. If you need sub-second fan-out to multiple consumers with replay, Redis Streams or NATS JetStream. If you're building something where event ordering, long-term retention, and consumer group isolation are all hard requirements at scale, Kafka (or a Kafka-compatible layer like Redpanda) is probably worth the tax.
The key is not defaulting to Kafka because it's what you've seen at previous companies. Match the infrastructure to the actual throughput and ordering requirements you have today, with some honest projection of where you'll be in six months.
One thing teams often miss
Switching later is painful but possible, if you design your producers and consumers around a thin abstraction layer from day one. Even something as simple as a publish(event) function that wraps your current backend means you can swap Redis Streams for Kafka without rewriting every service. Most teams don't do this and end up with Kafka client code scattered everywhere, which makes the initial bet much harder to unwind.
Build for where you are. Leave a door open for where you might go.
Top comments (0)