Picture this: you've got five microservices that all need to know when an order is placed. The payments service, the inventory service, the email service, the analytics pipeline, and some recommendation engine that data science bolted on last quarter. You could have the order service call each one directly. But then adding a sixth consumer means changing the order service. And if the email service is down for thirty seconds, do you just lose that event?
This is the shape of problem that event streaming solves. Once you have more than a couple of services that need the same data, you need something in the middle that can accept writes fast, store them durably, and let multiple consumers read independently at their own pace. That's Kafka.
🧠 Kafka isn't a message queue
This is the single most useful reframe. A traditional message queue (RabbitMQ, SQS) works like a to-do list: a message goes in, one consumer picks it up, the message gets deleted. Done.
Kafka doesn't do that. Kafka is an append-only log. A producer writes a message to the end of the log, and it stays there. Consuming a message doesn't delete it. Nothing deletes it except the retention policy (time-based or size-based, you configure it). A topic might keep seven days of data regardless of whether anyone read it.
And here's the part that changes everything: multiple independent consumer groups can read the same topic at completely different positions. Your analytics pipeline can be three hours behind your real-time notification service, and that's fine. They don't interfere with each other. They don't compete for messages. Each group maintains its own position in the log.
So Kafka is a distributed commit log with pub/sub semantics on top. Not a queue.
Topics, partitions, and offsets
A topic is just a named feed of messages. "order-events", "user-signups", whatever you want. But a topic isn't one big log. It's split into partitions, and this is where the real design lives.
Each partition is its own append-only, ordered sequence. Every message that lands in a partition gets a monotonically increasing number called an offset. Offset 0, 1, 2, 3, and so on. Ordering is guaranteed within a single partition. Not across the whole topic. This is a point people miss constantly.
Topic: order-events (3 partitions)
Partition 0: [0] [1] [2] [3] [4] [5] →
Partition 1: [0] [1] [2] [3] →
Partition 2: [0] [1] [2] [3] [4] [5] [6] [7] →
Each partition lives on a broker (a Kafka server). Partitions can be replicated across brokers for fault tolerance. One replica is the leader, the others are followers. Modern Kafka can coordinate all of this without ZooKeeper using KRaft mode, but the mental model is the same either way.
Producers and partitioning
When a producer sends a message, it goes to one specific partition. Which one? That depends on the message key.
If you provide a key (say, an order ID), Kafka hashes it and maps it to a partition. Same key always goes to the same partition. This gives you per-key ordering, which is usually what you actually want. All events for order #4521 arrive in order.
If you send with a null key, Kafka spreads messages across partitions round-robin. Good for throughput, but you lose any ordering guarantees.
Producers also control durability via the acks setting:
-
acks=0: don't wait for any confirmation. Fastest, least safe. -
acks=1: wait for the leader to write it. Faster, but you can lose data if the leader dies before followers catch up. -
acks=all: wait for the in-sync replicas (ISR) to confirm. Safest, and it's the default in recent Kafka versions.
Here's a minimal producer with kafkajs:
const { Kafka } = require("kafkajs");
const kafka = new Kafka({ brokers: ["localhost:9092"] });
const producer = kafka.producer();
await producer.connect();
await producer.send({
topic: "order-events",
messages: [
{ key: "order-4521", value: JSON.stringify({ type: "created", total: 79.99 }) },
{ key: "order-4522", value: JSON.stringify({ type: "created", total: 24.50 }) },
],
});
// same key = same partition = ordered per order
await producer.disconnect();
⚡ Consumer groups and offsets
This is the mechanic people get wrong most often.
A consumer group is a set of consumers that cooperate to read a topic. Within one group, each partition is assigned to exactly one consumer. Not shared. Not split. One partition, one consumer in that group. Period.
So if you have a topic with 6 partitions and a consumer group with 3 consumers, each consumer handles 2 partitions. If you scale to 6 consumers, each gets 1 partition. Perfect parallelism. But if you add a 7th consumer? It sits idle. There's nothing for it to do. You can't have more active consumers than partitions in a group.
Each consumer group tracks its own offset per partition. That's how different groups stay independent. The "notifications" group can be at offset 450 on partition 2 while the "analytics" group is at offset 312 on the same partition. They don't know about each other.
When a consumer joins or leaves, Kafka triggers a rebalance that redistributes partitions among the remaining members. And consumers pull data from Kafka. Kafka doesn't push to consumers.
const { Kafka } = require("kafkajs");
const kafka = new Kafka({ brokers: ["localhost:9092"] });
const consumer = kafka.consumer({ groupId: "notifications-group" });
await consumer.connect();
await consumer.subscribe({ topic: "order-events", fromBeginning: false });
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
const event = JSON.parse(message.value.toString());
console.log(`Partition ${partition} | Offset ${message.offset} | ${event.type}`);
// offset is committed automatically after processing (default behavior)
},
});
Why partition count is the decision that haunts you
Partitions are simultaneously the unit of parallelism AND the unit of ordering. More partitions means more consumers can work in parallel. But it also means weaker global ordering (you only get order within a partition), more file handles on brokers, and longer leader elections.
And the gotcha here is: increasing partition count later breaks your key-to-partition mapping. If you had 6 partitions and messages with key "user-123" went to partition 2, bumping to 12 partitions means "user-123" now hashes to a different partition. Your per-key ordering guarantee is effectively reset.
So over-provision partitions from the start. It's way easier to have 12 partitions with 3 consumers today and scale to 12 consumers later than to start with 3 partitions and realize you've capped your throughput. You can't undo this decision cleanly.
🛠️ When Kafka is the wrong tool
Look, Kafka is powerful but it's heavy. If you need a simple work queue with per-message acknowledgment, dead-letter queues, and retry logic out of the box, you're fighting Kafka's design. It wasn't built for that.
| Kafka | RabbitMQ | SQS | |
|---|---|---|---|
| Model | Append-only log | Message queue | Message queue |
| Retention/Replay | Yes (time/size policy) | No (deleted on ack) | No (deleted on ack, 14d max) |
| Ordering | Per partition | Per queue (with caveats) | FIFO queues only |
| Per-message ack | No (offset-based) | Yes | Yes |
| Routing | Topic + partition key | Exchanges, bindings, routing keys | Queue per consumer |
| Ops burden | High (brokers, replication, monitoring) | Medium | None (managed) |
If your use case is "process each job once, retry failures, route to dead-letter after 3 attempts" then RabbitMQ or SQS is the better fit. Actually, SQS is probably the best fit if you're on AWS and don't want to operate anything. Kafka shines when you need durable event replay, multiple independent consumers reading the same stream, or high-throughput append-only ingestion.
📌 Takeaways
- Kafka is a distributed log, not a queue. Consuming doesn't delete messages.
- Ordering is per-partition only, and the message key is what pins related events to the same partition.
- Within a consumer group, one partition goes to one consumer. Extras sit idle.
- Each consumer group tracks offsets independently, so groups never interfere.
- Partition count caps your parallelism and can't be reduced. Pick generously.
Top comments (0)