The first time I used Kafka in production, I treated it like RabbitMQ: a message queue. Produce a message, consumer picks it up, done. The message disappears. That’s how queues work.
That mental model worked — until a downstream payment processor went down for 3 hours, and we had to replay everything it missed. I went to look for the messages.
They were still there.
That moment is when Kafka actually clicked for me. It is not a queue. The messages do not disappear when consumed. And once that distinction lands, every other design decision Kafka makes becomes logical rather than arbitrary.
The commit log — Kafka’s actual storage model
Kafka stores messages in an append-only commit log — a file on disk that only ever grows. Every message gets an offset: a sequential position number starting at zero. When a consumer reads a message, nothing is deleted. The message stays. The consumer’s position pointer simply advances.
This is why two completely independent systems can read the same Kafka topic without interfering with each other. Consumer Group A might be at offset 7. Consumer Group B might be at offset 4. They share the same underlying log but track their own positions independently. If you add a third consumer group next week — a new analytics pipeline, a new audit service — it starts at offset 0 and replays everything from the beginning.
In a traditional message queue, that’s impossible. The message was already consumed and discarded. In Kafka, the log is the source of truth, and consumers are just readers moving through it at their own pace.
Now here’s the counterintuitive part. Kafka writes to disk and is still faster than many in-memory message brokers for high throughput. The reason is sequential I/O. Random disk writes are slow — the write head has to seek across the platter. Sequential writes, appending to the end of a file, are fast. Kafka only ever appends. Combined with the OS page cache — which keeps recently written data in RAM automatically — and a zero-copy technique called sendfile() that transfers data directly from the page cache to the network socket without passing through application memory, Kafka achieves throughput that can reach hundreds of thousands of messages per second on commodity hardware.
The disk is not Kafka’s compromise. It is the design.
Topics, partitions, and the order guarantee everyone gets wrong
A topic is a named log — think of it like a database table, but for events. A partition is a shard of that topic stored on a specific broker. More partitions means more parallelism, which means higher throughput.
Here is the part that trips up almost every developer building on Kafka for the first time: Kafka guarantees message order within a partition. It does not guarantee order across partitions.
If you have a topic with three partitions and a producer sends ten messages for the same user, those messages could land on any partition — round-robin by default. If they scatter across partitions, a consumer might process them out of order.
The fix is the partition key. You set the message key to something stable — a user ID, an order ID, a transaction ID. Kafka hashes that key and routes all messages with the same key to the same partition, every time. Every message for user-1001 lands on partition 0. Every message for user-2002 lands on partition 1. Order is guaranteed per user, not globally — which is usually exactly what you need.
This is one of the most important production decisions you make when designing a Kafka-based system, and most tutorials mention it in a single sentence. Getting it wrong means debugging ordering issues at 2 am that only manifest under load, when multiple partitions are being consumed concurrently.
Consumer groups: how Kafka scales consumption
A consumer group is a set of consumers that share the work of reading a topic. Kafka assigns each partition to exactly one consumer in the group at any given time. With three partitions and three consumers, each consumer handles one partition. With three partitions and five consumers, two consumers sit idle — you cannot have more active consumers than partitions.
Each consumer group tracks its own offset for each partition, stored in an internal Kafka topic called __consumer_offsets. When a consumer restarts after a crash, it reads its last committed offset from that topic and resumes exactly where it left off. This is what makes Kafka fault-tolerant — not replication alone, but the combination of durable storage and tracked offsets.
The production pain point is rebalancing. When a consumer joins or leaves a group — whether through a deployment, a crash, or a scale event — Kafka reassigns partitions across the remaining consumers. During a rebalance, consumption pauses across the entire group. For most workloads, this pause lasts a few seconds. For high-throughput systems with many consumers and frequent deployments, rebalancing can become a significant source of latency spikes — sometimes called a “rebalancing storm.”
Kafka 4.0’s KIP-848 incremental cooperative rebalancing dramatically reduces this. Instead of stopping all consumers and reassigning all partitions, consumers now transfer partitions incrementally while other consumers continue reading. If you’re seeing rebalancing issues in an older Kafka version, upgrading is one of the most effective fixes available.
Replication — how Kafka survives broker failures
Each partition has one leader broker and zero or more follower brokers. All reads and writes go to the leader. Followers replicate the leader’s log passively, staying as close to current as possible. If the leader broker fails, Kafka elects a new leader from the in-sync replica set (ISR) — the brokers that are fully caught up with the leader.
The replication factor controls how many copies exist. A factor of three means you can lose two brokers and still serve traffic. Most production setups use three.
The acks producer setting is your durability dial:
// Fire and forget — fastest, but you can lose messages
props.put("acks", "0");
// Wait for leader only — fast, safer
props.put("acks", "1");
// Wait for all in-sync replicas — slowest, safest
props.put("acks", "all");
props.put("min.insync.replicas", "2");
In fintech systems, acks=all with min.insync.replicas=2 is the standard. You are waiting for at least two brokers to confirm the write before acknowledging the producer. The added latency — typically 10–30ms — is a fair price for knowing that a single broker failure cannot silently drop a payment event.
KRaft — ZooKeeper is finally gone
Before Kafka 4.0, every Kafka cluster depended on ZooKeeper — a separate distributed coordination service — to manage broker metadata, leader elections, and cluster configuration. This meant running and maintaining a separate system alongside Kafka, with its own failure modes and operational complexity.
In March 2025, Apache Kafka 4.0 shipped. ZooKeeper is gone. KRaft — Kafka’s built-in Raft-based consensus protocol — now manages all cluster metadata directly on a quorum of controller nodes. Simpler deployments, faster failover, fewer moving parts.
If you are setting up a new Kafka cluster today, there is no ZooKeeper to configure. If you are still running an older version with ZooKeeper, the migration path to KRaft is well-documented, and the upgrade is worth the effort. Any tutorial or architecture diagram that still shows ZooKeeper as a required component is outdated.
When NOT to use Kafka
This is the part that saves you from over-engineering.
Don’t use Kafka as a simple job queue. If you need tasks processed once by one worker — send an email, resize an image, process a webhook — RabbitMQ or Amazon SQS is simpler and operationally lighter. Kafka’s complexity is only justified when you need replay, multiple independent consumer groups, or throughput above what a traditional queue can handle.
Don’t use Kafka for request-reply patterns. Kafka is designed for one-way event streaming. If you need a response to a specific message, you are working against the grain. Use a proper RPC mechanism.
Don’t get your partition count wrong at the start. You can increase partitions later, but you cannot decrease them without recreating the topic. And increasing partitions changes the hash-to-partition mapping, which breaks the ordering guarantee for any keys that shift partitions. Plan your partition count upfront based on your expected throughput and consumer count.
Don’t skip idempotent producers. Without enable.idempotence=true, network retries can produce duplicate messages. Kafka acknowledges a write, the network drops the response, your producer retries, and the message lands twice. In a payment system, that means double charges. Enable idempotence from day one:
props.put("enable.idempotence", "true");
props.put("acks", "all");
props.put("retries", Integer.MAX_VALUE);
props.put("max.in.flight.requests.per.connection", "5");
This is not optional in fintech. It is the baseline. Kafka is durable not despite writing to disk, but because of it. The commit log is not a compromise — it is the design. Once that clicks, every other Kafka decision makes sense.
Top comments (0)