Kafka End-to-End, Past the Point Where Tutorials Stop

Most Kafka explainers stop at "it's a distributed commit log" and "it's fast because sequential disk I/O." True, but that's just the tip of the iceberg. This post is a full pass through Kafka's internals — topology, storage, replication, cluster metadata, the producer path, the consumer path, and the wire protocol — written for people who already know what a broker is and want the parts that are actually confusing.

One idea threads through almost everything below: Kafka refuses random access. A consumer never asks for a key, only "everything after offset N." That single constraint is why the storage engine looks the way it does, why durability is a replication story rather than a disk-flushing story, and why so much of the producer/consumer machinery exists purely to handle the edge cases that constraint creates.

What this covers

If you've done the "produce a message, consume a message" tutorial, you already have the vocabulary — you just haven't seen what's underneath it yet. This post is organized as five layers, each building on the one before:

• Part I — The physical model. What a broker, topic, and partition actually are, and what's really sitting inside a partition on disk.
• Part II — Speed and safety. How reads get fast (page cache, zero-copy) and how writes get durable (replication, not fsync).
• Part III — Coordination. How the brokers agree on all of the above — who's alive, who's leader, what the config is.
• Part IV — The client contracts. What a producer's acks setting and a consumer group's rebalance actually do, mechanically.
• Part V — Under the hood. The wire protocol tying all of it together.

Each section starts by naming the tutorial-level version of the idea, then goes past it — so if a section opens with something you already know cold, that's the signal you can skim to the "but" and pick up from there.

Here's the whole thing in one picture before we go part by part — a single record's path from producer to committed offset, with the KRaft controller quorum coordinating leadership off to the side:

Part I — The physical model

1. The basic shape: broker, topic, partition

Simply put "Within Broker you have topics. A topic is a stream of messages, and it's split into partitions for scalability." True, but which of these three words refers to an actual machine, because the rest of this post hinges on it:

• Broker = a physical or virtual server. It owns CPU, RAM, disk, and NIC — this is the actual hardware in the cluster.
• Topic = a logical name, like user-signups. It is not a file anywhere, and it doesn't live on any single broker.
• Partition = the real thing. A topic is split into some number of partitions, and each partition is an ordered, append-only sequence of records living as an actual directory on one specific broker's disk, replicated to a handful of other brokers for safety.

Everything about how Kafka scales falls out of that partition being the true unit of physical storage. A topic with one partition can only ever live on one broker, no matter how big your cluster is; give it ten partitions and Kafka can spread those ten physical directories across ten different brokers, so ten machines share the read/write load instead of one.

Also, leadership is assigned per-partition, not per-topic or per-broker.

Topic: orders  (3 partitions, replication factor 2)

           Partition 0        Partition 1        Partition 2
Leader:    Broker 1           Broker 2           Broker 3
Follower:  Broker 2           Broker 3           Broker 1

Every partition has one broker acting as its leader (handling all reads and writes for it) and some number of followers replicating it. Notice Broker 2 above is a leader for partition 1 and a follower for partition 0 at the same time — a busy topic's partitions get scattered across brokers on purpose, so no single server becomes a hotspot just because it hosts something popular.

This also dictates how producers actually route messages: a producer doesn't write to "a topic." It fetches cluster metadata (which broker leads which partition), hashes the record's key to pick a partition, and sends the record directly to whichever broker currently leads that partition.

With that physical model in place, everything downstream in this post (segment files, replication, rebalancing) is really about managing that one unit: the partition. Next: what's actually inside one.

2. The storage engine: segments, not trees

The tutorial version:"a partition is an append-only log." Also true, and also where most explanations stop — without ever opening the directory to see what's in it. Not one giant file but a set of segments. At any time there's exactly one active segment being appended to, plus a set of older, closed segments. Each segment is really three files sharing a base offset as their filename:

• .log — the actual records, appended sequentially, never rewritten in place.
• .index — a sparse mapping of offset → byte position in the .log file. Sparse means it doesn't index every message, only one entry every few KB (log.index.interval.bytes). Finding a specific offset is a binary search over this sparse array to locate the right block, followed by a short linear scan through that block to the exact record. That's not the O(log n) exact-key lookup a B-Tree gives you but more like it "get close, then walk a few KB". And that's fine, because Kafka only ever needs a starting point for sequential reads, never an arbitrary record.
• .timeindex — the same trick, but mapping timestamp → offset, which is what powers offsetsForTimes (seeking by wall-clock time instead of a raw offset).

New segments roll over when the active one hits log.segment.bytes or log.roll.ms, whichever comes first. This matters operationally: segment size directly controls how much data can be deleted or compacted in one unit, and how much a broker has to scan when it restarts and rebuilds its index files.

Retention works at the segment level, not the message level because a whole segment is deleted once every record in it is older than retention.ms (or the partition exceeds retention.bytes). That's deliberate: deleting whole files is cheap; deleting individual records from the middle of an immutable append-only file is not.

Log compaction (used for topics like _consumer_offsets and for building changelog) is a different, optional cleanup mode: instead of deleting by age, a background cleaner thread rewrites closed segments to keep only the latest record per key, dropping everything superseded and eventually removing keys marked with a null-value "tombstone" once it's safe to do so. It only touches closed segments — never the active one — and it triggers based on a dirty ratio (min.cleanable.dirty.ratio), not on a schedule. This is the one place Kafka does something LSM-flavored, but notice it's opt-in and narrow in scope, not the default write path.

Record batch format: producers don't send one record at a time instead they send record batches, and the batch itself (not each record) carries a header with a base offset, a CRC checksum, an attributes bitmap (compression codec, timestamp type, whether it's part of a transaction, whether it's a control batch), a producer ID, a producer epoch, and a base sequence number. Individual records inside the batch store only small deltas (relative offset, relative timestamp) against that header, which is why compression works so well here - repetitive structure compresses cheaply, and Kafka compresses the whole batch as one unit rather than record-by-record. Hold onto that producer ID and sequence number as they resurface in Part IV when we get to idempotence and transactions.

We now know what's on disk. Part II is about what happens when that disk gets read from and written to under load.

Part II — Speed and safety

3. The read path: page cache, zero-copy, and its asterisk

"Kafka is fast because of zero-copy." Here's the actual path: Kafka doesn't manage its own read cache. It deliberately leaves that job to the Linux page cache, and uses the sendfile() syscall to move bytes straight from page cache to network socket, bypassing the JVM heap and user space entirely. A conventional "read file, write to socket" path involves up to four context switches and four data copies; sendfile collapses that to two of each. If a consumer is reading data that was just produced, it's often still sitting in page cache, so the broker never touches disk for that read at all.

The asterisk that a lot of "Kafka is zero-copy" claims skip: this optimization disappears the moment TLS is enabled. Encrypting the bytes in flight requires pulling them into user space to run through the SSL engine before they hit the socket — there's no such thing as kernel-space encryption via sendfile. Since most production clusters run with TLS for good reasons, a lot of real-world deployments aren't actually getting the zero-copy benefit people assume they are. Kafka still performs fine here — the network, not the CPU, is usually the real ceiling — but it's worth knowing which optimization you traded away for encryption.

4. Replication and durability: LEO, high watermark, and what acks=all really promises

Usually, all we hear is "set acks=all and Kafka is durable." But this is where the mechanism might sounds backwards at first if you stop there: Kafka gets its durability guarantees from replication, not from forcing disk flushes (fsync) on every write like a typical DB write. Disk flushing is deliberately left to the OS's background threads, because blocking on physical disk for every message would tank throughput and can make a broker miss heartbeats and get evicted from the cluster. So trading a marginal durability gain for a much bigger availability loss.

To see how replication actually provides that safety, it helps to be precise about the offsets involved, because "offset" gets used loosely:

• Log End Offset (LEO) every replica (leader and followers) tracks its own LEO: the offset of the next record it will write. This is just "how far this specific copy has gotten."
• High Watermark (HW) the leader tracks the LEO of every follower via their fetch requests, and sets the HW to the minimum LEO among all replicas currently in the ISR (in-sync replica set). The HW is the offset up to which a record is guaranteed to exist on every in-sync copy.
• Consumers can only ever read up to the HW, never past it. So they never observe a record that could vanish in a leader failover.

With acks=all (the default since Kafka 3.0), a producer's write is acknowledged only once it's been replicated into the page cache of every broker in the required ISR set (min.insync.replicas), advancing the HW past that record — not once anything has hit a physical platter. The bet is that "every in-sync replica loses power at the same instant before any of them flushes" is astronomically rarer than "one disk hiccups," and in practice that bet holds.

A few honest caveats that make this less of a blanket guarantee than it sounds:

• If the ISR shrinks below min.insync.replicas, producers using acks=all simply start getting errors and you get backpressure, not silent data loss. That's the correct trade-off, but it surprises people who expect the cluster to keep accepting writes no matter what.
• Unclean leader election (unclean.leader.election.enable) allows an out-of-sync replica to become leader to preserve availability, at the cost of silently truncating any records that were never replicated to it. It defaults to disabled for exactly this reason — enabling it is explicitly choosing availability over durability during a bad outage.
• Eligible Leader Replicas, generally available since Kafka 4.0, tightens this further by tracking exactly which replicas are provably caught up to the HW and therefore safe to promote, closing some of the historical gap between "technically in the ISR" and "actually safe to elect as leader."

Reads are fast, writes are durable — but both of those depended on brokers agreeing on facts like "who's the leader" and "what's in the ISR." Part III is about how that agreement actually happens.

Part III — Coordination

5. Cluster metadata: how brokers agree on all of this

Tutorials tend to mention this only in passing "ZooKeeper coordinates the cluster" as background trivia rather than something with mechanics worth understanding. Everything in Part II about who leads which partition, which replicas are in the ISR, which brokers are even alive is itself a piece of state that the whole cluster has to agree on, and that agreement has to survive brokers crashing and restarting.

For most of Kafka's life, an external ZooKeeper ensemble stored this metadata and ran controller elections. As of Kafka 4.0, ZooKeeper mode has been removed entirely.not deprecated, gone. The only supported mode now is KRaft, Kafka's own Raft-based consensus protocol, where a set of dedicated controller nodes replicate cluster metadata (broker liveness, partition leadership, configs) through an internal Raft-replicated topic called _cluster_metadata, instead of relying on an external system. (Comment down below if you guys want me to explain about consensus protocols.) Controller failover is now sub-second instead of the several seconds typical of ZooKeeper, and there's one less distributed system to operate.
That's the server side settled - brokers agreeing on brokers. Part IV moves to the client side: what a producer's flags and a consumer group's membership actually do against all of this machinery.

Part IV — The client contracts

6. The producer path, end to end

.send(), maybe batch.size and linger.ms as performance knobs, and acks as a durability knob.

Batching risk window. Producers buffer records client-side before sending, controlled by batch.size and linger.ms. If the application process dies while records are sitting in that in-memory buffer — before anything reaches the network — they're gone, and no broker-side guarantee can protect data the broker never received.

acks describes what happens after the batch leaves the producer:

• acks=0 fire and forget, no wait for a response at all.
• acks=1 the leader acknowledges as soon as it has the batch; if the leader crashes before replicating, it's lost.
• acks=all the leader waits for the full ISR quorum before acknowledging, at the cost of latency. This has been the default since Kafka 3.0.

Idempotent producers (also default since Kafka 3.0, via enable.idempotence=true) solve a narrower but very common problem: retries creating duplicates. When enabled, the broker hands the producer a Producer ID (PID), and every batch carries a sequence number that strictly increases per (PID, partition) the same fields we flagged in the record batch header back in Part I. The broker remembers the last sequence number it accepted for each (PID, partition) pair; if a retried batch arrives with a sequence number it's already seen, the broker just discards the duplicate and returns success, rather than appending it again. This is a single-producer-session, single-partition guarantee — it doesn't help across a producer restart, and it doesn't help if you need atomic writes across multiple partitions.

Transactions are built on top of idempotence to solve exactly that gap — atomic writes across multiple partitions (the classic "consume, transform, produce, and commit the input offset" pattern, where you need all of it to become visible together or not at all). A transactional.id gives the producer a stable identity across restarts (same PID, bumped producer epoch), and a server-side transaction coordinator tracks the transaction's state durably in an internal topic, __transaction_state. Committing or aborting writes a control record (a marker) into every partition involved — these markers actually consume real offsets, which is why you sometimes see offsets "skip" in a transactional topic; that gap is a marker, not lost data. Consumers configured with read_committed only read up to the Last Stable Offset (LSO) — the point before any still-open transaction — so LSO ≤ HW ≤ LEO always holds, and a long-running transaction can visibly stall downstream consumers by holding the LSO back. The epoch bump also fences "zombie" producers: if an old instance wakes up after being presumed dead (say, after a long GC pause) and tries to write with a stale epoch, the coordinator and brokers reject it outright.

7. The consumer path, end to end

The tutorial version: "consumer groups split up partitions, and rebalance when a consumer joins or leaves." True, and this is the section where "confusing" usually means the mental model is a version or two out of date.

Consumer groups exist to let multiple consumer processes cooperatively split up a topic's partitions, with a hard rule: within one group, a partition is assigned to at most one consumer at a time (a consumer can own multiple partitions, but not vice versa). Different consumer groups are entirely independent — each gets its own full copy of every message at its own pace.

Two different flavors of "offset" show up on the consumer side and get conflated constantly:

• The High Watermark, from Part II — a replication-safety marker set by the leader, not something any individual consumer controls.
• The committed offset — a consumer group's own bookmark of how far it has read. This is stored inside Kafka itself, in an internal topic called _consumer_offsets, managed by whichever broker is acting as that group's group coordinator. (Historically this lived in ZooKeeper; it hasn't for a long time now, well before ZooKeeper was removed entirely.)

Rebalancing has genuinely changed, not just been tweaked. The "classic" protocol works like this: consumers join via JoinGroup, the broker's group coordinator picks one member as the group leader, that member runs a client-side assignment algorithm (Range, RoundRobin, Sticky, or CooperativeSticky), and ships the resulting plan back through the coordinator via SyncGroup. The original version of this was "stop the world" — every consumer paused during a rebalance. CooperativeStickyAssignor improved that by making revocations incremental, but there was still a group-wide synchronization barrier underneath it.

KIP-848, generally available since Kafka 4.0, replaces this by moving assignment logic onto the broker's group coordinator entirely. Consumers become "thin": they declare their subscription, send continuous heartbeats (no more multi-round JoinGroup/SyncGroup handshake), and accept whatever partitions the coordinator's reconciliation loop hands them, incrementally. There's no elected client-side leader anymore, and no group-wide pause — only the specific partitions that need to move are affected, while unrelated consumers in the group keep processing the whole time. It's opt-in today via group.protocol=consumer and expected to become the default in a future release, but it's already the more scalable path, especially for large consumer groups where the old synchronization barrier was a real operational pain point.

Producers and consumers both ultimately talk to brokers over a connection. Part V closes with what's actually moving across that connection.

Part V — Under the hood

8. The wire protocol

This is the layer tutorials almost never mention, because it's invisible right up until something goes wrong. Kafka skips HTTP, gRPC, and WebSockets in favor of a custom binary protocol directly over raw TCP, minimizing framing overhead. Clients and brokers negotiate which versions of each request/response API they both support via an ApiVersions handshake at connection time, which is how Kafka has evolved its protocol for over a decade without breaking every client on every upgrade.

Consumer fetches are a form of long polling, tuned by two settings. fetch.min.bytes tells the broker not to bother responding until at least this much new data is available; fetch.max.wait.ms caps how long it'll wait before responding anyway, even with less. Producers, meanwhile, pipeline requests over the same persistent TCP connection — sending the next batch without waiting for the previous one's acknowledgment to return — to keep the connection saturated. Follower replication reuses this identical fetch/long-poll mechanism rather than maintaining a separate internal replication protocol, which is a nice simplification: a follower is, from the leader's point of view, just another consumer with a very tight polling loop.

---