Apache Kafka Architecture in 3 Minutes

#architecture #backend #kafka #systemdesign

Unlike traditional message queues that delete messages after consumption, Apache Kafka is a distributed, append-only commit log. Events are persisted to disk and can be replayed at any time.

Here is a quick breakdown of how it works.

1. Core Primitives

Topic: A logical stream of events (e.g., orders).
Partition: Slices of a topic spread across brokers to enable horizontal scaling.
Offset: A sequential, immutable ID (0, 1, 2...) assigned to every record in a partition.
Broker: A Kafka server node that stores partitions and serves client requests.

2. How Data Flows

[ Producer ] ──(publishes)──► [ Partition Leader ]
                                     │
                 ┌───────────────────┴───────────────────┐
                 ▼ (pull replication)                    ▼ (pull replication)
     [ Follower Replica (ISR) ]             [ Follower Replica (ISR) ]
                                     │
                                     ▼
                          [ Consumer Group (Pulls) ]

Partition Leader: Every partition has one leader broker that handles all writes.
Followers (ISR): In-Sync Replicas pull messages from the leader to stay synchronized.
High Watermark (HW): Messages are only visible to consumers after all ISR replicas have confirmed the write.
Consumer Groups: Each partition in a topic is consumed by only one worker per consumer group, guaranteeing ordered, conflict-free parallel reads.

3. How Producers Choose Partitions

Partition routing happens on the client side, not on the broker:

Explicit Partition: Sent directly if specified.
Message Key provided: Hashed via abs(MurmurHash2(key)) % total_partitions (records with the same key always go to the same partition).
No Key: Handled by the Sticky Partitioner, which fills an in-memory batch for one partition before rotating to the next.

4. Why Kafka is Incredibly Fast

Sequential Disk I/O: Appending records to the end of a log file is as fast as sequential memory access.
OS Page Cache: Kafka relies on OS RAM caching rather than JVM heap, avoiding garbage collection pauses.
Zero-Copy (sendfile): Data is transferred directly from the OS page cache to the network card (NIC), bypassing user-space CPU memory copying.

5. Tracking Progress (Offsets)

Brokers do not track individual message acknowledgments. Consumers manage their own state:

Fetch Position: Next offset requested in memory.
Committed Offset: Checkpoint saved in the internal __consumer_offsets topic. When a consumer restarts, it resumes from this exact checkpoint.

Quick Cheat Sheet

Component	Responsibility
Broker	Stores partition logs on disk
Partition	Unit of parallelism & ordering
Leader / ISR	Leader serves writes; ISR maintains fault tolerance
Consumer Group	Parallel consumption with automatic rebalancing
Zero-Copy	Page cache ➔ NIC for high throughput

📖 Full Deep Dive & Architecture Reference:

For the comprehensive engineering guide, diagrams, and multi-cluster setups, check out the original post:

👉 Apache Kafka Architecture & Core Concepts

DEV Community