In Apache Kafka, topics are logical categories, but partitions are the physical reality where distributed streaming happens. Every throughput benchmark, ordering guarantee, consumer group rebalance, and fault-tolerant failover in Kafka is governed directly by partition mechanics.
Understanding partitions is the single most critical concept for software engineers and systems architects building on Kafka. In this guide, we break down how partitions store data on disk, how producers route records to them, how replication guarantees durability, and how partitions establish the hard physical ceiling on consumption parallelism.
1. What is a Partition?
At its core, a Kafka partition is an ordered, immutable, append-only commit log of event records. Rather than storing an entire topic as a single monolithic queue on one server, Kafka divides topics into independent partitions distributed across different brokers in the cluster.
Partitions serve three foundational architectural purposes:
- Horizontal Scalability: Spreading a topic's partitions across multiple physical brokers distributes disk storage and network I/O across the entire cluster, allowing data volume to scale far beyond the hardware capacity of any single machine.
-
Strict Chronological Ordering: Kafka does not guarantee global ordering across an entire topic; it guarantees strict, total chronological ordering within a single partition. Records appended to a partition receive monotonically incrementing 64-bit offsets (
0, 1, 2, 3...) that remain permanent and unalterable. - The Fundamental Unit of Parallelism: Partitions dictate the concurrency ceiling for both write throughput from producers and read throughput across consumer groups. You cannot scale consumer parallelism beyond the total number of partitions in a topic.
2. How Records Land in Partitions: Producer Routing
When an application invokes producer.send(record), the Kafka client library determines the destination partition locally before transmitting any bytes across the network. The producer evaluates routing through a three-tiered logic flow:
producer.send(record)
│
├── Explicit Partition Specified?
│ ├── YES ──▶ Route to Target Partition (Bypasses All Partitioner Logic)
│ └── NO ──▶ Is Message Key Provided?
│ ├── YES (key != null) ──▶ abs(MurmurHash2(key)) % NumPartitions
│ └── NO (key == null) ──▶ Sticky Partitioner (Batch to Single Partition)
│
▼
RecordAccumulator Buffer (Partition-Specific Memory Pool)
Keyed Messages: Deterministic Ordering Guarantees
When a producer attaches a message key (such as userId, orderId, or deviceId), the default partitioner passes the serialized key bytes through MurmurHash2, computing abs(MurmurHash2(key)) % total_partitions. Because the algorithm is deterministic, records sharing the exact same key always land in the same partition.
This deterministic hashing enables event-driven systems to enforce strict per-entity causal ordering. For example, all lifecycle events for order_id = 98124 (e.g., OrderCreated, PaymentAuthorized, OrderShipped) land in the same partition in chronological sequence, guaranteeing that downstream consumers never process a shipment before payment authorization.
The Partition Expansion Trap: If you increase the partition count of an existing topic (e.g., from 4 to 8), the mathematical modulo formula (
hash % 8) immediately redistributes keys across the new partition layout. While historical records remain in their original partitions, subsequent events for that key route to the newly calculated partition, breaking chronological ordering during the transition.
Unkeyed Messages: The Sticky Partitioner
When records are produced without a key (key = null), modern Kafka clients invoke the Sticky Partitioner. Rather than cycling individual messages round-robin across partitions—which creates tiny, fragmented network packets—the sticky partitioner temporarily binds to a single partition until an entire memory buffer fills (governed by batch.size) or a timeout expires (governed by linger.ms).
Once the batch flushes over TCP to the broker, the partitioner rotates to another random partition. This strategy dramatically improves batch compression ratios, lowers client CPU usage, and slashes network overhead by consolidating thousands of individual events into dense, efficient socket requests.
Producer Buffer Volatility & Crash Recovery
Because Kafka producers use a dual-thread architecture where records stage in an in-memory RecordAccumulator buffer before a background Sender I/O thread flushes them, any abrupt process termination (such as an unhandled OutOfMemoryError, OS kernel panic, or SIGKILL) causes permanent data loss of all un-flushed records before they ever reach a broker.
This volatility is not a defect in Kafka, but an intentional distributed systems trade-off: batching writes in memory delivers hundreds of thousands of events per second, whereas synchronously blocking each send() call on broker disk commits would collapse throughput to a crawl. While graceful shutdowns (like Kubernetes SIGTERM signals) allow JVM shutdown hooks to invoke producer.close() and safely drain pending buffers, sudden crashes terminate the process instantly without running hooks.
To protect critical data against process death, production systems never rely solely on client-side memory; instead, they persist events to an upstream transactional database or Write-Ahead Log (WAL) via patterns like the Transactional Outbox before calling producer.send(), enabling reliable recovery upon restart.
3. Physical Storage Internals: How Partitions Live on Disk
Kafka partitions do not exist in a relational database or shared memory table. On a broker's physical filesystem, every partition is represented as a dedicated directory named <topic_name>-<partition_index> (for example, orders-0 or telemetry-3).
Log Segments and File Composition
Inside the partition directory, the commit log is split into manageable files called Log Segments. Segment rollover occurs when the active segment reaches a size threshold (configured by segment.bytes, defaulting to 1 GB) or an age threshold (segment.ms, defaulting to 7 days). Each segment consists of tightly coupled files sharing identical base names corresponding to the segment's base offset:
Log Retention Policies: Delete vs. Compact
Because partition logs are immutable, older segments are safely retired by background cleaner threads based on topic retention settings:
-
Delete Policy (
cleanup.policy = delete): Under the default policy, closed segments older thanretention.ms(e.g., 7 days) or exceedingretention.bytesare simply deleted from disk in their entirety. The active writing segment is never deleted, regardless of age. -
Compact Policy (
cleanup.policy = compact): Used for stateful changelogs and database CDC feeds, background cleaner threads scan closed segments and retain only the single most recent record for each distinct message key, pruning obsolete intermediate updates while preserving latest state indefinitely.
4. Replication, High Availability & ISR Mechanics
To survive hardware crashes without data loss, each partition is provisioned with a Replication Factor (typically 3 in production). Replicas of the same partition are distributed across separate physical brokers:
The Partition Leader is the authoritative broker serving incoming write requests from producers and coordinating read traffic for consumers. The remaining brokers act as Follower Replicas, continuously issuing FetchRequest calls to the leader to mirror uncommitted record batches onto their local disks.
The In-Sync Replicas (ISR) Set
The leader maintains a dynamic set of healthy nodes called the In-Sync Replicas (ISR) set. A follower remains in the ISR only if it continuously fetches records and keeps up with the leader within a configured lag window (replica.lag.time.max.ms, default 30 seconds). If a follower encounters network partitions or disk stalls exceeding this threshold, the leader evicts it from the ISR and notifies the KRaft controller.
Configuring Durability: Producer Acks & min.insync.replicas
Balancing message durability against throughput in Kafka begins with the producer's acks configuration, which dictates how many broker confirmations are required before a write is marked successful:
-
acks=0: Maximum throughput—fires records without waiting for broker responses (zero durability). -
acks=1: Waits only for the partition leader to write locally (vulnerable to data loss if the leader crashes before replication). -
acks=all(or-1): Mission-critical pipelines mandate this setting. The producer blocks until every active member of the ISR set successfully writes the record to disk, ensuring maximum durability at the cost of additional network round-trip latency.
5. Consumption Mechanics & Parallelism Boundaries
Partitions form the absolute physical boundary of consumer scalability. Inside a Kafka consumer group, partition assignment follows a strict architectural law: each partition can be assigned to at most ONE consumer instance within the same group at any given time.
The 1-to-1 Partition Allocation Law
If a topic has 4 partitions and your consumer group contains 2 consumer instances, each consumer processes events from 2 partitions. If you scale your consumer group to 4 instances, each consumer processes exactly 1 partition in parallel. However, if you add a 5th consumer instance, that extra consumer will sit completely idle, receiving zero partitions until an active consumer crashes or leaves the group.
Topic: telemetry-events (4 Partitions) Consumer Group: analytics-workers (5 Instances)
┌───────────────┐ ┌──────────────────────────────────┐
│ Partition 0 │ ══════════════════════════▶ │ Consumer 1 (Assigned P0) │
│ Partition 1 │ ══════════════════════════▶ │ Consumer 2 (Assigned P1) │
│ Partition 2 │ ══════════════════════════▶ │ Consumer 3 (Assigned P2) │
│ Partition 3 │ ══════════════════════════▶ │ Consumer 4 (Assigned P3) │
└───────────────┘ │ Consumer 5 (IDLE - Standby) │
└──────────────────────────────────┘
Consumer Lag: The Vital Health Metric
Because consumers maintain their own reading positions via committed offsets, they process data at their own pace without impacting producers. The difference between a partition's current High Watermark and a consumer group's last committed offset is known as Consumer Lag:
Consumer Lag = High Watermark (HW) - Committed Offset
A growing consumer lag indicates that downstream applications cannot keep up with incoming write velocity, serving as the primary alert signal for scaling consumer instances or optimizing slow downstream database writes.
6. Partition Sizing & Capacity Planning Best Practices
While partitions enable horizontal scaling, creating thousands of unnecessary partitions introduces severe operational penalties. Production capacity planning requires balancing write throughput targets against cluster overhead:
- Throughput Sizing Rule of Thumb: As a baseline, budget roughly 5 to 10 MB/sec of write throughput and 10 to 20 MB/sec of read throughput per partition on modern NVMe SSD storage. A topic requiring 100 MB/sec ingestion typically needs between 10 and 20 partitions.
- The Cost of Over-Partitioning: Each partition directory on disk maintains multiple open file descriptors (one for each active and closed segment index file). Running tens of thousands of partitions per broker exhausts Linux operating system file handles, inflates JVM heap requirements on KRaft controllers, and elongates broker restart times while indexes are verified.
-
Latency Trade-Offs: While more partitions increase consumer parallelism, they also distribute producer batch buffers across more queues, requiring higher message arrival rates to satisfy
batch.sizethresholds before records flush over the network.
Related Guides & Deep Dives
- Kafka Consumer Internals & Rebalance Protocols — Master the 1-to-1 partition allocation law, CooperativeSticky rebalances, and High Watermark consumption limits.
- Kafka Topics: Architecture & Design — Sizing partition capacity formulas, retention cleanup policies, and tombstone log compaction.
- Kafka Broker Architecture & Storage — Discover how brokers manage physical segment storage, sparse index binary searches, and KRaft consensus.
Top comments (0)