Kafka Fundamentals: Topics, Partitions, and Consumer Groups Explained the Hard Way

Most developers can wire up a Kafka producer and consumer, get a message flowing end to end,
and call it done. But there are much more into it, for example, why do we go with 6 partitions instead of
3, how partitions are created, what is relationship between consumers and topic partitions etc - they seem to have no real answer.

In this blog I will cover some of very fundamentatl elements of kafka, such as, what a partition actually is,
how a message ends up on one, and how consumer groups split the work. Then I will set up a kafka broker in Docker and
break it on purpose, which will explain something very useful. Along the way I also pick up a few patterns that keep showing up once you
start building real things on top of Kafka.

The Basic Terms, in Plain English

Before any commands, it helps to get the terminologies straight. Most blogs use them loosely, and that's usually where the
confusion starts.

Cluster: a bunch of Kafka servers working as one system. You talk to "the cluster", not to any one machine in it.

Broker: a single Kafka server. It stores partitions and handles reads/writes for whatever it owns. Real clusters
run at least 3 brokers so they can survive one going down; on your laptop you'll usually just run one.

Producer / Consumer: not separate infrastructure, just code in your app. A producer sends messages, a consumer
reads them.

Topic: a named stream, like orders. Here's the twist that takes a while to understand: a topic isn't one
queue. It's a set of partitions.

Partition: the concept that takes the longest to grasp, so it gets its own section below.

Consumer group: a name you give a bunch of consumers so they can split up the work of reading a topic. More on
this further down.

Offset: just a number that goes up by one for every message, marking its position inside a partition. Offsets
are per-partition, not shared across the whole topic. There's no per-message ack, no visibility timeout like AWS SQS -
each consumer just remembers how far it's read.

Why Partitions Matter

A lot of intro material treats a topic like a plain queue: stuff goes in one end, a consumer takes it out the other,
done. That picture is wrong enough to cause real problems later. A Kafka topic is actually a durable log split into
several independent pieces. Nothing gets deleted when it's read, several consumers can each replay the whole thing at
their own pace, and how it's split up is something you decide and have to live with. That splitting is the partition,
and it's where most of the early confusion comes from - so that's next.

Partitions Aren't Copies

A very common mistake is, assuming partitions as copies of the same stream, like read replicas of a database. They
aren't. That's what replicas are - copies of one partition spread across brokers, purely for fault tolerance,
which is a different topic I'll get to another time.

Partitions are shards. Each one holds a different slice of the topic, with no overlap. It's the same idea as
sharding a database table:

Topic: "orders" with 3 partitions

Partition 0:  [order-101][order-104][order-109] ...   ← different events
Partition 1:  [order-102][order-105][order-107] ...   ← different events
Partition 2:  [order-103][order-106][order-108] ...   ← different events

order-101 only ever lives in one partition. Put all the partitions together and you get the full topic; on their
own, none of them looks like any of the others. Two things fall out of this:

• Order is only guaranteed inside a single partition. Messages in partition 0 stay in order relative to each other. A message in partition 0 and one in partition 1 have no ordering relationship at all - Kafka makes no promises there.
• Different consumers can read different partitions at the same time. This is really the whole reason Kafka can scale the way it does.

How Kafka Decides Which Partition a Message Goes To

This comes down to the partition key, something you choose when you send a message. Kafka hashes it and always
maps that hash to the same partition. Same key in, same partition out, every time - as long as the partition count
doesn't change.

Say you've got a customer-orders topic. A few ways to key it:

• By customer_id: every event for one customer lands on the same partition, so ordering within a customer (say CREATED before SHIPPED) is guaranteed. Different customers still spread out across partitions, so you keep the parallelism.
• By order_id: good when a single order's events need to stay in order, but you don't care how one customer's different orders relate to each other.
• No key at all: Kafka spreads messages around over time for max throughput, with no ordering guarantee (more on the batching behaviour behind this a bit further down). Fine for something like clickstream events where order genuinely doesn't matter.

The question to actually ask yourself: which entity's events need to stay in order relative to each other? Whatever
that is, that's your key.

One thing worth knowing upfront: if you key by customer_id and one customer is way more active than the rest, their
partition turns into a hot spot - it's pinned to one broker and one consumer thread, so it takes all the load while
everyone else's partition sits comfortably. Real production headache, and the only way around it is thinking about key
design before you hit it.

How Many Partitions Should You Actually Use

This is a separate call from the key, and mixing the two up is a common next mistake. The key decides where a given
message goes. The count decides how many shards the topic has, full stop - it's about capacity and parallelism,
not about your business domain.

The rule that actually matters: only one consumer in a group can read a given partition at a time. So the
partition count is a hard cap on how much you can parallelise. A topic with 4 partitions will never be processed by
more than 4 consumers in the same group - deploy a fifth and it just sits there doing nothing.

A rough way to size it:

target_throughput = 100 MB/sec
single_partition_throughput ≈ 10 MB/sec (hardware/network dependent)

partitions needed ≈ 100 / 10 = 10 (minimum, then round up for headroom)

A few other things feed into the number: how slow your per-message processing is (slower downstream calls need more
partitions to keep up), how much you expect to grow (bumping the partition count later changes the key-to-partition
mapping and breaks ordering guarantees you already had, so most people just over-provision early), and broker
overhead (more partitions means more file handles and replication traffic per broker, and there's a real ceiling).

There's no "business" answer to partition count, and that's the thing to unlearn. The domain logic belongs in the
key. The count is just infra sizing, decided once, and annoying to change later.

Consumer Groups: How the Work Actually Gets Split Up

A consumer group is just a name you give a set of consumers that want to share the work of reading a topic. Any
consumer that starts up with the same group ID joins that group, and Kafka handles splitting the topic's partitions
between them.

This is where the partition-count rule from above actually bites: Kafka gives each partition to exactly one consumer
in the group at a time. More consumers than partitions and some sit idle. More partitions than consumers and some
consumers end up handling several.

Every time a consumer joins or drops out, Kafka triggers a rebalance - it works out a fresh split of the partitions
across whoever's left and hands out ownership again. That's the whole trick behind scaling up: start another instance
of your consumer with the same group ID, and Kafka does the rest. No code change needed.

There's a less obvious perk to grouping this way too: broadcasting. Two different consumer groups reading the same
topic each get their own full copy of every message. Say billing-service is one group and analytics-service is
another - both see everything, both track their own offsets, and neither one affects the other. Inside a group, Kafka
acts like a queue: one message, one consumer. Across groups, it acts like pub/sub: every group gets everything. Same
underlying log, both patterns at once.

Trying It on a Local Broker

Got a Kafka container running, with the scripts under /opt/kafka/bin added to PATH so I didn't have to type the
full path every time:

export PATH=$PATH:/opt/kafka/bin

Created two topics so I could compare them side by side:

kafka-topics.sh --create --topic orders-1p \
  --bootstrap-server localhost:9092 \
  --partitions 1 --replication-factor 1

kafka-topics.sh --create --topic orders-12p \
  --bootstrap-server localhost:9092 \
  --partitions 12 --replication-factor 1

--describe on orders-12p shows all 12 partitions, all led by broker 1 (single-broker setup, so nothing interesting
happening on the replication side yet):

Topic: orders-12p   PartitionCount: 12   ReplicationFactor: 1
    Partition: 0   Leader: 1   Replicas: 1   Isr: 1
    Partition: 1   Leader: 1   Replicas: 1   Isr: 1
    ...

Watching the Consumer Limit Play Out

Started three console consumers, all in the same group, pointed at the 1-partition topic:

kafka-console-consumer.sh --topic orders-1p \
  --bootstrap-server localhost:9092 \
  --group test-group-1p \
  --formatter-property print.partition=true

Then sent a few messages. Only one of the three consumers got anything. The other two just sat there, connected
and doing nothing - which is exactly the point. Checking the group directly backs it up:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group test-group-1p

GROUP           TOPIC       PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID
test-group-1p   orders-1p   0          22              22              0   console-consumer-...

One row. One partition, one consumer doing anything. The other two are members of the group, but they own nothing.

Ran the same test against the 12-partition topic, giving all three consumers a few seconds to settle before
producing anything, and this time the split was clean:

console-consumer-A → partitions 0, 1, 2, 3
console-consumer-B → partitions 4, 5, 6, 7
console-consumer-C → partitions 8, 9, 10, 11

Exactly what you'd expect: 12 partitions, 3 consumers, 4 each.

Why 10 Unkeyed Messages All Landed in One Partition

This one threw me for a minute. With all three consumers sitting evenly assigned, I sent 10 messages with no key and
expected them to spread out. Instead, all 10 landed in the same partition, and only one consumer terminal lit up.

My first thought was that the rebalance was broken. It wasn't - --describe still showed a perfectly even split. What
was actually happening: when you produce without a key, Kafka doesn't round-robin each message individually. The
producer uses what's called a sticky partitioner - it picks one partition and sticks with it for a whole batch (until
the batch fills up or linger.ms runs out), then switches to a different one for the next batch. I'd typed those 10
lines quickly into the console producer, so they got batched together and shipped as one unit to a single partition.

That's not a quirk, it's a real thing to know about before you hit it in production. Sticky partitioning trades
perfectly even spread for fewer, bigger batches and better throughput - it evens out over many batches, not
necessarily over a handful of messages typed by hand. Key your messages explicitly, though, and you get the same
result every time, no batching surprises involved.

Patterns That Show Up Once You Build on Top of This

Once partitions, keys, and consumer groups make sense, most of what you see in real Kafka systems turns out to be a
small handful of patterns built on top of them:

• Dead letter queue (DLQ). A message that keeps failing after a few retries goes to a separate <topic>-dlt topic instead of blocking the partition or getting silently dropped. The main consumer keeps moving, and the failed messages sit somewhere you can actually go look at them. I walk through building one in Building Kafka Producer-Consumer Using Go and Docker.
• Retry with backoff. The simple version: retry a failed message a few times with a short delay before giving up and sending it to the DLQ (also in the post above). At bigger scale, that retry often moves to its own topic (<topic>-retry-30s, <topic>-retry-5m) so a slow downstream doesn't hold up the main partition while messages wait around.
• Idempotent consumer. Kafka's default guarantee is at-least-once, so every consumer has to assume a message might show up twice - say, after a rebalance like the one above, or a commit that got retried. The fix lives on the consumer side, not the broker: keep track of a message's unique key (or a hash of its contents) somewhere, and skip anything you've already handled.
• Transactional outbox. The problem: a service can't save to its own database and publish a Kafka event as one all-or-nothing step. One can succeed while the other fails, and now the database and Kafka disagree - either the order got saved but nobody heard about it, or Kafka says the order exists when it was never actually saved. The fix: don't publish to Kafka directly. Save the event as a row in an outbox table instead, in the same database transaction as the real write. That part is now truly atomic - both rows are saved, or neither is. Getting the event into Kafka happens afterward, as a separate step:
  • Polling - a background job checks the outbox table now and then, publishes anything new, and marks it done.
  • CDC via Kafka Connect - watches the database's own internal change log for new rows and streams them to Kafka automatically, no polling needed.

On cleanup: don't delete a row the moment it's published - just mark it published, and let a separate scheduled
job purge old published rows later (say, once a day). That way a crash right after publishing doesn't lose the
record before you're sure Kafka got it, and the table doesn't grow forever either.

The database write stays the single source of truth and stays atomic; publishing becomes a "happens shortly after"
step instead of a "must happen right now" one - the database and Kafka are eventually consistent, not instantly
consistent - and the event can't be lost since it sits safely in the outbox table until it's confirmed published.
The cost: it's at-least-once, not exactly-once, so consumers still need to be
idempotent (as above), there's a small delay before events show up, and you're now running an extra table plus a
poller or CDC pipeline.

• Fan-out through separate consumer groups. Already covered above - every group reading a topic gets its own full copy of the stream. It's what lets billing-service and analytics-service both read orders independently, without either one's speed or downtime touching the other.

None of these are switches you flip in Kafka itself - they're just conventions you build into your producers and
consumers on top of the same partitions, keys, and groups covered above.

What I Actually Walked Away With

The mental model that finally stuck after all this:

Concept	What it actually is
Partition	A shard - a different, non-overlapping slice of the topic's events
Partition key	Decides which partition a given message lands on
Partition count	A capacity and parallelism decision, fixed at topic creation, expensive to change later
Consumer group	A set of consumers sharing the work of a topic; one partition → one active consumer at a time within a group
Offset	Per-partition read position, tracked separately by each consumer group
At-least-once delivery	The default guarantee - consumers need to be idempotent, not assume single delivery

Next up, when I get to it: multi-broker replication and leader election, and how retention and compaction actually
work.

Further reading

• Building Kafka Producer-Consumer Using Go and Docker - a hands-on Go implementation of the DLQ and retry patterns described above
• Spring Boot Kafka Producer-Consumer with Docker - the same producer-consumer shape built with Spring Boot
• Getting Started with Docker and Go Lang - containerizing the services these patterns run in