A traditional message queue deletes a message once it has been delivered and acknowledged. It sounds like tidy housekeeping. In practice it causes three problems you have probably met.
You cannot add a second independent reader. Marketing wants every order event too, but the fulfilment workers already share each message between them. A second view needs a fan-out topology, publishing twice, or another copy of the data somewhere.
You cannot replay. A bug in the fulfilment consumer mishandled nine hours of orders. The messages are gone, so the fix is a one-off backfill script against the database, rebuilding events that used to exist.
The broker tracks every message's fate: delivered, acknowledged, redelivered, dead-lettered. That bookkeeping is a large part of what makes queues hard to scale.
What a log changes
A Kafka topic is an append-only log kept according to a retention policy, not consumed away. Reading does not delete. A consumer group records only how far it has read in each partition, its offset. Adding a reader means adding a group with its own offsets. Replaying nine hours means moving a group's offsets back nine hours.
The price of storing one number
Within a traditional consumer group, a partition's progress is a single integer. That is what makes fan-out and replay cheap, and it is also why the group cannot say "this one record failed, deliver just that one again" while moving past it. One stuck record blocks its partition, which is why Kafka designs end up with a dead-letter topic.
| Queue | Kafka consumer group | Kafka share group | |
|---|---|---|---|
| Second independent reader | needs fan-out or double publishing | yes, a new group | yes |
| Replay nine hours | no, the messages are gone | yes, reset the offsets | yes, from the retained log |
| Acknowledge one record | yes | no, progress is one number per partition | yes |
| Ordering | lost with competing consumers | per partition | weakened |
| Parallelism ceiling | number of consumers | number of partitions | beyond the partition count |
Share groups, available from Kafka 4.2, bring back per-record acknowledgement, and with it per-record bookkeeping and weaker ordering. They are a different trade, not a newer version of the same thing.
Think Like an Engineer
A team wants both fan-out and per-record acknowledgement on one topic. They can have it with two groups of different kinds on the same log: a consumer group for the reader that needs per-key order and replay, and a share group for the worker that needs to skip failures. What no single group can offer is both, because skipping a record and keeping that key's order are the same decision seen from two sides.
The useful question about a topic is which of those columns its readers need. A topic carrying changes to an order's state needs per-partition ordering. A topic of independent work items may not.
Originally published at engineertoai.com. The full lesson is free, with no sign-up: Why a Log, and Not a Queue.
Top comments (1)
The comparison table is the real takeaway here. The per-record acknowledgement that share groups bring back is powerful, but it is worth noting that it reintroduces the per-message bookkeeping that queues struggle with, so the ordering and parallelism tradeoffs deserve a second look before adopting it.