Every few months an engineering team reaches the same fork. Two services need to talk without being wired directly together, and someone asks whether to reach for Kafka or RabbitMQ. The answers you find online are almost always a feature checklist, and a checklist is the wrong tool for this decision. It lines up two systems side by side and compares their columns as if they were built to do the same job, when the whole point is that they were not. One is a broker built to hand individual pieces of work to individual workers with a great deal of control over each one. The other is a durable log built to move enormous streams of events and let many independent readers replay them. Comparing their feature tables tells you almost nothing about which one your problem wants.
The question is never which broker is better. It is what shape your problem has, and once you can see the shape the choice makes itself.
This piece works the problem the way a staff engineer would work it in a design review. We start with the three delivery shapes that every broker is an implementation of, then the constraints that actually decide the call, then the arithmetic, because the numbers push you toward one answer well before any feature does. After that we go under the hood of both systems, down to exchanges and bindings, partitions and offsets, consumer groups and rebalances, replication and in-sync replicas, and what exactly-once really guarantees and where it stops. We finish with the real tradeoff, one failure mode worth rehearsing, a decision matrix, and the handful of patterns that transfer to any messaging problem you will ever design. One note on numbers first. Every throughput, latency, or storage figure here is an industry-typical estimate meant to drive the reasoning, not a measured metric from any specific deployment.
Three shapes, not two
Put the two product names aside for a minute. There are really three patterns for moving messages between parts of a system, and every broker on the market implements one or two of them.
A queue hands each job to one worker, pub/sub copies each event to every subscriber, and a log keeps every message for any reader to replay.
The first shape is a message queue, which is a work list. A producer puts a job on it, and exactly one worker takes that job and does it. Transcode this video, send this email, charge this card. Add more workers and they share the load, each still doing any given job once. This is a task queue, and RabbitMQ and Amazon SQS are the classic examples. The second shape is a pub/sub topic, which is a broadcast. A producer publishes an event once, and every subscriber receives its own independent copy. An order was placed, so the receipt service, the inventory service, and the analytics service each react on their own. The third shape is an event stream, which is a durable append-only log. Producers append to it, the log retains messages for a configured window of hours or days or forever, and consumers read from any position and can rewind and replay. This is Kafka, and also Pulsar, Redpanda, and Kinesis. The log is the source of truth and consumers are views over it.
RabbitMQ is an excellent message queue that can also do pub/sub through its routing. Kafka is an event stream that can imitate the other two shapes. They overlap in the middle, which is exactly why the comparison feels muddy, but their centers of gravity sit in different places, and everything below is a consequence of where those centers are. One honest footnote before we go deeper. Recent RabbitMQ versions added Streams, a persistent replayable log addressed by offset, so RabbitMQ can now reach into the third shape too. It does not move its center of gravity and it is not why most teams pick it, but the clean split is a little less clean than it used to be.
The decision only matters at the edges
For a large fraction of systems, either broker would be fine, and arguing about it is a waste of a design review. The choice starts to matter at the edges, where one of a few specific constraints becomes load-bearing. Name those constraints explicitly before you draw any boxes, because they are what actually select the tool.
The functional shape of the work and a few non-functional constraints, not a feature table, are what decide the broker.
Ask first what the unit of work is. If it is a discrete task that one worker should perform once and then forget, with retries and per-item routing, that is a queue's job. If it is an event that several independent consumers each need to see, and that a new consumer might need to replay from history to rebuild its own state, that is a log's job. Then ask about the non-functional constraints that break ties. Throughput ceiling, because a workload that peaks in the hundreds of thousands or millions of messages per second lives in Kafka territory and will fight a per-message broker. Retention and replay, because if you need to reprocess last week's events after a bug fix, you need a system that still has them. Ordering scope, because a system that must process events for one entity in strict order has different needs than one that does not care. Per-message control, because a workload that needs priorities, per-message expiry, and precise redelivery wants RabbitMQ's bookkeeping, and a workload that just wants raw flow does not. Get those four answers and the architecture reads as a series of responses to them rather than a pile of components.
Do the arithmetic first
Before choosing anything, put real numbers on the workload, because scale is the constraint that most often makes the decision for you. Take a concrete platform. Two very different flows run through it. A command flow processes orders, where each message is a job that a worker executes with retries and a dead-letter path for poison messages, peaking around 20,000 jobs per second. An event flow carries user activity and device telemetry that many teams consume, peaking around 1,000,000 events per second at roughly 1 KB each.
A task flow in the tens of thousands per second and an event flow near a million per second are two different problems, and the arithmetic assigns each to a different broker.
Those two rates are not a rounding error apart. They are almost two orders of magnitude apart, and that gap is the whole point. The command flow at 20,000 per second with per-message acknowledgement, priorities, and a dead-letter exchange is squarely a RabbitMQ workload, and 20,000 messages per second is comfortable for a well-tuned RabbitMQ cluster. The event flow is a different animal. One million events per second at 1 KB each is about 1 GB per second of ingest. Retain it for seven days and the raw log is roughly 605 TB, and at a replication factor of three that is close to 1.8 PB on disk. If a single consumer processes around 10,000 events per second, you need at least 100 consumers reading in parallel, which as we will see means at least 100 partitions. No amount of tuning makes a per-message broker comfortable at that volume, because the per-message overhead that gives RabbitMQ its control is exactly what caps its throughput. The arithmetic has already split the platform in two, and neither broker is wrong. They are answers to different lines of the sizing table.
How RabbitMQ actually routes
To judge RabbitMQ fairly you have to see that it is not really a queue. It is a routing engine with queues at the edges. Producers never publish to a queue directly. They publish to an exchange, and the exchange uses bindings to decide which queues receive a copy of the message.
A producer publishes to an exchange, bindings decide which queues get a copy, and consumers read from queues, so the producer knows nothing about who consumes.
There are four exchange types, and they are the source of RabbitMQ's flexibility. A direct exchange routes a message to the queue whose binding key exactly matches the message's routing key. A topic exchange matches on patterns, so a binding of order.eu.* catches order.eu.created but not order.us.created. A fanout exchange ignores keys entirely and copies to every bound queue, which is how you get pub/sub. A headers exchange matches on message header attributes instead of a routing key. This indirection is the entire design. The producer publishes one message with a routing key and knows nothing about who consumes it, and an operator can add a queue, bind it with a pattern, and start receiving a filtered slice of traffic without touching the producer.
Layer onto that the per-message features that make RabbitMQ so good at task work. Per-message acknowledgement, so a message is redelivered if a worker crashes before confirming it. A prefetch limit that controls how many unacknowledged messages a consumer may hold at once. Message priorities, per-message time-to-live, and dead-letter exchanges that catch messages which fail too many times so they can be inspected rather than lost. For durability across broker failures, modern RabbitMQ uses quorum queues, which replicate each queue across nodes with a Raft consensus group and have largely replaced the older mirrored-queue approach. All of that bookkeeping is exactly what task workloads want, and it is exactly the overhead that caps how fast RabbitMQ can go. Hold that thought, because it is the crux of the comparison.
How Kafka actually stores data
Kafka's model looks almost too plain next to RabbitMQ's routing, and that plainness is the source of its scale. A topic is split into partitions. Each partition is an ordered, append-only log of messages, and each message carries a monotonically increasing offset that is simply its position in that log.
A topic is partitions, each partition is an ordered append-only log addressed by offset, and reading never deletes, so every consumer group tracks its own position independently.
Walk through what that means because it is the core of everything else. Producers append to partitions, and which partition a message lands in is decided by the partitioner, which by default hashes the message key and takes the result modulo the partition count. A consumer group reads the topic, and Kafka assigns each partition to exactly one consumer in that group. The decision that changes everything is that reading does not delete. Messages stay until a retention policy, by age or by size, ages them out, or until log compaction keeps only the latest message per key. Each consumer group independently tracks its own committed offset, the position it has processed up to, stored in an internal topic called __consumer_offsets. Under the hood the log is a set of segment files written sequentially, and Kafka leans on the operating system page cache and a zero-copy send path to move bytes from disk to network without copying them through the application, which is a large part of why it is fast.
That one decision, keep the data and let each reader track its own position, gives Kafka three things a queue cannot do. Many consumer groups read the same log without affecting one another. A group can rewind to an earlier offset and replay. And a brand-new consumer can start from the beginning and rebuild its entire state from history. None of that is possible when a message vanishes the moment it is acknowledged. Kafka also dropped its old ZooKeeper dependency in favor of KRaft, a built-in Raft quorum that manages cluster metadata, which simplifies operations and speeds up how fast the cluster reacts to broker changes.
Consumer groups and the rebalance
The consumer group is how Kafka scales reads, and it comes with one hard constraint and one sharp edge. The constraint is that a partition is owned by exactly one consumer in a group at a time, so your maximum parallelism in a group equals your partition count. Twelve partitions can feed at most twelve consumers, and a thirteenth sits idle. Partition count is therefore a capacity decision you make deliberately up front, not a knob you discover under load.
Each partition is owned by exactly one consumer in a group, so parallelism is capped at the partition count and a rebalance reshuffles ownership when membership changes.
The sharp edge is the rebalance. When a consumer joins or leaves the group, or is presumed dead because it stopped calling poll within the configured interval, Kafka reassigns partition ownership across the surviving members. Older versions did this with a stop-the-world protocol that paused the entire group on every rebalance, which meant a single slow consumer could stall the lot. The modern cooperative-sticky assignor is far gentler, moving only the partitions that must move and letting the rest keep consuming through the change. In production, frequent rebalances are one of the most common causes of consumer lag, and they are usually triggered by long garbage-collection pauses or by processing that takes longer than the poll interval, so the consumer looks dead when it is only busy. This is why partition count and per-message processing time are decisions you size for peak deliberately, because getting them wrong shows up as lag and repeated rebalances rather than a clean error.
Ordering, guaranteed inside a partition and never across
Kafka does not give you global ordering. It gives you total ordering within a single partition and no ordering at all across partitions. This is the fact that trips up nearly everyone the first time they debug an out-of-order event that the design supposedly made impossible.
Messages with the same key hash to the same partition and are read in order, but two keys on different partitions have no ordering relationship at all.
It follows directly from the partitioner. Because the same key always hashes to the same partition, every message for a given key lands on the same ordered log and is consumed in the order it was produced. Key your messages by the entity whose order actually matters, the user id, the order id, the account number, and you get per-entity ordering for free. What you do not get is any guarantee that a message with key A was seen before a message with key C, because they live on different partitions consumed independently and in parallel. If you genuinely need one global order across everything, you need a single partition, which means a single consumer and no horizontal scaling, and that trade between ordering scope and throughput is one of the most important decisions in any streaming design.
Two caveats keep this honest. Ordering within a partition holds because the idempotent producer is enabled by default. Without it, a producer allowed more than one in-flight request per connection can have a retried batch land out of order on the same partition. And because the partition is chosen by the key hash modulo the partition count, adding partitions later remaps existing keys to different partitions, so per-key ordering is only stable while the partition count is fixed. Size partitions for peak up front, because growing them later quietly breaks the ordering you were relying on.
Delivery semantics, the trade between losing and duplicating
Make it reliable is not a single setting. It is a choice about what happens when a process crashes mid-flight, and that choice comes down to one thing, whether you record your position before or after you do the work.
Commit before processing and you can lose a message, commit after and you can duplicate one, and the safe default is at-least-once with idempotent handlers.
At-most-once commits the offset first, then processes. It is the fastest and never produces duplicates, but if the process dies after committing and before finishing, that message is gone. That is fine for metrics or logs where a small gap does not matter. At-least-once processes first, then commits. It never loses a message, but if the process dies after doing the work and before committing, the message is redelivered and processed again on restart. This is the default in most setups and it is safe as long as your handler is idempotent, meaning processing the same message twice produces the same result as processing it once. In practice that usually means deduplicating on a business key or making the write a conditional upsert. Exactly-once is the one everyone wants and most people misunderstand, and it deserves its own section because the boundary where it holds is as important as the guarantee itself.
RabbitMQ frames the same trade differently but faces the same physics. A message is delivered, the consumer does the work, and the consumer acknowledges. If the consumer crashes before acknowledging, RabbitMQ requeues the message and another consumer gets it, which is at-least-once with the same demand for idempotent handlers. There is no way around the fundamental choice, only different words for it.
What exactly-once really means
Kafka reaches exactly-once processing by stacking two mechanisms, and understanding where the guarantee stops is as important as understanding that it exists.
The idempotent producer removes retry duplicates and transactions make the writes and the offset commit atomic, but only inside the Kafka boundary.
The first mechanism is the idempotent producer. Each producer is assigned a producer id, and every message carries a per-partition sequence number. If a network hiccup makes the producer retry a send, the broker recognizes a sequence number it has already written and silently drops the duplicate. This removes the duplicates created by retries, it is enabled by default in modern Kafka, and it costs essentially nothing. The second mechanism is transactions. A producer with a transactional id can write to several partitions and commit its consumer offsets as a single atomic unit, so either every write and the offset advance together or none of them do. Consumers set their isolation level to read-committed so they never see the messages from a transaction that later aborted. Put together, a consume-transform-produce loop inside Kafka can process each input exactly once, with no duplicates and no losses, even across a crash.
The limit is the part people miss. Exactly-once holds inside the Kafka boundary, for Kafka-to-Kafka pipelines using Kafka's own transactions and offset management. The moment your consumer writes to an external system, a relational database, a payment provider, a third-party API, that external write is not part of the Kafka transaction. To keep the end-to-end guarantee you have to make the external write idempotent yourself, usually with a deduplication key, or coordinate a two-phase commit that most teams should avoid. So the honest senior statement is that Kafka gives you exactly-once within its own boundary, and everywhere the data leaves that boundary you are back to at-least-once plus idempotency. Anyone who tells you exactly-once is a checkbox has not shipped it.
Durability, replication and in-sync replicas
A single broker holding your only copy of the log is a data-loss incident waiting for a disk to fail. Durability in Kafka comes from replication, and the settings that control it are where a lot of quiet data loss hides.
Each partition has a leader and followers, the in-sync replicas are those caught up to the leader, and acks with min.insync.replicas set the durability floor.
Each partition has one leader and a configured number of follower replicas. Producers and consumers talk to the leader, and followers pull from it to stay current. The replicas that are fully caught up form the in-sync replica set, the ISR. Two settings decide how safe a write is. The producer's acks setting says how many replicas must confirm a write before it is considered done. With acks set to all, the leader waits for every in-sync replica to have the message before acknowledging. On its own that is not enough, because if the ISR has shrunk to just the leader, acks all still only means the leader has it. The second setting, min.insync.replicas, sets the floor. With a replication factor of three and min.insync.replicas of two, a write is only accepted when at least two replicas have it, so you can lose one broker with no data loss and no accepted write ever living on a single machine.
The tradeoff is latency and availability against durability. Waiting for more replicas is slower, and if too many replicas fall out of the ISR, writes are rejected rather than risked, which trades availability for safety. The common production baseline is a replication factor of three, min.insync.replicas of two, and acks all, which tolerates one broker failure cleanly. RabbitMQ quorum queues make the analogous choice with a Raft majority, where a write is committed once a majority of the queue's replicas have it. In both systems the shape of the answer is the same, keep more than one copy and refuse to acknowledge a write that only one machine has seen.
Push versus pull, and backpressure
A difference that matters more in practice than it looks on paper is how messages get from broker to consumer. RabbitMQ pushes, Kafka pulls, and that single distinction shapes how each system behaves when a consumer falls behind.
RabbitMQ pushes messages to consumers up to a prefetch limit, while Kafka consumers pull batches at their own pace, which is natural backpressure.
RabbitMQ pushes messages to consumers as they arrive, up to the prefetch limit, the number of unacknowledged messages a consumer is allowed to hold. Prefetch is the pressure valve. Set it too high and a slow consumer gets buried under messages it cannot process, hurting fairness and memory. Set it too low and consumers idle between acknowledgements, hurting throughput. When the broker itself is under memory pressure it raises a flow-control alarm and pushes back on producers over the TCP connection. Tuning prefetch per workload is a real and ongoing operational task. Kafka inverts this. Consumers pull, calling poll to fetch a batch of records whenever they are ready for more. A slow consumer simply polls less often, its offset lags behind the log end, and the broker does nothing special because the data is sitting on disk regardless. Backpressure is a natural consequence of the pull model rather than a mechanism you configure. Lag becomes an observable number, the distance between the log-end offset and the consumer's committed offset, which is the single most useful health metric a Kafka operator watches. The push model gives RabbitMQ low latency for individual messages and fine control over fairness, and the pull model gives Kafka simple, robust behavior under load at the cost of a little latency from batching.
The real tradeoff, throughput versus per-message control
Strip away the mechanics and the whole comparison reduces to a single axis. RabbitMQ spends effort per message to give you control over each one, and Kafka refuses to spend that effort so it can move a flood.
RabbitMQ trades throughput for fine per-message control, Kafka trades per-message control for raw throughput and replay, and your workload picks a corner.
Everything RabbitMQ does well, per-message acknowledgement, priorities, per-message expiry, precise redelivery and dead-lettering, arbitrary routing through exchanges, requires the broker to track the state of individual messages. That bookkeeping is genuine value for task workloads, and it is genuine cost at scale, because tracking millions of individual message states per second is expensive. Kafka does almost none of it. A partition is an offset in a file, a consumer group is a handful of committed offsets, and the broker mostly appends bytes and serves ranges of bytes. That is why Kafka moves gigabytes per second per cluster while a single RabbitMQ queue tops out far lower, and it is also why Kafka cannot easily give you a per-message priority or expire one message in the middle of a log. The systems did not make different quality choices. They made opposite bets about where to spend effort. Put your workload on the axis. Discrete tasks with rich control and modest volume sit at RabbitMQ's corner. High-volume event streams that need replay and fan-out sit at Kafka's. Most real platforms have both kinds of work and are best served by running both brokers for the jobs each is built for, rather than forcing one tool to imitate the other.
A failure mode worth rehearsing, the poison message
A design is only as good as its behavior on a bad day, so rehearse one concrete failure. A poison message is a message that a consumer cannot process, because it is malformed, references data that no longer exists, or trips a bug in the handler. On its own it is minor. The danger is how each system amplifies it if you have not planned for it.
A poison message on a Kafka partition blocks every message behind it, so route repeated failures to a dead-letter topic instead of retrying forever.
In Kafka the poison message sits at a specific offset on a partition, and the consumer will not advance its committed offset past a message it cannot process. If the handler retries forever, the consumer is stuck on that offset, the partition stops making progress, and lag on that partition grows without bound while every message behind the poison one waits. Because ordering is per partition, you cannot simply skip ahead and come back later without giving up the ordering guarantee. The fix is a deliberate policy. Try the message a bounded number of times, and once it exceeds the limit, produce it to a dead-letter topic along with the error and the original offset, commit past it, and keep the partition moving. The poison message is then quarantined for a human or a repair job to inspect, and the live stream is healthy. RabbitMQ handles the same situation with its dead-letter exchange, where a message that is rejected too many times or exceeds its time-to-live is routed to a separate queue automatically. The lesson is identical in both. Never retry a failing message without a bound, and always have a place to put the ones that will never succeed, because an unbounded retry turns one bad message into a stalled pipeline.
The senior answer, choose by shape
Now the decision resolves cleanly, because the shape of the work selects the tool. Line the common cases up against what each system is built for and the arguments stop.
Match the shape of the work to the broker built for it, and the choice stops being a debate about feature tables.
If the unit of work is a discrete task that one worker should do once, with retries, priorities, and a dead-letter path, that is RabbitMQ. Background jobs, order processing, email and notification sending, request-response over messaging, anything where you care about individual messages and volume is in the thousands or tens of thousands per second. If the unit of work is an event that many independent consumers need, that you might replay from history, or that arrives in the hundreds of thousands or millions per second, that is Kafka. Activity streams, telemetry and metrics pipelines, change-data-capture, event sourcing, and anything feeding a data lake or a stream processor. When you need strict per-entity ordering at scale, Kafka gives it to you per key through partitioning. When you need rich routing and per-message control, RabbitMQ gives it to you through exchanges and acknowledgements. And the honest answer for most large platforms is both, one broker for commands and tasks and one for events, each doing the job it was designed for, rather than a monoculture that fights its own workload half the time.
The patterns that transfer
Strip away the two product names and what remains is a set of decisions that show up in every messaging system you will ever design.
These four ideas outlast Kafka and RabbitMQ and apply to any system that moves messages between services.
First, identify the shape before the tool. Whether the work is a task, a broadcast, or a replayable stream determines the answer far more than any feature comparison, so name the shape first and let it choose. Second, do the arithmetic before the architecture, because a workload's throughput and retention needs frequently make the decision on their own, and the numbers are cheaper to compute than a migration is to reverse. Third, reliability is a choice about crash behavior, not a switch, so decide deliberately whether you can tolerate loss or duplication, default to at-least-once with idempotent handlers, and know exactly where any exactly-once guarantee stops. Fourth, always bound retries and always have a dead-letter path, because the difference between a resilient pipeline and a stalled one is whether a message that can never succeed has somewhere to go. Learn to see which of these levers your own problem sits on, and the Kafka versus RabbitMQ question stops being a debate you memorize the answer to and becomes a method you can apply to any broker, including the ones that do not exist yet.
I teach system design this way, from first principles with real diagrams and the trade-offs that only surface in production, as an interactive course at systemdesign.academy. The foundation lessons are free and need no signup.
Read the free lessons: https://systemdesign.academy
Top comments (0)