Kafka vs. RabbitMQ vs. SQS: The Message Queue That Started at 400,000 Messages a Second

Kafka vs. RabbitMQ vs. SQS: The Message Queue That Started at 400,000 Messages a Second

Kafka exists because RabbitMQ couldn't keep up. LinkedIn's engineers, back when the company was building the activity feed and metrics pipeline behind its own platform, found their existing message broker setup topping out two orders of magnitude below what they actually needed, and built Kafka specifically to publish more than 400,000 messages a second. Over a decade later, the gap is still real and still measured: a 2024 Confluent benchmark clocked Kafka sustaining roughly 605,000 messages per second on a three-broker cluster, while a widely cited 2026 comparison puts classic RabbitMQ queues around 40,000 to 50,000 messages per second per node, with RabbitMQ's newer Streams feature closing much of that gap on tuned clusters.

None of that makes RabbitMQ a worse technology, it makes it a different one, solving a different problem than raw throughput. And neither answers the third option most backend teams actually reach for first: Amazon SQS, a fully managed queue that trades control for zero operational overhead. This guide compares all three honestly, what each one actually optimizes for, where these real numbers come from, and how to decide before you've built a message queue architecture around the wrong one.

If messaging and distributed systems fundamentals are still new ground, our System Design course covers what this comparison builds on.

This post originally appeared on the Ciphemic Academia blog.

The Short Version

• Kafka is a distributed, append-only log built for extremely high throughput and replayable event streams, at the cost of real operational complexity.
• RabbitMQ is a traditional message broker built for flexible routing and reliable per-message delivery, simpler to run at smaller scale, with lower raw throughput in its classic queue mode.
• SQS is AWS's fully managed queue service, built to remove operational overhead entirely, at the cost of some control and features the other two offer natively.

If you want one default: start with SQS if you're on AWS and want to avoid running infrastructure. Reach for RabbitMQ when you need flexible routing and don't yet have Kafka-scale throughput needs. Reach for Kafka specifically once you have a measured, real need for high-throughput event streaming or replay, the way LinkedIn did.

What All Three Are Actually Trying to Solve

Before the differences, the shared job, since this is what any message queue exists to do:

• Decoupling producers from consumers: a service that sends a message shouldn't need to know or care which service, or how many, eventually processes it
• Absorbing load spikes: a queue lets a slow consumer catch up without the producer blocking or failing
• Guaranteeing delivery: messages shouldn't vanish if a consumer crashes mid-processing
• Enabling asynchronous work: background jobs, notifications, and cross-service communication all depend on not forcing everything to happen synchronously

The real difference is architecture: a log you can replay, a broker that routes intelligently, or a managed queue you don't have to think about at all, and that architectural choice is exactly what LinkedIn's 400,000-messages-a-second number and RabbitMQ's per-node ceiling both trace back to.

Kafka

What it actually is: a distributed, append-only log, split into partitions, where producers write messages and consumers read them at their own pace by tracking an offset, a position in the log. Unlike a traditional queue, messages aren't removed once consumed; they stay for a configured retention period, which means multiple independent consumers can replay the same stream from different points.

A small example (producing and consuming, conceptually):

producer.send("orders-topic", orderEvent)

# consumer tracks its own offset in the partition
consumer.subscribe("orders-topic")
for message in consumer.poll():
    process(message)
    consumer.commit_offset()

Where it shines:

• Extremely high throughput: this is the entire reason Kafka exists, LinkedIn's original 400,000-messages-a-second figure and Confluent's more recent 605,000-messages-per-second benchmark both measure exactly this strength
• Replayable event log: since messages persist for a retention window instead of disappearing on consumption, multiple services (billing, analytics, fraud detection) can independently read the same event stream without copying data between them
• Strong ordering guarantees per partition: messages within a single partition are strictly ordered, which matters for event sourcing and audit-style use cases
• Built for streaming, not just queuing: Kafka's ecosystem (Kafka Streams, Kafka Connect) extends naturally into real-time stream processing, not just message passing

Where it struggles:

• Real operational complexity: running Kafka well requires genuine expertise, partition management, broker tuning, and historically JVM-specific operational knowledge, a cost one team's own migration write-up specifically flagged as a reason they chose RabbitMQ Streams instead
• Higher latency under light load: Kafka trades per-message latency for batch throughput; published comparisons consistently show RabbitMQ winning on raw per-message latency at lower volumes, precisely because Kafka's batching exists to serve throughput, not speed per message
• Operational floor is higher: a single RabbitMQ node is a legitimate production deployment for a small service; Kafka's minimum viable setup for real resilience typically starts at multiple brokers

Who this suits: systems with a genuine, measured need for very high throughput or event replay, exactly the problem LinkedIn built Kafka to solve, and increasingly common at companies running real-time analytics, activity feeds, or event-sourced architectures.

RabbitMQ

What it actually is: a traditional message broker built on the AMQP protocol, where producers publish messages to exchanges, which route them to queues based on configurable rules, and consumers receive and acknowledge each message individually. Once a message is acknowledged, it's removed from the queue, a fundamentally different model from Kafka's persistent log.

A small example (publishing with routing, conceptually):

channel.exchange_declare(exchange='orders', exchange_type='topic')
channel.basic_publish(exchange='orders', routing_key='order.created', body=order_json)

# a specific queue only receives messages matching its binding
channel.queue_bind(exchange='orders', queue='billing_queue', routing_key='order.created')

Where it shines:

• Flexible, powerful routing: AMQP's exchange types (direct, topic, fanout) let you route messages to exactly the right consumers based on real routing logic, something Kafka's simpler topic-and-partition model doesn't natively offer
• Lower operational floor: a single, well-configured RabbitMQ node is a genuinely reasonable production setup for smaller services, with a built-in management UI that lowers the barrier to running it confidently
• Strong per-message delivery guarantees: RabbitMQ tracks and acknowledges every individual message, which suits workloads where guaranteed, precise delivery matters more than raw volume
• Closing the throughput gap where it matters: RabbitMQ's newer Streams feature, a Kafka-like append-only log mode, reaches into the million-messages-per-second range on tuned clusters, narrowing a gap that used to be far more one-sided

Where it struggles:

• Lower throughput in classic queue mode: the widely cited 40,000-to-50,000-messages-per-second-per-node figure for classic queues is real, and it's precisely the ceiling that pushed companies like the AppDirect engineering team to replace RabbitMQ with Kafka once their data volume outgrew it
• No native long-term replay in classic mode: once a message is consumed and acknowledged in a standard queue, it's gone, so replaying historical events requires the newer Streams feature specifically, not RabbitMQ's original model
• Scaling requires more deliberate work at very high volume: RabbitMQ can scale, but matching Kafka's highest-throughput numbers requires specific tuning and the Streams feature, not the default configuration

Who this suits: systems that need flexible, rule-based routing, moderate-to-high throughput workloads, and teams who want a lower operational floor than Kafka while keeping more control than a fully managed service offers.

SQS

What it actually is: Amazon's fully managed queue service. There's no broker to run, no cluster to tune, no servers to patch, you create a queue through the AWS console or API, and AWS handles the infrastructure entirely. SQS offers two queue types: Standard (near-unlimited throughput, at-least-once delivery, no strict ordering) and FIFO (strict ordering and exactly-once processing, with a lower throughput ceiling).

A small example (sending and receiving, conceptually):

sqs.send_message(QueueUrl=queue_url, MessageBody=json.dumps(order))

messages = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10)
for msg in messages['Messages']:
    process(msg)
    sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=msg['ReceiptHandle'])

Where it shines:

• Zero operational overhead: no servers, no cluster management, no capacity planning, which removes an entire category of engineering work that both Kafka and RabbitMQ require
• Scales automatically: Standard queues handle very high throughput without any manual tuning or provisioning on your part, AWS absorbs that scaling work entirely
• Tight AWS ecosystem integration: native integration with Lambda, SNS, and the rest of AWS's services means less custom plumbing for teams already building on AWS
• Pay-per-use pricing: you pay for requests made, not for idle broker capacity sitting unused, which can be genuinely cheaper for bursty or moderate-volume workloads

Where it struggles:

• Less architectural flexibility: no custom routing logic like RabbitMQ's exchanges, and no replayable log like Kafka's, messages are simply consumed and gone, closer to RabbitMQ's model than Kafka's
• FIFO queues trade throughput for ordering: SQS FIFO queues cap out at a few thousand messages per second without batching, a real ceiling compared to Kafka's partition-based throughput
• Less control over delivery semantics: you're working within AWS's model, which is simpler but genuinely less tunable than self-managing Kafka or RabbitMQ's finer-grained configuration options
• Vendor lock-in: SQS is AWS-specific, so a multi-cloud or cloud-agnostic architecture needs a different answer

Who this suits: teams already on AWS who want to avoid running message infrastructure entirely, moderate-throughput workloads without complex routing needs, and smaller teams where operational simplicity outweighs fine-grained control.

Side-by-Side Comparison

	Kafka	RabbitMQ	SQS
Model	Distributed, persistent log	Traditional broker with routing	Fully managed queue
Peak throughput (published figures)	~605K msgs/sec (3-broker benchmark)	~40–50K msgs/sec/node (classic queues)	Near-unlimited (Standard), lower for FIFO
Message replay	Yes, native and core to the design	No (classic queues); yes via Streams	No
Routing flexibility	Basic (topics, partitions)	Strong (exchanges, routing keys)	Basic
Operational overhead	High, real infrastructure to run	Moderate, lower floor than Kafka	None, fully managed
Ordering guarantees	Per-partition	Per-queue, configurable	FIFO queues only, with throughput trade-off
Best for	High-throughput event streaming, replay	Flexible routing, moderate-scale reliability	Zero-ops simplicity on AWS

How This Fits a Career Path

• Backend Engineer (general): understanding when and why a system needs a message queue at all, before picking a specific one, is the more foundational skill most interviews actually probe for
• Platform or infrastructure engineer: Kafka expertise is close to a standard expectation for high-scale, event-driven architectures; our CI/CD Pipelines course covers the surrounding infrastructure skills this role assumes
• Cloud-focused or AWS-heavy roles: SQS fluency, including when to reach for it instead of self-managed infrastructure, is directly relevant and frequently tested
• System design interviews: be ready to justify a queue choice using the same reasoning LinkedIn, AppDirect, and the RabbitMQ Streams teams above actually used, a specific, measured throughput or routing requirement, not a default preference

How to Choose Without Overthinking It

1. Start with your actual throughput requirement, measured, not guessed. If you're nowhere near tens of thousands of messages per second, the gap between these three matters far less than other factors like operational cost and team familiarity.
2. If you're on AWS and want to avoid running infrastructure, start with SQS. It removes an entire category of operational work, and for most moderate-throughput backend workloads, it's genuinely sufficient.
3. Reach for RabbitMQ when routing logic matters. If your system needs messages intelligently routed to different consumers based on content or type, RabbitMQ's exchange model does real work SQS and Kafka don't offer as naturally.
4. Reach for Kafka only once you have Kafka-scale problems. LinkedIn built it because RabbitMQ topped out two orders of magnitude below their actual need. If you're not near that kind of volume, or you don't need event replay, you're very likely taking on real operational complexity for a problem you don't have yet.

A note on honesty: the specific throughput numbers above come from different benchmarks, run on different hardware, at different points in time, LinkedIn's original figure is now over a decade old, and RabbitMQ's Streams feature has closed much of the historical gap since. Treat these as evidence that a real difference exists, not as numbers that will reproduce exactly on your own infrastructure. Benchmark your actual workload before committing.

Common Mistakes When Learning Messaging Systems

• Reaching for Kafka by default because it's the most talked-about option. Its operational complexity is real and needs to be paying for a genuine throughput or replay requirement, not just reputation.
• Underestimating SQS because it's "just AWS." For a large share of real backend workloads, SQS's zero-operational-overhead model is a genuinely better fit than self-managing a broker.
• Ignoring RabbitMQ's routing strength. Teams focused purely on throughput numbers sometimes miss that RabbitMQ solves a routing problem the other two don't address as naturally.
• Assuming message delivery guarantees are the same across all three. At-least-once, exactly-once, and ordering guarantees differ meaningfully between Kafka, RabbitMQ, and SQS's Standard versus FIFO queues; read the actual guarantees, don't assume.
• Migrating before measuring. AppDirect's own migration write-up describes a careful, gradual, dual-pipeline rollout, not a sudden cutover, precisely because getting this decision wrong under production load is expensive to undo.

Frequently Asked Questions

Should a beginner learn Kafka, RabbitMQ, or SQS first?

SQS, if you're working within AWS, since it requires the least infrastructure knowledge to get started and still teaches core queueing concepts. RabbitMQ is a strong second step for understanding routing-based messaging, with Kafka's log-based model as a distinct third concept once the fundamentals are solid.

Is Kafka always faster than RabbitMQ?

In raw sustained throughput, consistently yes, which is the entire reason Kafka was built. But RabbitMQ has historically posted lower per-message latency at smaller scale, and its newer Streams feature narrows the throughput gap considerably on tuned clusters.

Can SQS replace Kafka for event streaming?

Not directly. SQS doesn't offer Kafka's persistent, replayable log model; once a message is processed, it's gone. For genuine event-sourcing or multi-consumer replay needs, Kafka (or RabbitMQ Streams) solves a problem SQS isn't designed for.

Why did LinkedIn build Kafka instead of just scaling RabbitMQ?

Because the throughput gap they measured was described as two orders of magnitude, not a tuning problem solvable within their existing broker's architecture. That's a genuinely different scale of need than most teams will ever hit, which is worth remembering before assuming you need the same solution.

Does RabbitMQ Streams make Kafka unnecessary?

Not entirely. Streams narrows the throughput gap significantly, but Kafka's broader ecosystem (Kafka Connect, Kafka Streams, its maturity at extreme scale) still gives it real advantages for teams already investing deeply in event-driven architecture.

How do interviewers evaluate this topic?

Usually by asking you to design a system and justify your queue choice for its specific requirements, throughput, ordering, routing, operational cost, rather than testing for a fixed favorite. Being able to explain why LinkedIn needed Kafka, and why most systems don't need Kafka's exact scale, signals real understanding.

Measure Your Actual Throughput Before You Choose

The real lesson from LinkedIn's 400,000-messages-a-second origin story isn't "always pick Kafka," it's that the decision followed a specific, measured bottleneck their existing broker genuinely couldn't handle. Explore the System Design course to build the skills that let you make that same evidence-based call on your own architecture.