Message queues are one of those topics where the debate sounds technical but the real question is simple: which one fits your problem?
Most teams pick based on what they've heard of, what a senior engineer used at a previous job, or what the latest conference talk recommended. None of those are good reasons.
The right message queue becomes invisible infrastructure that just works. The wrong one becomes a constant source of production incidents and engineering frustration.
Here's a clear breakdown of all three — what each one is actually optimized for, where each one breaks down, and how to make the decision without guessing.
Before Comparing: Understand the Fundamental Difference
Kafka, RabbitMQ, and SQS are not interchangeable. They solve different problems at a fundamental level.
RabbitMQ is a message broker. It routes messages from producers to consumers and deletes them once they're acknowledged. The message is gone when it's processed.
Kafka is an event log. It stores messages durably in an ordered, append-only log. Consumers read from the log at their own pace. Messages persist — and can be replayed.
SQS is a managed queue. AWS runs it for you. No infrastructure, no operations, no configuration. Messages are hidden during processing and deleted when acknowledged.
These are not the same thing wearing different clothes. Choosing the wrong one means fighting the tool instead of building features.
RabbitMQ — The General-Purpose Broker
RabbitMQ's defining feature is flexible routing. Producers publish to an exchange. The exchange routes messages to queues based on binding rules — direct match, topic pattern, fanout, or headers.
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
# Exchange routes messages by routing key pattern
channel.exchange_declare(exchange='orders', exchange_type='topic')
# Route EU priority orders to a dedicated queue
channel.queue_bind(
exchange='orders',
queue='orders.eu.priority',
routing_key='orders.eu.*'
)
# Publish — broker decides where it goes
channel.basic_publish(
exchange='orders',
routing_key='orders.eu.priority',
body='{"order_id": "123", "region": "eu"}',
properties=pika.BasicProperties(
delivery_mode=2, # Persistent — survives broker restart
priority=9 # Per-message priority
)
)
An orders.eu.priority routing key can land in three different queues by pattern match, configured at the broker, with no producer or consumer changes. Neither Kafka nor SQS has anything comparable.
Where RabbitMQ wins:
- Task queues — background jobs, email sending, report generation. Each message is processed once by one worker. Classic work queue pattern.
- Complex routing — messages need to go to different queues based on content, priority, or region. RabbitMQ handles this natively.
- Low latency — sub-millisecond delivery for time-sensitive operations.
- Per-message control — TTL, priority, delayed delivery, dead-letter exchanges. Mature, fine-grained control.
Where RabbitMQ breaks down:
- Replay — messages are deleted after acknowledgment. A new service that needs last month's events cannot have them. Streams help but it's not what RabbitMQ was built for.
- Massive throughput — degrades when queues grow very long. Not the right tool for millions of events per second.
- Operational complexity — self-hosted RabbitMQ requires monitoring, patching, clustering, and on-call response.
Self-hosted RabbitMQ costs $200–$600/month for a small cluster. Amazon MQ (managed RabbitMQ) starts around $300/month for development, scaling to $1,000+ for production with high availability.
Pick RabbitMQ when: you need a task queue, flexible routing, or low-latency message delivery and you don't need event replay.
Kafka — The Event Streaming Platform
Kafka is not a message queue. It's a distributed, append-only event log.
Messages are written to topics, split into partitions. Consumers read from partitions at their own pace using offsets. Kafka doesn't delete messages when consumed — they persist for a configured window (default 7 days, configurable indefinitely).
from confluent_kafka import Producer, Consumer
# Producer
producer = Producer({'bootstrap.servers': 'localhost:9092'})
producer.produce(
topic='user-events',
key='user-123', # Same key → same partition → ordered
value='{"event": "purchase", "amount": 49.99}',
callback=lambda err, msg: print(f"Delivered: {msg.offset()}")
)
producer.flush()
# Consumer — reads from offset, not destructive
consumer = Consumer({
'bootstrap.servers': 'localhost:9092',
'group.id': 'analytics-service',
'auto.offset.reset': 'earliest' # Can replay from beginning
})
consumer.subscribe(['user-events'])
while True:
msg = consumer.poll(timeout=1.0)
if msg:
process_event(msg.value())
consumer.commit() # Advance offset — message still in log
The key difference: multiple independent consumer groups can read the same events simultaneously. Your analytics service, your recommendation engine, and your fraud detection system can all consume the same user-events topic independently, at their own pace, without interfering with each other.
Where Kafka wins:
- Event replay — new service needs last month's data? Read from offset 0. No re-ingestion, no backfill jobs.
- High throughput — millions of events per second. Purpose-built for this.
- Multiple consumers — many services consuming the same event stream independently.
- Event sourcing — rebuilding application state from an ordered event log.
- Stream processing — real-time pipelines with Kafka Streams or Flink.
Where Kafka breaks down:
- Task queues — no per-message ack, no priority, no delay, and head-of-line blocking within a partition. Teams end up writing a retry-topic ladder to simulate what RabbitMQ does natively.
- Operational overhead — Kafka clusters require significant expertise to run well. Use managed services (Confluent Cloud, Amazon MSK) unless you have a dedicated platform team.
- Overkill at small scale — if you have 10 events per second, Kafka's complexity is not justified.
The best practice: start with managed services like Confluent Cloud or Amazon MSK to reduce operational overhead while learning Kafka.
Pick Kafka when: you need event replay, high-throughput streaming, or multiple independent services consuming the same event stream.
SQS — Managed Simplicity
SQS is the easiest to operate and the hardest to outgrow incorrectly.
AWS runs everything. No servers, no clusters, no configuration. You create a queue, send messages, and poll for them. It scales automatically.
import boto3
import json
sqs = boto3.client('sqs', region_name='us-east-1')
QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123456789/orders'
# Send
sqs.send_message(
QueueUrl=QUEUE_URL,
MessageBody=json.dumps({
'order_id': '456',
'user_id': 'usr-789',
'total': 99.99
}),
MessageAttributes={
'EventType': {
'StringValue': 'order.created',
'DataType': 'String'
}
}
)
# Receive and process
response = sqs.receive_message(
QueueUrl=QUEUE_URL,
MaxNumberOfMessages=10,
WaitTimeSeconds=20, # Long polling — reduces empty responses
VisibilityTimeout=30 # Message hidden for 30s during processing
)
for message in response.get('Messages', []):
try:
body = json.loads(message['Body'])
process_order(body)
# Delete only after successful processing
sqs.delete_message(
QueueUrl=QUEUE_URL,
ReceiptHandle=message['ReceiptHandle']
)
except Exception as e:
# Don't delete — message returns to queue after VisibilityTimeout
logger.error(f"Processing failed: {e}")
The idempotency requirement: SQS Standard queues deliver at-least-once. Messages will occasionally be delivered more than once.
This is the single most common SQS production bug — it surfaces as mysterious double-charges and duplicate emails long after launch.
Design your consumers to be idempotent from day one. Processing the same message twice must produce the same result as processing it once.
def process_order(order: dict):
order_id = order['order_id']
# Check idempotency before processing
if redis.get(f"processed:order:{order_id}"):
return # Already processed — skip
# Process the order
create_fulfillment(order)
send_confirmation_email(order)
# Mark as processed
redis.setex(f"processed:order:{order_id}", 86400, "1")
Where SQS wins:
- Zero operational overhead — no servers to manage, no clusters to monitor, no patches to apply.
- AWS-native integration — Lambda triggers, SNS fan-out, EventBridge routing. First-class AWS citizen.
- Reliable at any scale — handles spikes automatically without pre-provisioning.
- FIFO queues — exactly-once processing with strict ordering when you need it.
Where SQS breaks down:
- No replay — messages can be up to 256KB, and you can't replay messages once they're deleted.
- Cost at scale — SQS pricing is per-request. At 10,000 messages/second, you're looking at a five-figure monthly bill.
- No routing — no exchange model, no topic pattern matching. SNS + SQS can simulate fan-out but it's not native.
- Vendor lock-in — deeply AWS-specific. Migrating off is painful.
Pick SQS when: you're on AWS, need zero ops, and your use case is straightforward queue-based processing without replay.
The Decision Framework
Do you need event replay?
(New services reading historical events,
event sourcing, rebuilding state)
│
└── YES → Kafka
(Confluent Cloud or MSK in production)
Do you need complex routing?
(Different queues based on message content,
priority queues, dead-letter routing)
│
└── YES → RabbitMQ
(CloudAMQP or Amazon MQ for managed)
Are you on AWS and need zero ops?
(Simple task queue, Lambda integration,
no replay required)
│
└── YES → SQS
(Standard for throughput, FIFO for ordering)
High throughput + multiple consumers
no replay needed?
│
└── Consider both Kafka and RabbitMQ Streams
Real-World Combinations That Work
Most production systems use more than one broker for different workloads.
SaaS product:
- SQS → background jobs (email, webhooks, report generation)
- Kafka → event log (user activity, audit trail, analytics pipeline)
E-commerce platform:
- RabbitMQ → order processing tasks (payment, fulfillment, notifications)
- Kafka → event streaming (inventory updates, recommendation engine, fraud detection)
Microservices at scale:
- Kafka → event backbone between services
- SQS → simple async jobs within a service boundary
- RabbitMQ → internal task dispatch with complex routing requirements
The One-Line Summary for Each
Kafka — use it when you need an event log that multiple systems can read, replay, and process independently.
RabbitMQ — use it when you need a task queue with flexible routing and fine-grained message control.
SQS — use it when you're on AWS, need zero infrastructure overhead, and don't need replay.
The mistake isn't picking the wrong one for your use case — it's applying one choice uniformly across every async workload in your system.
This post is part of OutworkTech's backend engineering series. Related reading: How to Automate Repetitive Business Processes and How to Handle 1M+ Users Without Breaking Your System.
OutworkTech builds and scales backend systems, APIs, and SaaS infrastructure for companies that need engineering depth without the overhead. If your async architecture needs a second opinion — let's talk.
Top comments (0)