Kafka Is Overkill for Most of Us — Here's What I Actually Run in Production Instead

TL;DR: The trap is always the same: you spin up Kafka with Docker Compose in three hours, get a producer pushing messages, feel like a genius — then Monday rolls around and you're staring at a ZooKeeper ensemble that's split-brained, a consumer group stuck in perpetual rebalancing, and

📖 Reading time: ~34 min

What's in this article

1. The Real Problem: Kafka's Ops Burden Hits You on Day 2, Not Day 1
2. The Four Alternatives I've Actually Run Under Load
3. RabbitMQ: The Boring Choice That Actually Works
4. NATS JetStream: The One I Reach for on New Projects Now
5. Redpanda: Drop-in Kafka Replacement That Doesn't Lie About It
6. Apache Pulsar: Powerful, But Respect the Complexity Tax
7. Side-by-Side: What Actually Matters for HA Microservices
8. When to Pick What: My Actual Decision Tree

---

The Real Problem: Kafka's Ops Burden Hits You on Day 2, Not Day 1

The trap is always the same: you spin up Kafka with Docker Compose in three hours, get a producer pushing messages, feel like a genius — then Monday rolls around and you're staring at a ZooKeeper ensemble that's split-brained, a consumer group stuck in perpetual rebalancing, and broker logs screaming about under-replicated partitions. Day 1 is great. Day 2 through Day 30 is where the real ops cost shows up.

The JVM overhead alone should give small teams pause. A 3-node Kafka cluster with default configs routinely chews through 6–8GB of RAM before a single application message flows through it. Each broker wants at least 4GB heap to stay stable under moderate load, ZooKeeper wants another 1–2GB, and suddenly your "lightweight" message bus costs more RAM than your actual microservices. I've watched teams on $200/month bare-metal nodes blow their entire memory budget just getting Kafka to stop throwing OutOfMemoryError during log compaction.

Before picking an alternative, be precise about what your pipeline actually needs. High-availability in a microservices context means three specific things, not a vague uptime promise:

• At-least-once delivery: messages survive broker restarts and network partitions — your consumer must be idempotent, but no message gets silently dropped
• Consumer group rebalancing: when a service instance dies or scales, other instances pick up its partitions/queues without manual intervention and without losing offset position
• Dead-letter handling: poison messages that fail repeatedly go somewhere observable, not into a black hole that silently halts your pipeline

Kafka checks all three boxes, which is why it became the default recommendation. The problem is that doing all three correctly requires tuning min.insync.replicas, acks=all, enable.idempotence=true, proper DLQ topic setup, and consumer group session timeout values — and getting any of those wrong under load produces failure modes that are genuinely hard to diagnose at 2am. KRaft mode (Kafka without ZooKeeper, stable since Kafka 3.3) removes one layer of that complexity, but you're still managing JVM heap, log retention, and partition rebalancing by hand.

This guide is specifically useful if you're running fewer than 10 services, you're self-hosting on a budget (think 3 VMs or a small Kubernetes cluster with real memory constraints), or your team got burned by a broker going sideways during peak traffic and you don't have a dedicated platform engineering team to absorb that incident. The alternatives below aren't downgrades — some of them handle specific HA scenarios better than Kafka does at small scale, with a fraction of the operational surface area. For broader context on reducing tooling overhead across your entire stack, the Ultimate Productivity Guide: Automate Your Workflow in 2026 covers complementary patterns worth pairing with a leaner messaging setup.

The Four Alternatives I've Actually Run Under Load

RabbitMQ — Still My First Reach

I've run RabbitMQ on production pipelines handling 50K+ messages/minute and the thing that keeps pulling me back is how sane its ops story is. You get a real web UI at port 15672 out of the box, rabbitmq-diagnostics actually tells you what's wrong, and the AMQP 0-9-1 protocol has client libraries in every language worth naming. For event-driven microservices with moderate throughput and complex routing — fan-out, dead-letter queues, priority queues — nothing else gives you this flexibility with so little config ceremony. The 3.12+ release brought significant queue memory improvements that fixed the one thing I used to complain about most.

# Spin up a 3-node cluster locally for testing
docker network create rabbitmq-net

docker run -d --net rabbitmq-net --hostname rabbit1 \
  -e RABBITMQ_ERLANG_COOKIE='secret' \
  -e RABBITMQ_DEFAULT_USER=admin \
  -e RABBITMQ_DEFAULT_PASS=admin \
  --name rabbit1 rabbitmq:3.13-management

# Join rabbit2 to rabbit1 after it starts
docker exec rabbit2 rabbitmqctl stop_app
docker exec rabbit2 rabbitmqctl join_cluster rabbit@rabbit1
docker exec rabbit2 rabbitmqctl start_app

The honest downside: RabbitMQ's HA clustering model uses a mirrored queue approach that does not scale horizontally the way Kafka partitions do. Once you're hitting sustained high-throughput scenarios where consumers need to replay old messages, you'll feel the ceiling. Quorum queues (available since 3.8, mature by 3.10) fixed the mirror-split-brain problem but added write amplification. I've also watched it fall over under rapid reconnection storms from misconfigured services — the connection/channel model means 500 services reconnecting simultaneously is a real incident.

NATS JetStream — What I Switched To When RabbitMQ's Clustering Started Hurting

The thing that caught me off guard about NATS JetStream was how small the binary is and how fast it gets to "working cluster." You're talking a single Go binary under 20MB, no external dependencies, Raft-based clustering that just works when you point three nodes at each other. JetStream adds persistence and at-least-once delivery on top of core NATS, which is pure fire-and-forget pub/sub. For microservice architectures where services are ephemeral and you need low-latency messaging (sub-millisecond in local benchmarks) without the JVM overhead, this is where I land now for greenfield projects.

# nats-server.conf for a 3-node JetStream cluster
server_name: n1
listen: 0.0.0.0:4222

jetstream {
  store_dir: /data/jetstream
  max_memory_store: 1GB
  max_file_store: 10GB
}

cluster {
  name: mycluster
  listen: 0.0.0.0:6222
  routes: [
    nats-route://n2:6222
    nats-route://n3:6222
  ]
}

# Create a durable stream from CLI
nats stream add ORDERS \
  --subjects "orders.*" \
  --retention limits \
  --max-age 24h \
  --replicas 3

JetStream's consumer model is genuinely different from RabbitMQ — push and pull consumers are both first-class, and the subject hierarchy gives you routing without a separate exchange/binding config layer. Where it bites you: the operational tooling is still maturing compared to RabbitMQ's 15 years of ecosystem. The NATS CLI is excellent but Grafana dashboards and alerting rules aren't as plug-and-play. Also, JetStream stream limits behave differently than you'd expect under backpressure — test your DiscardNew vs DiscardOld policy choice before production or you'll be confused why messages are silently dropped.

Redpanda — Kafka Wire Protocol, No JVM, Actually Ships as One Binary

I was skeptical when I first read "Kafka-compatible without Kafka" because that claim usually means "compatible until you hit the edge cases." Redpanda mostly holds up. It speaks the Kafka protocol at API versions that matter in practice, which means your existing kafka-python, confluent-kafka, or librdkafka-based code connects without modification. The JVM elimination matters more than the marketing suggests — no GC pauses, no heap tuning, no 6GB RAM floor just to get the broker to start. A single Redpanda node on a $6/month VPS with 1GB RAM can handle workloads that would make Kafka complain about heap exhaustion.

# docker-compose for a single-node Redpanda (dev/staging)
services:
  redpanda:
    image: docker.redpanda.com/redpandadata/redpanda:v23.3.11
    command:
      - redpanda start
      - --smp 2
      - --memory 1G
      - --overprovisioned
      - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
    ports:
      - "9092:9092"

  console:
    image: docker.redpanda.com/redpandadata/console:v2.4.3
    ports:
      - "8080:8080"
    environment:
      KAFKA_BROKERS: redpanda:29092

The trade-off I keep coming back to: Redpanda's free self-hosted tier is genuinely uncapped, but the clustering story for multi-node HA requires more careful network configuration than its docs suggest. Raft consensus means a 3-node cluster tolerates one failure — same as Kafka's default ISR — but Redpanda's partition rebalancing under node loss is faster in my experience. Where I wouldn't use it today: if you're already deep in the Kafka ecosystem with Kafka Streams, KsqlDB, or Kafka Connect connectors that use internal topics, some rough edges appear. The Connect-compatible layer is improving but not equivalent.

Apache Pulsar — Technically Impressive, Operationally Heavy

Pulsar's architecture is genuinely clever: separate the storage layer (BookKeeper) from the broker layer so you can scale them independently. That design choice means a broker crash doesn't lose in-flight messages because the data was never on the broker — it's on the bookies. Multi-tenancy is baked in at the namespace level, which makes it appealing for platform teams managing pipelines for multiple internal teams. The geo-replication is also first-class rather than a bolt-on. I respected all of this.

# The "simple" Pulsar standalone still pulls multiple containers
docker run -it -p 6650:6650 -p 8080:8080 \
  --mount source=pulsardata,target=/pulsar/data \
  --mount source=pulsarconf,target=/pulsar/conf \
  apachepulsar/pulsar:3.2.0 \
  bin/pulsar standalone

# But a real HA cluster requires ZooKeeper (or etcd), 
# minimum 3 BookKeeper nodes, minimum 2 brokers
# That's 8+ processes minimum before you've written a line of app code

Here's why I stepped back: operating Pulsar means operating ZooKeeper (yes, you can use etcd in newer versions, but most production deployments still use ZK), BookKeeper, and the broker layer as three separate things with separate failure modes and separate tuning. The Pulsar Helm chart for Kubernetes is 600+ lines and touches more knobs than I want to own without a dedicated platform team. The Java-heavy stack also means GC tuning is back on the table — I've seen BookKeeper GC pauses cause latency spikes in otherwise healthy clusters. If you're a team of 3 running microservices and you don't have someone who wants to become a Pulsar specialist, the operational surface area will eventually bite you. I'd revisit this at scale where the storage/compute separation actually pays for itself.

RabbitMQ: The Boring Choice That Actually Works

The thing that surprises most people switching from Kafka is how immediately useful RabbitMQ is. No ZooKeeper, no broker IDs to coordinate, no topic partition math. You're pushing messages in minutes, not hours. I've bootstrapped it on a $6 DigitalOcean droplet during an on-call incident and it held up fine for the task. That kind of accessibility is genuinely underrated.

Getting it running takes one command:

# Docker is the path I actually recommend — version pinning matters here
docker run -d \
  --hostname rabbit1 \
  --name rabbit \
  -p 5672:5672 \
  -p 15672:15672 \
  rabbitmq:3.13-management

Or on Debian/Ubuntu bare metal: apt install rabbitmq-server and it's running as a systemd service immediately. The management UI lands at http://localhost:15672 (guest/guest by default — change this before you do anything else). Seriously, before you declare a single queue.

Stop Using Classic Mirrored Queues

Half the RabbitMQ HA tutorials floating around still show you classic mirrored queues with ha-mode: all. That's deprecated as of 3.8 and will be removed. Quorum queues are the actual HA primitive now — they use the Raft consensus algorithm, tolerate (n-1)/2 node failures in a cluster, and have much better consistency guarantees. If you're setting up a new pipeline and someone pastes you a policy with ha-mode, close that tab.

Here's a real Python declaration using pika that I'd actually put in production:

import pika

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost')
)
channel = connection.channel()

# x-queue-type quorum is non-negotiable for HA
# x-delivery-limit prevents poison messages from looping forever
# x-dead-letter-exchange routes failed messages somewhere inspectable
channel.queue_declare(
    queue='orders.processing',
    durable=True,
    arguments={
        'x-queue-type': 'quorum',
        'x-delivery-limit': 5,           # after 5 nacks, route to DLX
        'x-dead-letter-exchange': 'orders.dlx',
        'x-dead-letter-routing-key': 'orders.failed',
    }
)

# Declare the DLX separately — it's just a normal exchange
channel.exchange_declare(
    exchange='orders.dlx',
    exchange_type='direct',
    durable=True
)
channel.queue_declare(queue='orders.failed', durable=True,
    arguments={'x-queue-type': 'quorum'})
channel.queue_bind('orders.failed', 'orders.dlx', 'orders.failed')

connection.close()

The x-delivery-limit field is one I wish I'd known about earlier. Without it, a consumer that keeps crashing will nack the same message indefinitely, burning CPU and clogging your queue. Set it, pair it with a DLX, and you get a dead-letter queue for free where you can inspect and replay problem messages manually.

The Management UI Is Actually Your Observability Layer

I went in expecting the management UI to be a toy. It's not. The 3.13 UI shows per-queue message rates (publish/deliver/ack), consumer utilization as a percentage, memory and disk usage per node, and connection-level breakdown. For a majority of debugging sessions — figuring out why a consumer is falling behind, checking if a DLX is filling up, confirming message TTLs are firing — I haven't needed to reach for Grafana or Datadog. That's a real operational win, especially early in a project before you've built out your full observability stack.

Where RabbitMQ Will Fight You

Two genuine rough edges worth knowing upfront. First: the Federation and Shovel plugins for multi-datacenter setups are powerful but the mental model isn't obvious. Federation federates exchanges or queues between brokers loosely (good for pub/sub across DCs). Shovel moves messages between queues more literally (good for migration or forwarding). The docs technically cover it, but understanding which one you actually need and then debugging why messages aren't flowing across DCs will cost you a full day if you've never touched them. Budget for that honestly.

Second ceiling to know: RabbitMQ's throughput sweet spot is roughly 20k–50k messages/sec on reasonable hardware with quorum queues. Past that you start tuning prefetch counts, connection pooling, and flow control — and you're fighting the tool. More importantly, RabbitMQ is fundamentally a work queue and routing broker, not a log. You cannot replay messages from an offset the way Kafka or Redpanda lets you. Once a message is acked, it's gone. If your pipeline needs "replay the last 6 hours of events for a new consumer," RabbitMQ is the wrong answer. For everything else — task queues, fanout to microservices, request/reply patterns, retry-with-DLX workflows — it's hard to beat the simplicity-to-reliability ratio.

NATS JetStream: The One I Reach for on New Projects Now

The thing that genuinely surprised me about NATS JetStream was the ops burden — or the near-total lack of it. I've stood up Kafka clusters, I've fought RabbitMQ's Erlang cookie clustering nonsense, and then I downloaded a single 20MB binary and had a durable, HA message stream running in under 10 minutes. That gap in complexity is not subtle.

# Download, unzip, run. That's actually it.
curl -L https://github.com/nats-io/nats-server/releases/download/v2.10.14/nats-server-v2.10.14-linux-amd64.zip -o nats.zip
unzip nats.zip
./nats-server-v2.10.14-linux-amd64/nats-server --jetstream

# Output you'll see:
# [1] Starting nats-server
# [1] Version:  2.10.14
# [1] JetStream enabled
# [1] Listening for client connections on 0.0.0.0:4222

No JVM. No Zookeeper. No broker topology to reason about before you can send your first message. For greenfield microservices projects where you don't want to hand the new team a Kafka runbook on day one, this is a real argument.

The JetStream config that actually makes it HA

Running with --jetstream flag is fine for local dev, but for production you want a proper config with explicit storage limits and clustering. Here's the server.conf I use as a starting point for a 3-node setup. The max_file_store is the one people forget — without it, JetStream will happily eat your disk.

# node1.conf
server_name: node1
listen: 0.0.0.0:4222

jetstream {
  store_dir: /data/nats
  max_memory_store: 1GB   # RAM-backed storage for hot streams
  max_file_store: 20GB    # disk cap — set this or regret it
}

cluster {
  name: orders-cluster
  listen: 0.0.0.0:6222
  routes: [
    nats-route://node2:6222
    nats-route://node3:6222
  ]
}

# node2.conf and node3.conf are identical
# except server_name and their own routes pointing back

# Start all three — they discover each other via routes
nats-server -c node1.conf &
nats-server -c node2.conf &
nats-server -c node3.conf &

# Verify cluster formed (needs nats CLI: https://github.com/nats-io/natscli)
nats server list

Compare this to RabbitMQ clustering, where you're dealing with rabbitmqctl join_cluster, Erlang cookies that have to match across nodes, and quorum queue configuration that's separate from mirroring policies. With NATS, the cluster block in one config file and matching routes is genuinely all there is.

Pull consumers are what you actually want for microservices

NATS has two consumer models: push (server delivers to a subject) and pull (client asks for messages explicitly). For microservices, pull wins every time — your service controls its own fetch rate, back-pressure is natural, and you don't get a flood of messages to a consumer that's halfway through a slow DB write. Creating one takes 30 seconds:

# Create the stream first
nats stream add ORDERS \
  --subjects "orders.*" \
  --storage file \
  --replicas 3 \
  --retention limits \
  --max-age 24h

# Then create a durable pull consumer
nats consumer add ORDERS processor \
  --pull \
  --ack explicit \
  --deliver all \
  --filter "orders.created"

# Your service fetches in batches — no message is lost if the pod dies
# because ack is explicit and the message stays in stream until acked
nats consumer next ORDERS processor --count 10

The --ack explicit flag is the important one. Without it, NATS defaults to ack-none mode, which is fine for fire-and-forget but useless if you care about delivery guarantees. With explicit ack, a crashed pod means those messages get redelivered to another instance — which is exactly what you want from a HA pipeline.

The gotcha nobody warns you about: subject namespace collisions

JetStream was added onto core NATS, and this shows when you mix both in the same service. Core NATS uses subjects for pub/sub. JetStream also uses subjects, but JetStream subjects map to streams based on your config. If you publish to orders.created with a plain NATS publisher in one service and you have a JetStream stream configured to capture orders.*, the message gets persisted — but if another part of your codebase also subscribes to orders.created with a plain core NATS subscription, they're operating in completely different delivery semantics. One gets the message once and it's gone. The other gets it durably with replay. I spent a debugging afternoon on this before I just made a rule: if a stream is configured for a subject pattern, nothing in the codebase touches those subjects with core NATS APIs. Pick one model per subject namespace.

Stream replay and time-based delivery — the Kafka feature I actually missed

The feature I missed most moving away from Kafka was offset-based replay. Kafka lets consumers rewind by offset or timestamp and reprocess. JetStream covers this properly with start time delivery, and it's cleaner to invoke than I expected:

# Replay from a specific point in time — great for reprocessing after a bug fix
nats consumer add ORDERS reprocessor \
  --pull \
  --ack explicit \
  --deliver by_start_time \
  --start-time "2024-11-01T00:00:00Z" \
  --filter "orders.created"

# Or from a sequence number if you logged it
nats consumer add ORDERS reprocessor \
  --pull \
  --ack explicit \
  --deliver by_start_sequence \
  --start-sequence 4821

This is the feature that moves JetStream from "lightweight pub/sub with persistence" into "actual event sourcing infrastructure." For any service that needs audit replay, backfill after a deployment, or debugging a specific time window of bad data — this works. The one constraint: your stream's --max-age setting determines how far back you can go. Default is unlimited but unbounded retention will eat your disk, so size this intentionally against your replay requirements.

Redpanda: Drop-in Kafka Replacement That Doesn't Lie About It

The thing that surprised me most about Redpanda is how honest the "drop-in replacement" claim actually is. I've been burned by "compatible" tools that quietly break on edge cases — Redpanda genuinely speaks the Kafka wire protocol at port 9092, which means your existing kafka-python producers, confluent-kafka consumers, and kcat debugging commands work without a single line change. Not "mostly work" — actually work.

Getting it running on bare metal or a VM is one command away from a usable broker:

curl -1sLf 'https://dl.redpanda.com/nzc4AAMHNXH6oesDH7buNuDkN3U/redpanda/cfg/setup/bash.deb.sh' | sudo bash && sudo apt install redpanda

For local dev where you just want to validate a pipeline without spinning up a JVM, single-node mode is genuinely useful:

# --overprovisioned tells Redpanda not to fight you over CPU scheduling
# --reserve-memory 0M skips the default headroom reservation — fine for dev, not for prod
sudo rpk redpanda start --overprovisioned --smp 1 --memory 1G --reserve-memory 0M

After that, bootstrap.servers=localhost:9092 in your consumer config and you're producing and consuming. No JVM to install. No ZooKeeper to babysit. No KRaft migration to plan around — Redpanda just skips all of that because the coordination layer is built in C++ from scratch. The operational difference this makes is real: a Kafka cluster where ZooKeeper goes sideways at 2am is a very different incident than a Redpanda cluster that just... keeps running. And because it's C++, you get predictable tail latencies without GC pauses spiking p99 every few minutes. That matters a lot in microservice architectures where one slow consumer can back up an entire pipeline.

The honest trade-off you need to know before committing: Redpanda Community is genuinely open source and free, and it covers most production use cases — replication, rack awareness, SASL/TLS auth, Schema Registry, and the HTTP Proxy API are all in there. But Tiered Storage (offloading old segments to S3/GCS to keep broker disk small) and some cluster management features require Redpanda Enterprise. Check their pricing page before assuming your architecture works on Community — if you're planning on infinite retention with cheap object storage, that's an enterprise feature.

I'd reach for Redpanda specifically when a team has existing Kafka expertise and consumer code they don't want to rewrite, but the ops burden of managing JVM heap tuning, ZooKeeper quorums, and broker GC logs is killing velocity. You keep 100% of your tooling — Kafka UI, Burrow for consumer lag, whatever you've built — and trade the JVM for a process that uses a fixed, predictable amount of memory. That's a genuinely rare situation where the migration cost is near-zero and the operational upside is significant.

Apache Pulsar: Powerful, But Respect the Complexity Tax

The thing that genuinely surprised me about Pulsar wasn't any benchmark — it was realizing that the brokers don't store any data at all. Zero. They're pure compute. Apache BookKeeper owns persistence entirely, which means you can scale your brokers and your storage layers independently. That's either a brilliant architectural decision or a DevOps tax you'll be paying for years, depending on who's on your team.

To actually evaluate it without committing to the full cluster setup, standalone mode is your friend. The snippet below is the one that actually works — I burned two hours on variations that partially started before landing on this:

# Pull the official image — this is Pulsar 3.1.x LTS line
docker run -it \
  -p 6650:6650 \
  -p 8080:8080 \
  --name pulsar \
  apachepulsar/pulsar:3.2.0 \
  bin/pulsar standalone

# Verify the broker is up and topics are accessible
curl http://localhost:8080/admin/v2/persistent/public/default

# Produce a quick test message
bin/pulsar-client produce persistent://public/default/test-topic \
  --messages "hello from standalone"

# Consume it back
bin/pulsar-client consume persistent://public/default/test-topic \
  --subscription-name my-sub \
  --num-messages 1

Standalone mode bundles ZooKeeper and BookKeeper in-process, so that single container is actually running three logical components. Don't let that fool you into thinking production looks anything like this. But for playing with the admin API, testing client libraries, or validating your schema registry setup — it's genuinely useful.

Geo-replication is where Pulsar earns real respect. In Kafka you're essentially stitching together MirrorMaker2 configs and hoping the lag doesn't spiral during a partition event. Pulsar treats geo-replication as a first-class namespace-level configuration. You define a replication cluster list on the namespace, and Pulsar handles async replication between data centers automatically. I've seen teams run services across three regions with this and the operational model stays coherent — one logical topic, multiple physical clusters, transparent failover. No third-party connector, no custom consumer offset management.

Here's the honest accounting of what production actually requires. A minimum viable HA cluster means:

• 3 Pulsar brokers (stateless, but still 3 JVMs with 2–4GB heap each)
• 3 BookKeeper bookies — this is non-negotiable for write quorum
• 3 ZooKeeper nodes for coordination metadata
• Optionally a Pulsar Proxy if you want clean external access

That's 9–10 JVM processes before your first application message gets produced. On a $20/month VPS stack this is physically impossible. On a managed Kubernetes cluster you're looking at real CPU and memory budget before your services even exist. StreamNative (the commercial Pulsar company) offers a cloud-managed version that abstracts this, but the free tier is limited enough that you're evaluating it, not running production on it.

My honest take: if your organization has a platform engineering team whose job is owning infrastructure, Pulsar's architecture pays dividends — especially that geo-replication model. If you're three developers who also write the frontend and handle on-call, you will spend more time babysitting BookKeeper ensembles than shipping features. The complexity isn't fake complexity — it solves real distributed systems problems — but it demands dedicated human attention. Teams without that shouldn't choose Pulsar because the docs told them it was powerful.

Side-by-Side: What Actually Matters for HA Microservices

The question I get asked most often is "which one should we use?" — and the honest answer is that four follow-up questions eliminate three of the four options immediately. Let me give you the comparison first, then the shortcuts.

Comparison: What the Numbers Actually Look Like

| Feature                  | RabbitMQ 3.13     | NATS JetStream 2.10 | Redpanda 24.x      | Apache Pulsar 3.x     |
|--------------------------|-------------------|----------------------|--------------------|------------------------|
| Min nodes for real HA    | 3 (quorum queues) | 3 (Raft cluster)     | 3 (Raft-based)     | 3 broker + 3 bookie    |
| Protocol                 | AMQP 0-9-1        | NATS (custom)        | Kafka wire (0.9+)  | Pulsar binary + Kafka  |
| Message replay           | Limited (streams) | Yes (JetStream)      | Yes (full log)     | Yes (full log)         |
| Idle RAM per node        | ~150–300 MB       | ~30–80 MB            | ~500 MB–1.5 GB     | ~1–2 GB (broker alone) |
| DLQ out of the box       | Yes               | Yes (JetStream)      | Manual (consumer)  | Yes                    |
| Operational complexity   | 2/5               | 2/5                  | 3/5                | 5/5                    |

Pulsar's operational complexity score of 5 isn't me being dramatic — you're running two separate distributed systems simultaneously (brokers + BookKeeper). A single Pulsar deployment that you'd trust in production has six nodes minimum, and that's before you add ZooKeeper (pre-3.x) or the new Oxia metadata store. The memory footprint per node is also not optional overhead — the JVM heap for a BookKeeper bookie under real load starts at 1 GB just to avoid constant GC pauses. Pulsar is architecturally clever, but "free" in ops cost it is not.

The Fast Decision Tree

Before reading another benchmark, answer these in order:

1. Do your consumers already speak Kafka protocol? If you have existing Kafka client code — using librdkafka, confluent-kafka-python, the Java KafkaConsumer — go Redpanda. Zero code changes, drop-in replacement, and you skip Kafka's ZooKeeper/KRaft operational burden. I switched a team from managed Confluent to self-hosted Redpanda in an afternoon because the consumer group semantics are byte-for-byte identical.
2. Do you have zero appetite for JVM operations? NATS JetStream. The entire server binary is ~20 MB, the 30–80 MB idle RAM figure above is real, and the cluster config is genuinely this short:

# nats-server.conf — minimal 3-node JetStream cluster
server_name: "node-1"
listen: 0.0.0.0:4222

jetstream {
  store_dir: /data/nats
  max_memory_store: 1GB
  max_file_store: 20GB
}

cluster {
  name: "mycluster"
  listen: 0.0.0.0:6222
  routes: [
    nats://node-2:6222
    nats://node-3:6222
  ]
}

1. Are you already running RabbitMQ with classic mirrored queues? Don't migrate. Upgrade to quorum queues (available since RabbitMQ 3.8, mature since 3.10) and you get Raft-based replication with proper leader election. The classic mirrored queue HA model had a real split-brain risk — quorum queues eliminate that. One policy change and you're done:

# Apply quorum queue policy via management API
rabbitmqctl set_policy ha-quorum ".*" \
  '{"queue-mode":"lazy","x-queue-type":"quorum"}' \
  --apply-to queues

The Node Count Lie You'll Hear

Every tool in this table needs three nodes minimum for real fault tolerance. Not two. I've seen people deploy RabbitMQ as a 2-node cluster thinking they get HA — what they actually get is a cluster that stops accepting writes the moment one node dies, because quorum requires a majority and 1-of-2 isn't a majority. The math doesn't care about your budget. With three nodes, you survive one node failure and keep quorum (2-of-3). With NATS JetStream, the Raft group needs an odd number ≥ 3 for the same reason. Redpanda's partition replication factor of 3 means you're paying for three brokers before a single topic is truly fault-tolerant. Anyone who quotes you a "2-node HA setup" is describing a system that fails silently under the exact conditions HA is supposed to handle.

The one scenario where RabbitMQ edges out NATS for microservices is complex routing logic — topic exchanges, header-based routing, per-message TTLs, and built-in DLQ with requeue semantics are all native. NATS JetStream's DLQ equivalent (advisory subjects + consumer nak with backoff) works, but you're assembling it from parts rather than flipping a switch. Redpanda has no native DLQ at all — you build it at the consumer level or use Kafka Streams, which adds its own dependency. For a pipeline where poison-pill messages are a real operational concern, that distinction matters on your first 2 AM incident.

When to Pick What: My Actual Decision Tree

The thing that actually determines which tool you should pick isn't feature lists — it's your current operational reality. I've watched teams spend three sprints evaluating message brokers when the answer was obvious from day one if they'd just been honest about their constraints.

Greenfield project, small team (2–5 devs), self-hosted

NATS JetStream, full stop. The operational surface area is tiny — single binary, no ZooKeeper, no JVM tuning, no separate schema registry. A team of two can run this without a dedicated platform engineer. You get persistence, consumer groups, exactly-once delivery, and key-value store built in. The Helm chart gets you to production-ready in an afternoon. I switched to this from a heavier setup specifically because I wanted to sleep at night without PagerDuty alerts about broker rebalancing.

Existing Kafka clients you can't touch right now

Redpanda. Don't migrate your producers and consumers — migrate the broker. Redpanda speaks the Kafka wire protocol natively, so your existing librdkafka-based clients, Kafka Streams apps, and connector configs mostly just work against a different bootstrap address. The practical upside: you drop the JVM, you drop ZooKeeper (or KRaft), and you get meaningfully lower tail latencies. The gotcha is that some Kafka-specific admin APIs have subtle behavior differences, so run your integration tests against Redpanda before you cut over production.

# just change the bootstrap server — your producer code stays identical
kafka-console-producer \
  --bootstrap-server redpanda-0:9092 \
  --topic my-existing-topic

You need flexible routing logic

RabbitMQ and nothing else on this list. If you have requirements like "route to queue A if header region=eu, fanout to all subscribers if type=alert, dead-letter after 3 retries to a separate exchange" — that's RabbitMQ's entire reason for existing. Topic exchanges, header exchanges, fanout, direct, dead-letter exchanges, per-message TTL. No other tool in the free tier matches this routing model. The trade-off is real though: AMQP topology management becomes infrastructure-as-code you have to maintain, and the management UI makes it deceptively easy to create routing tangles that are a nightmare to debug six months later.

Multi-region active-active with geo-replication

If geo-replication is a hard architectural requirement, Pulsar is the honest answer — it was designed for this use case and its tiered storage + geo-replication model is genuinely mature. But Pulsar has a real ops tax: you're managing BookKeeper alongside the brokers, and the learning curve for a small team is steep. If your team doesn't have someone who's run distributed storage systems before, Redpanda's geo-replication (available in their self-managed tier) is the safer bet. You give up some of Pulsar's flexibility and get back your sanity.

Kubernetes operator-managed deployments

All four tools have Kubernetes operators, but the maturity gap is real. The RabbitMQ Cluster Operator (maintained by the RabbitMQ core team) handles rolling upgrades, TLS rotation, and user management cleanly — it's been in production widely since the 1.x days. The NATS Helm chart from nats-io is similarly solid and actively maintained. Pulsar's operator ecosystem is more fragmented; StreamNative maintains one, but you'll hit rough edges. Redpanda's operator is improving fast but is younger than the others.

Hard stops that will actually hurt you

• NATS JetStream without persistent storage: The default storage backend is in-memory. If your pod restarts, your stream history is gone. Always configure file storage explicitly in your JetStream config — store_dir: /data/jetstream backed by a PVC. This is not in the quickstart guide prominently enough and it will bite you.
• RabbitMQ classic mirrored queues on 3.x: Classic mirrored queues are deprecated and the removal timeline is real. If you're setting up anything new, use quorum queues. If you have existing mirrored queue configs, migrating to quorum queues is not trivial — the semantics differ, especially around x-max-length and overflow behavior. Do it before you're forced to.

# NATS JetStream - always set this in your server config
jetstream {
  store_dir: /data/jetstream   # back this with a persistent volume
  max_memory_store: 1GB
  max_file_store: 20GB
}

# RabbitMQ - declare quorum queues, not classic mirrored
rabbitmq.queues.declare({
  durable: true,
  arguments: { 'x-queue-type': 'quorum' }  # not x-ha-policy
})

Config Snippets That Took Me Hours to Get Right

RabbitMQ Quorum Queue with Dead-Letter Exchange

The thing that got me the first time was declaring the dead-letter exchange before the quorum queue — RabbitMQ will silently accept a queue declaration that references a non-existent DLX, then just drop dead-lettered messages into the void. Here's the full Python pika block that actually works, including the TTL and overflow policy I always want in production:

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()

# Declare DLX first — order matters or messages silently vanish
channel.exchange_declare(
    exchange="orders.dlx",
    exchange_type="direct",
    durable=True
)

# Dead-letter queue bound to the DLX
channel.queue_declare(
    queue="orders.dead",
    durable=True,
    arguments={"x-queue-type": "classic"}  # DLQ doesn't need quorum
)

channel.queue_bind(
    queue="orders.dead",
    exchange="orders.dlx",
    routing_key="orders.dead"
)

# The actual quorum queue with DLX wired in
channel.queue_declare(
    queue="orders.processing",
    durable=True,
    arguments={
        "x-queue-type": "quorum",
        "x-dead-letter-exchange": "orders.dlx",
        "x-dead-letter-routing-key": "orders.dead",
        "x-delivery-limit": 5,          # quorum-specific: max redelivery attempts
        "x-overflow": "reject-publish",  # backpressure instead of silent drops
        "x-max-length": 100000
    }
)

connection.close()

The gotcha with Docker: set RABBITMQ_ERLANG_COOKIE to the same value across all nodes or your cluster will never form — nodes just refuse to connect with no helpful error in the logs. Also, RABBITMQ_DEFAULT_VHOST must be explicitly set if you're doing anything beyond the default /, because pika's default connection won't match your app's vhost and you'll get an obscure ACCESS_REFUSED error on publish.

NATS JetStream 3-Node Cluster — The server.conf That Actually Works on v2.10.x

NATS documentation shows you the routes config, but never shows you the full working file with JetStream storage limits alongside cluster config. I burned two hours discovering that max_file_store is per-server, not per-cluster — so size it accordingly. This is the exact config I use for all three nodes (change server_name and the listen/routes addresses per node):

# node1/server.conf — replicate with node2/node3 changing server_name + routes

server_name: "nats-node1"
listen: "0.0.0.0:4222"
http: "0.0.0.0:8222"   # monitoring endpoint — don't skip this

jetstream {
  store_dir: "/data/jetstream"
  max_mem_store: 1GB
  max_file_store: 10GB    # per-node limit, not cluster-wide
}

cluster {
  name: "pipeline-cluster"
  listen: "0.0.0.0:6222"
  routes: [
    "nats-route://nats-node2:6222"
    "nats-route://nats-node3:6222"
  ]
}

# Raise these — defaults are embarrassingly low for any real pipeline
max_connections: 10000
max_payload: 8MB
write_deadline: "10s"

Then create your stream with replication explicitly set — JetStream defaults to R1 (no replication) even in a cluster, which defeats the entire point:

nats stream add ORDERS \
  --subjects "orders.>" \
  --storage file \
  --replicas 3 \
  --max-msgs 5000000 \
  --max-bytes 5GB \
  --max-age 72h \
  --retention limits \
  --discard old \
  --server nats://nats-node1:4222

The Docker gotcha here: NATS_SERVER_NAME does nothing. You must set server_name in the config file or pass --name as a CLI flag. Relying on the env var means all your nodes share the same name, the cluster forms, but then behaves unpredictably under failover.

Redpanda Single-Broker for Local Dev with Kafka-Compatible Listeners

Redpanda's advertised listener setup is the most copy-pasted-wrong config I've seen in Slack threads. The classic mistake: setting advertised_kafka_api to localhost and then wondering why your containerized app can't reach the broker — the broker happily accepts the connection then sends back localhost as the leader address in metadata, which is unreachable from inside another container.

# redpanda.yaml — single broker, local dev, works with existing Kafka clients

redpanda:
  data_directory: /var/lib/redpanda/data
  empty_seed_starts_cluster: true

  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: internal
    - address: 0.0.0.0
      port: 9093
      name: external

  advertised_kafka_api:
    - address: redpanda          # Docker service name — not localhost
      port: 9092
      name: internal
    - address: localhost         # for connecting from your host machine
      port: 9093
      name: external

  admin_api:
    - address: 0.0.0.0
      port: 9644

  developer_mode: true           # disables fsync — never use in production
  auto_create_topics_enabled: true

rpk:
  kafka_api:
    brokers:
      - localhost:9093           # rpk itself uses the external listener
  admin_api:
    addresses:
      - localhost:9644

The environment variable that trips everyone up: REDPANDA_ADVERTISE_KAFKA_ADDRESS looks like it should override advertised_kafka_api, but on versions before 23.2 it's flat-out ignored when a config file is also mounted. Either use the config file or the env var — mixing them means one silently wins and you don't know which.

The One Environment Variable Per Tool That Bites Everyone

• RabbitMQ: RABBITMQ_ERLANG_COOKIE — must be identical across all nodes. If it's missing from even one node, clustering fails silently on older versions (pre-3.12 just logs "nodedown" with no explanation).
• NATS: JS_DEFAULT_REPLICAS doesn't exist — it's a trap people expect to work. You must set replicas per-stream at creation time. No env var shortcut exists.
• Redpanda: REDPANDA_DEVELOPER_MODE — setting this to true in prod-like environments disables all fsync calls. Data survives restarts in most cases, but you will lose messages on a hard crash and there's no warning in the broker logs that it's running in this mode.
• All three on Docker: memory limits without matching the broker's own heap settings will get you OOM-killed with no useful error. Set RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-rabbit vm_memory_high_watermark 0.6", cap NATS JetStream mem store below the container limit, and for Redpanda set --memory in the rpk flags or the process will try to claim 80% of host RAM.

What I'd Tell Myself Before Starting This Journey

The thing that would have saved me the most time: I spent two weeks tuning RabbitMQ throughput before realizing my Python consumers were the actual bottleneck. They were doing synchronous database writes inside the message handler. The broker was sitting at 4% CPU while my consumer lag grew. Profile your consumer processing time, check your DB connection pool exhaustion, look at your downstream API latency — do all of that before you touch a single broker config knob.

Every tool covered here — Redpanda, RabbitMQ, NATS JetStream, Pulsar — exposes Prometheus metrics out of the box or with minimal config. Redpanda has a /metrics endpoint on port 9644 by default. RabbitMQ needs the rabbitmq_prometheus plugin enabled, which is one command:

# Enable the plugin, then metrics are available at :15692/metrics
rabbitmq-plugins enable rabbitmq_prometheus

# Verify it's live
curl -s http://localhost:15692/metrics | grep rabbitmq_queue_messages_ready | head -5

The mistake I see constantly is teams wiring up observability after their first production incident. By then you're flying blind during the postmortem. Stand up a Grafana dashboard on day one, even if it's ugly. The community dashboards for all four of these brokers are on grafana.com — search by tool name and import the dashboard JSON directly. You want consumer lag, publish rate, and DLQ depth on a single screen before you deploy anything to production.

HA and disaster recovery are not interchangeable, and confusing them will burn you. HA means your pipeline keeps processing if one node dies. DR means you can recover your data and resume processing after a datacenter-level failure, potentially with an RPO measured in minutes or hours. Redpanda with 3-node replication handles HA. DR requires cross-region replication, backup snapshots, and a tested runbook. I've watched teams declare their setup "highly available" because they had a replica, then lose 40 minutes of messages when their cloud provider's AZ went dark because they had no cross-AZ consumer group configured. Decide which one you actually need on day one, because the architecture looks different.

Never migrate brokers under production load. I cannot stress this enough. Before you flip DNS from your old broker to the new one, run these specific tests in staging with production-volume traffic replayed from your existing logs:

• Consumer group rebalancing: kill one consumer instance while the group is processing at peak throughput. Measure how long partition reassignment takes and whether any messages get double-processed.
• Message replay: reset consumer offsets to 2 hours ago and replay. Watch for duplicate processing, ID collisions, and whether your idempotency keys actually hold up.
• DLQ routing: intentionally poison 5% of messages and confirm they land in the dead-letter queue, not silently dropped. Then verify your DLQ consumer can drain and reprocess them.
• Graceful shutdown: send SIGTERM to a consumer mid-batch. Check whether in-flight messages get requeued or lost.

If any of those four tests reveals a surprise in staging, it would have been a production incident. The DNS flip is five seconds of work; the testing is two days. Do the two days.

---

Disclaimer: This article is for informational purposes only. The views and opinions expressed are those of the author(s) and do not necessarily reflect the official policy or position of Sonic Rocket or its affiliates. Always consult with a certified professional before making any financial or technical decisions based on this content.

---

Originally published on techdigestor.com. Follow for more developer-focused tooling reviews and productivity guides.