DEV Community

Cover image for Messaging Systems: MQTT vs AMQP vs RabbitMQ vs Kafka vs NATS vs Apache Pulsar
Shiv Rai (S_RAI)
Shiv Rai (S_RAI)

Posted on AI-assisted

Messaging Systems: MQTT vs AMQP vs RabbitMQ vs Kafka vs NATS vs Apache Pulsar

#webdev #api #systemdesign #tutorial

What are messaging systems?

Messaging systems enable applications to communicate by exchanging messages through shared infrastructure rather than communicating directly with each other.

In direct communication models, a sender typically expects the receiver to be available and responsive at the time of the interaction. Messaging systems remove this dependency by introducing intermediaries that can receive, store, route, and deliver messages independently of the sender and receiver.

This approach exists because distributed systems are inherently asynchronous. Services fail, networks become unreliable, workloads spike unexpectedly, and systems evolve independently. Messaging infrastructure helps absorb these realities by decoupling producers from consumers.

Common building blocks include:

  • Brokers that receive, route, and deliver messages
  • Queues that distribute work among consumers
  • Topics that publish messages to multiple subscribers
  • Consumers that process messages independently of producers

Common use cases include:

  • Background job processing
  • Microservice communication
  • Event distribution
  • Data pipelines
  • IoT telemetry
  • Workflow orchestration
  • Stream processing
  • System integration

This article covers: MQTT, AMQP, RabbitMQ, Kafka, NATS, Apache Pulsar

Understanding the Technologies

Not all technologies in this category solve the same problem or exist at the same architectural layer. Some define how messages are transmitted, while others provide the infrastructure that stores, routes, and delivers those messages.

Technology Category Primary Focus
MQTT Messaging Protocol Lightweight communication for constrained devices and unreliable networks
AMQP Messaging Protocol Standardized broker-based messaging and routing
RabbitMQ Message Broker Traditional queue-based messaging and work distribution
Kafka Event Streaming Platform Durable event streams and large-scale data pipelines
NATS Messaging System Lightweight, low-latency messaging and service communication
Apache Pulsar Messaging & Streaming Platform Unified messaging and event streaming infrastructure

Some choices are complementary rather than competing.

For example:

  • MQTT messages are often transported through MQTT brokers.
  • RabbitMQ commonly implements AMQP concepts and protocols — specifically the AMQP 0-9-1 dialect, not the later OASIS-standardized AMQP 1.0. AMQP is the protocol specification; RabbitMQ is one concrete, operable broker that implements it.
  • Kafka and Pulsar are typically chosen for event-streaming workloads rather than traditional work queues.
  • NATS focuses on lightweight messaging rather than long-term event storage.

Decision Tree

  • MQTT is typically chosen when devices operate on unreliable, low-bandwidth, or battery-constrained networks.
  • AMQP is a protocol standard rather than a product you deploy directly; in practice, most teams get AMQP-style messaging by choosing RabbitMQ, which implements the widely-adopted AMQP 0-9-1 dialect.
  • RabbitMQ is often the default choice for traditional queues, task distribution, and workflow orchestration.
  • Kafka is most compelling when messages become long-lived streams of data that must be replayed and processed repeatedly.
  • NATS is optimized for lightweight, low-latency service communication.
  • Pulsar becomes attractive when both messaging and streaming capabilities are needed within the same distributed platform. 
flowchart TD

A[Need Messaging Infrastructure] --> B{Communicating with<br/>devices over unreliable<br/>or constrained networks?}

B -->|Yes| MQTT[MQTT]

B -->|No| C{Need durable event streams,<br/>replay, and data pipelines?}

C -->|Yes| D{Need large-scale multi-tenancy,<br/>geo-distribution, or both messaging<br/>and streaming in one platform?}

D -->|Yes| PULSAR[Apache Pulsar]
D -->|No| KAFKA[Kafka]

C -->|No| E{Need traditional queues,<br/>routing, and work distribution?}

E -->|Yes| RABBIT[RabbitMQ<br/>implements AMQP 0-9-1]

E -->|No| G{Need lightweight,<br/>low-latency service messaging?}

G -->|Yes| NATS[NATS]
G -->|No| RABBIT

Comparison

Aspect MQTT AMQP RabbitMQ Kafka NATS Apache Pulsar
Category Protocol Protocol Broker Platform Event Streaming Platform Messaging System Messaging & Streaming Platform
Primary Model Pub/Sub messaging Broker-based messaging standard Queues and routing Distributed event log Lightweight messaging and pub/sub Unified messaging and streaming
Messaging vs Streaming Messaging Messaging Messaging Streaming-first Messaging Messaging + Streaming
Message Persistence Optional Protocol-dependent Strong persistence support Core design principle Optional Core design principle
Delivery Guarantees Configurable QoS levels Protocol-defined guarantees Strong acknowledgment model Strong durability and replay Configurable depending on deployment mode Strong durability and acknowledgment model
Throughput Profile Device-oriented workloads Moderate Moderate Very high High Very high
Latency Profile Low Moderate Low to moderate Low Very low Low
Scalability Model Device fleets Broker-dependent Clustered broker scaling Distributed partitions Distributed messaging fabric Distributed storage and brokers
Ordering Model Limited and workload-dependent Implementation-dependent Queue ordering Partition ordering Subject ordering Partition ordering
Operational Complexity Low Varies by implementation Medium High Low High
Ecosystem Maturity Mature Mature standard Very mature Very mature Mature Mature
Typical Use Cases IoT, telemetry, edge devices Enterprise interoperability Background jobs, workflows, task queues Event streaming, analytics, CDC, data platforms Service communication, cloud-native systems Multi-tenant platforms, messaging and streaming
Strengths Efficient on constrained networks Standardized interoperability Mature queueing and routing model Replayability, durability, ecosystem, scale Simplicity, speed, low operational overhead Combines queueing and streaming capabilities
Weaknesses Not intended for large-scale stream processing Requires a broker implementation Less suited for event-streaming workloads More complex than traditional messaging systems Not designed as a long-term event log More operationally complex than simpler alternatives

Messaging vs Streaming

If you need... Usually Start With
Background jobs and work queues RabbitMQ
Task distribution RabbitMQ
Enterprise broker-based messaging RabbitMQ / AMQP
Device communication MQTT
Service-to-service messaging with minimal latency NATS
Durable event streams and replay Kafka
Unified messaging and streaming Pulsar
  • If messages are consumed once and discarded -> Messaging
  • If messages become a durable history that multiple consumers may replay-> streaming

MQTT

MQTT (Message Queuing Telemetry Transport) was created in 1999. At the time, many devices operated under severe constraints:

  • Oil Pipeline Sensors
  • Satellite Links
  • Industrial Equipment
  • Remote Monitoring Systems

These environments often had:

  • High latency
  • Expensive bandwidth
  • Intermittent connectivity
  • Low-power hardware

HTTP was considered too heavy. The goal of MQTT was:

  • Minimal Bandwidth
  • Minimal Power Usage
  • Minimal Complexity

MQTT thinks in topics and subscriptions

The model breaks down into:

  • Publishers
  • Topics
  • Subscribers

Instead of:

Device A ---> Device B
Enter fullscreen mode Exit fullscreen mode

you get:

Device A
    |
 Publish
    |
temperature/room1
    |
 MQTT Broker
    |
 Subscribe
    |
Device B
Enter fullscreen mode Exit fullscreen mode

Example

Publisher

client.publish(
  "temperature/room1",
  "24.5"
)
Enter fullscreen mode Exit fullscreen mode

Subscriber

client.subscribe(
  "temperature/room1"
)
Enter fullscreen mode Exit fullscreen mode

Whenever a message arrives:

def on_message(msg):
    print(msg.payload)
Enter fullscreen mode Exit fullscreen mode

Output:

24.5
25.1
24.8
Enter fullscreen mode Exit fullscreen mode

Quality of Service (QoS)

MQTT lets each subscription choose its own delivery guarantee:

  • QoS 0 – at most once (fire and forget)
  • QoS 1 – at least once (may deliver duplicates)
  • QoS 2 – exactly once (highest overhead)

Constrained devices often default to QoS 0 or 1 to save bandwidth and battery, reserving QoS 2 for messages that can't tolerate loss or duplication.

Pros and Cons

Pros Cons
Extremely lightweight Requires broker infrastructure
Excellent for IoT Poor fit for request/response APIs
Low bandwidth usage Topic design can become complex
Supports unreliable networks Harder to debug than HTTP
QoS delivery guarantees Security must be carefully managed
Publisher/subscriber decoupling Not ideal for CRUD applications
Huge industry adoption Less suitable for large analytics pipelines

AMQP

AMQP (Advanced Message Queuing Protocol) was first developed around 2003–2006. A stable specification shipped in 2008, with formal standardization through OASIS following in 2012. Large enterprises were building systems that relied heavily on messaging:

  • Banking
  • Trading
  • Payments
  • Order Processing
  • Enterprise Integration

Every vendor had its own protocol which often couldn't communicate with each other. AMQP aimed to create a standard protocol for reliable enterprise messaging.

The goal was:

  • Guaranteed Delivery
  • Reliable Routing
  • Transactions
  • Interoperability

AMQP thinks in a standardized routing model

Core assumption: Routing behavior should be defined by the protocol itself, not by whichever broker happens to run it — so producers, consumers, and brokers from different vendors can all agree on the same messaging semantics.

This is a different mental model from "call a specific broker's API." AMQP is a wire-level specification: it describes exchanges, bindings, and queues as protocol concepts that any compliant broker must implement the same way.

Messages
+
Routing Rules
Enter fullscreen mode Exit fullscreen mode

Example

Producer

channel.basic_publish(
    exchange="orders",
    routing_key="created",
    body="Order #123"
)
Enter fullscreen mode Exit fullscreen mode

Consumer

channel.basic_consume(
    queue="order-service"
)
Enter fullscreen mode Exit fullscreen mode

Producer sends:

Order #123
Enter fullscreen mode Exit fullscreen mode

Consumer receives:

Order #123
Enter fullscreen mode Exit fullscreen mode

Exchanges and Routing

AMQP's "sophisticated" routing comes from exchanges sitting between producers and queues. A producer publishes to an exchange, not directly to a queue, and the exchange decides where the message goes based on its type:

  • Direct – routes by exact routing-key match
  • Topic – routes by pattern match (e.g., orders.*)
  • Fanout – broadcasts to every bound queue
  • Headers – routes based on message header values

This is what lets a single published message reach multiple queues, or be filtered before it ever reaches a consumer.

Pros and Cons

Pros Cons
Very reliable delivery More complex than MQTT
Powerful routing via exchanges Higher operational overhead
Durable queues and acknowledgements Heavier protocol footprint
Supports many messaging patterns Not ideal for large event streams
Mature RabbitMQ ecosystem Infrastructure can become complex
Strong enterprise adoption Less suitable for constrained devices
Good for workflows and background jobs Overkill for simple applications

Kafka

Apache Kafka was created at LinkedIn around 2010 and open-sourced in 2011. LinkedIn's infrastructure was generating massive amounts of events:

Page Views
Clicks
User Activity
Logs
Metrics
Analytics
Enter fullscreen mode Exit fullscreen mode

Different systems needed access to the same data. This created tight coupling and scalability problems. Kafka was designed to answer, "How can thousands of systems continuously produce and consume events at massive scale?" The goal was:

  • Store Events
  • Replay Events
  • Process Events
  • Scale Horizontally

Kafka is not primarily a message queue.

Kafka is an event streaming platform.

Kafka thinks in event logs

Kafka's model centers on immutable event streams.

Core assumption: Events are valuable records that should be stored, not merely delivered.

Instead of:

Producer
   |
Queue
   |
Consumer
   |
Delete Message
Enter fullscreen mode Exit fullscreen mode

Kafka's approach:

Producer
   |
Append Event
   |
Log
   |
Consumer Reads
Enter fullscreen mode Exit fullscreen mode

The event remains in the log.

This is the most important concept in Kafka.

Example

Producer

producer.send(
    "orders",
    {
      "orderId": 123,
      "status": "created"
    }
)
Enter fullscreen mode Exit fullscreen mode

Consumer

for msg in consumer:
    print(msg.value)
Enter fullscreen mode Exit fullscreen mode

Output:

{
  "orderId": 123,
  "status": "created"
}
Enter fullscreen mode Exit fullscreen mode

The key difference: The event remains stored even after consumption. Another consumer can read it later.

Partitions and Consumer Groups

A Kafka topic is split into partitions, each an ordered, append-only log. Partitioning is what lets Kafka scale horizontally — different partitions can live on different brokers and be written and read in parallel.

Consumers read partitions in consumer groups: Kafka assigns each partition to exactly one consumer within a group, so the group as a whole processes a topic in parallel while each individual partition stays strictly ordered.

Pros and Cons

Pros Cons
Massive throughput Operational complexity
Event replay capability Higher resource requirements
Durable event storage Not suited for RPC
Excellent scalability Event schema management can be difficult
Decouples producers and consumers Eventual consistency challenges
Strong ecosystem and tooling Can be overkill for small systems
Ideal for analytics and event-driven systems Steeper learning curve

NATS

NATS was created by Derek Collison and first released around 2011. Around the early 2010s, distributed systems (Microservices, Containers, Cloud Platforms, Service Meshes) were becoming increasingly common. Existing messaging systems often fell into two categories:

  1. RabbitMQ → Feature Rich
  2. Kafka → Massive Scale

But many teams wanted:

Simple
Fast
Reliable
Cloud Native
Enter fullscreen mode Exit fullscreen mode

without operating a large messaging platform.

NATS was designed to answer: What's the simplest possible way for distributed services to communicate?

The philosophy was:

Small Protocol
Small Server
Low Latency
High Simplicity
Enter fullscreen mode Exit fullscreen mode

NATS thinks in subjects

Core assumption: Services should communicate through lightweight, hierarchical subjects rather than pre-declared queues or topics.

Subjects are just dot-separated strings, and wildcards let a subscriber match many subjects at once without the publisher and subscriber agreeing on a fixed topic list in advance:

orders.created     # exact subject
orders.*           # matches orders.created, orders.cancelled, etc.
orders.>           # matches orders.created, orders.us.created, and deeper
Enter fullscreen mode Exit fullscreen mode

Example

Publisher

nc.Publish(
  "orders.created",
  []byte("123")
)
Enter fullscreen mode Exit fullscreen mode

Subscriber

nc.Subscribe(
  "orders.created",
  func(msg *nats.Msg) {
    fmt.Println(
      string(msg.Data)
    )
  },
)
Enter fullscreen mode Exit fullscreen mode

Output:

123
Enter fullscreen mode Exit fullscreen mode

JetStream

By default, core NATS messages are fire-and-forget — if no one is subscribed when a message is published, it's gone. JetStream is NATS's built-in persistence layer: it adds durable storage, message replay, and at-least-once delivery on top of core NATS subjects. It's what lets NATS take on some Kafka-like use cases, at the cost of some of the operational simplicity that makes core NATS attractive in the first place.

Pros and Cons

Pros Cons
Extremely low latency Smaller ecosystem
Operationally simple Less suited for analytics workloads
Lightweight deployment JetStream increases complexity
Excellent for microservices Fewer integrations than Kafka
Supports pub/sub and RPC Smaller talent pool
Cloud-native friendly Not ideal for long-term event storage
Easier to run than Kafka/RabbitMQ Less proven for huge data platforms

Apache Pulsar

Apache Pulsar was originally developed at Yahoo around 2013 and open-sourced in 2016. Yahoo operated some of the world's largest messaging and data systems:

Mail
News
Advertising
Search
Analytics
Enter fullscreen mode Exit fullscreen mode

At that scale, traditional messaging systems began showing limitations. Particularly:

Storage Scaling
Broker Scaling
Multi-Tenancy
Geo Replication
Enter fullscreen mode Exit fullscreen mode

Kafka was already successful, but Yahoo wanted a different architecture. Most messaging platforms looked like:

Broker
  |
Storage
Enter fullscreen mode Exit fullscreen mode

Pulsar introduced:

Broker
  |
Storage Layer
Enter fullscreen mode Exit fullscreen mode

as separate components.

Pulsar thinks in event streams with separated compute and storage

Core assumption: Messaging and storage should scale independently.

Instead of:

Broker
  |
Stores Data
Enter fullscreen mode Exit fullscreen mode

Pulsar:

Broker
  |
Reads/Writes
  |
Storage System
Enter fullscreen mode Exit fullscreen mode

This creates more flexibility at scale. In practice, it means brokers can be added or removed to handle more traffic without touching how data is stored, and storage capacity can grow independently to hold more history — something a coupled broker-and-storage design can't do without rebalancing both at once.

Example

Producer

Producer<String> producer =
    client.newProducer()
        .topic("orders")
        .create();

producer.send(
    "Order Created"
);
Enter fullscreen mode Exit fullscreen mode

Consumer

Consumer<String> consumer =
    client.newConsumer()
        .topic("orders")
        .subscriptionName("billing")
        .subscribe();
Enter fullscreen mode Exit fullscreen mode

Receive:

Message<String> msg =
    consumer.receive();
Enter fullscreen mode Exit fullscreen mode

Looks very similar to Kafka but architecturally different.

Pros and Cons

Pros Cons
Independent compute and storage scaling More complex architecture
Built-in multi-tenancy Smaller ecosystem
Native geo-replication Fewer experienced operators
Supports queues, streams, and pub/sub More moving parts
Durable storage via BookKeeper Higher operational complexity
Strong cloud-native support Kafka tooling ecosystem is larger
Excellent for large organizations Often overkill for smaller systems

RabbitMQ

RabbitMQ was first released in 2007 by Rabbit Technologies and is one of the most widely adopted implementations of AMQP. Specifically, RabbitMQ is built around AMQP 0-9-1, an earlier and more widely deployed dialect than the later OASIS-standardized AMQP 1.0 — this is why RabbitMQ isn't automatically wire-compatible with every "AMQP" broker.

Before RabbitMQ, many applications communicated directly:

Service A
    |
Service B
Enter fullscreen mode Exit fullscreen mode

This created problems:

  • Tight Coupling
  • Retries
  • Downtime Propagation
  • Traffic Spikes

Teams wanted a reliable middle layer that could:

  • Store Messages
  • Route Messages
  • Retry Delivery
  • Guarantee Processing

RabbitMQ answers: How can services exchange messages reliably without needing to be online at the same time?

Instead of:

Producer
    |
Consumer
Enter fullscreen mode Exit fullscreen mode

you get:

Producer
    |
RabbitMQ
    |
Consumer
Enter fullscreen mode Exit fullscreen mode

The broker acts as a buffer and router.

RabbitMQ thinks in exchanges you actually operate

Core assumption: The abstract routing model AMQP describes needs a real, operable broker behind it — with clustering, a management UI, plugins, and tuning knobs. RabbitMQ is that broker: it implements AMQP's exchange-and-queue routing model and adds its own operational tooling, protocol plugins (including MQTT and STOMP support), and ecosystem on top.

Example:

OrderCreated
Enter fullscreen mode Exit fullscreen mode

Producer sends the message.

RabbitMQ decides:

Email Service
Analytics Service
Inventory Service
Enter fullscreen mode Exit fullscreen mode

should receive it.

Example

Producer

channel.basic_publish(
    exchange="orders",
    routing_key="created",
    body="Order #123"
)
Enter fullscreen mode Exit fullscreen mode

Consumer

channel.basic_consume(
    queue="email-service",
    on_message_callback=handler
)
Enter fullscreen mode Exit fullscreen mode

Message:

Order #123
Enter fullscreen mode Exit fullscreen mode

flows through RabbitMQ.

Pros and Cons

Pros Cons
Mature and battle-tested More complex than NATS
Powerful routing via exchanges Not optimized for event streaming
Strong reliability guarantees Scaling can become complex
Excellent worker queue support Lower throughput than Kafka
Durable queues and retries Ordering can be difficult at scale
Large ecosystem and tooling More operational overhead
Supports multiple messaging patterns Not ideal for long-term event retention

Key Takeaway

  • Choose MQTT for device communication.
  • Choose RabbitMQ (the practical way to get AMQP-style messaging) for traditional messaging and queues.
  • Choose NATS for lightweight, low-latency messaging.
  • Choose Kafka for durable event streams and data platforms.
  • Choose Pulsar when messaging and streaming must coexist within the same platform.

Top comments (0)