Celery Tasks on RabbitMQ 4.0: Classic Queues vs Quorum Queues

When building distributed systems with Celery and RabbitMQ, one of the most common pitfalls is tasks disappearing. This usually happens in clustered setups when using classic queues. RabbitMQ 4.x introduced some changes and new queue types, which affects Celery users in production. In this post, we'll explore:

• How RabbitMQ cluster connections work
• Differences between classic and quorum queues
• Limitations of each queue type
• Why tasks sometimes disappear on classic queues in a 3-node cluster
• How to fix these issues

---

RabbitMQ Cluster Architecture (n/2 + 1)

A RabbitMQ cluster is a peer-to-peer network of nodes where each node can accept connections and coordinate with other nodes. In production, cluster operations often rely on a majority quorum rule:

Minimum nodes to operate = floor(n/2) + 1

For example, in a 3-node cluster:

• n = 3
• (3 / 2) + 1 = 2 nodes
• At least 2 nodes must be up and reachable for the cluster to accept writes or form a majority.

This ensures consistency and reliability, especially for replicated queues.

RabbitMQ 3-node cluster diagram

        +------------+
        | Node A     |
        | (rabbitmq1)|
        +------------+
           |    \
           |     \
           |      \
        +------------+        +------------+
        | Node B     |--------| Node C     |
        | (rabbitmq2)|        | (rabbitmq3)|
        +------------+        +------------+

• Each node is a disc node (stores persistent queues).
• Messages in classic queues are stored on the node where the queue is declared.
• Cluster nodes maintain peer-to-peer links; (n/2)+1 nodes are needed to maintain consistency in operations like leader election.

---

Classic Queues in RabbitMQ 4.x

Classic queues are the original queue type in RabbitMQ:

• Messages are stored on the node where the queue is created
• By default, they are not replicated across nodes (unless mirrored, but mirrored queues are deprecated and removed from 4.x ((https://www.rabbitmq.com/docs/3.13/ha)[Read more]))
• Suitable for single-node setups or non-critical workloads

What changed in RabbitMQ 4.0

• classic_mirrored_queue_version feature is removed in 4.0
  • This means old mirrored queues are deprecated
  • Classic queues are now always node-local unless manually mirrored via legacy options

Why tasks disappear in a classic queue

In a 3-node cluster, consider a task published to a classic queue:

• Queue lives on Node A
• Cluster requires (n/2)+1 = 2 nodes to operate correctly
• If Node A becomes temporarily unreachable (network glitch, maintenance), Celery can still publish tasks to Node B
• Node B cannot deliver the task because queue master is on Node A
• As a result, tasks appear lost, especially in short outages

Example scenario

Producer publishes task -> Node B
Queue master       -> Node A
Cluster majority    -> 2 nodes required (A + B)
Node A unavailable for short time
Result: Task cannot be delivered, disappears

The main reason for disappearing tasks is that classic queues exist on a single node, while the cluster is enforcing majority (n/2)+1. If the node hosting the queue is unreachable, even briefly, the task cannot be routed to the master.

---

Quorum Queues

Quorum queues were introduced to replace mirrored queues and solve the reliability problems of classic queues in clusters.

How quorum queues work

• Messages are replicated across multiple nodes
• Writes require majority acknowledgment (consensus) to be considered successful
• Safe against node failures, because tasks exist on multiple nodes

Quorum queue diagram in 3-node cluster

             +----------------------+
             |      Task Producer    |
             +----------+-----------+
                        |
                        v
             +----------------------+
             | Quorum Queue Leader   |
             |       Node A         |
             +----------+-----------+
                        |
           -------------------------------
           |                             |
           v                             v
+----------------------+       +----------------------+
| Quorum Queue Follower |       | Quorum Queue Follower |
|       Node B          |       |       Node C          |
+----------------------+       +----------------------+

• A task is only removed when leader + majority followers confirm
• Even if Node A goes down, Node B or C can continue serving the queue

---

Limitations of quorum queues

Limitation	Explanation	How to mitigate
Slower than classic queues	Each write requires majority acknowledgment	Use quorum queues only for critical tasks
No priorities	Quorum queues do not support task priorities	Use separate queues for priority handling
Memory pressure	Replication consumes memory on all nodes	Monitor memory and adjust ram nodes if needed
Prefetching	Workers must fetch 1 task at a time for reliability	worker_prefetch_multiplier = 1 in Celery

---

Celery configuration for quorum queues

broker_url = "amqp://admin:admin@rabbitmq1:5672//"

task_default_queue = "celery"
task_default_queue_type = "quorum"

task_acks_late = True
task_reject_on_worker_lost = True
worker_prefetch_multiplier = 1

• Ensures tasks are not lost even if nodes fail
• Compatible with a RabbitMQ cluster
• Works reliably with countdown/ETA tasks when paired with separate queues

---

Classic vs Quorum: Summary Table

Feature	Classic Queue	Quorum Queue
Persistence	Node-local, disk	Node-local, replicated to majority
Reliability	Low in clusters	High, survives node failures
Supports ETA / countdown	Yes	Yes
Priorities	Yes	No
Throughput	High	Lower than classic
Recommended for production	❌ Only small setups	✅ Critical tasks

---

Key Takeaways

1. Classic queues are node-local in RabbitMQ 4.0, so tasks can disappear if the master node is unavailable.
2. RabbitMQ clusters enforce (n/2)+1 majority for operations, which interacts poorly with classic queues on a single node.
3. Quorum queues replicate tasks across nodes, making task loss nearly impossible.
4. Celery settings like task_acks_late and worker_prefetch_multiplier must be tuned with quorum queues.
5. Countdown / ETA tasks should be isolated in separate queues to avoid blocking main workers.

---

Recommended Production Setup (3-node cluster)

Celery workers -> connect to all 3 nodes
Default queue -> Quorum queue
Countdown queue -> Quorum queue, separate worker
Classic queues -> Only for dev or non-critical workloads

This configuration ensures tasks are never lost, even during node failure or cluster partitions.

This setup is robust and production-ready for Celery + RabbitMQ 4.0 clusters.