Your quorum dead-letter queue can drop the dead letters: x-delivery-count travels along

Originally published at warrenops.io.

A dead-letter queue is supposed to be the place where failed messages are safe until someone looks at them. On RabbitMQ 4 with quorum queues and default settings, that is not quite true: a message that died because it was delivered too often can arrive in the DLQ with its delivery budget already used up, and the first consumer or tool that crashes while holding it, or even just peeks at it, makes RabbitMQ drop it for good. Here is what happens, measured on RabbitMQ 4.3.6, and three ways to make the DLQ safe again.

The short version as a video (1:42, captions, no sound needed):

Delivery limits in 30 seconds

Quorum queues count how often a message was delivered and not settled. Past the queue's delivery limit (x-delivery-limit, or delivery-limit in a policy) RabbitMQ stops redelivering it: the message is dead-lettered with reason delivery_limit if the queue has a dead-letter exchange, and dropped otherwise. Since RabbitMQ 4.0 every quorum queue has a limit of 20 unless you set one, which is good: a poison message no longer loops forever.

What counts changed in 4.0 as well:

What happens to a delivered message	3.13	4.x
basic.nack / basic.reject with requeue=true	counts	does not count
management UI "Get messages" with requeue	counts	does not count
the channel or connection closes while it is unacknowledged (consumer crash, pod killed, network drop)	counts	counts
basic.reject with requeue=false (dead-lettered)	not measured	counts
put back (any of the above, or a peek with requeue) while its count is already above the queue's limit	not measured	dropped, or dead-lettered on
put back (any of the above, or a peek with requeue) while its count is already above the queue's limit	not measured	dropped, or dead-lettered on

The count is visible on the message as the header x-delivery-count (RabbitMQ 4 adds x-acquired-count, every acquisition including requeues).

What we measured

A work queue and a dead-letter queue, both quorum queues, no limits set, so both have RabbitMQ 4's default of 20:

ch.exchange_declare('lim.dlx', 'fanout', durable=True)
ch.queue_declare('lim.dlq', durable=True, arguments={'x-queue-type': 'quorum'})
ch.queue_bind('lim.dlq', 'lim.dlx')
ch.queue_declare('lim.work', durable=True,
                 arguments={'x-queue-type': 'quorum', 'x-dead-letter-exchange': 'lim.dlx'})
ch.basic_publish('', 'lim.work', b'order-4711')

Then a consumer that crashes on this message every time: fetch it without acknowledging and close the connection, until the work queue gives up.

while message_count('lim.work') > 0:
    conn = pika.BlockingConnection(params)
    conn.channel().basic_get('lim.work', auto_ack=False)
    conn.close()          # the consumer "crashed" while holding the message

The results:

abandoned deliveries on lim.work until it gave up: 21
in lim.dlq: reason delivery_limit, x-delivery-count 21
left in lim.dlq after ONE abandoned delivery there: 0

The message reached the DLQ, as it should, with reason delivery_limit. But it brought its count of 21 along, and the DLQ's own limit is 20. The next abnormal return, a single one, made the DLQ give up on it too, and since the DLQ has no dead-letter exchange of its own, the message was dropped.

It does not even take a crash. In a second run we only looked: one basic.get followed by basic.nack with requeue=true, and once more through the management UI's "Get messages" with requeue. Both times the DLQ was empty afterwards. On RabbitMQ 4.3.6 a message whose count is already above the queue's limit is dropped the moment it is put back, abnormally or not; a count equal to the limit survives. So the usual first step of an investigation, peeking at the DLQ, is what loses the message.

It does not even take a crash. In a second run we only looked: one basic.get followed by basic.nack with requeue=true, and once more through the management UI's "Get messages" with requeue. Both times the DLQ was empty afterwards. On RabbitMQ 4.3.6 a message whose count is already above the queue's limit is dropped the moment it is put back, abnormally or not; a count equal to the limit survives. So the usual first step of an investigation, peeking at the DLQ, is what loses the message.

Why the count travels

The delivery count is not just a header the queue writes for you to read: it is part of the message's annotations, and dead-lettering keeps them. The DLQ continues counting from where the work queue stopped. A header that a publisher sets is a different thing: we published a fresh message with x-delivery-count: 5 into a queue with a limit of 5, and RabbitMQ ignored it. The next delivery showed 1, and the message survived five abandoned deliveries. So the carried count comes from dead-lettering, not from the header itself.

It is not only delivery_limit messages. In the same test with explicit limits, a message that had been abandoned twice and then rejected arrived in the DLQ with a count of 3, and with a DLQ limit of 5 it was dropped after three more failures, not five. Every message rejected by a consumer arrives with at least 1.

Who abandons a delivery in a DLQ?

More often than you would think:

• a consumer on the DLQ that processes dead letters (an alerting bridge, an archiver, a "retry later" service) and crashes or is redeployed while holding messages with prefetch;
• a script that reads the DLQ with basic.get and dies, or is stopped with Ctrl+C, before it acknowledges or nacks;
• a replay tool whose connection drops in the middle of a run;
• a pod eviction or a network blip during any of the above.

On 3.13 it was worse (every requeue counted, so even looking at messages in the management UI cost deliveries), but 4.x's default limit makes the carry-over bite with no configuration at all.

Three fixes

All three kept the message through three abandoned deliveries in the DLQ in our test.

1. No delivery limit on the DLQ. A DLQ is not consumed in a loop, so it does not need one. By policy:

rabbitmqctl set_policy dlq-no-limit '\.dlq$' '{"delivery-limit": -1}' --apply-to quorum_queues

or with x-delivery-limit: -1 when you declare the queue; a negative value means unlimited. Only one policy applies to a queue: if one matches your DLQs already, add the key there.

2. A much higher limit, if you want a backstop anyway: x-delivery-limit: 100.

3. A classic queue as DLQ. Classic queues have no delivery limit. You lose the replication of a quorum queue, which matters less for a queue that is mostly written and rarely read; decide by how much you care about dead letters surviving the loss of a node.

And in whatever reads the DLQ: settle every message you fetch, ack what you have handled and nack with requeue=true what you have not. On 4.x a requeue costs nothing, except for a message that is already above the queue's limit: the requeue drops it as well. Fix the limit before anyone reads the queue.

Checking your own queues

• Do not peek first. Check the queue: rabbitmqctl list_queues name type arguments policy shows which DLQs are quorum queues and whether a limit is set. A quorum DLQ with a limit can lose messages to the peek itself; set delivery-limit: -1 first, then look.
• After that, x-delivery-count next to x-death on a message tells you how much of its budget it used before it died.
• No x-delivery-limit and no delivery-limit policy means 20 on RabbitMQ 4.
• Replaying the message, by publishing it anew to its original exchange (a shovel does that too), gives it a fresh count. Dead-lettering it once more into another quorum queue does not.

Checklist

• Quorum DLQs: delivery-limit: -1 by policy, or classic DLQs.
• Consumers and scripts on DLQs settle every message they fetch, also on shutdown.
• Take the limit off quorum DLQs before anyone peeks at them: on 4.x a peek drops what arrived above the limit.
• Replay to reset the budget; dead-lettering a message on into another quorum queue keeps the count.

---

Where Warren fits

Warren shows each message's remaining budget in a quorum DLQ ("1 of 20 left") and warns on the queue page when dead letters are one or two failed deliveries away from being dropped, saying whether RabbitMQ would drop them or dead-letter them again. Its replays publish the messages anew with a fresh count and strip the stale x-delivery-count, and it reads nothing from a quorum queue where that can cost messages until you confirm: on 3.13 any quorum queue with a limit, on 4.x a quorum dead-letter queue with a limit, whose over-limit arrivals a peek would drop. Free for one broker.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation