Dead letters in the RabbitMQ management UI: what it can do, and where it bites

Originally published at warrenops.io.

Every RabbitMQ operator has the management UI open, and when a dead-letter queue fills up, it is the first place anyone looks. Each queue page has a small corner for messages: Get messages, Move messages, Publish message and Purge. It is enough for a lot of days. It also has a few defaults and side effects that are easy to miss in the middle of an incident. Here is what each button does, measured on RabbitMQ 4.3.6 with classic and quorum queues.

The same task in the management UI and in Warren, side by side (2:02, captions, no sound needed):

Get messages: four ack modes, one safe default

Get messages fetches messages with basic.get and then settles them according to the Ack Mode you pick. The form starts at one message, and payloads are cut at 50,000 bytes.

Ack Mode	What happens to the message
Nack message requeue true (default)	put back, marked redelivered
Automatic ack	removed from the queue
Reject requeue true	put back, marked redelivered
Reject requeue false	removed, or dead-lettered again if the queue has a dead-letter exchange

The two "put back" modes look the same. On a classic queue they are: both put the message back where it was. On a quorum queue they are not:

quorum DLQ, three messages m1 m2 m3, then one Get messages of m1:
  Nack message requeue true   -> m1 m2 m3, x-delivery-count of m1 unchanged
  Reject requeue true         -> m2 m3 m1, x-delivery-count of m1 + 1

Measured on 4.3.6, also with plain AMQP: basic.nack with requeue=true does not count against a quorum queue's delivery limit, basic.reject with requeue=true does, and it moves the message to the back. Every Reject requeue true spends one delivery of the message's budget.

The two "removed" modes are the dangerous ones on a dead-letter queue. A DLQ usually has no dead-letter exchange of its own, so Reject requeue false does not move the message anywhere: it is gone, exactly like Automatic ack.

Use the default. Nack message requeue true is the only mode that neither removes a message nor costs it a delivery.

Quorum dead-letter queues on RabbitMQ 4: look before you peek

Even the default has one trap. Since RabbitMQ 4.0 every quorum queue has a delivery limit of 20 unless you set one, and a message that a quorum work queue dead-letters for delivery_limit arrives in a quorum DLQ with its count already at 21. On 4.3.6, a message whose count is above the queue's limit is dropped the moment it is put back, and Get messages with requeue puts it back. One look and it is gone.

Before anyone opens Get messages on a quorum DLQ, take the limit off:

rabbitmqctl set_policy dlq-no-limit '\.dlq$' '{"delivery-limit": -1}' --apply-to quorum_queues

The quorum article has the measurements and two other fixes.

Reading why a message died

Get messages shows the headers of each message as nested tables. The x-death table is the incident report: the queue the message died in, the reason (rejected, expired, maxlen, delivery_limit), how often, and the exchange and routing keys it was originally published with. The x-death article goes through it field by field.

What the UI does not do is look across messages. With five messages you read them one by one. With 300 messages and three different exceptions, you are scrolling a page of raw payloads and nested tables, looking for the dozen that are different. There is no search inside a queue and no grouping.

Move messages: one queue in, one queue out

Move messages appears on the queue page once the rabbitmq_shovel and rabbitmq_shovel_management plugins are enabled. It asks for one thing, a destination queue, and creates a dynamic shovel named Move from <queue> with these settings:

src-delete-after: queue-length    # move what is there now, then delete the shovel
ack-mode:         on-confirm      # a message leaves the source once the destination has it
dest-queue:       <what you typed>
dest-add-forward-headers: false

That makes it a safe move: nothing is lost on the way, and the shovel removes itself when it is done. What it does not do matters more for dead letters:

• Everything goes to one queue. A DLQ that collects from several queues or routing keys cannot be sent back to where each message came from. The original exchange and routing key are in each message's x-death; Move messages does not read them.
• All or nothing. It moves every message that is in the queue when it starts. There is no selection: the twelve poison messages go along with the 288 that only need a second try.
• The death headers travel along. In our test the moved messages arrived with x-death, x-first-death-*, x-last-death-* and, from a quorum queue, the old x-delivery-count as a plain header. A consumer that reads the x-death count to decide when to give up sees the old count on arrival. If yours does, strip the headers when you replay instead.
• No pacing. The shovel moves as fast as the broker lets it, with up to 1,000 messages in flight. A consumer that just recovered gets the whole backlog at once.

When the destination is the one queue the messages should go to, and you trust all of them, it is the right button.

Publish message: replaying one message by hand

To send a single message back to its route, people copy the payload from Get messages into Publish message on the exchange. Two things get lost on the way. The form sets properties and headers you type in, and its own help says it: "Only long string headers can be set here." Numbers, booleans and nested headers do not survive. And the original stays in the DLQ until you remove it with a second, destructive Get messages. Copy, publish, check, then remove, in that order.

Purge: final

Purge deletes every message in the queue. The UI asks once, in its own words: "Are you sure? Messages cannot be recovered after purging." There is no copy, no list of what was in the queue and no record of who did it. If a queue might hold anything you need later, export it first (Get messages with Nack message requeue true and a high count, then save the page, or a script).

Who did what

The management UI keeps no record of messages fetched, moved, published or purged. If the postmortem asks who replayed what at 3:12, the answer is in someone's shell history or nowhere.

Checklist

• Get messages: keep the default, Nack message requeue true. Reject requeue true costs a delivery on a quorum queue; Automatic ack and Reject requeue false remove the message.
• Quorum DLQ on RabbitMQ 4: set delivery-limit: -1 before the first peek.
• Move messages: one destination queue, all messages, death headers included. Fine for a straight move, wrong for "back to where each one came from".
• Publish message: string headers only; remove the original afterwards, not before.
• Purge: export first if anything in the queue could matter.

---

Where Warren fits

Warren covers the message corner of the management UI for dead-letter work, and leaves configuration to it. It peeks with a requeue that costs nothing and asks before it reads a quorum queue where a peek could drop messages; it groups a DLQ by exception and searches it; it replays selected messages to the exchange and routing key in each one's x-death, with the death headers stripped, throttled and stoppable; it keeps a copy of discarded and purged messages for seven days, and an audit log of who did what. Free for one broker. A side-by-side table is on Warren vs. the RabbitMQ management UI.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation