DEV Community: Viktor

Replay only some RabbitMQ dead letters: by exception, header or routing key

Viktor — Mon, 19 Oct 2026 05:58:30 +0000

Originally published at warrenops.io.

The tax service was down for twenty minutes and is back. orders.dlq holds 230 messages: 41 of them failed with TaxServiceUnavailableException, the rest are missing orders and declined payments that a replay would only send round again. RabbitMQ has no "move the messages where header X equals Y": a queue hands out messages from the head, one after the other, and the management UI's "Move messages" and a shovel take all of them. Here is how to take only the 41, measured on RabbitMQ 4.3.6 with classic and quorum queues.

The loop that never ends

The obvious script reads a message, checks it, and puts the ones it does not want straight back:

method, props, body = ch.basic_get("orders.dlq", auto_ack=False)
if not wanted(props):
    ch.basic_nack(method.delivery_tag, requeue=True)   # do not do this

A message requeued this way goes back to the head of the queue, where it was. The next basic_get returns the same message again, and the script spins on the first unwanted message forever. Classic and quorum queues behave the same.

Hold the ones you skip

The way through the queue is to keep the skipped messages unacknowledged while you read on. As long as a delivery is outstanding on the channel, basic_get hands out the next message. At the end one basic_nack gives all of them back:

import sys, pika

QUEUE = "orders.dlq"
DEATH = ("x-death", "x-first-death-", "x-last-death-", "x-delivery-count")

def wanted(method, headers):
    # your filter; Spring's RepublishMessageRecoverer writes x-exception-*
    exc = headers.get("x-exception-type", "")
    return exc.endswith("TaxServiceUnavailableException")

def route(method, headers):
    if headers.get("x-death"):  # dead-lettered by the broker
        death = headers["x-death"][0]
        return death["exchange"], death["routing-keys"][0]
    # republished by Spring's recoverer
    return headers["x-original-exchange"], headers["x-original-routingKey"]

conn = pika.BlockingConnection(pika.URLParameters(sys.argv[1]))
ch = conn.channel()
ch.confirm_delivery()  # 1.

total = ch.queue_declare(QUEUE, passive=True).method.message_count  # 2.
moved = 0
try:
    for _ in range(total):
        method, props, body = ch.basic_get(QUEUE, auto_ack=False)  # 3.
        if method is None:
            break
        headers = dict(props.headers or {})
        if not wanted(method, headers):
            continue  # 4. hold it, unacked
        exchange, routing_key = route(method, headers)
        props.headers = {k: v for k, v in headers.items()
                         if not k.startswith(DEATH)}
        ch.basic_publish(exchange, routing_key, body, props,
                         mandatory=True)  # raises if not routed
        ch.basic_ack(method.delivery_tag)  # only after the confirm
        moved += 1
finally:
    ch.basic_nack(0, multiple=True, requeue=True)  # 5. hand the rest back
conn.close()
print(f"moved {moved} of {total}")

We ran it against a dead-letter queue with nine messages: six dead-lettered by the broker, three republished by a Spring-style recoverer without x-death. It moved the three that matched to the work queue, without death headers, and left the other six in the dead-letter queue in their original order. Same result on a classic and a quorum queue. With a filter that raised an exception on the fifth message, the one message already moved stayed moved and the other eight were back in the dead-letter queue, their delivery count unchanged.

What each numbered comment is for:

Confirms. With confirm_delivery(), pika's basic_publish waits for the broker and raises if the message was not taken; with mandatory=True it also raises when no queue is bound for the routing key. Ack the original only after that.
A snapshot of the length. Read at most as many messages as the queue held when you started. Without it, a queue that keeps receiving dead letters while you read never runs empty, and the script never stops.
No auto-ack. With auto_ack=True a skipped message is gone the moment you look at it.
Hold, do not nack. The skipped message stays unacknowledged on this channel until step 5.
One nack at the end, also when something fails. basic_nack(0, multiple=True) returns every delivery still outstanding on the channel, in the finally block so that an exception in your filter or a failed publish hands the messages back the same way. The skipped messages go back to the head of the queue in their old order.

Afterwards every message that was skipped is marked redelivered. That is harmless in a dead-letter queue, but worth knowing if a consumer treats the flag as suspicious.

Why not just close the connection?

Closing the channel also returns every unacknowledged message, and many scripts rely on it. On a classic queue that is fine. On a quorum queue it is not the same thing as a nack:

How the skipped messages go back (quorum queue, 4.3.6)	`x-delivery-count`	29 scans of one message, default delivery limit
`basic_nack(requeue=True)`	unchanged	message still there
connection closed with the message unacked	+1 per close	message gone at scan 22

RabbitMQ 4 gives every quorum queue a delivery limit of 20. A return by channel close counts as a failed delivery; an explicit nack does not. Twenty-one scans that end with conn.close(), or twenty-one crashes of the script, and the message is dropped, or dead-lettered further if the dead-letter queue has a dead-letter exchange of its own. On a quorum dead-letter queue that is fed by a quorum queue the margin is smaller still, because the message arrives with the deliveries it already used up: in our test a single reject on the work queue brought it in with x-delivery-count 1, and the quorum DLQ article shows a poison message arriving with 21. So end every scan with the nack, in a finally block as above, and never leave the messages to the channel close.

Filters that work

What you can filter on depends on who put the message into the dead-letter queue:

Dead-lettered by the broker (reject, expiry, queue length): x-death[0] has reason, queue, exchange and routing-keys. The exception is not there, because the broker never saw it. Your own publish headers are kept, so a header like x-tenant or x-order-type works.
Republished by the consumer (Spring's RepublishMessageRecoverer, MassTransit's _error queues): the exception sits in the headers, x-exception-type and x-exception-message for Spring, MT-Fault-ExceptionType for MassTransit, and there is no x-death. Spring keeps the route in x-original-exchange and x-original-routingKey.
The body. Possible, but parse defensively: some dead letters are dead because the body does not parse.

Exception messages usually contain ids ("Order 4711 not found"). Compare the exception type, or the message with digits replaced, not the full text.

Or split them before they mix

If the same question comes up every week, let the broker sort the dead letters as they arrive. A dead-lettered message keeps its routing key unless the queue sets x-dead-letter-routing-key, so a topic exchange as dead-letter exchange can send each kind to its own queue:

ch.exchange_declare("orders.dlx", "topic", durable=True)
ch.queue_declare("orders.dlq.created", durable=True)
ch.queue_bind("orders.dlq.created", "orders.dlx", "orders.created")
ch.queue_declare("orders.dlq.cancelled", durable=True)
ch.queue_bind("orders.dlq.cancelled", "orders.dlx", "orders.cancelled")
ch.queue_declare("orders.work", durable=True, arguments={"x-dead-letter-exchange": "orders.dlx"})

Then "replay the created orders" is "replay one queue", and "Move messages" or a shovel can do it. Two cautions from the test: a catch-all binding with # next to the specific ones receives a copy of every message, so the two queues overlap; and a routing key the bindings do not cover is dropped silently. Bind every key you use, or put an alternate exchange on the dead-letter exchange. Splitting works for routing keys and your own headers (a headers exchange), not for the exception, which the broker does not know.

Checklist

Never nack a skipped message right away; hold it, and nack everything still held at the end.
End the scan with that nack in a finally block, not with a connection close, above all on quorum queues.
Read at most the queue length you saw at the start.
Confirms and mandatory=True on the publish, ack after the confirm.
Filter on what is actually there: x-death for broker dead letters, exception headers for republished ones.
If you filter by the same thing every time, split the dead-letter queue by routing key instead.

Where Warren fits

Warren does this selection in the browser. On a dead-letter queue's page you group the messages by exception, reason, queue, routing key or number of deaths and click a group to narrow the table. Then you replay the ticked rows, or every message in the whole queue that matches a reason or a search text; Warren checks that with a dry run first, so the replay takes exactly the messages you saw. On the Failures page one row is one exception across all dead-letter queues, and Replay on that row replays it in every queue you tick. Each message goes to its original route with confirms, and the ones you did not pick stay where they were. Free for one broker.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation

Replay one message first: a canary for RabbitMQ dead letters

Viktor — Mon, 12 Oct 2026 08:00:04 +0000

Originally published at warrenops.io.

The fix is merged, the pipeline is green, and 143 messages wait in orders.dlq. Replaying all of them at once is a bet: that the new version runs on every consumer replica, that the fix covers these payloads and not only the one in the ticket, and that the consumer does not fail on something else further down. If the bet is lost, 143 messages die a second time, the alerts fire again, and the dead-letter queue now mixes old failures with new ones. Sending one message first and waiting a moment costs a couple of minutes and settles the question for almost nothing.

The idea as a video (canary replay and the incident report, captions, no sound needed):

Before you send anything

A canary only means something if its message is processed during the wait. Check two things on the work queue first:

A consumer is attached. rabbitmqctl list_queues name messages consumers or the queue's page in the management UI. With no consumer the canary sits in the queue and every wait "passes".
The work queue is not backed up. If 20,000 messages are ahead of the canary, it is processed after the wait, not during it. Wait until the backlog is gone, or publish the canary later.

Then pick the canary from the failure you mean to fix. If the dead-letter queue holds two different exceptions, one canary passing says nothing about the other; send one per exception.

Send one message, tagged

The replay is the same as for a whole batch, stopped after the first message: read it from the dead-letter queue, publish it to its original route with a header that names this replay, wait for the publisher confirm, then acknowledge the original.

import uuid, pika

DEATH = ("x-death", "x-first-death-", "x-last-death-", "x-delivery-count", "x-acquired-count")
replay_id = str(uuid.uuid4())

ch.confirm_delivery()
method, props, body = ch.basic_get("orders.dlq", auto_ack=False)
death = props.headers["x-death"][0]  # broker dead-lettering; Spring or MassTransit error queues keep the route elsewhere
headers = {k: v for k, v in (props.headers or {}).items() if not k.startswith(DEATH)}
headers["x-replay-id"] = replay_id
props.headers = headers
ch.basic_publish(death["exchange"], death["routing-keys"][0], body, props, mandatory=True)
ch.basic_ack(method.delivery_tag)  # confirm_delivery made the publish wait for the broker
print("canary", props.message_id, "replay", replay_id)

Your own headers survive dead-lettering on classic and quorum queues, so if the canary dies again it comes back carrying x-replay-id, together with a fresh x-death. That pair is what tells its return apart from a new failure that arrived in the meantime. The details are in Did the replay work?.

How long to wait

Long enough for the consumer's whole retry chain to run out, plus the time to reach the canary. The chain is usually the larger part:

Retry set-up	When a failing message reaches the DLQ
No retries, `basic.reject` with `requeue=false`	within a second of delivery
Spring `RetryTemplate` / `@Retryable`, 3 attempts, backoff 1 s × 2	after about 3 s (1 + 2), plus processing
Broker-side retry: reject into a wait queue whose TTL dead-letters back, 5 rounds of 30 s	after about 2.5 minutes
MassTransit `UseDelayedRedelivery` with 5, 15, 30 minutes	after about 50 minutes

Read the numbers off your own configuration. Two minutes covers in-process retries with short backoffs; a delayed-redelivery set-up needs a canary that waits an hour, or a canary sent before lunch and the rest afterwards.

Watch for it

During the wait nothing else is replayed, so the dead-letter queue changes only through new failures and through the canary coming back. Check its depth; when it grows, look at what was added:

def came_back(ch, queue, replay_id, depth_before):
    depth = ch.queue_declare(queue, passive=True).method.message_count
    if depth <= depth_before:
        return False
    # returns arrive at the tail: read the whole queue and put everything back
    found, tags = False, []
    for _ in range(depth):
        method, props, _ = ch.basic_get(queue, auto_ack=False)
        if method is None:
            break
        tags.append(method.delivery_tag)
        h = props.headers or {}
        found = found or (h.get("x-replay-id") == replay_id and "x-death" in h)
    for tag in tags:
        ch.basic_nack(tag, requeue=True)
    return found

Take depth_before right after the canary left the queue, and call this every few seconds until the wait is over. Reading and putting back marks messages as redelivered. On a quorum dead-letter queue with a delivery limit, check what a requeue costs on your RabbitMQ version before you peek at all.

If your consumer sends failures to a different queue than the one the message came from (a parking queue after the last retry, a Spring .dlq next to a broker DLQ), watch that queue too.

Decide

The canary came back. Stop. The other 142 stay where they are, which is exactly where you want them while you read the new exception on the canary. Nothing is mixed up: one message died twice, and it says so in its headers.
It did not come back within the wait. Replay the rest, ideally throttled, so a consumer that has only just recovered is not flooded. Tag them with the same replay id and keep looking for returns over the next hours, as for any replay.
New messages arrived but none is the canary. Something else is failing right now. Find out what before the 142 join them.

When a passed canary proves nothing

The consumer acknowledged and dropped it. "Not back" means not seen in a dead-letter queue. If the consumer logs a business id, check that the canary's id shows up as processed.
The canary was not representative. It was the one payload the fix was written for. If the batch holds several customers, tenants or message versions, a canary from each is cheap.
The failure depends on load. A canary does not hit a rate limit, a connection pool or a lock that 142 messages at full speed do. That is what throttling the rest is for.
The wait was shorter than the retry chain. The canary is still in its third retry when the rest goes out. Check the table above against your own settings.

Checklist

A consumer is attached to the work queue and it is not backed up.
One canary per exception, tagged with a replay id, death headers stripped.
Wait longer than the consumer's full retry chain.
Back in the queue with your replay id and a new x-death: stop. Not back: replay the rest, throttled.
Keep watching for returns after the rest went out.

Where Warren fits

Warren's replay dialog has a Canary option: it replays the first message of the selection alone, watches the source queue for it during the wait (2 minutes by default, 5 seconds to 30 minutes), and replays the rest only if it has not died back by then, at the rate you set. The replay's page says "Canary out" with the time the wait ends, then "Canary passed" or "Canary died again"; a canary that died stops the replay and the audit log records why. The Failures page offers the same per exception, and its incident report lists every replay of that failure with how its canary went. Free for one broker.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation

Dead letters in the RabbitMQ management UI: what it can do, and where it bites

Viktor — Thu, 08 Oct 2026 07:32:38 +0000

Originally published at warrenops.io.

Every RabbitMQ operator has the management UI open, and when a dead-letter queue fills up, it is the first place anyone looks. Each queue page has a small corner for messages: Get messages, Move messages, Publish message and Purge. It is enough for a lot of days. It also has a few defaults and side effects that are easy to miss in the middle of an incident. Here is what each button does, measured on RabbitMQ 4.3.6 with classic and quorum queues.

The same task in the management UI and in Warren, side by side (2:02, captions, no sound needed):

Get messages: four ack modes, one safe default

Get messages fetches messages with basic.get and then settles them according to the Ack Mode you pick. The form starts at one message, and payloads are cut at 50,000 bytes.

Ack Mode	What happens to the message
Nack message requeue true (default)	put back, marked `redelivered`
Automatic ack	removed from the queue
Reject requeue true	put back, marked `redelivered`
Reject requeue false	removed, or dead-lettered again if the queue has a dead-letter exchange

The two "put back" modes look the same. On a classic queue they are: both put the message back where it was. On a quorum queue they are not:

quorum DLQ, three messages m1 m2 m3, then one Get messages of m1:
  Nack message requeue true   -> m1 m2 m3, x-delivery-count of m1 unchanged
  Reject requeue true         -> m2 m3 m1, x-delivery-count of m1 + 1

Measured on 4.3.6, also with plain AMQP: basic.nack with requeue=true does not count against a quorum queue's delivery limit, basic.reject with requeue=true does, and it moves the message to the back. Every Reject requeue true spends one delivery of the message's budget.

The two "removed" modes are the dangerous ones on a dead-letter queue. A DLQ usually has no dead-letter exchange of its own, so Reject requeue false does not move the message anywhere: it is gone, exactly like Automatic ack.

Use the default. Nack message requeue true is the only mode that neither removes a message nor costs it a delivery.

Quorum dead-letter queues on RabbitMQ 4: look before you peek

Even the default has one trap. Since RabbitMQ 4.0 every quorum queue has a delivery limit of 20 unless you set one, and a message that a quorum work queue dead-letters for delivery_limit arrives in a quorum DLQ with its count already at 21. On 4.3.6, a message whose count is above the queue's limit is dropped the moment it is put back, and Get messages with requeue puts it back. One look and it is gone.

Before anyone opens Get messages on a quorum DLQ, take the limit off:

rabbitmqctl set_policy dlq-no-limit '\.dlq$' '{"delivery-limit": -1}' --apply-to quorum_queues

The quorum article has the measurements and two other fixes.

Reading why a message died

Get messages shows the headers of each message as nested tables. The x-death table is the incident report: the queue the message died in, the reason (rejected, expired, maxlen, delivery_limit), how often, and the exchange and routing keys it was originally published with. The x-death article goes through it field by field.

What the UI does not do is look across messages. With five messages you read them one by one. With 300 messages and three different exceptions, you are scrolling a page of raw payloads and nested tables, looking for the dozen that are different. There is no search inside a queue and no grouping.

Move messages: one queue in, one queue out

Move messages appears on the queue page once the rabbitmq_shovel and rabbitmq_shovel_management plugins are enabled. It asks for one thing, a destination queue, and creates a dynamic shovel named Move from <queue> with these settings:

src-delete-after: queue-length    # move what is there now, then delete the shovel
ack-mode:         on-confirm      # a message leaves the source once the destination has it
dest-queue:       <what you typed>
dest-add-forward-headers: false

That makes it a safe move: nothing is lost on the way, and the shovel removes itself when it is done. What it does not do matters more for dead letters:

Everything goes to one queue. A DLQ that collects from several queues or routing keys cannot be sent back to where each message came from. The original exchange and routing key are in each message's x-death; Move messages does not read them.
All or nothing. It moves every message that is in the queue when it starts. There is no selection: the twelve poison messages go along with the 288 that only need a second try.
The death headers travel along. In our test the moved messages arrived with x-death, x-first-death-*, x-last-death-* and, from a quorum queue, the old x-delivery-count as a plain header. A consumer that reads the x-death count to decide when to give up sees the old count on arrival. If yours does, strip the headers when you replay instead.
No pacing. The shovel moves as fast as the broker lets it, with up to 1,000 messages in flight. A consumer that just recovered gets the whole backlog at once.

When the destination is the one queue the messages should go to, and you trust all of them, it is the right button.

Publish message: replaying one message by hand

To send a single message back to its route, people copy the payload from Get messages into Publish message on the exchange. Two things get lost on the way. The form sets properties and headers you type in, and its own help says it: "Only long string headers can be set here." Numbers, booleans and nested headers do not survive. And the original stays in the DLQ until you remove it with a second, destructive Get messages. Copy, publish, check, then remove, in that order.

Purge: final

Purge deletes every message in the queue. The UI asks once, in its own words: "Are you sure? Messages cannot be recovered after purging." There is no copy, no list of what was in the queue and no record of who did it. If a queue might hold anything you need later, export it first (Get messages with Nack message requeue true and a high count, then save the page, or a script).

Who did what

The management UI keeps no record of messages fetched, moved, published or purged. If the postmortem asks who replayed what at 3:12, the answer is in someone's shell history or nowhere.

Checklist

Get messages: keep the default, Nack message requeue true. Reject requeue true costs a delivery on a quorum queue; Automatic ack and Reject requeue false remove the message.
Quorum DLQ on RabbitMQ 4: set delivery-limit: -1 before the first peek.
Move messages: one destination queue, all messages, death headers included. Fine for a straight move, wrong for "back to where each one came from".
Publish message: string headers only; remove the original afterwards, not before.
Purge: export first if anything in the queue could matter.

Where Warren fits

Warren covers the message corner of the management UI for dead-letter work, and leaves configuration to it. It peeks with a requeue that costs nothing and asks before it reads a quorum queue where a peek could drop messages; it groups a DLQ by exception and searches it; it replays selected messages to the exchange and routing key in each one's x-death, with the death headers stripped, throttled and stoppable; it keeps a copy of discarded and purged messages for seven days, and an audit log of who did what. Free for one broker. A side-by-side table is on Warren vs. the RabbitMQ management UI.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation

Did the replay work? Finding the RabbitMQ dead letters that come back

Viktor — Wed, 07 Oct 2026 06:10:11 +0000

Originally published at warrenops.io.

The fix is deployed, 143 messages go back from the dead-letter queue to their original exchange, the broker confirms every one of them, and the incident is closed. Ten minutes later the dead-letter queue holds 5 messages again. Are they five of the 143, or five new failures? Without a little preparation nobody can tell, because a publish confirm only says the broker took the message, not that the consumer processed it. Here is how to make a replay answer that question, measured on RabbitMQ 4.3.6.

The short version as a video (0:58, captions, no sound needed):

What a confirmed replay tells you

A replay done right reads each message from the dead-letter queue, publishes it to its original route with publisher confirms, and acknowledges the original only after the confirm. That guarantees nothing is lost on the way. It says nothing about what happens next:

the consumer processes the message, which is what you hoped;
the consumer fails on it again, and it is dead-lettered back, because the fix did not cover this case;
the message sits in the work queue because the consumer is not running yet.

The management UI's "Move messages", a shovel, or a script that loops basic.get and basic.publish all stop at the confirm. To see the second case you have to recognise a message when it comes back.

Tag what you replay

Give every replayed message a header that names the replay, and remove the death headers so the message starts fresh:

import uuid, pika

DEATH = ("x-death", "x-first-death-", "x-last-death-", "x-delivery-count", "x-acquired-count")
replay_id = str(uuid.uuid4())

ch.confirm_delivery()
while (got := ch.basic_get("orders.dlq", auto_ack=False)) != (None, None, None):
    method, props, body = got
    death = props.headers["x-death"][0]  # broker dead-lettering; Spring or MassTransit error queues keep the route elsewhere
    headers = {k: v for k, v in (props.headers or {}).items() if not k.startswith(DEATH)}
    headers["x-replay-id"] = replay_id
    props.headers = headers
    ch.basic_publish(death["exchange"], death["routing-keys"][0], body, props, mandatory=True)
    ch.basic_ack(method.delivery_tag)  # confirm_delivery made the publish wait for the broker
print("replay", replay_id)

Keep message_id and the body as they are: they are what lets you match a returned message to the one you sent, also when two replays of the same message overlap.

What survives the second death

We published a message with an application header, rejected it into a dead-letter queue, replayed it with x-replay-id (once with the death headers stripped, once with them kept), and rejected it again. Classic and quorum queues behaved the same:

After the second dead-lettering	Result
your own headers (`x-app`, `x-replay-id`)	kept, unchanged
`x-death` when the replay stripped it	a fresh entry, `count` 1
`x-death` when the replay kept it	also a fresh entry, `count` 1: the broker does not continue a count a publisher sent along
`x-delivery-count` (quorum)	starts again at 1

Two things follow. Application headers ride along through dead-lettering, so the replay id is still there when the message comes back. And x-death cannot count replays: whatever the message went through before, after a republish RabbitMQ starts from scratch. Inside a broker-side retry loop (reject into a wait queue whose TTL dead-letters back to the work queue) the count does go up, 1, 2, 3, as before; it is only the republish that resets it. If you want to know how often a message has been replayed, count it in a header of your own, for example x-replay-count, increased on every replay.

A message that came back is therefore one that carries your replay id and a death: a new x-death from the broker, or, for consumers that republish failures themselves (Spring's RepublishMessageRecoverer, which copies the original headers), fresh exception headers. The id alone is not enough, because a replayed message still waiting in the work queue carries it too.

Look for them

Returned messages arrive at the tail of the dead-letter queue. A peek through the management API takes the head, so look early, while the queue is short, or read deep enough:

rabbitmqadmin -f raw_json get queue=orders.dlq ackmode=ack_requeue_true count=500 \
  | jq --arg id "$REPLAY_ID" '[.[] | select(.properties.headers["x-replay-id"] == $id
        and .properties.headers["x-death"] != null)] | length'

ack_requeue_true puts the messages back. They are marked redelivered afterwards, and on a quorum queue of RabbitMQ 3.13 every such peek counts as a delivery against the queue's delivery limit; on 4.x a requeue no longer counts.

When to look: returns usually show up within minutes, because the consumer fails on the message as soon as it gets it. Look once after a few minutes and again after an hour or two. Failures that depend on time (a nightly job, a token that expires, a downstream that is only down at peak) need a second look the next day.

Reading the result

None came back. Good, with a caveat: "not back" means not seen in a dead-letter queue. A consumer that acknowledges and drops a message it cannot handle looks the same. If your consumers log a business id, a spot check of five messages in the consumer log settles it.
A few came back, all with the same exception. The fix covered most cases but not this one. Group the returned messages by exception before replaying them again; replaying them unchanged produces the same result.
Most came back. The fix is not deployed where the consumer runs, or the consumer fails on something the replay itself changed, for example a header your code relied on and the replay removed.
They come back again and again. A message replayed three times without success is not waiting for the next deploy. Park it and read it. Your own replay counter tells you when that point is reached.

Checklist

Every replay gets an id, written into each message as a header.
Strip the death headers on replay; count replays in a header of your own, because x-death restarts after a republish.
A message with your replay id and a new death came back. The id alone does not mean that.
Look after a few minutes, after an hour or two, and the next day.
Group what came back by exception before you replay it again.

Where Warren fits

Warren does this bookkeeping for every replay: each message carries x-warren-replay-id, every read of a queue notices messages that died again, and Warren reads the source queue itself 5 minutes, 30 minutes, 2 hours and a day after the replay. The replay's page then says "5 of 143 replayed messages died again: 5 back in orders.dlq" and marks them; the audit log shows "5 back" next to the count. Free for one broker.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation

MassTransit on RabbitMQ: what ends up in _error and _skipped, and how to move it back

Viktor — Wed, 07 Oct 2026 06:00:49 +0000

Originally published at warrenops.io.

A consumer throws, MassTransit retries, and after the last attempt the message is gone from the endpoint. It is not lost: it sits in a queue next to it, shipping-label_error, with the exception in its headers. RabbitMQ's dead-lettering played no part in that, which is why the usual DLQ advice (x-death, dead-letter exchanges) does not quite fit. Here is what MassTransit does, what the headers say, and how to put the messages back once the bug is fixed.

The short version as a video (1:17, captions, no sound needed):

The topology MassTransit declares

For a receive endpoint shipping-label on RabbitMQ, MassTransit declares a queue shipping-label and an exchange of the same name bound to it. Message types get exchanges of their own (Shop.Contracts:CreateShippingLabel), bound to the endpoint's exchange when a consumer on that endpoint handles the type. Two more pairs appear the first time they are needed:

shipping-label_error: messages whose consumer threw, after all retries.
shipping-label_skipped: messages that arrived at the endpoint but that no consumer there handles, typically after a consumer was removed or renamed while publishers kept sending.

builder.Services.AddMassTransit(x =>
{
    x.AddConsumer<CreateShippingLabelConsumer>();
    x.UsingRabbitMq((context, cfg) =>
    {
        cfg.Host("rabbitmq", "/", h =>
        {
            h.Username("shipping");
            h.Password(builder.Configuration["RabbitMq:Password"]);
        });
        cfg.ReceiveEndpoint("shipping-label", e =>
        {
            e.ConfigureConsumer<CreateShippingLabelConsumer>(context);
        });
    });
});

MassTransit publishes the message itself into the error queue, through the shipping-label_error exchange, and acknowledges the original. The broker never dead-lettered anything, so there is no x-death header and no dead-letter exchange on the queue. Tools and scripts that look for x-death find nothing to work with.

What the headers tell you

The error queue keeps MassTransit's JSON envelope as the body (application/vnd.masstransit+json: messageId, conversationId, sourceAddress, destinationAddress, messageType and your message) and adds headers that answer most of the postmortem:

Header	Says
`MT-Reason`	`fault` for the error queue, `skip` for the skipped queue
`MT-Fault-ExceptionType`	the exception, e.g. `Shop.Shipping.CarrierUnavailableException`
`MT-Fault-Message`	its message
`MT-Fault-StackTrace`	the stack trace
`MT-Fault-ConsumerType`, `MT-Fault-MessageType`	which consumer failed on which message type
`MT-Fault-RetryCount`	how many retries ran before it gave up
`MT-Fault-Timestamp`, `MT-Host-*`	when, and on which machine and MassTransit version

Group the error queue by MT-Fault-ExceptionType and the incident usually explains itself: five CarrierUnavailableExceptions from one outage of the carrier, two InvalidOperationExceptions that are real bugs in the data. Those two groups want different treatment.

MassTransit also publishes a Fault<CreateShippingLabel> event when this happens. Subscribing to it is the cleanest way to alert from inside your code; watching the depth of *_error queues is the way to alert from outside it.

Retry, redelivery, and when a message reaches the error queue

cfg.ReceiveEndpoint("shipping-label", e =>
{
    // second-level: back to the broker, delivered again after 5, 15 and 30 minutes
    e.UseDelayedRedelivery(r => r.Intervals(TimeSpan.FromMinutes(5), TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(30)));
    // first-level: in memory, a few quick attempts
    e.UseMessageRetry(r => r.Intervals(200, 1000, 3000));
    e.ConfigureConsumer<CreateShippingLabelConsumer>(context);
});

UseMessageRetry retries inside the consumer, in memory, while the message stays unacknowledged; keep it to seconds. UseDelayedRedelivery hands the message back to the broker and has it delivered again later; it needs a scheduler (cfg.UseDelayedMessageScheduler() with RabbitMQ's delayed-message-exchange plugin) and counts the rounds in the header MT-Redelivery-Count. Only when both are used up does the message go to _error, with MT-Fault-RetryCount saying how far it got.

Two consequences for a replay later. A transient failure (a carrier down for twenty minutes) should rarely reach the error queue if the redelivery intervals outlast typical outages; if it does, the intervals were too short. And a message that still carries MT-Redelivery-Count from its first life starts the next one with part of its budget spent.

Moving the messages back

The bug is fixed, the carrier is back. The messages belong in shipping-label again. What to get right:

Send them to the endpoint's exchange, shipping-label, routing key empty. It is bound to the queue, so the message arrives exactly where MassTransit would deliver it. Publishing to the message type's exchange instead would reach every endpoint that subscribes to the type, which is a second delivery for all of them.
Leave the body alone. MassTransit dispatches on the envelope's messageType. Unwrap the JSON to "simplify" it and the consumer no longer recognises the message, which then lands in _skipped.
Remove the fault and redelivery headers: MT-Reason, MT-Fault-* and MT-Redelivery-Count, so the message starts with a full retry budget and a later fault is not mixed up with this one.
Confirm before you acknowledge, and go slowly if the consumer just recovered.
Skipped messages are different: replaying them to the same endpoint skips them again. Deploy the consumer for their type first, or send them to the endpoint that has it.

The management UI's "Move messages" (shovel plugin) covers point 1 for a whole queue, and nothing else: no selection by exception, headers untouched, no record of what was moved. A small tool with RabbitMQ.Client 7 does all of it:

// RabbitMQ.Client 7: with confirmation tracking, BasicPublishAsync returns once the broker confirmed
var factory = new ConnectionFactory { Uri = new Uri(rabbitUri) };
await using var connection = await factory.CreateConnectionAsync("error-replay");
await using var channel = await connection.CreateChannelAsync(
    new CreateChannelOptions(publisherConfirmationsEnabled: true, publisherConfirmationTrackingEnabled: true));

const string errorQueue = "shipping-label_error";
var endpoint = errorQueue[..^"_error".Length];
var moved = 0;
while (moved < limit && await channel.BasicGetAsync(errorQueue, autoAck: false) is { } result)
{
    var props = new BasicProperties(result.BasicProperties);
    props.Headers = result.BasicProperties.Headers?
        .Where(h => h.Key != "MT-Reason" && h.Key != "MT-Redelivery-Count" && !h.Key.StartsWith("MT-Fault-"))
        .ToDictionary(h => h.Key, h => h.Value);

    // to the endpoint's exchange, body untouched: MassTransit dispatches on the envelope's messageType
    await channel.BasicPublishAsync(endpoint, routingKey: "", mandatory: true, basicProperties: props, body: result.Body);
    await channel.BasicAckAsync(result.DeliveryTag, multiple: false); // only after the confirm
    moved++;
}

Messages you do not want to move (the two real bugs) need a filter before the publish: read MT-Fault-ExceptionType and leave those unacknowledged; they go back to the error queue when the channel closes. Add a delay between publishes for a large queue, and log what you moved where.

Checklist

Know your endpoints' _error and _skipped queues and watch their depth, or subscribe to Fault<T>.
First-level retry for seconds, delayed redelivery for outages, so only real failures reach _error.
Group the error queue by MT-Fault-ExceptionType before you move anything.
Replay to the endpoint's exchange, body untouched, MT-Fault-*, MT-Reason and MT-Redelivery-Count removed, confirmed before acknowledged.
Skipped messages need their consumer first.

Where Warren fits

Warren recognises MassTransit's _error and _skipped queues as dead-letter queues, groups them by MT-Fault-ExceptionType, and says in one sentence which consumer faulted after how many retries. A replay goes to the endpoint's exchange with the fault and redelivery headers removed, the envelope untouched, confirmed message by message and throttled if you want, and every replay is in the audit log. Free for one broker.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation

Your quorum dead-letter queue can drop the dead letters: x-delivery-count travels along

Viktor — Tue, 06 Oct 2026 08:44:55 +0000

Originally published at warrenops.io.

A dead-letter queue is supposed to be the place where failed messages are safe until someone looks at them. On RabbitMQ 4 with quorum queues and default settings, that is not quite true: a message that died because it was delivered too often can arrive in the DLQ with its delivery budget already used up, and the first consumer or tool that crashes while holding it, or even just peeks at it, makes RabbitMQ drop it for good. Here is what happens, measured on RabbitMQ 4.3.6, and three ways to make the DLQ safe again.

The short version as a video (1:42, captions, no sound needed):

Delivery limits in 30 seconds

Quorum queues count how often a message was delivered and not settled. Past the queue's delivery limit (x-delivery-limit, or delivery-limit in a policy) RabbitMQ stops redelivering it: the message is dead-lettered with reason delivery_limit if the queue has a dead-letter exchange, and dropped otherwise. Since RabbitMQ 4.0 every quorum queue has a limit of 20 unless you set one, which is good: a poison message no longer loops forever.

What counts changed in 4.0 as well:

What happens to a delivered message	3.13	4.x
`basic.nack` with `requeue=true`	counts	does not count
`basic.reject` with `requeue=true`	counts	counts, and the message goes to the back
management UI "Get messages" in its default mode, "Nack message requeue true"	counts	does not count
the channel or connection closes while it is unacknowledged (consumer crash, pod killed, network drop)	counts	counts
`basic.reject` with `requeue=false` (dead-lettered)	not measured	counts
put back (any of the above, or a peek with requeue) while its count is already above the queue's limit	not measured	dropped, or dead-lettered on
put back (any of the above, or a peek with requeue) while its count is already above the queue's limit	not measured	dropped, or dead-lettered on

The count is visible on the message as the header x-delivery-count (RabbitMQ 4 adds x-acquired-count, every acquisition including requeues).

What we measured

A work queue and a dead-letter queue, both quorum queues, no limits set, so both have RabbitMQ 4's default of 20:

ch.exchange_declare('lim.dlx', 'fanout', durable=True)
ch.queue_declare('lim.dlq', durable=True, arguments={'x-queue-type': 'quorum'})
ch.queue_bind('lim.dlq', 'lim.dlx')
ch.queue_declare('lim.work', durable=True,
                 arguments={'x-queue-type': 'quorum', 'x-dead-letter-exchange': 'lim.dlx'})
ch.basic_publish('', 'lim.work', b'order-4711')

Then a consumer that crashes on this message every time: fetch it without acknowledging and close the connection, until the work queue gives up.

while message_count('lim.work') > 0:
    conn = pika.BlockingConnection(params)
    conn.channel().basic_get('lim.work', auto_ack=False)
    conn.close()          # the consumer "crashed" while holding the message

The results:

abandoned deliveries on lim.work until it gave up: 21
in lim.dlq: reason delivery_limit, x-delivery-count 21
left in lim.dlq after ONE abandoned delivery there: 0

The message reached the DLQ, as it should, with reason delivery_limit. But it brought its count of 21 along, and the DLQ's own limit is 20. The next abnormal return, a single one, made the DLQ give up on it too, and since the DLQ has no dead-letter exchange of its own, the message was dropped.

It does not even take a crash. In a second run we only looked: one basic.get followed by basic.nack with requeue=true, and once more through the management UI's "Get messages" with requeue. Both times the DLQ was empty afterwards. On RabbitMQ 4.3.6 a message whose count is already above the queue's limit is dropped the moment it is put back, abnormally or not; a count equal to the limit survives. So the usual first step of an investigation, peeking at the DLQ, is what loses the message.

Why the count travels

The delivery count is not just a header the queue writes for you to read: it is part of the message's annotations, and dead-lettering keeps them. The DLQ continues counting from where the work queue stopped. A header that a publisher sets is a different thing: we published a fresh message with x-delivery-count: 5 into a queue with a limit of 5, and RabbitMQ ignored it. The next delivery showed 1, and the message survived five abandoned deliveries. So the carried count comes from dead-lettering, not from the header itself.

It is not only delivery_limit messages. In the same test with explicit limits, a message that had been abandoned twice and then rejected arrived in the DLQ with a count of 3, and with a DLQ limit of 5 it was dropped after three more failures, not five. Every message rejected by a consumer arrives with at least 1.

Who abandons a delivery in a DLQ?

More often than you would think:

a consumer on the DLQ that processes dead letters (an alerting bridge, an archiver, a "retry later" service) and crashes or is redeployed while holding messages with prefetch;
a script that reads the DLQ with basic.get and dies, or is stopped with Ctrl+C, before it acknowledges or nacks;
a replay tool whose connection drops in the middle of a run;
a pod eviction or a network blip during any of the above.

On 3.13 it was worse (every requeue counted, so even looking at messages in the management UI cost deliveries), but 4.x's default limit makes the carry-over bite with no configuration at all.

Three fixes

All three kept the message through three abandoned deliveries in the DLQ in our test.

1. No delivery limit on the DLQ. A DLQ is not consumed in a loop, so it does not need one. By policy:

rabbitmqctl set_policy dlq-no-limit '\.dlq$' '{"delivery-limit": -1}' --apply-to quorum_queues

or with x-delivery-limit: -1 when you declare the queue; a negative value means unlimited. Only one policy applies to a queue: if one matches your DLQs already, add the key there.

2. A much higher limit, if you want a backstop anyway: x-delivery-limit: 100.

3. A classic queue as DLQ. Classic queues have no delivery limit. You lose the replication of a quorum queue, which matters less for a queue that is mostly written and rarely read; decide by how much you care about dead letters surviving the loss of a node.

And in whatever reads the DLQ: settle every message you fetch, ack what you have handled and nack with requeue=true what you have not. On 4.x a nack with requeue costs nothing (a reject with requeue does count), except for a message that is already above the queue's limit: the requeue drops it as well. Fix the limit before anyone reads the queue.

Checking your own queues

Do not peek first. Check the queue: rabbitmqctl list_queues name type arguments policy shows which DLQs are quorum queues and whether a limit is set. A quorum DLQ with a limit can lose messages to the peek itself; set delivery-limit: -1 first, then look.
After that, x-delivery-count next to x-death on a message tells you how much of its budget it used before it died.
No x-delivery-limit and no delivery-limit policy means 20 on RabbitMQ 4.
Replaying the message, by publishing it anew to its original exchange (a shovel does that too), gives it a fresh count. Dead-lettering it once more into another quorum queue does not.

Checklist

Quorum DLQs: delivery-limit: -1 by policy, or classic DLQs.
Consumers and scripts on DLQs settle every message they fetch, also on shutdown.
Take the limit off quorum DLQs before anyone peeks at them: on 4.x a peek drops what arrived above the limit.
Replay to reset the budget; dead-lettering a message on into another quorum queue keeps the count.

Where Warren fits

Warren shows each message's remaining budget in a quorum DLQ ("1 of 20 left") and warns on the queue page when dead letters are one or two failed deliveries away from being dropped, saying whether RabbitMQ would drop them or dead-letter them again. Its replays publish the messages anew with a fresh count and strip the stale x-delivery-count, and it reads nothing from a quorum queue where that can cost messages until you confirm: on 3.13 any quorum queue with a limit, on 4.x a quorum dead-letter queue with a limit, whose over-limit arrivals a peek would drop. Free for one broker.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation

Dead letters in Spring Boot: why RabbitMQ messages end up in the DLQ, and how to replay them

Viktor — Mon, 05 Oct 2026 07:10:01 +0000

Originally published at warrenops.io.

A @RabbitListener throws. What happens next depends on three Spring AMQP settings most services never set on purpose: whether the message is requeued, whether it is retried, and what the last retry does with it. Get them wrong and one bad message loops forever or silently disappears. Get them right and it waits in a dead-letter queue until you have fixed the bug. Here is the whole chain, and how to put the messages back afterwards.

What Spring does when a listener throws

By default the listener container catches the exception and rejects the message with requeue. RabbitMQ puts it back at the head of the queue, the same consumer gets it again a millisecond later, throws again, and so on. A single poison message keeps a consumer busy, fills the log, and holds back everything behind it.

Two exceptions to that: a MessageConversionException (the payload could not be turned into your type) is treated as fatal by the default error handler and rejected without requeue. And if your code throws AmqpRejectAndDontRequeueException, the message is rejected without requeue too. For everything else, switch the default off:

spring:
  rabbitmq:
    listener:
      simple:
        default-requeue-rejected: false

Now a failed message is rejected with requeue=false. Whether that means "gone" or "dead-lettered" depends on the queue.

Give the queue a dead-letter route

A rejected message only reaches a dead-letter queue if its queue has a dead-letter exchange. In Spring you declare it with the queue:

@Bean
Queue orders() {
    return QueueBuilder.durable("orders.process")
            .deadLetterExchange("orders.dlx")
            .deadLetterRoutingKey("orders.dlq")
            .build();
}

@Bean
DirectExchange deadLetterExchange() {
    return new DirectExchange("orders.dlx");
}

@Bean
Queue deadLetterQueue() {
    return QueueBuilder.durable("orders.dlq").build();
}

@Bean
Binding deadLetterBinding() {
    return BindingBuilder.bind(deadLetterQueue()).to(deadLetterExchange()).with("orders.dlq");
}

RabbitMQ then republishes every rejected, expired or overflowing message from orders.process to orders.dlx with routing key orders.dlq, and adds an x-death header that says which queue it came from, why, and its original exchange and routing key. (Reading x-death explains every field.)

One trap: queue arguments are fixed when a queue is created. If orders.process already exists without them, Spring's declaration fails with PRECONDITION_FAILED and the listener does not start. For existing queues use a policy instead, which can be added and changed at any time:

rabbitmqctl set_policy orders-dlx "^orders\.process$" \
  '{"dead-letter-exchange":"orders.dlx","dead-letter-routing-key":"orders.dlq"}' \
  --apply-to queues

Retry before giving up

Many failures are transient: a timeout, a lock, a service that restarts. Spring Boot retries inside the listener before it rejects:

spring:
  rabbitmq:
    listener:
      simple:
        default-requeue-rejected: false
        retry:
          enabled: true
          max-attempts: 4
          initial-interval: 1s
          multiplier: 2
          max-interval: 10s

Two things to know about this retry. It is stateless and in memory: the consumer thread sleeps between attempts while the message stays unacknowledged, and with a prefetch of 250 the other 249 wait too. Keep the total backoff short, seconds rather than minutes. And because the attempts never left the consumer, RabbitMQ does not see them: the message arrives in the DLQ with x-death count 1, although it failed four times.

For longer delays (a downstream system that is down for ten minutes), let the broker wait instead: a wait queue with a TTL whose dead-letter exchange points back at the work exchange. Then every cycle shows up in x-death, and the consumer can give up after a number of cycles by reading it:

@SuppressWarnings("unchecked")
static long deathsIn(Message message, String queue) {
    var deaths = (List<Map<String, Object>>) message.getMessageProperties().getHeaders().get("x-death");
    if (deaths == null) {
        return 0;
    }
    return deaths.stream()
            .filter(death -> queue.equals(String.valueOf(death.get("queue"))))
            .mapToLong(death -> ((Number) death.get("count")).longValue())
            .sum();
}

What the last attempt does: reject, or republish

When the retries are used up, a MessageRecoverer decides. The default rejects the message without requeue, so it is dead-lettered by the broker with x-death as described above. The alternative is RepublishMessageRecoverer:

@Bean
MessageRecoverer messageRecoverer(RabbitTemplate rabbitTemplate) {
    // failed messages go to orders.error with the exception in the headers
    return new RepublishMessageRecoverer(rabbitTemplate, "orders.error", "orders.process.failed");
}

It publishes a copy to an exchange of your choice and acknowledges the original. The copy carries x-exception-message, x-exception-stacktrace (trimmed to fit the frame size), x-original-exchange and x-original-routingKey. The broker never dead-lettered it, so there is no x-death.

	Reject (default)	RepublishMessageRecoverer
Where it lands	The queue's dead-letter exchange	The exchange you name
Why it failed	`x-death` reason `rejected`, no exception	The exception message and stack trace
Where it came from	`x-death`: queue, exchange, routing keys	`x-original-exchange`, `x-original-routingKey`
Needs topology	Dead-letter exchange on the queue	Error exchange and queue

If you want to know why a message failed without searching the logs, republish. If you want the broker to stay in charge (TTL, length limits and quorum delivery limits dead-letter the same way), reject. Many teams do both: rejected messages of a TTL wait queue, republished ones from the listener.

Putting the messages back

The bug is fixed, 300 messages wait in orders.dlq. A replay has to get four things right:

Acknowledge only after the broker confirmed the copy. Publish with confirms, wait, then ack the dead letter. Ack first and a broken connection loses the message.
Send it where it came from, not to one queue for all: the exchange and routing key from x-death, or the x-original-* headers after a republish.
Strip the bookkeeping. A consumer that gives up after three deaths rejects a replayed message on arrival if its old x-death is still there.
Go slowly if the consumer just recovered. Three hundred messages in a second can be the second incident.

With the client factory underneath Spring's and the plain channel API it looks like this; originalRoute reads the first x-death entry's exchange and routing key, or the x-original-* pair:

/** Moves up to [limit] messages from the DLQ back to where they came from. */
int replay(CachingConnectionFactory connectionFactory, String dlq, int limit) throws Exception {
    // a connection of its own from the underlying client factory: Spring's factory caches channels,
    // and confirm mode must not leak into that cache
    try (var connection = connectionFactory.getRabbitConnectionFactory().newConnection("dlq-replay");
         var channel = connection.createChannel()) {
        channel.confirmSelect();
        int moved = 0;
        while (moved < limit) {
            GetResponse response = channel.basicGet(dlq, false);
            if (response == null) {
                break; // the queue is empty
            }
            var props = response.getProps();
            Map<String, Object> headers = new HashMap<>(props.getHeaders() == null ? Map.of() : props.getHeaders());
            Route route = originalRoute(headers); // from x-death, or x-original-* after a republish
            headers.keySet().removeIf(name ->
                    name.startsWith("x-death") || name.startsWith("x-first-death") ||
                    name.startsWith("x-last-death") || name.startsWith("x-exception") ||
                    name.startsWith("x-original"));
            channel.basicPublish(route.exchange(), route.routingKey(), true,
                    props.builder().headers(headers).build(), response.getBody());
            channel.waitForConfirmsOrDie(5_000);              // the broker has it now
            channel.basicAck(response.getEnvelope().getDeliveryTag(), false); // only then remove it here
            moved++;
        }
        return moved;
    }
}

Not in this sketch, and worth adding before you run it in production: a return listener for unroutable messages (mandatory=true sends them back instead of dropping them), a rate limit, selecting messages by exception rather than taking the first N, and a record of what was moved where. Do not use RabbitTemplate.receive() for this: it acknowledges on receipt, before anything has been published.

Checklist

default-requeue-rejected: false, so a failure is rejected instead of looping.
A dead-letter exchange on every work queue, by argument for new queues, by policy for existing ones.
Short in-memory retries; long delays through a TTL wait queue that shows up in x-death.
RepublishMessageRecoverer if you want the exception next to the message.
Replays that confirm before they ack, route back to the original exchange, and strip the death headers.

Where Warren fits

Warren reads both conventions: x-death from broker dead-lettering and the x-exception-* and x-original-* headers of Spring's RepublishMessageRecoverer. It groups a dead-letter queue by exception, shows each message's story, and replays the ones you select to their original exchange and routing key, death and exception headers stripped, confirmed message by message and throttled if you want. Every replay is in the audit log. Free for one broker.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation · 4-minute walkthrough on YouTube

Alerting when a RabbitMQ queue has no consumers

Viktor — Fri, 02 Oct 2026 06:01:48 +0000

Originally published at warrenops.io.

A queue with messages and zero consumers is the quietest outage there is. Nothing errors. Producers publish happily. The dashboard shows the broker green. Then the TTL runs out, or the queue hits its limit, and an hour of orders is in a dead-letter queue. Here is how to get paged before that.

What to alert on

"No consumers" alone is not the alert. Plenty of queues have no consumers by design: retry-wait queues, parking-lot DLQs, queues of a batch job that runs at night. The signal is the combination:

consumers == 0
and messages_ready > 0 (something is waiting)
and it has been like that for a few minutes (a deploy restarts consumers; do not page on a 20-second gap)
and the queue is not on your list of intentionally consumer-less queues.

Two neighbours of this alert are worth setting up at the same time, because they catch the cases "consumers exist but do nothing":

Stuck consumers: consumers > 0, messages_ready > 0, and the ack rate is zero for five minutes. The consumer process is alive, holds the connection, and is deadlocked or stuck on a downstream call.
Growth: messages_ready grew by more than N in the last M minutes. Consumers are there but cannot keep up.

Where the numbers come from

Management HTTP API

The management plugin exposes everything you need per queue. Ask for only the columns you want, otherwise the response for a broker with a thousand queues is huge:

curl -s -u monitor:secret \
  'http://rabbit:15672/api/queues?columns=vhost,name,consumers,messages_ready,messages_unacknowledged,message_stats.ack_details.rate'

Relevant fields:

Field	Meaning
`consumers`	Number of consumers currently attached to the queue.
`messages_ready`	Messages waiting to be delivered. This is the "backlog".
`messages_unacknowledged`	Delivered but not yet acked. High and flat means consumers hold messages without finishing them.
`message_stats.ack_details.rate`	Acks per second, averaged over the last sampling interval. Zero with a backlog and consumers present is the stuck-consumer signal.
`consumer_utilisation`	Fraction of time consumers could take new messages. Low with a backlog means consumers are the bottleneck.
`idle_since`	Set when the queue had no activity at all since that time. Useful to ignore truly dead queues.

The monitoring user needs the monitoring tag, nothing more. Do not use an administrator account for a poller.

Prometheus plugin

Since RabbitMQ 3.8, rabbitmq_prometheus ships with the broker. By default /metrics is aggregated: you get totals per node, not per queue, which is useless for this alert. Per-queue values come from either:

/metrics/per-object, every metric for every object. Fine for a few hundred queues, heavy for thousands.
/metrics/detailed?family=queue_coarse_metrics&family=queue_consumer_count (3.9+), only the families you ask for. Metrics from this endpoint are prefixed rabbitmq_detailed_.
Or set prometheus.return_per_object_metrics = true in rabbitmq.conf to make /metrics per-object.

rabbitmq-plugins enable rabbitmq_prometheus
curl -s 'http://rabbit:15692/metrics/detailed?family=queue_coarse_metrics&family=queue_consumer_count' | grep orders

Prometheus alert rules

With the detailed endpoint scraped as its own job, three rules cover the three signals above. Adjust the prefix (drop detailed_) if you use per-object metrics.

groups:
- name: rabbitmq-queues
  rules:
  - alert: RabbitMQQueueNoConsumers
    expr: |
      rabbitmq_detailed_queue_consumers == 0
      and on (vhost, queue) rabbitmq_detailed_queue_messages_ready{queue!~".*(\\.wait|\\.dlq|\\.parking)$"} > 0
    for: 5m
    labels: {severity: page}
    annotations:
      summary: "{{ $labels.vhost }}/{{ $labels.queue }} has {{ $value }} messages and no consumer"

  - alert: RabbitMQQueueConsumersStuck
    expr: |
      rabbitmq_detailed_queue_consumers > 0
      and on (vhost, queue) rabbitmq_detailed_queue_messages_ready > 0
      and on (vhost, queue) rate(rabbitmq_detailed_queue_messages_acked_total[5m]) == 0
    for: 5m
    labels: {severity: page}

  - alert: RabbitMQQueueGrowing
    expr: |
      delta(rabbitmq_detailed_queue_messages_ready[15m]) > 1000
    for: 0m
    labels: {severity: warn}

The regex on queue excludes the queues that have no consumers on purpose; more on that below. Because the exclusion sits on the messages_ready side of the and, those queues never produce a result, whatever their consumer count.

Route severity: page to your on-call and warn to a channel. A queue growing at 2 a.m. may be a batch job; a queue with no consumer at 2 a.m. is not.

No Prometheus? Cron and curl

A twenty-line script on a box that can reach the management API does the job for a small setup. This one posts to a Slack incoming webhook when a queue has a backlog and no consumer, and keeps a marker file so it does not repeat itself every minute:

#!/usr/bin/env bash
set -euo pipefail
API=http://rabbit:15672; AUTH=monitor:secret
HOOK=https://hooks.slack.com/services/T000/B000/xxxx
IGNORE='\.(wait|dlq|parking)$'
STATE=/var/tmp/rabbit-noconsumer; mkdir -p "$STATE"

# "vhost/queue<TAB>ready" for every queue that is in trouble right now
ALERTING=$(curl -sf -u "$AUTH" "$API/api/queues?columns=vhost,name,consumers,messages_ready" \
  | jq -r --arg ign "$IGNORE" \
      '.[] | select(.consumers == 0 and .messages_ready > 0 and (.name | test($ign) | not))
           | "\(.vhost)/\(.name)\t\(.messages_ready)"')

# notify once per incident: a marker file per queue
while IFS=$'\t' read -r q ready; do
  [ -n "$q" ] || continue
  key="$STATE/$(printf '%s' "$q" | md5sum | cut -c1-32)"
  [ -e "$key" ] && continue
  printf '%s\n' "$q" > "$key"
  curl -sf -X POST -H 'content-type: application/json' "$HOOK" \
    -d "$(jq -nc --arg t ":rotating_light: $q has $ready messages and no consumer" '{text: $t}')"
done <<< "$ALERTING"

# recovered: drop markers for queues that are no longer alerting
for f in "$STATE"/*; do
  [ -e "$f" ] || continue
  grep -qF "$(cat "$f")" <<< "$ALERTING" || rm -f "$f"
done

Run it every minute from cron. The "for 5 minutes" part is missing here; add it by requiring the marker to be older than five minutes before posting, or accept the occasional deploy-time notification. Teams users swap the Slack payload for an Adaptive Card or a plain {"text": ...} to an incoming webhook.

Deciding which queues to exclude

Do this once and write it down, otherwise the alert gets muted the first week:

Retry-wait queues (TTL, dead-letter back to the work queue): never have consumers. Exclude by name pattern.
Dead-letter and parking-lot queues: no consumers, and messages in them are a different alert (see below). Exclude here, alert separately.
Batch queues consumed on a schedule: exclude, or alert only outside the schedule.
Everything else with a backlog and no consumer is an incident.

Name your queues so a regex can tell these apart. orders.process, orders.retry.wait, orders.dlq is a convention that makes every rule on this page a one-liner.

The companion alert: something landed in a DLQ

The no-consumer alert is the early warning. The late warning is messages_ready > 0 on a queue matching \.dlq$, because by then messages have already failed. Give that one a lower threshold than you think: one dead letter in a queue that is normally empty is news, three hundred is a postmortem. If you want fewer false alarms, alert on the rate of arrivals into the DLQ (message_stats.publish_details.rate on the DLQ, or rate(rabbitmq_detailed_queue_messages_published_total[5m])) rather than on the count.

Keep the history

Whatever alerts you build, keep the per-queue time series for at least a week. The question after a no-consumer page is always "since when", and the management UI only shows the last hour at any useful resolution. Prometheus gives you this for free. The cron script does not; if you go that route, at least append the numbers to a file per run.

Where Warren fits

If you do not run Prometheus, Warren's Team edition does this out of the box: alert rules match queues by regex on one or all clusters, with conditions for messages above a threshold, no consumers while messages wait, and growth within a window, each with a sustained-for time. Notifications go to Slack, Teams or a JSON webhook, repeat after a configurable interval and resolve when the queue recovers. Every queue is sampled every 30 seconds and kept for seven days, so "since when" has an answer.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation · 4-minute walkthrough on YouTube

Reading the x-death header: which queue rejected the message, and why

Viktor — Thu, 01 Oct 2026 06:28:52 +0000

Originally published at warrenops.io.

Every message in a dead-letter queue carries its own incident report. RabbitMQ writes it into the x-death header when it dead-letters the message. Most people have seen it in the management UI as a wall of nested tables and closed the tab. Here is how to read it.

Related, in a 1:13 video (captions, no sound needed): why the x-death count starts over after a replay.

Where it comes from

When RabbitMQ dead-letters a message, it does not just move it. It republishes the message to the queue's dead-letter exchange (set by the x-dead-letter-exchange queue argument or a dead-letter-exchange policy), optionally with a new routing key (x-dead-letter-routing-key). Before publishing, it modifies the headers:

It adds or updates an x-death header, an array of tables.
The first time a message is dead-lettered it also sets x-first-death-reason, x-first-death-queue and x-first-death-exchange. These are never changed afterwards.
Recent RabbitMQ versions also set x-last-death-reason, x-last-death-queue and x-last-death-exchange, updated on every dead-lettering.
If the message died because it expired and had a per-message TTL, the expiration property is removed (otherwise it would expire again in the DLQ) and preserved in the x-death entry as original-expiration.

Everything else, payload, properties, your own headers, is untouched.

The entry

A typical header, as the management UI or a client shows it:

x-death: [
  {
    "count": 3,
    "reason": "rejected",
    "queue": "orders.process",
    "time": 1759042215,
    "exchange": "orders",
    "routing-keys": ["order.created"]
  },
  {
    "count": 2,
    "reason": "expired",
    "queue": "orders.retry.wait",
    "time": 1759042155,
    "exchange": "orders.retry",
    "routing-keys": ["order.created"]
  }
]
x-first-death-exchange: orders
x-first-death-queue:    orders.process
x-first-death-reason:   rejected

Field	Meaning
`queue`	The queue the message was in when it was dead-lettered. This is the queue whose consumer rejected it, whose TTL ran out, or whose length limit hit. Not the queue it landed in.
`reason`	`rejected`: a consumer did `basic.reject` or `basic.nack` with `requeue=false`. `expired`: message or queue TTL ran out. `maxlen`: the queue was over `x-max-length` or `x-max-length-bytes` and the overflow policy is `drop-head` or `reject-publish-dlx`. `delivery_limit`: a quorum queue redelivered the message more often than its `delivery-limit` allows.
`time`	Unix timestamp (seconds) of the first time this queue dead-lettered the message for this reason. When `count` grows, `time` does not move.
`exchange`	The exchange the message was originally published to before it arrived in `queue`. Together with `routing-keys` this is where a replay sends it back.
`routing-keys`	The routing key(s) the message was published with. Usually one. Can be several if the message was CC'd or BCC'd.
`count`	How many times this message was dead-lettered from this queue for this reason. Entries are not appended per event; RabbitMQ finds the entry with the same `queue` and `reason` and increments it.
`original-expiration`	Only present when `reason` is `expired` and the message had a per-message `expiration`. The value that was removed.

Reading the array

The array is ordered most recent first: x-death[0] is the last thing that happened to the message. In the example above, the story reads:

The message was published to exchange orders with key order.created and landed in orders.process.
The consumer rejected it. It went to orders.retry.wait, a queue with a TTL and no consumers.
It expired there and came back to orders.process. Rejected again. Expired again. Rejected a third time.
After the third rejection (count: 3) the message ended up wherever you are looking at it now, presumably a final DLQ, because the consumer stopped requeueing after three attempts or the retry queue's TTL routed it elsewhere.

Note what the header does not tell you: which DLQ the message is in now (you know that, you are looking at it), and what went wrong in the consumer. For that you need the consumer's log at time, and time is the first rejection, not the last. If your retry cycle is 60 seconds and count is 3, the last rejection was about two minutes after time.

Same queue twice with different reasons

Because entries are keyed by queue and reason, a queue can appear twice: once with rejected, once with expired. That usually means a consumer was down for a while (messages expired) and then came back and rejected the same message. Both entries carry their own time, so you can tell when each phase started.

Why time does not update

People look at time, see a timestamp from three days ago, and conclude the message has been sitting in the DLQ for three days. It may have arrived a minute ago after its tenth cycle. To know when a message actually entered the DLQ you need something outside the header: a timestamp property your producer set (that is when it was published, also wrong), a message-level TTL trick, or a tool that remembers when it first saw the message.

What the header looks like for each cause

Reason	Typical pattern	What to check
`rejected`, count 1	Single entry, one queue	Consumer log at `time`. Parsing error or a business rule, usually deterministic. Fix, then replay.
`rejected`, count > 1	Alternating with an `expired` entry from a wait queue	Retry topology did its job. Either a transient failure that outlasted the retries, or a real bug. The consumer log at the last attempt tells which.
`expired`, from a work queue	Single entry, the exchange is the normal one	Nobody consumed in time. Consumer count on that queue, consumer throughput, was the TTL sane.
`maxlen`	Many messages with identical `time`, seconds apart	A burst the consumer could not absorb. Producer side, or scale consumers. These messages are usually fine to replay once the backlog is gone.
`delivery_limit`	Quorum queue, no `rejected` entry	The consumer crashed or its connection dropped repeatedly while holding the message, without ever rejecting it. Classic poison message or a consumer that dies on this payload.

Messages with no x-death at all

Not every message in a dead-letter queue was dead-lettered by RabbitMQ. Many frameworks catch the exception and republish the message to an error queue themselves. The broker never dead-letters it, so there is no x-death. Instead you get framework headers:

Spring AMQP (RepublishMessageRecoverer): x-exception-message, x-exception-stacktrace, x-original-exchange, x-original-routingKey.
MassTransit (_error queues): MT-Fault-Message, MT-Fault-ExceptionType, MT-Fault-StackTrace, MT-Reason.
NServiceBus: NServiceBus.ExceptionInfo.ExceptionType, NServiceBus.ExceptionInfo.Message, NServiceBus.FailedQ.

These are often more useful than x-death because they carry the actual exception. But the "where to send it back" question then has to be answered from x-original-* or NServiceBus.FailedQ, and there is no count. Your replay tooling should read both conventions.

How to look at it

For one message at a time, paste it into the x-death decoder: it tells the story above in plain words, also for the Spring, MassTransit and NServiceBus headers, and runs in your browser.

Management UI

Queue page → Get messages, ack mode Nack message requeue true, and a small count. The UI renders x-death as nested tables. It works for one or two messages; for fifty it is unreadable, and the peek marks every fetched message as redelivered.

CLI

rabbitmqadmin -f raw_json get queue=orders.dlq ackmode=ack_requeue_true count=5 \
  | jq '.[] | .properties.headers["x-death"]'

-f raw_json makes rabbitmqadmin print the messages as JSON instead of a table, and jq pulls out the header. ack_requeue_true puts the messages back; they are marked redelivered afterwards, which is a property of RabbitMQ and cannot be avoided when peeking.

Code

def death_story(headers):
    for d in headers.get("x-death", []):
        keys = ",".join(d.get("routing-keys", []))
        print(f"{d['count']}x {d['reason']:<14} in {d['queue']}  "
              f"(from {d['exchange']!r} / {keys}, first at {d['time']})")

In the Java client the values arrive as List<Map<String,Object>> with LongString values for strings; call toString() on them before comparing.

Two rules for your own consumers

In a retry loop, read the x-death count to decide when to give up. When a message cycles through a wait queue, the broker counts every round in the entry for your queue; look for the entry whose queue is your own queue, not blindly [0]. The count only covers dead-lettering inside the broker: after a republish (a replay, or a consumer that publishes the message again) RabbitMQ 4 starts again at 1, even if the publisher sent the old header along. To count across republishes, use a header of your own; application headers survive dead-lettering unchanged.
Strip the death headers when you deliberately republish for a fresh start. A replay tool should do the same. A message that keeps its x-death after a replay will be rejected on arrival by any consumer that follows rule one.

Where Warren fits

Warren shows each dead letter's x-death as a readable timeline: which queue, why, how often, when, and a plain-language verdict such as "rejected three times by orders.process after cycling through orders.retry.wait". Messages a framework republished itself are recognised by their x-exception-* or MT-Fault-* headers, and "dead since" is filled from when Warren first saw the message, which x-death cannot tell you.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation · 4-minute walkthrough on YouTube

How to replay dead-lettered messages in RabbitMQ

Viktor — Tue, 29 Sep 2026 14:09:33 +0000

Originally published at warrenops.io.

A dead-letter queue is filling up, the bug that caused it is fixed, and now 300 messages have to go back to where they came from. This is what your options are, what each one gets wrong, and the details that decide whether the replay works or produces a second incident.

What you are actually dealing with

RabbitMQ dead-letters a message when a consumer rejects it with requeue=false, when its TTL expires, when the queue is over its length limit, or (quorum queues) when the delivery limit is reached. The broker republishes the message to the queue's dead-letter exchange, and on the way it adds an x-death header: which queue it came from, why, when, the original exchange and routing keys. (Reading x-death is its own article.)

That header is the key to a correct replay. It tells you where the message was supposed to go. It is also the thing that trips up most home-made replays, because the header stays on the message unless you remove it, and consumers with retry logic often read it.

Three questions before you move anything:

Is the cause fixed? Replaying into a consumer that still fails just cycles the messages back into the DLQ, with a bigger x-death count each time.
Where do they go back to? To the original exchange with the original routing key (so bindings decide again), or directly to the queue they died in? The first is correct if the topology has changed; the second is what most people mean.
All of them, or some? A DLQ usually mixes several failure causes. Replaying everything replays the poison messages too.

Option 1: the management UI's "Move messages"

With the rabbitmq_shovel and rabbitmq_shovel_management plugins enabled, every queue page in the management UI has a Move messages panel: type a destination queue name, click, done. Under the hood it creates a temporary shovel that drains the queue into the destination via the default exchange.

rabbitmq-plugins enable rabbitmq_shovel rabbitmq_shovel_management

Good for: one queue, all messages, right now, you trust every message in there.

Not good for: anything selective. It moves everything, including the ten messages that are the actual poison. It does not strip x-death, so consumers that count retries may reject them on arrival. It publishes with no confirms visible to you and leaves no record of who moved what. And it only targets a queue by name, never the original exchange.

Option 2: a shovel you define yourself

A dynamic shovel gives you a bit more control than the button. src-delete-after: queue-length makes it stop after moving the messages that were in the queue when it started, so new dead letters arriving during the replay are not dragged along.

rabbitmqctl set_parameter shovel replay-orders \
  '{"src-protocol": "amqp091", "src-uri": "amqp://", "src-queue": "orders.dlq",
    "dest-protocol": "amqp091", "dest-uri": "amqp://",
    "dest-exchange": "orders", "dest-exchange-key": "order.created",
    "src-delete-after": "queue-length", "ack-mode": "on-confirm"}'

ack-mode: on-confirm is the important line: the shovel acknowledges a source message only after the destination confirmed it, so a crash mid-way duplicates at most one message and loses none.

Limits: one destination for the whole batch. If your DLQ collects messages from five different routing keys (common when one DLX serves a whole service), a shovel sends them all to the same place. And again: no selection, no header cleanup, no audit beyond the shovel's own log line.

Option 3: a script

This is what most teams end up with at 3 a.m., and it is fine if it gets four things right. In Python with pika:

import pika, json

conn = pika.BlockingConnection(pika.URLParameters("amqp://user:pass@rabbit/%2F"))
ch = conn.channel()
ch.confirm_delivery()                     # 1. publisher confirms

while True:
    method, props, body = ch.basic_get("orders.dlq", auto_ack=False)   # 2. never auto-ack
    if method is None:
        break
    headers = dict(props.headers or {})
    death = (headers.get("x-death") or [{}])[0]
    exchange = death.get("exchange", "")
    routing_key = (death.get("routing-keys") or [method.routing_key])[0]

    if not looks_replayable(body, headers):            # 3. select (your own check)
        continue                                         # leave it unacked, see 3.

    for k in list(headers):                             # 4. strip death bookkeeping
        if k.startswith("x-death") or k.startswith("x-first-death") or k.startswith("x-last-death"):
            del headers[k]
    props.headers = headers

    ch.basic_publish(exchange, routing_key, body, props, mandatory=True)
    ch.basic_ack(method.delivery_tag)                   # only after the confirm above returned

ch.basic_nack(0, multiple=True, requeue=True)          # puts every skipped message back
conn.close()

The four things, in order of how often they go wrong:

Confirms before acks. confirm_delivery() turns basic_publish into a call that raises when the broker did not accept the message. Ack the source message only after that. Ack first and a broker hiccup deletes messages you never republished.
No auto-ack. With auto_ack=True the message is gone the moment you fetched it. If the script dies between fetch and publish, so is the message.
Select, do not drain. Leave the messages you skip unacknowledged. While they are outstanding on the channel, the next basic_get returns the next message, and one basic_nack(0, multiple=True) at the end puts all of them back. Do not basic_nack each one with requeue=True right away: it goes back to the head of the queue and the next basic_get returns the same message again, forever. And do not leave them to the connection close: on a quorum queue that counts as a delivery, and after 21 such scans the default delivery limit drops the message (measured on RabbitMQ 4.3.6; details).
Strip the death headers. Otherwise the consumer sees a message with x-death[0].count = 3 and, if it counts retries, gives up immediately. Also look for application-level retry headers such as x-retry-count or Spring's x-exception-*; a replay is supposed to be a fresh start.

mandatory=True makes the broker return the message (basic.return) instead of silently dropping it when the routing key no longer matches any binding, which happens more often than you would think after a deploy renamed a queue. Register a return callback or the confirm will look like a success.

Flooding. Three hundred messages arriving in the same second at a consumer that was just restarted is how you get a second outage. Sleep between publishes, or replay in batches of fifty and watch the consumer's error rate in between.

Same idea in Java

With the RabbitMQ Java client: channel.confirmSelect(), channel.basicGet(queue, false), publish with channel.basicPublish(exchange, key, true, props, body), then channel.waitForConfirmsOrDie(), then channel.basicAck(tag, false). Spring AMQP users: RabbitTemplate with publisher-confirm-type: correlated and a ConfirmCallback, and take the messages with rabbitTemplate.receive(queue) inside a transaction or with manual acks via execute(ChannelCallback).

Option 4: do not replay by hand at all

Some failures are transient by nature: a downstream API was down for two minutes. For those, a retry topology handles the replay before anyone is paged: the consumer rejects into a wait queue with a per-queue TTL and no consumers, whose dead-letter exchange points back at the work queue. After N cycles (read x-death[0].count) the consumer routes the message to a final parking lot DLQ that a human looks at.

This does not remove the need for a manual replay; it removes the manual replays that were never necessary. The parking lot still fills up with the messages that failed for a real reason, and those you replay after the fix.

Checklist

	Move button	Shovel	Script
Select which messages	no	no	yes
Original exchange and key per message	no	no (one target)	yes
Strip x-death and retry headers	no	no	if you write it
Loss-free (confirm before ack)	yes	with `on-confirm`	if you write it
Throttle	no	no	if you write it
Who moved what, when	no	no	if you log it
Edit a message before replay	no	no	if you write it

Whatever you use: replay a handful first, watch the consumer, then the rest. And write down what you did. The postmortem will ask.

Where Warren fits

Warren is the script above as a tool: browse the DLQ, see the payload and why each message died, select some or the first N, replay them to their original exchange and routing key, throttled, with death and retry headers stripped and publisher confirms on every message. Every replay is recorded with who, when and the per-message outcome. Free for one broker.

The 4-minute walkthrough, from 200 dead letters to a throttled replay:

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation · 4-minute walkthrough on YouTube