Replay one message first: a canary for RabbitMQ dead letters

Originally published at warrenops.io.

The fix is merged, the pipeline is green, and 143 messages wait in orders.dlq. Replaying all of them at once is a bet: that the new version runs on every consumer replica, that the fix covers these payloads and not only the one in the ticket, and that the consumer does not fail on something else further down. If the bet is lost, 143 messages die a second time, the alerts fire again, and the dead-letter queue now mixes old failures with new ones. Sending one message first and waiting a moment costs a couple of minutes and settles the question for almost nothing.

The idea as a video (canary replay and the incident report, captions, no sound needed):

Before you send anything

A canary only means something if its message is processed during the wait. Check two things on the work queue first:

• A consumer is attached. rabbitmqctl list_queues name messages consumers or the queue's page in the management UI. With no consumer the canary sits in the queue and every wait "passes".
• The work queue is not backed up. If 20,000 messages are ahead of the canary, it is processed after the wait, not during it. Wait until the backlog is gone, or publish the canary later.

Then pick the canary from the failure you mean to fix. If the dead-letter queue holds two different exceptions, one canary passing says nothing about the other; send one per exception.

Send one message, tagged

The replay is the same as for a whole batch, stopped after the first message: read it from the dead-letter queue, publish it to its original route with a header that names this replay, wait for the publisher confirm, then acknowledge the original.

import uuid, pika

DEATH = ("x-death", "x-first-death-", "x-last-death-", "x-delivery-count", "x-acquired-count")
replay_id = str(uuid.uuid4())

ch.confirm_delivery()
method, props, body = ch.basic_get("orders.dlq", auto_ack=False)
death = props.headers["x-death"][0]  # broker dead-lettering; Spring or MassTransit error queues keep the route elsewhere
headers = {k: v for k, v in (props.headers or {}).items() if not k.startswith(DEATH)}
headers["x-replay-id"] = replay_id
props.headers = headers
ch.basic_publish(death["exchange"], death["routing-keys"][0], body, props, mandatory=True)
ch.basic_ack(method.delivery_tag)  # confirm_delivery made the publish wait for the broker
print("canary", props.message_id, "replay", replay_id)

Your own headers survive dead-lettering on classic and quorum queues, so if the canary dies again it comes back carrying x-replay-id, together with a fresh x-death. That pair is what tells its return apart from a new failure that arrived in the meantime. The details are in Did the replay work?.

How long to wait

Long enough for the consumer's whole retry chain to run out, plus the time to reach the canary. The chain is usually the larger part:

Retry set-up	When a failing message reaches the DLQ
No retries, basic.reject with requeue=false	within a second of delivery
Spring RetryTemplate / @Retryable, 3 attempts, backoff 1 s × 2	after about 3 s (1 + 2), plus processing
Broker-side retry: reject into a wait queue whose TTL dead-letters back, 5 rounds of 30 s	after about 2.5 minutes
MassTransit UseDelayedRedelivery with 5, 15, 30 minutes	after about 50 minutes

Read the numbers off your own configuration. Two minutes covers in-process retries with short backoffs; a delayed-redelivery set-up needs a canary that waits an hour, or a canary sent before lunch and the rest afterwards.

Watch for it

During the wait nothing else is replayed, so the dead-letter queue changes only through new failures and through the canary coming back. Check its depth; when it grows, look at what was added:

def came_back(ch, queue, replay_id, depth_before):
    depth = ch.queue_declare(queue, passive=True).method.message_count
    if depth <= depth_before:
        return False
    # returns arrive at the tail: read the whole queue and put everything back
    found, tags = False, []
    for _ in range(depth):
        method, props, _ = ch.basic_get(queue, auto_ack=False)
        if method is None:
            break
        tags.append(method.delivery_tag)
        h = props.headers or {}
        found = found or (h.get("x-replay-id") == replay_id and "x-death" in h)
    for tag in tags:
        ch.basic_nack(tag, requeue=True)
    return found

Take depth_before right after the canary left the queue, and call this every few seconds until the wait is over. Reading and putting back marks messages as redelivered. On a quorum dead-letter queue with a delivery limit, check what a requeue costs on your RabbitMQ version before you peek at all.

If your consumer sends failures to a different queue than the one the message came from (a parking queue after the last retry, a Spring .dlq next to a broker DLQ), watch that queue too.

Decide

• The canary came back. Stop. The other 142 stay where they are, which is exactly where you want them while you read the new exception on the canary. Nothing is mixed up: one message died twice, and it says so in its headers.
• It did not come back within the wait. Replay the rest, ideally throttled, so a consumer that has only just recovered is not flooded. Tag them with the same replay id and keep looking for returns over the next hours, as for any replay.
• New messages arrived but none is the canary. Something else is failing right now. Find out what before the 142 join them.

When a passed canary proves nothing

• The consumer acknowledged and dropped it. "Not back" means not seen in a dead-letter queue. If the consumer logs a business id, check that the canary's id shows up as processed.
• The canary was not representative. It was the one payload the fix was written for. If the batch holds several customers, tenants or message versions, a canary from each is cheap.
• The failure depends on load. A canary does not hit a rate limit, a connection pool or a lock that 142 messages at full speed do. That is what throttling the rest is for.
• The wait was shorter than the retry chain. The canary is still in its third retry when the rest goes out. Check the table above against your own settings.

Checklist

• A consumer is attached to the work queue and it is not backed up.
• One canary per exception, tagged with a replay id, death headers stripped.
• Wait longer than the consumer's full retry chain.
• Back in the queue with your replay id and a new x-death: stop. Not back: replay the rest, throttled.
• Keep watching for returns after the rest went out.

---

Where Warren fits

Warren's replay dialog has a Canary option: it replays the first message of the selection alone, watches the source queue for it during the wait (2 minutes by default, 5 seconds to 30 minutes), and replays the rest only if it has not died back by then, at the rate you set. The replay's page says "Canary out" with the time the wait ends, then "Canary passed" or "Canary died again"; a canary that died stops the replay and the audit log records why. The Failures page offers the same per exception, and its incident report lists every replay of that failure with how its canary went. Free for one broker.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation