DEV Community

Cover image for Six ways RabbitMQ loses a message without an error (measured on 4.3.6)
Viktor
Viktor

Posted on Originally published at warrenops.io

Six ways RabbitMQ loses a message without an error (measured on 4.3.6)

#rabbitmq #devops #messaging #python

Originally published at warrenops.io.

None of these throw an exception. The publisher gets its confirm, the consumer runs, the queue exists, and some messages are simply not there anymore, or arrive late, or only one of your consumers works. Every example below ran against a throwaway RabbitMQ 4.3.6 with Python and pika; the output is what the script printed.

1. auto_ack=True: one crash, 99 orders gone

A queue holds 100 orders. A consumer with auto_ack=True handles the first one and crashes on the second:

handled order-7001
crash while handling the second one (100 message(s) had arrived in the consumer)
auto.q holds 0 message(s)
Enter fullscreen mode Exit fullscreen mode

With automatic acknowledgement the broker considers a message done the moment it is sent. It also ignores the prefetch, so it pushes the whole queue into the consumer's buffer at once. When the process dies, everything in that buffer is gone.

The same crash with manual acks and basic_qos(prefetch_count=10):

handled order-7001, ack
crash while handling the second one (10 message(s) had arrived in the consumer)
auto.q holds 99 message(s)
Enter fullscreen mode Exit fullscreen mode

Fix: auto_ack=False, set a prefetch, and ack after the work is done.

2. Confirmed by the broker, delivered to nobody

A topic exchange shop with one binding, orders.created. The publisher has confirms on and sends with the key order.created, one letter short:

publish order-7101 to shop with key order.created: confirmed
shop.orders       holds 0 message(s)
Enter fullscreen mode Exit fullscreen mode

A publisher confirm says the broker took the message, not that any queue got it. A message no binding matches is dropped, and still confirmed. With mandatory=True the broker returns it instead, and pika raises:

publish order-7101 to shop with key order.created: RETURNED, NO_ROUTE
Enter fullscreen mode Exit fullscreen mode

Fix: publish with mandatory=True and handle the return, or give the exchange an alternate-exchange that catches whatever no binding matches. More on the return in NO_ROUTE 312 and the mandatory flag.

3. The second consumer does nothing

A backlog of 20 slow orders. Consumer A is running; consumer B is started to help. Neither sets a prefetch:

consumer A has 20 order(s) waiting for it
consumer B has 0 order(s) waiting for it
Enter fullscreen mode Exit fullscreen mode

Without basic_qos the prefetch is unlimited, so the broker sends A the whole backlog the moment it subscribes. B idles while A works through all 20 alone. With prefetch_count=1:

consumer A has 1 order(s) waiting for it
consumer B has 1 order(s) waiting for it
Enter fullscreen mode Exit fullscreen mode

Fix: always set a prefetch. 1 for slow jobs, a few dozen for fast ones.

4. The dead-letter queue deleted itself

A dead-letter queue declared with x-expires: 5000, copied from a temporary queue's arguments. Three orders fail and arrive in it as they should. Six seconds later:

exp.dlq           holds 3 message(s)
exp.dlq           does not exist
Enter fullscreen mode Exit fullscreen mode

x-expires deletes a queue that has had no consumers for that long, messages included, and none of them is dead-lettered. A dead-letter queue has no consumers almost by definition.

Fix: no x-expires, no x-message-ttl and no length limit on a dead-letter queue. It should keep messages until a person decides.

5. Priority 9, and it still waits its turn

Five reports at priority 0, then a password reset at priority 9, into a queue declared without x-max-priority:

consumer gets: report-1, report-2, report-3, report-4, report-5, PASSWORD-RESET
Enter fullscreen mode Exit fullscreen mode

A classic queue without x-max-priority ignores the message property completely. Declared with x-max-priority: 10:

consumer gets: PASSWORD-RESET, report-1, report-2, report-3, report-4, report-5
Enter fullscreen mode Exit fullscreen mode

Fix: set x-max-priority when you declare the queue; it cannot be added later. Quorum queues sort by priority without any argument, and there is a catch of their own: on 4.3.6 a message published without a priority was delivered as if it had priority 4, ahead of messages that explicitly carry 0 to 3.

6. Durable queue, restart, messages gone

A durable classic queue, three orders published with confirms but without delivery_mode=2, then the node restarts:

tr.q holds 3 message(s)
node restarted
tr.q              holds 0 message(s)
Enter fullscreen mode Exit fullscreen mode

Durable keeps the queue. A message survives a restart only if it is persistent as well.

Fix: publish with pika.BasicProperties(delivery_mode=2), or use quorum queues, which store every message on disk.

Checklist

  • Manual acks, after the work, with a prefetch.
  • mandatory=True or an alternate exchange on every exchange you publish to.
  • No x-expires, TTL or length limit on dead-letter queues.
  • x-max-priority at declaration if you rely on priorities.
  • Persistent messages, publisher confirms and durable queues, all three.

Twelve more traps of the same kind, from quorum delivery limits to the double ack, as one video (6:52, captions, no sound needed):

Where Warren fits

Most of these end with a message that is gone, and nothing can bring it back. The ones that end in a dead-letter queue are where Warren helps: it groups the dead letters by exception, shows what each one carried, and replays the right ones to their original route, with a dry run first and an audit log afterwards.

Try Warren in a minute (one compose file, demo broker with real dead letters included) · Documentation

Top comments (0)