When an order is packed, our order service publishes a label request to a topic exchange in RabbitMQ called shipping, with a routing key made of the event and the carrier, such as label.requested.dpd or label.requested.royalmail. Our carrier integration consumes from one queue bound to that exchange with the pattern label.requested.*, and books a label with whichever carrier the key names. The order service publishes with confirms switched on and treats a request as sent only when the broker has acknowledged it.
In October we added DHL's express service next to their standard one, and the new carrier code went into the routing key as dhl.express. For two days no DHL Express parcel got a label. About nineteen hundred orders sat packed in our warehouses while the order service logged every request as published and confirmed, and the carrier integration's queue looked normal, because nothing for DHL Express ever reached it.
In a topic exchange the words of a routing key are separated by dots. A star in a binding matches exactly one word and a hash matches zero or more. label.requested.dhl.express has four words, label.requested.* matches three, and no binding on the exchange matched the new key. RabbitMQ drops a message that routes to no queue, unless it was published as mandatory, in which case it is returned to the publisher, or the exchange has an alternate exchange to pass it to. We had neither.
The confirm did not help, because a confirm is not a receipt from a queue. For a message that routes nowhere, RabbitMQ sends the confirm as soon as the exchange has worked out that there is no queue to deliver it to. Our publisher had been waiting for exactly the acknowledgement that meant the message was gone.
Carrier codes in routing keys are now single words, dhl_express, and the code that builds the key refuses a dot. Every exchange we own has an alternate exchange set by policy, which leads to a queue called unroutable, and a single message in it pages the integration on call. The order service publishes with mandatory set, and a returned message is treated as a failed publish and stays in its outbox. We republished the nineteen hundred requests from that outbox the same afternoon.
A confirm told us the exchange had finished with each message. Dropping it was one of the ways an exchange can finish.
– Sergey Shinder
Top comments (0)