On a Saturday in March our order API started timing out on every checkout, and so did returns, stock updates and the driver app. Each of them publishes messages to the same RabbitMQ cluster, and at a quarter to eleven that cluster stopped accepting messages from anybody.
The management page showed a red disk alarm on one of the three nodes. RabbitMQ has a disk free limit, fifty megabytes by default, and when free space falls below it the node raises an alarm and blocks every connection that publishes. The alarm is cluster wide: if one node is short of disk, publishers on all nodes are blocked. Connections that only consume carry on, so a backlog can still be worked off.
The disk had been filled by a queue for our analytics loader. That consumer had been switched off on Wednesday for a schema change and nobody switched it back on. The queue had no length limit, so for three days it received a copy of every order event, and by Saturday it held about forty gigabytes on a node with fifty.
Being blocked is not an error. The broker simply stops reading from the publisher's socket, and the client library is told through a connection.blocked notification, which none of our services listened for. Each publish call waited on a full TCP buffer with no timeout, inside a request handler, so the API threads piled up behind it and every caller timed out. The alarm was protecting the broker, and our services took it as an invitation to wait for ever.
We purged the analytics queue, which the loader could rebuild from the database, and everything recovered within a minute. Every queue now has a policy with max-length-bytes, and the queues we can afford to lose use the reject-publish overflow, so a forgotten queue turns away its own messages instead of filling the disk. Analytics has moved to a small cluster of its own, because an alarm on a shared broker blocks everyone sharing it. Our publishers register a blocked listener and fail fast instead of hanging. We alert on any queue growing without consumers for an hour, and on free disk in gigabytes. And switching a consumer off now opens a ticket that stays open until it is switched back on.
A broker is shared backpressure. One queue nobody was reading decided how long every service we run would wait.
– Sergey Shinder
Top comments (0)