DEV Community

Cover image for One bad message can stall your whole queue. Dead-letter queues are the fix.
InfraSim
InfraSim

Posted on AI-assisted

One bad message can stall your whole queue. Dead-letter queues are the fix.

Here's a failure that looks impossible until you've seen it once.

You run an order-confirmation service. The web app drops an OrderPlaced message on a queue, and a pool of workers picks messages up, renders the email and sends it. It has handled millions of orders without a hiccup.

Then one order arrives with a currency field that's null. The template renderer throws. The worker never acknowledges the message, so after the visibility timeout the broker hands it out again. The next worker throws too. And the next.

If the queue preserves order (a Kafka partition, a FIFO message group), that one message is now at the head of the line and nothing behind it moves. Every confirmation email after it just waits. If it's a plain queue, the rest still flow, but the bad message keeps eating a worker slot every cycle, forever, and if your consumer crashes rather than catching the exception, it restarts and crash-loops the whole pool.

That message is a poison message: one that will never succeed, no matter how many times you try it.

Transient vs permanent failures

Retries exist because most failures are transient. The database was failing over, the network blipped, a downstream service was briefly overloaded. Try again in a moment and it works.

A poison message fails for a reason that lives inside the message: bad data, a schema your code doesn't understand, an edge case your logic can't handle. Retrying it changes nothing. You can't always tell the two apart from one error, so the safe default is to give every message a budget: a fixed number of attempts, and then it's out of the main queue.

Where does it go? Somewhere it can't hurt anyone, but where a human can find it.

The dead-letter queue

A dead-letter queue (DLQ) is a separate queue for messages that have used up their attempts. The main line keeps moving, and the failure is parked with enough context to fix it later.

Most brokers support it directly:

  • Amazon SQS has a redrive policy: after maxReceiveCount receives, a message moves to the queue you name as its DLQ.
  • RabbitMQ has dead-letter exchanges. A message is dead-lettered when a consumer rejects it without requeueing, when its TTL expires, or when the queue is over its length limit.
  • Kafka has no broker-level DLQ. It's a consumer-side pattern: on repeated failure you publish the record to a separate topic. Kafka Connect and Spring Kafka both ship helpers for it.

A setup that actually works

A DLQ by itself is just somewhere for messages to go. What makes it useful is everything around it:

  1. Retry with backoff before dead-lettering. Three to five attempts is plenty, spaced out with exponential backoff and a bit of jitter. Immediate retries hammer a dependency exactly when it's weakest, which is how a slow database becomes a down database.
  2. Skip retries for errors you know are permanent. A validation error or an unparseable payload will fail the same way every time. Send it straight to the DLQ.
  3. Keep the evidence. Store the original payload, the error, the attempt count and timestamps with the dead message. "Something failed" is not what you want to read at 3 a.m.
  4. Alert on it. Page or notify when DLQ depth goes above zero, or when its oldest message passes an age you care about. A DLQ nobody watches is just a slower way to lose data: the messages sit there until retention deletes them.
  5. Give it longer retention than the source queue. On some brokers, SQS standard queues among them, a message's age keeps counting from when it was first enqueued, not from when it was dead-lettered.
  6. Have a way back. Once the bug is fixed, you need to replay the messages (SQS calls this a redrive). Replaying means some messages get processed twice, so make consumers idempotent: sending the same confirmation twice should be impossible, or at least harmless.

Back to the order service: with a budget of three attempts and a DLQ, the null currency message fails three times, gets parked, and an alert fires. Every other customer gets their email. Someone fixes the template the next morning, redrives one message, and that customer gets theirs too, a few hours late but in one piece.

Retries are for bad luck. The dead-letter queue is for bad data. A lot of queue trouble starts when you mix the two up.


We drafted this article with help from an AI assistant and reviewed it before publishing.

Top comments (0)