One Bad Message Kept Killing My Queue: Poison Messages and Dead-Letter Queues in n8n

This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.

The queue that never drained

A workflow queue looked healthy from the outside: new jobs arriving, executions running, no errors on the dashboard. But the backlog never shrank. Every morning the queue was longer than the night before, and the same job kept appearing at the front.

The job itself was not special. It was one malformed message — a payload with a field type the workflow did not expect — and every time a worker picked it up, the workflow threw, the job went back to the queue, and the next worker picked it up again. The queue was not slow. It was stuck behind one message that could never succeed, and every retry was just the same failure wearing a new timestamp.

That is a poison message: an input that fails deterministically, on every attempt, forever. Retries do not fix it. Backoff only spaces out the failures. And if your queue processes in order, or your worker pool is small, one poison message can hold up everything behind it while looking, from the metrics, like ordinary load.

Retries cannot tell "not yet" from "never"

The root mistake was treating all failures as transient. A timeout, a 429, a brief provider outage — those are "not yet" failures, and retrying is the right answer. A schema violation, a missing required field, a value that can never parse — those are "never" failures. No amount of waiting changes them. The payload will be just as malformed on attempt fifty as it was on attempt one.

So the first fix is classification at the point of failure, not at the point of retry. Before a job goes back to the queue, the workflow asks one question: did this fail because the world was temporarily unavailable, or because this input is broken? Transient failures get retried with backoff. Permanent failures never go back to the queue at all.

In practice I classify by failure kind, not by error text: a validation error, a parse error, or a missing-field error is permanent; a timeout, a rate limit, or a 5xx from a provider is transient. Error text is a trap — providers reword messages, and a string match that worked last month silently stops matching. Classify on the shape of the failure: which node threw, what kind of error it was, whether the input itself was the problem.

A dead-letter queue is a decision, not a trash can

Permanent failures go to a dead-letter queue (DLQ): a separate store that holds the failed job, the full original payload, the error, the attempt count, and the time it failed. The DLQ is not where messages go to be forgotten. It is where they go to wait for a human decision, with all the evidence attached.

Three things make a DLQ useful instead of decorative. First, it keeps the original payload byte-for-byte, not a summary of it — a replay needs the exact input, and a truncated log line is not a replay. Second, it records why the job failed in a structured way, so a week later you can still tell "schema violation" from "provider rejected it." Third, it pages someone. A DLQ that nobody watches is just a slower trash can; mine sends one alert per new DLQ entry, deduplicated by failure kind, so a flood of identical poison messages produces one page, not a hundred.

Replay is the whole point

The DLQ earns its keep when the underlying problem gets fixed. Maybe the sender fixes their payload format. Maybe you deploy a workflow version that handles the new shape. Either way, the fix is only half the job — the failed messages still need to run.

So replay is a first-class operation, not a manual hack. A replay takes a DLQ entry, re-queues it as a new job with the same payload, and records the link between the original failure and the replay attempt. If the replay succeeds, the entry is marked resolved with the replay's job id. If it fails again, it goes back to the DLQ with a new attempt count — it does not vanish, and it does not loop forever, because replay attempts are counted separately from ordinary retries and capped.

Two rules keep replay safe. Replay one entry at a time, or a small batch, and watch the first few — replaying a hundred poison messages at once just recreates the original outage with better intentions. And never replay into the same queue position blindly: if the workflow version changed since the failure, the replay should run on the new version, because replaying against the old code reproduces the old failure exactly.

Cap retries, even for transient failures

Even transient failures need a ceiling. A provider that is down for a day should not produce a job that has retried four hundred times and will keep retrying for a week. I cap ordinary retries — a small number, with backoff — and when the cap is hit, the job goes to the DLQ too, marked as "retries exhausted" rather than "permanently broken." The distinction matters for the human reading it: one says "this input is bad," the other says "the world was broken for longer than we were willing to wait." Both need a decision, but they are different decisions.

The cap also protects the queue. Without it, a long outage fills the queue with jobs that are all waiting to retry, and when the provider recovers, the flood of retries competes with fresh work. With a cap, the backlog after an outage is bounded, and the DLQ holds the overflow in an orderly pile instead of a stampede.

The test is a poison message, not a happy path

Testing this means deliberately feeding the workflow a message that can never succeed, and asserting four things: it does not loop forever; it lands in the DLQ with the full payload and a structured reason; the main queue keeps draining the messages behind it; and a replay after a fix succeeds and marks the entry resolved. If any of those fails, the poison message is still in charge of your queue — you just have not noticed yet, because the backlog looks like load.

Count this at the provider side, not in n8n: record how many times each message was actually attempted, because a job that "retried three times" in n8n may have hit the provider more times than that if the failure happened after the side effect.

The principle

A queue is a promise: every message will either be processed or be accounted for. Retries keep the first half of that promise for transient failures. A dead-letter queue keeps the second half for everything else. Without one, a single bad message does not just fail — it fails forever, blocks everything behind it, and hides in your metrics as ordinary load. Give permanent failures a place to go, a reason they can be read later, and a replay path back — and the queue goes back to being a queue, instead of a monument to one bad payload.

---

I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.