Most queue failures are not "retry problems." They are routing problems. A transient timeout deserves another attempt; a malformed payload should leave the hot path; and a poison message must not hold every healthy job hostage. This short Node.js drill turns those decisions into executable assertions you can explain in a backend interview.
What is the invariant?
In an interview, start with the invariant instead of naming a broker:
Every delivery ends in exactly one observable outcome: acknowledged, retried with a bounded budget, or isolated for inspection.
That sentence is more useful than "we will use SQS" or "we will add retries." It gives you something to test.
A message should carry a stable identifier and an attempt count. The count must increase monotonically. A successful handler acknowledges the message. A retryable failure returns it to the queue, usually with backoff. A permanent failure, or a retry budget that has been exhausted, moves it to a dead-letter destination and raises an alert.
This is the core idea behind a dead-letter queue (DLQ). In Amazon SQS, a redrive policy uses maxReceiveCount\ to decide when a message is moved from the source queue to its DLQ. AWS describes the DLQ as a place to inspect and diagnose unconsumed messages, then redrive them after the cause is fixed. A DLQ is not a trash can and it is not a substitute for observability.
Which failures should retry?
The first design decision is classification.
Retry a failure when the next attempt could plausibly see a different world:
- a timeout or connection reset
- a temporary database outage
- a downstream 429 or 503
- a worker crash before the message was acknowledged
Do not retry just because an exception was thrown. A malformed payload, a missing required record, or a violated business rule will usually fail the same way forever. Retrying it burns worker capacity and can block an ordered lane.
The classifier does not need to predict the future perfectly. It needs to make the policy explicit and leave unknown errors visible. In production, you might attach an error code, a retry-after duration, and the queue name to the log entry.
A minimal routing function
Here is a dependency-free policy function. It does not pretend to be a complete broker client; it isolates the decision you should be able to defend on a whiteboard.
\`js
import assert from "node:assert/strict";
const MAX_ATTEMPTS = 3;
const retryableCodes = new Set(["ETIMEDOUT", "ECONNRESET", "429"]);
function route(message, process) {
const attempt = message.attempt + 1;
try {
process(message);
return {
action: "ack",
message: { ...message, attempt },
};
} catch (error) {
const retryable = retryableCodes.has(error.code);
const exhausted = attempt >= MAX_ATTEMPTS;
return {
action: retryable && !exhausted ? "retry" : "dead-letter",
message: {
...message,
attempt,
failureCode: error.code,
},
};
}
}
`\
There are two details worth calling out.
First, attempt\ is calculated before the handler runs. The failure record therefore tells you which delivery failed, even when the handler throws immediately.
Second, "retryable" is not enough. The retry budget is a second gate. A flaky dependency can be transient for hours; without a ceiling, your queue has an infinite loop disguised as resilience.
Test the boundaries, not just the happy path
A good interview answer includes the smallest test that would catch a bad policy. Add these assertions below the function:
\`js
const timeout = Object.assign(
new Error("upstream timed out"),
{ code: "ETIMEDOUT" },
);
const invalid = Object.assign(
new Error("missing account id"),
{ code: "INVALID_PAYLOAD" },
);
const base = {
id: "evt-42",
idempotencyKey: "order-81",
attempt: 0,
};
assert.equal(
route(base, () => { throw timeout; }).action,
"retry",
);
assert.equal(
route({ ...base, attempt: 2 }, () => { throw timeout; }).action,
"dead-letter",
);
assert.equal(
route(base, () => { throw invalid; }).action,
"dead-letter",
);
assert.equal(
route(base, () => {}).action,
"ack",
);
console.log("queue routing assertions passed");
`\
Run it with Node.js and you should see:
\
queue routing assertions passed
\\
The useful part is not the number three. It is that the policy has four observable exits:
| Input | Decision | Why |
|---|---|---|
| timeout on attempt 1 | retry | the dependency may recover |
| timeout on attempt 3 | dead-letter | the budget is exhausted |
| invalid payload | dead-letter | another delivery will not repair the data |
| successful handler | ack | the message can leave the source queue |
What changes in a real worker?
The function above returns an action. The worker has to make that action safe.
Backoff and visibility. A retry should not immediately hammer a failing dependency. Use exponential backoff with jitter, and make the broker's visibility timeout longer than the handler's expected processing window. A timeout that expires while the handler is still running can create duplicate deliveries.
Partial batch failure. If a broker delivers ten messages at once, do not fail the whole batch because one payload is bad. Acknowledge successful records, retry only the retryable records, and dead-letter the permanent ones. The exact API differs by broker, but the contract is the same.
Per-key ordering. A DLQ can intentionally break strict ordering. AWS warns against using a DLQ with FIFO workloads when moving one message out of the sequence would change the meaning of later operations. If ordering is a hard business invariant, pause the affected partition and escalate instead of silently skipping.
Metrics. Alert on DLQ depth, age of the oldest dead-lettered message, retry rate, and per-key lag. Total queue depth alone can look healthy while one partition is stuck behind a poison message.
Replay is a new delivery
A replay is not a magical undo. It is another delivery with another chance to create a side effect.
Before redriving a message, record:
- the original message id and failure reason;
- the code or data change that should make the replay succeed;
- a replay id or idempotency key;
- the operator and timestamp;
- the result of the replay.
The handler should use an idempotency key when it calls an external system. Otherwise, a message that succeeded at the payment provider but timed out before acknowledgement can be charged again on replay. Keep the original payload for diagnosis, and make the replay command rate-limited so a fixed bug does not create a second traffic spike.
For a small system, a manual redrive command plus a dashboard may be enough. For a high-volume system, put replay behind a separate worker with a canary batch, a maximum messages-per-second setting, and a kill switch.
How to say this in an interview
A concise answer can follow this sequence:
- "I will classify failures into retryable and permanent."
- "I will cap attempts and add backoff so a poison message cannot loop forever."
- "Permanent or exhausted messages go to a DLQ with an alert and enough metadata to diagnose them."
- "I will preserve idempotency across retries and replays."
- "I will call out the ordering trade-off: a DLQ may be wrong for a strict FIFO workflow."
- "I will test one transient failure, one exhausted retry, one permanent failure, and one success."
That answer demonstrates judgment, not just vocabulary. You are showing how the system behaves when the happy path disappears.
For a timed practice round, spend five minutes drawing the states, five minutes implementing route\, and five minutes explaining replay and ordering. A tool such as aceround.app - AI interview assistant can provide follow-up questions while you practice, but the important artifact is still your own failure policy and tests.
Sources
Disclosure: AI assistance was used for outlining and editing. The code and technical claims were reviewed and the assertions were run with Node.js 22.
Top comments (0)