A logistics webhook should be retried until the delivery decision is durable, while the receiver observes one business event. The least complex design is a durable work record, a bounded retry schedule, and an idempotency key derived from the event itself. A dead-letter queue is for work that needs a separate recovery policy; it is not a substitute for idempotency.
Short answer: write the delivery attempt before sending, use a stable event ID as the receiver's idempotency key, acknowledge the queue message only after the attempt state is durable, and move exhausted work to a dead-letter path. A timeout after the remote system accepted the request is still an ambiguous result. Retrying it is correct only when the receiver deduplicates it.
This matters in logistics because one shipment update can trigger labels, customer notifications, or a carrier handoff. A duplicate notification is noisy. A duplicate handoff can be expensive. The example keeps the queue generic so the design can live behind a self-hosted broker, a managed queue, or an existing worker library.
Duplicates are inevitable.
How should teams handle failed background tasks using a dead-letter path?
Treat a timeout as “outcome unknown,” not as “delivery failed.” That distinction determines the rest of the design. The worker cannot inspect the remote side after its connection disappears, so it records the attempt, waits according to policy, and sends the same event ID again. The receiver decides that the second request is a replay and returns the original result.
I initially treated every failed background task as safe to retry immediately. The useful correction is to classify the failure first: a rejected payload, a temporary outage, and an unknown network outcome need different handling. The trade-off is deliberate. More state makes the worker harder to explain, but one boolean loses the information needed to recover a shipment safely.
Here is the boundary I would write down before implementing the worker. An event named \'shipment.delivered\' is created once, with an immutable ID such as \'evt_42\'. The producer records it as pending and publishes a message containing only that ID. The first worker call reaches the carrier endpoint, but the socket closes before the response arrives. The worker records unknown and schedules the next attempt; it does not invent a new event ID. The second call sends the same ID. If the carrier had already committed the update, its idempotency record returns the saved result. If it had not, the carrier applies the update and stores the key. In both cases, the local ledger can mark the event delivered after a confirmed response. If the second call is rejected because the payload is invalid, another retry adds no value, so the record goes to the dead-letter path with the payload error attached. This is why the event ID belongs in both the ledger and the outbound request. The queue message can be duplicated, delayed, or redelivered; identity survives those transport behaviors.
Keep the retry decision boring.
There are three useful states in the ledger: pending, delivered, and dead. An attempt can also carry unknown as a diagnostic outcome, but it should not become a fourth business result. A delivery is either accepted by the receiver or still needs another attempt.
The data flow is short: a shipment event creates one ledger row, a queue message points to that row, a worker claims the row, and the receiver sees the immutable event ID. The queue controls when work runs. The ledger controls what work is allowed to do. Keeping those responsibilities separate prevents a broker acknowledgement from being mistaken for business success.
The design has limits. It cannot guarantee exactly-once effects at an endpoint that does not support idempotency, and it cannot make a slow carrier API responsive. For a low-value notification, dropping after a short retry window may be the right choice; for a handoff, retaining the dead-letter record and requiring review is safer. Choose the recovery cost before choosing the queue settings.
A runnable Node.js implementation
The following TypeScript uses an in-memory store so the control flow is easy to run and test. In production, replace the store with a transactional database and replace publish with the queue client's publish operation. The interfaces are intentional: they make the delivery policy testable without binding it to one broker.
type DeliveryState = "pending" | "delivered" | "dead";
type AttemptOutcome = "accepted" | "rejected" | "unknown";
interface ShipmentEvent {
id: string;
shipmentId: string;
status: "packed" | "in_transit" | "delivered";
occurredAt: string;
}
interface DeliveryRecord {
event: ShipmentEvent;
state: DeliveryState;
attempts: number;
nextAttemptAt: number;
lastOutcome?: AttemptOutcome;
}
interface DeliveryStore {
get(eventId: string): Promise<DeliveryRecord | undefined>;
put(record: DeliveryRecord): Promise<void>;
markDelivered(eventId: string): Promise<void>;
markDead(eventId: string, outcome: AttemptOutcome): Promise<void>;
}
interface WebhookClient {
send(event: ShipmentEvent, idempotencyKey: string):
Promise<"accepted" | "rejected">;
}
const MAX_ATTEMPTS = 6;
const RETRY_DELAYS_MS = [30_000, 120_000, 600_000, 1_800_000, 7_200_000];
function nextDelay(attempts: number): number {
return RETRY_DELAYS_MS[Math.min(attempts, RETRY_DELAYS_MS.length - 1)];
}
async function deliver(
eventId: string,
store: DeliveryStore,
client: WebhookClient,
now = Date.now(),
): Promise<"ack" | "retry" | "dead-letter"> {
const record = await store.get(eventId);
if (!record || record.state !== "pending") return "ack";
const attempt = record.attempts + 1;
record.attempts = attempt;
try {
const result = await client.send(record.event, record.event.id);
if (result === "accepted") {
await store.markDelivered(eventId);
return "ack";
}
record.lastOutcome = "rejected";
} catch {
record.lastOutcome = "unknown";
}
if (attempt >= MAX_ATTEMPTS) {
await store.markDead(eventId, record.lastOutcome ?? "unknown");
return "dead-letter";
}
record.nextAttemptAt = now + nextDelay(attempt - 1);
await store.put(record);
return "retry";
}
The important ordering is easy to miss. On acceptance, markDelivered must be durable before the queue message is acknowledged. On an unknown result, the record is persisted before the message is made available again. If the process exits between those operations, the broker can redeliver the message; the state check makes that replay harmless.
A real store needs one more guard: claim the record with a compare-and-set or a short lease. Two workers can receive the same message during a visibility timeout or after a consumer restart. Both may reach the remote endpoint, so the receiver's idempotency key remains the final protection. A lock reduces duplicate attempts; it cannot erase the uncertainty created by a network timeout.
For teams using a background task library, this boundary is the part worth preserving when the library changes: the task payload carries an event ID, while the ledger owns state and timing.
Why a dead-letter path does not solve duplicates
A dead-letter queue answers a routing question: where should a message go after the normal consumer cannot process it? It does not answer an identity question: are these two deliveries the same shipment event? Those questions need different fields and different operators.
For this workflow, dead-lettering is appropriate after six attempts, after a schema validation failure, or after a policy-defined non-retryable response. The exact count is a policy choice, not a reliability law. Six attempts gives the example enough time to cover short outages while keeping the recovery queue visible; a team with a different outage budget should choose a different threshold and record why.
Do not dead-letter every HTTP error. A malformed payload should be isolated quickly. A temporary overload should use backoff. A timeout should be labeled unknown. A receiver response that confirms the event was accepted should mark the ledger delivered even if the worker's local request completed late. These classifications are more useful than a single failed: true flag.
Queue acknowledgement has the same boundary. Consumer acknowledgement tells the broker that the consumer has finished its handling contract; it does not prove that a remote HTTP server has committed the shipment update. Acknowledging before the ledger write creates loss on process exit. Acknowledging after the remote call but before the ledger write creates a duplicate risk on retry. The durable ledger write closes that gap.
Testing the ugly outcomes
I would start with the test that is easiest to omit: make the fake receiver record the event, then throw a timeout. On the next invocation, the receiver should see the same event ID and the ledger should end in delivered, not in two business records. The test is small, but it forces the system to model an accepted request whose response never arrived.
it("retries an ambiguous response with the same idempotency key", async () => {
const seen: string[] = [];
let calls = 0;
const client: WebhookClient = {
async send(event, key) {
seen.push(event.id + ":" + key);
calls += 1;
if (calls === 1) throw new Error("socket timeout");
return "accepted";
},
};
const first = await deliver("evt_42", store, client, 1_700_000_000_000);
const second = await deliver("evt_42", store, client, 1_700_000_120_000);
expect(first).toBe("retry");
expect(second).toBe("ack");
expect(seen).toEqual(["evt_42:evt_42", "evt_42:evt_42"]);
});
Add tests for a duplicate queue message, a worker crash after the HTTP response, a non-retryable 4xx response, an expired lease, and a dead-letter replay. For the replay test, reset the record to pending only through an explicit operator action. A replay should preserve the original event ID and increment an audit counter; editing the event into a new identity makes it impossible to explain what happened later.
The receiver should enforce its own idempotency record with a uniqueness constraint on the event ID. If it stores the response associated with that key, repeated requests can receive the same response without repeating the business mutation. The sender still needs the ledger because it owns scheduling, attempt history, and dead-letter decisions.
Operating the workflow without guesswork
Watch four signals: pending age, retry rate, dead-letter count, and delivery latency. Pending age catches a stuck consumer. Retry rate catches a remote dependency or payload problem. Dead-letter count catches work that needs a human decision. Delivery latency shows whether the backoff policy is delaying normal shipments. Keep the event ID in structured logs, but avoid putting sensitive shipment details into log messages.
Cost enters through retention and attempts rather than a clever queue choice. Store enough history to explain a dispute, then archive old attempt details. Bound the payload size and put large documents behind an object reference. Keep the policy in configuration, but version the policy with the record so a later replay does not silently use a new retry schedule.
Before shipping, verify that the producer creates the ledger row and the initial queue message in one recoverable workflow, that workers can safely process a duplicate message, and that an operator can inspect and replay a dead record. Confirm the receiver's idempotency retention exceeds the maximum retry window. Finally, inject process termination at each boundary: before sending, after sending, after marking delivered, and before acknowledging the queue message.
The decision rule is straightforward: use the queue for scheduling, the ledger for durable intent, and the event ID for identity. Route only exhausted or non-retryable work to the dead-letter path. That gives a logistics system a clear recovery story without pretending that a network timeout has a knowable outcome.
Further reading
- RabbitMQ, “Consumer Acknowledgements and Publisher Confirms”: https://www.rabbitmq.com/docs/confirms
Top comments (0)