Short answer: treat every delayed cleanup as an at-least-once event, record an idempotency decision before side effects, and expose retry state as a timeline that an on-call engineer can read in one glance.
That rule fits a marketplace job such as “remove abandoned carts after 24 hours.” A web request should enqueue the work and return. A worker can then retry a timeout, a throttled downstream call, or a process restart without pretending delivery is exactly once. The duplicate is expected; the duplicate effect is not.
How do webhook retries move through a delayed queue?
Queues usually promise at-least-once delivery. The broker marks a message complete only after the worker acknowledges it. If the process dies after deleting a cart but before that acknowledgement, the same message can arrive again. A delayed job adds another clock: its scheduled timestamp, its visibility time, and the time it finally runs may all differ.
This is why “the queue delivered twice” is an incomplete incident report. The useful question is: which attempt crossed which boundary?
For a marketplace cleanup, I log one event per attempt with a stable job key, for example cart:8f31:expire. The event includes scheduled_at, started_at, finished_at, attempt, and outcome. A retry that never reached the database is different from a retry that committed a deletion and lost its acknowledgement. The dashboard needs to show that distinction.
Keep the state small and explicit. Six states cover most operational conversations:
| State | Meaning | Operator action |
|---|---|---|
| scheduled | Waiting for its due time | Check clock skew and backlog age |
| leased | A worker owns the attempt temporarily | Check lease duration and worker heartbeat |
| committed | The business effect and idempotency record were saved | Treat later deliveries as no-ops |
| retryable | The attempt failed before a safe commit | Inspect error class and backoff |
| dead-lettered | Retry budget was exhausted | Replay only after fixing the cause |
| cancelled | The cart was already paid, restored, or manually protected | Verify the cancellation reason |
Short labels beat a wall of logs.
How should Node.js workers prove idempotency before a side effect?
Use a durable key derived from the business event, not from the delivery attempt. attempt=4 is not an identity. cart:8f31:expire is.
The worker below uses a database transaction and a uniqueness constraint on event_key. The important sequence is: claim the key, perform the guarded mutation, then commit both records together. If a process exits after the commit, a redelivery finds the key and returns duplicate without touching the cart again.
type CleanupEvent = {
eventKey: string;
cartId: string;
scheduledFor: string;
};
type CleanupResult = "committed" | "duplicate" | "cancelled";
export async function processCleanup(
db: { transaction<T>(fn: (tx: any) => Promise<T>): Promise<T> },
event: CleanupEvent,
): Promise<CleanupResult> {
return db.transaction(async (tx) => {
const claim = await tx.idempotency.insertIfAbsent({
eventKey: event.eventKey,
createdAt: new Date().toISOString(),
});
if (!claim.inserted) return "duplicate";
const cart = await tx.carts.findForUpdate(event.cartId);
if (!cart || cart.status !== "abandoned") {
await tx.idempotency.markOutcome(event.eventKey, "cancelled");
return "cancelled";
}
await tx.carts.markExpired(event.cartId);
await tx.idempotency.markOutcome(event.eventKey, "committed");
return "committed";
});
}
The uniqueness constraint is the real guardrail. An in-memory Set disappears on restart and cannot coordinate two workers. A Redis-only lock can help reduce contention, but it is not a substitute for a durable commit record. Your mileage may vary with transaction isolation and database engine; test the exact crash window you rely on.
Retry policy should follow the failure class. Timeouts, connection resets, and HTTP 429 responses are usually transient. A malformed cart identifier, an authorization failure, or a rejected state transition is not fixed by waiting.
I use bounded exponential backoff with jitter, then send the event to a dead-letter stream. The worker emits structured fields rather than a prose-only message:
const retryable = new Set(["ETIMEDOUT", "ECONNRESET", "HTTP_429"]);
function nextDelay(attempt: number, seed: number): number {
const ceiling = Math.min(15 * 60_000, 1_000 * 2 ** attempt);
return Math.floor(ceiling * (0.5 + seed * 0.5));
}
function classify(errorCode: string): "retryable" | "permanent" {
return retryable.has(errorCode) ? "retryable" : "permanent";
}
The metric that catches customer impact is not total retries. It is oldest_due_job_age_seconds, split by outcome, plus the count of keys whose first attempt is still uncommitted. Alert on a sustained age threshold and on a rise in duplicate outcomes. A duplicate spike often means acknowledgements are timing out, not that the business rule suddenly changed. During a busy sale, I also graph the p95 delay between scheduled_for and started_at, the number of active leases, the dead-letter rate, and the fraction of events that finish as cancelled; those four lines let an operator distinguish a slow queue from a flood of invalid work before paging the database team.
I once chased a noisy “cleanup failures” alert that was actually a clock problem: the scheduler emitted UTC timestamps while a local parser treated them as wall time. The error rate looked normal; the backlog age was absurd. Put the timestamp and timezone in every event. Tiny detail. Big difference.
Can an observability-first runbook prevent duplicate webhook processing?
Think of the flow as a line of custody: scheduler → queue → lease → transaction → acknowledgement. At each arrow, record one timestamp and one correlation key. When an operator sees two effects, the trace should answer whether there were two deliveries, two business keys, or one key claimed twice.
For webhook-triggered cleanup, store the provider's event ID alongside your own eventKey. Verify the webhook signature before enqueueing, and validate any URL fetched by a worker against an explicit allowlist. OWASP's SSRF guidance is relevant here because a retry can otherwise repeat an unsafe request many times, amplifying the impact.
Runbook steps should be boring and reversible: pause new scheduling, inspect the oldest five keys, sample one committed transaction, and replay only dead-lettered events whose idempotency records are absent or explicitly reset. Never bulk-delete the ledger to “unstick” a queue. That erases the evidence needed to explain the next duplicate.
Where does this pattern stop fitting?
The catch is that a database-backed ledger adds storage, retention, and schema work. It is not suitable when the side effect cannot participate in a transaction and has no idempotency key of its own; in that case, use an outbox plus a compensating workflow, or choose a downstream API that accepts a client token. Stick with a simpler one-shot scheduler when missed work is harmless and the job has no external effect.
Also set a retention policy. Keeping every key forever increases query cost and makes incident review harder. Retain enough history to cover the maximum retry horizon and your audit needs, then archive it with the same correlation ID. I am not sure one universal horizon exists; payment reversals and cart expiry have very different business clocks.
The practical decision rule is narrow: if a delayed marketplace job changes durable state, design for redelivery, make the commit observable, and make replay a normal operation. Exactly-once delivery is a slogan. Exactly-once effect is an application property you can test.
Top comments (0)