Short answer: treat a webhook retry queue as an at-least-once transport, and make the durable business change idempotent with a stable event identity and a database transaction.
The deciding constraint is the gap between doing work and acknowledging it. A worker can commit a change, lose its process before the acknowledgement is recorded, and receive the same job again. No reasonable retry policy can infer whether that first change happened. Delayed jobs help recovery; they do not close that gap.
For an AI application, that distinction matters twice. A repeated state change can corrupt customer state, while a repeated call to a model or external API can consume latency and usage budget without producing a new result. Keep the queue simple, but make its boundaries explicit.
Why do webhook retry queues create duplicate processing with delayed jobs and at-least-once delivery?
There are usually two independent retry loops. The webhook sender retries when it does not receive a timely success response. The receiving system may then retry its queued work after a timeout, a process restart, or an unavailable dependency. Both loops are valid, and either can place the same logical event in front of a handler more than once.
Consider a delivery carrying evt_482. The receiver accepts it and places it in durable work storage. A worker charges ahead: it writes the new subscription status, commits the transaction, and is interrupted before its queue acknowledgement becomes durable. On recovery, the queue sees an unacknowledged attempt and makes evt_482 runnable again. The second worker is not seeing a new customer action. It is seeing an unresolved delivery outcome.
That is why "exactly once" is a misleading promise for most distributed systems. The sender, queue, worker, database, and any downstream service do not share one atomic commit. The useful target is one durable business effect for each logical event, even though execution can repeat.
A scheduled retry introduces a separate ordering question. If an older account update waits in a delayed set while a newer update completes, blindly applying the older message later can roll state back. Put a producer version, sequence, or monotonic timestamp in the event when the domain has an ordering rule. The transaction that claims the event should also reject a stale transition.
The small failures are the dangerous ones. A 200 response proves that a hop responded; it does not prove that an intended state exists after a crash boundary.
Put the idempotency boundary around the business effect
Start with an event key that is stable across every delivery. A producer-supplied event ID is ideal. When the protocol does not offer one, derive a deterministic key from immutable business identifiers, the event type, and a source version. Generating a random key on receipt defeats deduplication because each retry looks unrelated.
Then claim that key in durable storage. An in-memory Set vanishes on restart and cannot coordinate multiple Node.js processes. A read followed by an insert is also insufficient: two workers can both observe an absent row before either write wins. A unique constraint or equivalent conditional insert gives the database one clear arbitration point.
One key, one ledger.
Here is the full decision path worth testing before calling a handler safe. A signed delivery with evt_482 arrives twice because the sender did not observe the receiver's first response; both requests validate to the same identity, and both attempts try to enqueue work. Queue-level deduplication may collapse those attempts, but it is an efficiency optimization rather than proof, since one request may have reached the queue through a different path or after queue history has been cleaned up. A worker claims evt_482 and writes the account status. If that worker finishes normally, it acknowledges the job. If it exits after the transaction and before the acknowledgement, another worker can later receive the same job. Its conditional claim sees the existing event record and does no second business mutation. If version 12 has already changed the account, an older delayed version 11 must likewise fail the state transition check even if it has an event identity never seen before. Finally, if applying the change creates outbound work, that work is written as an outbox record in the same transaction. A separate dispatcher can make repeated delivery attempts visible against that outbox identity. Each step has a different purpose: request authentication establishes trust, event identity groups repeated deliveries, a unique claim stops duplicate effects, a version rule protects ordering, and an outbox prevents a committed state change from being forgotten before follow-up work is scheduled. Folding them into a single "retry" setting makes incidents hard to explain.
This focused TypeScript sketch uses generic interfaces deliberately. The queue can reduce redundant scheduling with an event-derived job key, but the transaction owns correctness because redelivery can originate upstream of the queue as well.
type Delivery = {
eventId: string;
accountId: string;
version: number;
status: "active" | "paused";
};
type Queue = {
enqueue(name: string, delivery: Delivery, options: {
key: string;
attempts: number;
delayMs: number;
}): Promise<void>;
};
type Transaction = {
claimEvent(eventId: string): Promise<boolean>;
updateAccountIfNewer(input: Delivery): Promise<void>;
};
declare const queue: Queue;
declare const database: { transaction<T>(work: (tx: Transaction) => Promise<T>): Promise<T> };
export async function acceptDelivery(delivery: Delivery): Promise<void> {
await queue.enqueue("apply-account-status", delivery, {
key: delivery.eventId,
attempts: 6,
delayMs: 1_000,
});
}
export async function applyDelivery(delivery: Delivery): Promise<void> {
await database.transaction(async (tx) => {
if (!(await tx.claimEvent(delivery.eventId))) return;
await tx.updateAccountIfNewer(delivery);
});
}
The required behavior of claimEvent is a durable, unique claim. updateAccountIfNewer enforces the domain's ordering rule in the same transaction. In some domains, such as adding an immutable audit record, version ordering is unnecessary; in others, such as a mutable subscription state, it is central. Don't invent an ordering rule where the producer cannot support one.
External side effects need a further decision. A database transaction cannot atomically commit an email, payment request, or model call owned by another system. Persist an outbox record with the business mutation, then have a worker deliver that record using the same idempotency identity when the destination supports it. If the destination has no idempotency mechanism, the remaining ambiguity should be documented and monitored rather than hidden behind a retry count.
Which retry failures should become delayed work?
Retry only failures that can change with time. A transient network timeout, a rate limit, or a temporary dependency overload may merit exponential backoff with jitter. Invalid signatures, malformed payloads, and prohibited state transitions will not become valid because they wait longer. Record them as terminal outcomes with a reason an operator can inspect.
Set a finite attempt budget. Without one, a bad event becomes permanent background activity and can repeat expensive downstream calls. The right limit depends on the business deadline, the retry schedule, and the recovery expectation; it is not a universal queue setting. I'm not sure a generic default will fit a payment update and a low-priority analytics event equally well, because their consequences and freshness requirements differ.
For inbound webhooks, authenticate before accepting work, validate the payload shape, write the acceptance record durably, and return promptly. For outbound webhooks whose destination is user-controlled, the dispatcher is also an SSRF boundary. Restrict allowed schemes and destinations, validate resolved addresses, and control network egress as OWASP recommends. URL parsing alone is not a security policy.
The catch is operational ownership. A database-backed work table is often enough for modest volume, but its leases, polling behavior, and cleanup remain application work. A separate queue can give delayed work clearer lifecycle states, but it introduces another service to monitor. A workflow runtime can fit long-lived, multi-step compensation, yet it adds a mental model that is excessive for one database update. Choose the smallest system whose acknowledgement and recovery behavior the on-call engineer can explain.
| Shape | Fits best when | Responsibility that stays with the team |
|---|---|---|
| Database work table | Work volume is modest and state lives in one database | Leasing, polling, and cleanup |
| Dedicated queue | Delayed work needs explicit lifecycle states | Queue operations and worker capacity |
| Workflow runtime | Work has timers, compensations, and several durable steps | Runtime model and operational visibility |
Test the acknowledgement gap, then measure the outcome
Happy-path tests do not exercise the reason queues redeliver. Use a real database in an integration test and run two handlers for the same event identity concurrently. Assert that one durable transition results. Then repeat the handler after the first transaction has committed, simulate a delayed version 10 arriving after version 11, exhaust the attempt budget for an invalid event, and verify that replay retains its original identity.
The test should distinguish duplicate execution from duplicate effect. The former is expected under at-least-once delivery. The latter is a defect.
Production metrics should make that distinction visible. Track oldest runnable-job age, ready and delayed counts, attempt distribution, terminal outcomes, worker duration, and the number of duplicate claims. Pair those transport signals with a business completion marker. Queue depth alone is weak evidence: a large delayed backlog may be planned, while a small set of old runnable jobs may indicate an urgent processing gap.
Carry the event ID, job identifier, attempt number, and a non-secret destination identifier through logs and traces. Do not log signing secrets or whole sensitive payloads. Also count calls per logical event, particularly where a handler invokes token-metered work. The ledger claim belongs before an expensive action, and reusable output should be persisted when the business operation allows it.
Before copying this design, measure duplicate executions, duplicate business effects, recovery time, oldest-job age, and calls per event. Those numbers reveal whether a delayed-job policy is buying recovery or just postponing an unsafe handler.
References
- OWASP SSRF Prevention Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html
- BullMQ documentation: https://docs.bullmq.io/
Top comments (0)