Short answer: persist each logistics event before delivery, give it a stable idempotency key, and schedule the next attempt for five minutes later only after a retryable outcome; a public HTTPS endpoint and a durable queue are necessary, but neither prevents duplicate deliveries by itself.
Map the delivery boundary
Start with the failure you need to survive. A shipment-status webhook can leave your process, reach the carrier, and still lose the response. If the worker crashes at that exact boundary, the queue can't know whether the carrier committed the update. The correct guarantee is therefore at-least-once transport with idempotent handling, not an imagined exactly-once network call.
| Mechanism | Pick this when | Main trade-off |
|---|---|---|
| Database-backed scheduler | Delivery state already lives in a transactional database and moderate polling delay is acceptable | Simple recovery model, but polling and claim contention need care |
| Durable message broker | The team already operates broker consumers and needs high delivery throughput | Acknowledgements and redelivery are explicit, but business idempotency still belongs in application storage |
| In-process timer | The task is disposable and losing it during a deploy is acceptable | Tiny setup cost, but no durable recovery after process exit |
For shipment updates that matter, the in-process option is usually out. A five-minute setTimeout looks convincing in a local Node.js example, then disappears during a rolling deployment. Use it for cache refreshes or best-effort nudges, not for a customs-clearance event that another system must record.
Timers forget.
A database scheduler is a strong pick when enqueueing the webhook must share a transaction with the shipment update. The broker option fits teams that already have consumer supervision, dead-letter handling, and queue metrics. RabbitMQ, for example, documents consumer acknowledgements separately from publisher confirms: the former covers delivery from broker to consumer, while the latter covers publishing into the broker. Those mechanics close important loss windows. They don't prove that an HTTPS receiver processed a request only once.
Priority is different from time. A priority queue changes which ready message is consumed first; it is not, by itself, a clock that makes a task eligible in five minutes.
How should a Node.js task queue schedule a delayed webhook retry after 5 minutes?
Model time as data. Store nextAttemptAt, then let a worker atomically claim only records whose time has arrived. The useful diagram-in-words is: shipment transaction -> outbox row -> due-task claim -> HTTPS attempt -> delivery receipt or next due time. Every arrow can run twice except the transaction that records a successful receipt.
Keep three identifiers separate. eventId names the logistics fact, such as a shipment entering out_for_delivery. deliveryId names this destination's delivery of that fact. attemptId names one network attempt. The receiver gets the stable deliveryId as its idempotency key; logs and traces get the changing attemptId. Reusing the attempt identifier as the idempotency key is a subtle mistake because every retry then looks new to the receiver.
Trace one ambiguous delivery. At 10:00:00, the worker claims a due shipment event and starts a 60-second lease. It sends the request with delivery key del_8f31; the receiving warehouse commits the status change, but the connection closes before the worker reads the response. At 10:00:10, the worker records a transport failure and sets nextAttemptAt to 10:05:10. The second attempt carries the same delivery key and a new attempt identifier. The receiver finds del_8f31, skips the already-applied transition, and returns its stored success. The sender can now mark the task delivered. This sequence — commit, lost response, delayed retry, duplicate suppression — is the case both sides must pass before deployment, because a queue acknowledgement alone can't resolve what happened inside the receiver's transaction.
For a fixed five-minute policy, compute the next timestamp from the time the retryable result is recorded. Store it. Don't ask a sleeping worker to remember it.
Duplicates are normal.
Implement the idempotent delivery state machine
The core interface below is deliberately storage-neutral. Its claimDue operation must atomically move a due record from scheduled to in_flight, typically with a lease so another worker can recover it after a crash. markDelivered must be conditional: once a delivery is recorded, a late worker can't schedule it again.
type DeliveryStatus = "scheduled" | "in_flight" | "delivered" | "failed";
type WebhookTask = {
deliveryId: string;
eventId: string;
endpoint: string;
payload: unknown;
attemptCount: number;
nextAttemptAt: Date;
status: DeliveryStatus;
};
type DeliveryStore = {
claimDue(now: Date, leaseMs: number): Promise<WebhookTask | null>;
markDelivered(deliveryId: string, attemptId: string, status: number): Promise<void>;
reschedule(
deliveryId: string,
attemptId: string,
nextAttemptAt: Date,
status: number | null,
): Promise<void>;
markFailed(deliveryId: string, attemptId: string, status: number | null): Promise<void>;
};
const FIVE_MINUTES_MS = 5 * 60 * 1_000;
const CLAIM_LEASE_MS = 60_000;
const MAX_ATTEMPTS = 8;
function requirePublicHttpsEndpoint(raw: string): URL {
const endpoint = new URL(raw);
if (endpoint.protocol !== "https:") {
throw new Error("Webhook endpoint must use HTTPS");
}
// Resolve and reject private, loopback, and link-local addresses before dispatch.
return endpoint;
}
function isRetryable(status: number): boolean {
return status === 408 || status === 429 || status >= 500;
}
async function deliverOne(store: DeliveryStore, now = new Date()): Promise<boolean> {
const task = await store.claimDue(now, CLAIM_LEASE_MS);
if (!task) return false;
const endpoint = requirePublicHttpsEndpoint(task.endpoint);
const attemptId = crypto.randomUUID();
let status: number | null = null;
try {
const response = await fetch(endpoint, {
method: "POST",
headers: {
"content-type": "application/json",
"idempotency-key": task.deliveryId,
"x-webhook-attempt-id": attemptId,
},
body: JSON.stringify({
eventId: task.eventId,
deliveryId: task.deliveryId,
data: task.payload,
}),
signal: AbortSignal.timeout(10_000),
});
status = response.status;
if (response.ok) {
await store.markDelivered(task.deliveryId, attemptId, response.status);
return true;
}
if (!isRetryable(response.status) || task.attemptCount + 1 >= MAX_ATTEMPTS) {
await store.markFailed(task.deliveryId, attemptId, response.status);
return true;
}
} catch (error: unknown) {
if (task.attemptCount + 1 >= MAX_ATTEMPTS) {
await store.markFailed(task.deliveryId, attemptId, status);
return true;
}
}
await store.reschedule(
task.deliveryId,
attemptId,
new Date(now.getTime() + FIVE_MINUTES_MS),
status,
);
return true;
}
The endpoint check in the snippet is only the policy boundary. A production dispatcher also has to resolve DNS and reject private, loopback, and link-local destinations, then protect against DNS rebinding at connection time. Otherwise, accepting an arbitrary “public HTTPS URL” creates a server-side request forgery path. An allowlist of partner hosts is cleaner when the logistics network is known in advance.
The receiver needs its own atomic rule: insert the idempotency key and apply the shipment transition in one transaction. If the key already exists, return the stored successful result without applying the transition again. This is where duplicate prevention actually happens. The sender's delivered flag reduces extra calls, but it can't settle the ambiguous case where the receiver commits and the response vanishes.
Keep the state transitions narrow. Retry 408, 429, transport failures, and server-error responses. Treat most other client-error responses as terminal because waiting five minutes won't repair a malformed signature or unknown shipment identifier. A 429 response may include explicit retry timing; if the integration contract permits it, that server-provided delay should override the fixed interval. The exact status policy is contract-specific, so verify it with each receiving team rather than copying a generic matrix unchanged.
No magic here.
Test and measure queue health
A webhook dashboard should separate task age from request latency. Request latency answers “was this attempt slow?” Oldest-due-task age answers “are deliveries waiting past their promised time?” Track due tasks, in-flight leases, attempt outcomes by status class, terminal failures, and the time from shipment event creation to confirmed delivery. Alert on sustained age and terminal failures; a count of retries alone can rise during a legitimate partner rate limit. The 10-second request timeout, 60-second claim lease, and 5-minute retry delay represent three different clocks, so graph them independently.
Use deliveryId as the join key across queue records and receiver receipts, with attemptId on each request log. Never put the full shipment payload into a metric label. High-cardinality tracking belongs in logs or traces, while metrics keep bounded dimensions such as destination group and outcome class. This split makes the crisp before/after visible: before, “webhooks are failing”; after, “34 due deliveries for carrier group B are more than ten minutes late, and 429 dominates the latest attempts.”
Test the ugly boundary. Arrange for a fake endpoint to record the idempotency key and then close the connection before the worker receives a response. The worker should reschedule after five minutes, the receiver should observe a second request with the same key, and the shipment transition should still exist once. Also test a worker exit after claiming, two workers racing for one due task, a deploy while tasks are pending, a terminal client error, and exhaustion at the eighth attempt.
Choose the operating limit
This design is not suitable when every event must arrive at an exact wall-clock instant; polling cadence, queue contention, network delay, and receiver throttling all add variance. Use a scheduler with a stated timing contract when precision matters. Stick with an existing durable broker when the team already operates it well, and prefer a transactional database scheduler when atomic creation beside shipment state is the dominant requirement.
The catch is operational ownership. Leases need expiry and recovery, idempotency records need a retention policy longer than the sender's retry horizon, and terminal failures need a replay process that preserves the original deliveryId. The right retention window might be longer than the active retry period when operators can replay old failures. These aren't optional cleanup tasks. They are the mechanism that keeps a five-minute retry from becoming a duplicate-delivery incident.
Top comments (0)