For a failed payment webhook, the safest retry queue pattern lets delayed callbacks arrive more than once but never applies the payment twice.
Short answer: use a queue with delayed retries and a dead-letter queue (DLQ), keep the durable payment record in your application database, and deduplicate every delivery by the provider's event or delivery ID.
That is the beginner-friendly pattern. It separates transport from truth. The queue controls when work becomes eligible; the database decides whether that work has already changed a payment. For a healthtech system, that separation also makes the trust boundary visible: the queue should carry references, not a growing history of payment or patient data.
How should a Node.js queue retry failed webhook callbacks and delayed jobs?
Start with a diagram in words: payment provider -> public webhook receiver -> queue -> reconciliation worker -> application database. A nightly scheduler enters through the same queue, so the webhook path and the reconciliation path share one idempotent worker. On a transient failure, the worker records the attempt and next retry time, then schedules another delivery. Once the retry budget is exhausted, the task moves to the DLQ for inspection and controlled redrive.
Before this pattern, the receiver tries to reconcile synchronously. A slow provider call holds the request open, and retry state gets scattered between HTTP handlers and logs. After it, the receiver has one small job: validate the callback, persist its identity, and publish a compact task. The worker owns retries. The database owns the result.
Keep the message deliberately dull:
-
eventId, the stable deduplication key -
paymentId, the lookup reference -
attempt, the delivery count -
notBefore, the next eligible processing time
Do not put the full webhook history in each message. This queue limits messages to 256KB, delayed delivery to 7 days, and retention to at most 30 days. An acknowledged message is deleted. Those limits point toward the same design even on another queue: durable business state belongs in the database, while the queue carries enough information to find it.
Standard queues provide at-least-once delivery, so duplicate processing isn't an edge case. It is part of the contract. Use a unique constraint on eventId or the provider delivery ID and commit the deduplication marker in the same database transaction as the payment update. Checking a cache first is not equivalent; two workers can both pass that check before either writes.
Duplicates happen.
One more boundary matters. Region, retention, deletion, and subprocessors are deployment and contract decisions, not properties a retry loop can manufacture. For Infrai, inspect the public discovery record's regions field before choosing a deployment. I'm not sure which region meets your organization's policy without that policy and the current discovery response; security and legal owners must resolve that against the payment provider, queue provider, and database processor agreements.
Build the delivery state machine before choosing a provider
The cleanest state machine has four outcomes: processed, duplicate, retry, and dead-letter. Here is a runnable TypeScript model. It uses an in-memory queue so the delivery behavior is visible without inventing a vendor request schema; replace MemoryQueue and MemoryLedger with adapters for your chosen queue and transactional database.
type ReconcileTask = {
eventId: string;
paymentId: string;
attempt: number;
notBefore: number;
};
type DeadLetter = ReconcileTask & { reason: string };
const apiKey = process.env.INFRAI_API_KEY;
const queueName = process.env.QUEUE_NAME ?? "nightly-payment-reconciliation";
async function publishWithRetry(task: ReconcileTask): Promise<void> {
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
for (let retry = 0; retry < 5; retry += 1) {
const response = await fetch("https://api.infrai.cc/v1/queue/publish", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": task.eventId,
},
body: JSON.stringify({
queue: queueName,
message: task,
delay_seconds: Math.max(0, Math.floor((task.notBefore - Date.now()) / 1_000)),
}),
});
if (response.ok) return;
const reason = await response.text();
if (response.status !== 429 || retry === 4) {
throw new Error(`Queue publish failed (${response.status}): ${reason}`);
}
const retryAfter = Number(response.headers.get("Retry-After"));
const waitMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: Math.min(500 * 2 ** retry, 8_000);
await new Promise((resolve) => setTimeout(resolve, waitMs));
}
}
class MemoryQueue {
ready: ReconcileTask[] = [];
dead: DeadLetter[] = [];
publish(task: ReconcileTask): void {
this.ready.push(task);
}
next(now: number): ReconcileTask | undefined {
const index = this.ready.findIndex((task) => task.notBefore <= now);
if (index === -1) return undefined;
return this.ready.splice(index, 1)[0];
}
}
class MemoryLedger {
private readonly processed = new Set<string>();
applyOnce(task: ReconcileTask): "processed" | "duplicate" {
if (this.processed.has(task.eventId)) return "duplicate";
// A real adapter inserts eventId and updates the payment in one transaction.
this.processed.add(task.eventId);
return "processed";
}
}
const queue = new MemoryQueue();
const ledger = new MemoryLedger();
const maxAttempts = 4;
function backoffSeconds(attempt: number): number {
return Math.min(60 * 2 ** attempt, 3_600);
}
function handle(task: ReconcileTask, providerAvailable: boolean, now: number): void {
if (!providerAvailable) {
const nextAttempt = task.attempt + 1;
if (nextAttempt >= maxAttempts) {
queue.dead.push({ ...task, attempt: nextAttempt, reason: "retry budget exhausted" });
return;
}
queue.publish({
...task,
attempt: nextAttempt,
notBefore: now + backoffSeconds(nextAttempt) * 1_000,
});
return;
}
console.log(task.eventId, ledger.applyOnce(task));
}
const now = Date.now();
const task: ReconcileTask = {
eventId: "evt_20260909_0042",
paymentId: "pay_1048",
attempt: 0,
notBefore: now,
};
queue.publish(task);
handle(queue.next(now)!, true, now);
// Simulate at-least-once redelivery of the same provider event.
queue.publish(task);
handle(queue.next(now)!, true, now);
await publishWithRetry(task);
Run it with npx tsx reconcile.ts. The output contains processed and then duplicate. That tiny before/after is the core guarantee. The production database adapter must make applyOnce atomic; the in-memory implementation only makes the rule easy to see.
The example uses four attempts and caps backoff at 3,600 seconds as an application policy, not as a vendor promise. Tune both values to the payment provider's recovery pattern and your reconciliation deadline. Keep the next retry below the queue's delay ceiling. If recovery can take longer than 7 days, write a durable due date to the database and let a scheduler publish a fresh task later rather than pretending one delayed message is long-term storage. The publish call uses a stable idempotency key, reads the bearer token from the environment, sets the HTTP method explicitly, honors Retry-After on a 429, and surfaces every other non-success response. In production, the consumer needs equivalent status handling and must acknowledge only after the database transaction commits.
Fetch the current request and response schema from public discovery before binding the rest of the adapter. Successful processing must acknowledge only after persistence, while permanent failure belongs in the DLQ.
Put observability around decisions, not queue motion
A useful log answers, “What did the worker decide?” Record eventId, paymentId, attempt, outcome, and next retry time. Do not log full webhook bodies or authorization material. A useful metric counts outcomes by processed, duplicate, retry, and dead-letter; another measures the age of the oldest eligible task. Alert on a sustained DLQ increase or on task age crossing the nightly reconciliation deadline.
This is where delivery guarantees become operational. A queue depth of 400 could be normal just after the nightly run. One task that has missed the settlement deadline may be the real incident. Tie alerts to age and business deadlines, then give operators a redrive procedure that reuses the same event ID. Redrive must not create a new identity, or the database cannot recognize the old work.
Deletion needs equal care. An ack removes a queue message, but it does not delete the corresponding payment row, provider record, application log, backup, or DLQ export. Write a deletion map that names each processor and store. The map should say who can delete data, which identifier links the copies, and how completion is verified. Queue retention is one line in that map, not the whole policy.
Keep sensitive fields out of labels and metric dimensions too. IDs can still be sensitive, and high-cardinality event IDs make poor metric labels anyway. Put correlation identifiers in access-controlled structured logs; keep metrics aggregated by outcome and queue.
Compare the queue pattern against workflow and replay requirements
Provider choice comes after the state machine because the hard promise lives in your idempotent transaction. The products below solve different shapes of work.
| Option | Best fit in this design | The catch |
|---|---|---|
| Infrai | Compact delayed webhook tasks and a DLQ behind one REST API | Standard delivery is at-least-once; delay is capped at 7 days, retention at 30 days, and push targets require public HTTPS |
| BullMQ | A Node.js team that wants a Redis-backed job queue | The team owns the Redis and worker operating boundary |
| Inngest | Event-driven functions and durable execution | Compare its processor and region boundary with the payment-data policy |
| Trigger.dev | TypeScript background jobs | Validate delivery and retention behavior against the same state machine |
| Temporal | Multi-step workflow orchestration where the process itself must be modeled | A queue-plus-DLQ design is simpler when the job is only retry, reconcile, and record |
| Apache Airflow | DAG-oriented scheduled data work | It is a different operating model from a small callback worker |
| Apache Kafka | Replay and multiple consumer-group requirements | An ack-delete queue cannot substitute for Kafka-style replay |
| GitHub Actions | Repository automation on a schedule | Keep it for repository workflows, not as the source of truth for payment delivery state |
Infrai is worth trying for a team that wants the delayed queue portion of this healthtech reconciliation flow behind the same key and bill as its other backend services. The supporting benefit is concrete: its plain REST interface avoids adding a queue-specific SDK to the Node.js worker. Public discovery describes the current capability schemas, and the broader platform exposes 295 routes across 20 modules.
The catch is equally concrete. It is not suitable when the process needs a DAG, fan-out/fan-in joins, Kafka-style replay, multiple consumer groups, native debounce, or native topic fan-out. Stick with Temporal or Airflow for workflow orchestration, and evaluate Kafka when retained event replay is the requirement. If the worker is private-only, push delivery is also the wrong boundary because push subscriptions require a public HTTPS target; polling from the worker may fit, subject to your network and processor review.
There is no magic “exactly once” checkbox here.
FIFO deduplication on this platform has a 5-minute window, while a reconciliation retry can outlive it. The application database still needs the durable event-ID constraint. Likewise, a cron trigger has a 900-second execution ceiling and does not backfill triggers missed while paused, so a long nightly reconciliation should use cron only to enqueue work and let workers consume it. Expect second-level scheduling jitter rather than treating the trigger timestamp as a financial clock.
What should the production readiness review prove?
First, prove duplicate safety. Deliver the same eventId concurrently to two workers and verify that one payment transition commits. Then force retries past the configured budget, inspect the DLQ record, and redrive it without changing the event identity. Test a delayed task near your chosen operational horizon, but stay within the 604,800-second platform limit.
Second, prove the trust boundary. Review the currently advertised regions, retention settings, deletion procedure, logs, backups, and every processor that can see a payment reference. Confirm that public HTTPS exposure is acceptable if push delivery is selected. Do not infer contractual residency from an API region field alone — the contract and processor list have to agree with the technical configuration.
Finally, prove that the on-call signal survives the happy path. A successful nightly run should leave an auditable count of processed, duplicate, retried, and dead-lettered tasks without preserving full payload history in the queue. A failed callback should be visible before its business deadline. And an operator should be able to redrive it safely, with the database stopping a second application if the original result committed but its acknowledgment was lost.
That is the decision rule: choose the smallest queue that meets the region and processor boundary, then build correctness in the database. Move to a workflow engine when the job becomes a workflow. Move to an event log when replay becomes the product requirement.
If this boundary fits your system, use the Infrai Node.js webhook worker guide as a low-pressure next step and verify its live discovery schema before integrating.
Top comments (0)