A delayed webhook retry should survive a deploy, run after the request is gone, and produce one business effect even when delivery is attempted five times. For a small Node.js SaaS, start with a database-backed cron poller when the workload is modest; move the same job contract behind a message queue when backlog latency or parallelism becomes a product requirement. The deciding feature is the idempotency key, not the timer.
| Option | Pick this when | The trade-off |
|---|---|---|
| Database + cron poller | A few thousand due rows, minute-level delay is fine, and the team already operates Postgres | You must claim rows safely and watch for a slow poller |
| Message queue with delayed messages | Retry timing needs seconds, workers scale independently, or bursts are normal | Visibility timeouts, dead-letter policy, and queue metrics become part of your service |
| Hybrid (cron enqueues) | Cleanup is periodic but delivery is bursty | Two durability boundaries need one trace ID and one replay story |
What should a Node.js SaaS use for delayed webhook retries?
Treat the scheduler as a wake-up signal. The durable record is the source of truth. A webhook_attempts row can hold event_id, attempt, next_attempt_at, status, and a unique idempotency_key. Cron selects rows whose next_attempt_at <= now(), while a queue carries the same row identifier. Neither mechanism should contain the only copy of the payload.
The fintech cleanup case makes this concrete. Every night, a worker finds expired authorization records and emits a partner webhook. The HTTP request that created the authorization is long gone. If the worker crashes after the partner accepts the message but before the local update, the next run must send the same key and safely converge. HMAC signing, as specified in RFC 2104, protects authenticity; it does not make a duplicate harmless.
Pick cron or a queue by failure shape
Cron is a good fit when “within one minute” is an honest service level and the due set is bounded. Keep the interval short enough for that promise, then claim work in a transaction. A row lock with SKIP LOCKED lets several processes drain different rows without inventing a second scheduler.
The catch is operational visibility. A cron process can be alive while its query is stuck, so alert on oldest due age, claim duration, and the count of rows past their deadline. I once spent an afternoon staring at a green process metric while the oldest due row had crossed 47 minutes; the missing metric was the incident.
Keep it boring.
A queue earns its keep when thousands of retries arrive together or when each tenant needs independent concurrency. The worker acknowledges only after the outbound request and idempotent state transition are durable. Use exponential backoff with a cap, and put exhausted messages in a dead-letter stream that a human can inspect and replay.
Priority queues can separate fraud-review callbacks from routine cleanup, but priority is not free: RabbitMQ's documentation notes that more priorities consume more resources and can reduce throughput. That is a capacity decision, not a reason to hide business urgency in a magic number.
The failure shape matters more than the brand of scheduler. Imagine a card processor releases 18,000 authorizations at midnight. A one-minute poll sees a wall of due rows. With a bounded batch and SKIP LOCKED, the database drains them predictably, but the oldest row may wait through several intervals. A queue can fan that burst across workers, yet now a visibility timeout can redeliver a message while the first worker is still waiting on a partner. The idempotency key absorbs that race; the alert tells you whether the race is becoming normal. For a quiet tenant, the queue's extra moving parts buy nothing. For a noisy tenant, a single global cron loop can turn one customer's burst into everyone else's delay. Partition by tenant only when the service-level objective requires it, because every partition adds capacity and fairness policy to test.
A TypeScript contract that makes either scheduler safe
Here is the small part I want both implementations to share. The endpoint is intentionally generic; replace it with the partner's documented URL and signing rules.
type Delivery = {
eventId: string;
attempt: number;
idempotencyKey: string;
body: string;
signature: string;
};
async function deliver(d: Delivery, send: (body: string, headers: Record<string, string>) => Promise<number>) {
const code = await send(d.body, {
'content-type': 'application/json',
'idempotency-key': d.idempotencyKey,
'x-signature': d.signature
});
if (code >= 200 && code < 300) return { state: 'delivered' as const };
if (code === 409) return { state: 'delivered' as const }; // partner already applied this key
if (code === 429 || code >= 500) return { state: 'retry' as const };
return { state: 'dead-letter' as const };
}
The database update must be conditional: UPDATE webhook_attempts SET status = 'delivered' WHERE idempotency_key = $1 AND status <> 'delivered'. If two workers race, one update wins and the other observes zero changed rows. That is the invariant to test.
My test matrix includes a crash after the partner response, a duplicate queue message, a clock jump, and a retry after a 429. I also record a correlation ID across the scheduler, worker, and partner response. Short logs are useful. Missing context is not.
Limits and a practical handoff
A cron poller is not suitable when the contract promises sub-second delivery, tenant-level rate limits, or burst elasticity; choose a queue then. A queue is not suitable when the team cannot own dead-letter review and visibility-timeout alerts; stick with the transactional poller until that operating habit exists.
I'm not sure any timer choice stays cheapest as volume grows, because database I/O, egress, and on-call time move at different rates. Measure oldest-due age and duplicate-effect count for two weeks, then change the transport without changing the idempotency contract. That keeps the cleanup job boring, which is exactly what a fintech control should be.
Top comments (0)