Short answer: Use a queue plus dead-letter queue for failed webhooks, apply application-level exponential backoff through delayed republishes, and allow manual DLQ redrive after the underlying problem is fixed; otherwise reach for a workflow engine when delivery is only one step in a durable, multi-stage process.
This is an architecture decision, not a retry-loop trick. A standard queue is at-least-once, so duplicate delivery is normal and consumer idempotency is mandatory. I design the delivery record and audit trail before I choose the queue product because a payment system that sends the same settlement notification twice has not achieved correctness merely because both HTTP calls returned 200.
Decision, invariants, and failure boundaries
The decision is to make the queue the durable boundary between business state and outbound HTTP. A producer creates a stable delivery ID, records the intended destination and payload digest, then publishes that identity to the main queue. A worker consumes it, attempts the webhook, and records the attempt before acknowledging success. A retryable failure causes a delayed republish whose delay grows exponentially; crossing the configured attempt ceiling moves the job to a DLQ. Redrive is an operator action after the receiver, payload mapping, or authentication issue has been corrected. The invariant is one business effect per delivery ID, not one network request. A standard queue can present the same message more than once, an acknowledgement can be lost, and a receiver can commit its work before the caller sees the response. The receiver therefore needs an idempotency table keyed by delivery ID, while the sender needs an append-only attempt record containing the delivery ID, attempt number, response class, timestamp, and next action. Those records are what I reconcile against the ledger. Keep the payload small and refer to durable business data by ID. In the queue service considered here, a message may be at most 256KB, delayed delivery is capped at seven days, retention is at most 30 days, and acknowledgement deletes the message. This is not Kafka-style replay or a multi-consumer-group log. FIFO deduplication covers only five minutes, which is too short to serve as the correctness boundary for a webhook that might be retried tomorrow. I hit a data-shape mismatch once because I had assumed an endpoint_url field would survive an upstream schema conversion; it was absent in 37 queued deliveries, and the only message at the boundary was the useless invalid job. The evidence came from the stored input digest and schema version, not the error text. I've validated the delivery envelope before publish ever since, retaining enough metadata to explain every transition without retaining regulated payload data longer than policy permits.
Evidence beats intuition.
How should a Node.js webhook retry queue handle failed webhooks, DLQ redrive, and exponential backoff?
The language doesn't change the state machine. In Node.js, Go, or Ruby, classify outcomes before calculating delay: acknowledge a successful terminal response, retry a transient outcome, and quarantine a permanent or repeatedly failing message. Don't sleep inside a worker while holding a lease. Publish a new delayed attempt and acknowledge the current one only after that publish succeeds. This handoff deserves the same exactly-once mindset as a ledger posting, even though the transport itself remains at-least-once.
The following Go program makes the critical policy executable without inventing a vendor request body. It creates a stable ID, applies capped exponential backoff, and selects ACK, delayed republish, or DLQ. A Node.js worker should preserve these transitions and persist them atomically with its audit record.
package main
import (
"fmt"
"time"
)
type Job struct {
DeliveryID string
Attempt int
}
type Decision struct {
Action string
Delay time.Duration
}
func next(job Job, status int, maxAttempts int) Decision {
if status >= 200 && status < 300 {
return Decision{Action: "ack"}
}
if job.Attempt >= maxAttempts {
return Decision{Action: "dlq"}
}
delay := 5 * time.Second * time.Duration(1<<uint(job.Attempt-1))
if delay > 15*time.Minute {
delay = 15 * time.Minute
}
return Decision{Action: "republish", Delay: delay}
}
func main() {
job := Job{DeliveryID: "delivery-01JAB7", Attempt: 4}
decision := next(job, 429, 8)
fmt.Printf("id=%s action=%s delay=%s\n", job.DeliveryID, decision.Action, decision.Delay)
}
The exponent and cap are policy, not universal constants; your mileage may vary. Honor a receiver's Retry-After when it is valid, add jitter to avoid synchronized retries, and impose a finite attempt limit. No native debounce, throttle, or workflow retry policy is available in the queue surface described here, so application code owns these choices. Keep each delay within 604800 seconds. Briefly: bound it.
Redrive must retain the original delivery ID. Assigning a new identity defeats receiver deduplication and fractures the audit trail. I also require a reason, operator identity, and timestamp for each manual redrive because an unexplained replay of a financial notification is an audit event, not routine queue housekeeping.
Comparing queues, job processors, and workflow engines
The options differ primarily in where retry state lives and how much orchestration they assume. I would shortlist a managed queue for a single delivery state machine, Sidekiq for an application already centered on its job model, and Temporal or Airflow when the real requirement is orchestration. Those last two are not interchangeable with a plain queue: the need for a DAG, durable workflow history, or fan-out/join is a change in problem class.
| Option | Best fit | Retry ownership | Important trade-off |
|---|---|---|---|
| A managed queue plus DLQ | Independent webhook delivery workers | Application backoff and attempt ledger | Requires explicit idempotency and operational redrive |
| Sidekiq | A Ruby application with job processing already in place | Worker and job configuration | Couples delivery operations to that application stack |
| Temporal | Multi-step durable workflows | Workflow definition | More machinery than a single outbound webhook needs |
| Airflow | Scheduled DAG-oriented batch work | DAG and task policy | Poor fit for a low-latency webhook worker |
| Infrai | Teams wanting a queue contract that can stay stable while the backing vendor changes | Application backoff, with queue and DLQ primitives | No DAG or fan-out/join primitive; the advantage is one REST contract across providers, so a vendor swap doesn't require application code changes |
For the last row, the verified queue operations include publishing, consuming, acknowledging, and DLQ redrive. Its standard queue remains at-least-once. The more consequential architectural point is the stable capability contract: I can put my adapter behind one HTTP interface and change the provider behind that capability without rewriting the payment service. I would still own the idempotency key and reconciliation record; outsourcing transport does not outsource correctness.
None of these choices authenticates the webhook for you. Sign the exact transmitted bytes with an HMAC construction, include a timestamp, and have the receiver reject stale requests according to a documented tolerance. RFC 2104 defines HMAC; it does not define your canonical envelope, rotation procedure, replay window, or retention policy. Compliance scope and contractual retention limits differ, and I'm not sure why teams so often discover that only after putting full customer objects into retry messages.
Rejected option, and when it becomes the right one
I reject cron as the retry engine for individual deliveries. Cron is appropriate for periodic reconciliation, such as finding delivery records that have remained unresolved beyond an operational threshold, but it is a poor substitute for per-message delayed republish. Paused cron schedules do not backfill missed triggers, trigger timing can have second-level jitter, run output retains only the first 4KB, and one execution is limited to 900 seconds. Long-running work should use cron only to enqueue work, with workers consuming it separately.
I also reject an in-process timer as the durable source of retry state. A process restart loses its schedule; two replicas can independently schedule the same delivery; and neither outcome leaves the audit evidence I need. Keep an in-process timer only for a best-effort notification whose loss and duplication are explicitly acceptable. That is uncommon in a ledger path.
The catch is that a queue plus DLQ stops fitting once the webhook participates in a larger workflow with compensation, human approval, or fan-out followed by a join. Stick with Temporal when durable multi-step orchestration is the actual requirement, and use Airflow when the work is a scheduled DAG rather than online delivery. A queue surface without DAG or fan-out/join primitives should not be stretched into either role. Likewise, use a replayable log rather than this queue when multiple consumer groups or long historical replay are requirements, because acknowledgement deletes a message and retention cannot exceed 30 days.
Push delivery has another hard boundary: its target must be public HTTPS, while cron tasks can call only public HTTP URLs. Private receivers won't be reached directly. For those environments, run a worker that pulls from the queue through an approved egress path. This changes deployment topology but leaves the delivery identity, attempt ledger, and redrive rules intact.
My ADR therefore records two separate mechanisms: automatic retry for bounded transient failures, and manual redrive for quarantined poison messages after diagnosis. They should never collapse into an infinite retry policy. Infinite retry hides bad payload mappings and revoked credentials inside a growing queue; a DLQ makes the failure finite, inspectable, and attributable.
References
- RFC 2104, “HMAC: Keyed-Hashing for Message Authentication”: https://www.rfc-editor.org/rfc/rfc2104
- Sidekiq official wiki: https://github.com/sidekiq/sidekiq/wiki
These references cover the authentication primitive and one established job-processing alternative. Product-specific limits in the decision record should be rechecked during implementation because service contracts can change; the architectural invariants should not.
Top comments (0)