The Simplest Node.js Architecture for Failed HTTP Webhook Retries

Short answer: use a durable queue to hold each failed webhook job and its next eligible delivery time, let a small HTTP worker perform one attempt at a time, and move exhausted jobs to a dead-letter queue (DLQ); use cron only for a periodic reconciliation scan, or as the entire retry mechanism when the workload is small enough that coarse scheduling and database polling are acceptable.

That division follows from the constraint, not from a preference for infrastructure. A webhook attempt can produce an ambiguous result: the receiver may apply the request while the sender fails to observe the response. The sender therefore needs durable state, a stable delivery identifier, an attempt history, and a policy for deciding when uncertainty becomes an operator-visible exception. A timer alone can't provide those properties. A database-backed queue can, as can a dedicated queue system.

Keep the design boring.

What should own delayed retries for failed webhook jobs: queue or cron?

The queue should own normal redelivery because the relevant clock belongs to each job. Job A may be ready now, job B may need another delay, and job C may have exhausted its policy. A cron expression owns a process wake-up time; it does not, by itself, represent those three independent states. Once a cron implementation adds attempt_count, next_attempt_at, row claiming, terminal state, and replay metadata to a table, the table has become the queue even if the team still calls the process a cron job.

Cron remains useful for a different question: did an event that should have created a delivery job fail to create one? A reconciliation scan can compare the authoritative event record with the delivery ledger and enqueue missing work. This is deliberately separate from retrying known jobs. The retry path handles recorded failures; reconciliation detects absent records.

For a simple Node.js service, the resulting topology has only four responsibilities:

1. The application commits the business event and a delivery intent durably.
2. A dispatcher makes the intent available as a queued job.
3. An HTTP worker claims one job, sends one request, records one outcome, and either completes or reschedules it.
4. A DLQ holds terminal jobs for inspection and controlled replay.

The application and dispatcher may use Node.js, while the behavioral contract stays language-independent. BullMQ documents delayed jobs, retry attempts, and backoff strategies for Node.js systems, so it is one concrete reference for these primitives rather than a requirement for the architecture. A transactional database table with workers that claim due rows can implement the same abstract contract, although the team then owns the concurrency, wake-up, retention, and operational behavior.

Decision	Cron over a durable table	Durable delayed queue
Small volume and coarse timing	Often sufficient	May add unnecessary operations
Per-job delay and backoff	Must be modeled in table state	Native scheduling concept
Worker concurrency	Must be designed and tested	Usually part of the queue contract
Terminal failures	Requires an explicit terminal table or state	Requires an explicit DLQ policy
Reconciliation for missing jobs	Good fit	Still needs an external source of truth

The table is a decision aid, not a ranking. If five-minute polling meets the delivery objective and the team already operates the database, cron plus a carefully claimed table can be the simpler system. If retries need independent timing, controlled concurrency, or prompt wake-up without constant polling, a delayed queue better matches the shape of the problem.

Model an attempt as an auditable state transition

Exactly-once should be treated as a design mindset, not as a claim that an HTTP request can happen only once. The practical target is stronger and more testable: every delivery job has one stable identifier; every attempt appends an immutable outcome; the receiver can recognize a repeated logical delivery; and a replay from the DLQ preserves the original identity. This doesn't make the network certain. It makes uncertainty visible and recoverable.

An adequate job record contains the delivery ID, destination, immutable payload reference or payload, attempt count, next eligible time, status, and creation time. The audit record for an attempt should also retain its start time, completion time, response class, and a bounded diagnostic reason. Sensitive headers and secrets don't belong in that record. Retention, access control, and redaction must follow the system's applicable compliance regime; there is no universal retention duration that can be inferred from queue mechanics alone.

The state machine can remain compact:

package delivery

import "time"

type State string

const (
    Pending State = "pending"
    Leased  State = "leased"
    Sent    State = "sent"
    Dead    State = "dead"
)

type Job struct {
    DeliveryID string
    Endpoint   string
    Body       []byte
    Attempt    int
    NextRunAt  time.Time
    State      State
}

Transitions, rather than queue reads, are the unit to test. pending -> leased must be exclusive for a lease interval. A successful response permits leased -> sent. A retryable outcome permits leased -> pending with a new NextRunAt. Exhaustion or a permanent policy decision permits leased -> dead. If a worker loses its lease, another worker may try the same logical delivery, so the delivery identifier must not change.

This is where payment-oriented systems need discipline. An idempotency key prevents duplication only when the receiver defines and enforces the associated behavior; the sender cannot manufacture that guarantee alone. The contract should specify key scope, retention, payload equality rules, and the response to a key reused with different content. I'm not sure a generic retention recommendation would help here, because it depends on the receiver's replay window and compliance obligations. Resolve it in the integration contract, then test beyond the longest sender retry horizon.

Authentication is a separate concern from idempotency. RFC 2104 defines HMAC, which can authenticate a message with a shared secret. A signed webhook design must define the exact bytes covered by the MAC and how keys are managed; it should not reconstruct JSON before verification because a byte-level signature contract depends on the signed representation. The snippet below demonstrates the narrow HMAC operation without pretending to define a complete wire protocol:

package delivery

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
)

func Sign(secret, body []byte) string {
    mac := hmac.New(sha256.New, secret)
    _, _ = mac.Write(body)
    return hex.EncodeToString(mac.Sum(nil))
}

func Verify(secret, body []byte, signature string) bool {
    want, err := hex.DecodeString(signature)
    if err != nil {
        return false
    }

    mac := hmac.New(sha256.New, secret)
    _, _ = mac.Write(body)
    return hmac.Equal(mac.Sum(nil), want)
}

Short code, strict contract.

Put policy at the HTTP worker boundary

The worker should do one attempt, classify what it observed, persist that observation, and return control to the queue. It shouldn't sleep until the next retry — a sleeping process is neither durable scheduling nor an audit trail. Delayed availability belongs in persisted queue state.

Classification is a policy decision. A connection failure is ambiguous because no response was observed. An HTTP response can be mapped to success, retry, or terminal failure according to the receiver contract. Avoid a blanket rule that every 4xx is permanent or every non-2xx is retryable; integrations may define throttling, authentication, conflict, and validation responses differently. The worker needs a small adapter for that contract rather than an ever-growing global switch statement.

The following Go interface shows the boundary. The values in ExamplePolicy are illustrative configuration choices, not universal defaults.

package delivery

import "time"

type Outcome string

const (
    Accepted  Outcome = "accepted"
    Retryable Outcome = "retryable"
    Terminal  Outcome = "terminal"
)

type Observation struct {
    Outcome    Outcome
    StatusCode int
    Reason     string
}

type RetryPolicy struct {
    MaxAttempts int
    Delays      []time.Duration
}

var ExamplePolicy = RetryPolicy{
    MaxAttempts: 5,
    Delays: []time.Duration{
        30 * time.Second,
        2 * time.Minute,
        10 * time.Minute,
        30 * time.Minute,
    },
}

func NextDelay(policy RetryPolicy, completedAttempts int) (time.Duration, bool) {
    if completedAttempts >= policy.MaxAttempts || completedAttempts < 1 {
        return 0, false
    }
    index := completedAttempts - 1
    if index >= len(policy.Delays) {
        return 0, false
    }
    return policy.Delays[index], true
}

For the production path, the persistence boundary matters more than the backoff arithmetic. Record the attempt before acknowledging the queue message. If rescheduling and recording are separate operations, define how reconciliation finds a recorded retry decision with no corresponding queued job. If the queue supports an atomic retry transition, use it; otherwise keep an outbox-like intent that a dispatcher can recover. The invariant is straightforward: each terminal decision is recorded, and each nonterminal decision eventually corresponds to available work.

Backoff should be bounded, and jitter can spread retries that became eligible together. The actual series must follow the receiver's capacity and the delivery objective. Don't bury those values in worker code: configuration should be versioned or copied into the job so an auditor can determine which policy governed a historical attempt. Also cap concurrent deliveries per destination. A global worker limit protects the sender, while a destination limit prevents one failing endpoint from consuming the entire pool.

Observability should describe job state rather than merely process health. Useful signals include oldest eligible job age, attempts by outcome, lease recovery count, destination-level saturation, jobs entering the DLQ, and reconciliation gaps. Logs should carry delivery_id and attempt so an operator can reconstruct the sequence without joining on an error message. Payload bodies, shared secrets, and signature values should be excluded or redacted.

Testing needs the same focus on ambiguity. Exercise duplicate claims, a worker stopping after the receiver acts but before acknowledgment, rescheduling persistence failure, lease expiry, a repeated DLQ replay, and two deliveries for the same business event. Then verify the audit sequence and downstream idempotency, not just the final queue depth. Fast unit tests can cover classification and policy; integration tests should cover the queue and persistence transitions; a deployment test should drain one worker version while another begins claiming jobs.

A DLQ is an exception workflow, not extra storage

A job should enter the DLQ with its stable delivery ID, immutable payload reference, full attempt history, terminal reason, policy version, and timestamps. Replaying it should create a new execution record linked to the same logical delivery, not quietly erase the dead-letter record or invent a fresh idempotency identity. Operators then have evidence of both the original exhaustion and the later decision to retry.

Don't automate every replay.

A terminal validation result may require correcting upstream data or changing the receiving contract before another attempt. An authentication result may require credential rotation. A long receiver outage may justify a controlled batch replay with destination concurrency limits. These cases need an owner, a reason code, and a recorded action; a button labeled “retry all” is operationally convenient but weak as an audit mechanism.

Cron fits beside this workflow as a low-frequency control. It can compare committed business events or outbox intents with delivery jobs, find leases that exceeded their recovery policy, and report terminal jobs that have no owner. It should not silently convert every anomaly into another send. In a financial workflow, detection and correction are distinct privileges for good reason — the first can be broad and automated, while the second may require approval and a preserved explanation.

Roll out the simplest architecture that meets the constraint

Start with the contract and state transitions, because changing queue technology won't repair an unstable delivery identity. Add a durable delivery ID and attempt ledger first. Next, move retry timing out of process memory and into NextRunAt or a delayed-job facility. Then add explicit terminal state and a replay procedure. Finally, introduce reconciliation against the authoritative event source and alert when it finds a gap.

During migration, run the old cron scanner as a narrow safety check while the worker owns new retries. It should report discrepancies rather than race the worker to send the same job. Once the discrepancy rate is understood and the replay procedure has been exercised, reduce the scanner to its intended reconciliation role.

The catch is operational ownership. A delayed queue adds another stateful component, so it is not suitable when webhook delivery is best-effort, volume is low, timing is coarse, and a claimed database table already meets recovery needs; stick with cron plus durable rows in that case. Conversely, a cron-only scanner stops being simple when it accumulates per-job clocks, leases, backoff, concurrency partitions, terminal-state handling, and replay tooling. At that point, name the table what it is and operate it as a queue.

The selection rule is therefore compact: choose cron for periodic discovery, choose a durable queue for independently scheduled attempts, and require the same idempotency and audit invariants from either implementation. Node.js is an implementation detail. The durable state machine is the architecture.

References

• RFC 2104: HMAC keyed-hashing for message authentication
• BullMQ documentation