Short answer: A Node.js webhook queue consumer should persist the next eligible delivery time from Retry-After, acknowledge only after that state commits, and let a regional scheduler republish the work rather than sleeping in the worker.
A queue receipt is not an audit record. A process can restart, a lease can expire, and a receiver can observe a request even when the sender never receives its response. For payment- or ledger-adjacent events, those are ordinary states, not edge cases. The design needs a durable record of intent, a stable idempotency key, and a reconciliation path; a timer held in a consumer provides none of them.
Short-lived work is different.
No sleep. No guesswork.
How should a Node.js webhook queue consumer process 429 Retry-After across US and EU SaaS?
First, parse Retry-After at the HTTP boundary. HTTP permits either a delay in seconds or an HTTP date, so the sender should record an absolute next_attempt_at after applying its documented local cap. A small randomized spread can prevent thousands of deliveries from becoming eligible on the same tick. The exact cap is a policy decision: receivers are allowed to ask for time, while the sender needs a bounded, reviewable rule for work that cannot progress automatically.
Then make one transaction carry the facts that matter: the delivery identifier, attempt number, response class, calculated eligibility time, and an append-only audit event. Commit it before acknowledging the queue message. If the consumer exits after the commit but before acknowledgement, the queue can redeliver; the delivery-state transition must recognize that duplicate. If it exits before the commit, the queue delivery remains available and the durable record has not advanced. This is an exactly-once mindset applied to the business transition, without pretending that HTTP or queues offer exactly-once transport.
The distinction becomes important during a timing race: a receiver may accept a request, the sender may lose the response, the queue lease may lapse, and a second worker may obtain the same message, all while the original business event remains a single obligation. The sender cannot infer a safe final state from one network observation. It must retain the delivery identity, send the same idempotency key on the repeated attempt, and preserve both observations in the audit trail so that reconciliation can distinguish an unresolved transport outcome from a newly created event. That is slower to describe than a retry loop. It is the part that makes the loop defensible.
A stable idempotency key belongs to the originating business event and is reused for every webhook attempt. The receiving endpoint may or may not implement deduplication, so the sending system must still reconcile its own records. Don't generate a new key for a retry: that converts one obligation into two indistinguishable operations.
A useful delivery row has fields such as delivery_id, region, payload_ref, state, attempt_count, next_attempt_at, lease_owner, lease_expires_at, and idempotency_key. Keep payload material in its assigned regional store where residency policy requires it; the scheduling record can carry a reference instead. The audit log should append transitions rather than overwrite them, because an operator eventually has to answer why an event was deferred and whether it was sent again.
Make delayed republish a database state transition
A delayed republish is often described as a queue feature, but the portable part of the design is the state machine. A scheduler claims rows whose next_attempt_at has arrived, leases a bounded batch, and publishes compact messages to the queue in the same region. The consumer reloads the payload, preserves the idempotency key, records the attempt, and sends the HTTP request. Its next action follows the durable state, rather than a callback chain inside one process.
PostgreSQL's FOR UPDATE SKIP LOCKED can distribute claims across schedulers without making each worker wait on a row another worker has already leased. PostgreSQL also documents that SKIP LOCKED yields an inconsistent view, which is why it is appropriate for work distribution and inappropriate for reconciliation or balance-like assertions. A reconciliation query should examine overdue rows, expired leases, and business events with no final delivery state without relying on skip-locked selection.
Cron is a reasonable clock edge for a short claim-and-publish pass. It is not the source of truth. If one invocation is missed, the following invocation must find every already-eligible row, which is why eligibility resides in the database. For a low-volume service, a periodic scan produces a simple operational model. A continuous poller can reduce the delay floor, but it adds another always-on process and more pressure to measure lease behavior carefully.
Here is the narrow contract that matters. The code is Go to make the transaction boundary explicit; a Node.js consumer should implement the same SaveDeferral contract with its database client.
package delivery
import (
"context"
"errors"
"net/http"
"strconv"
"time"
)
type Message struct {
DeliveryID string
Attempt int
Region string
IdempotencyKey string
}
type Store interface {
SaveDeferral(context.Context, Message, time.Time) error
}
func RetryAt(header http.Header, now time.Time, maxDelay time.Duration) (time.Time, error) {
raw := header.Get("Retry-After")
if seconds, err := strconv.Atoi(raw); err == nil && seconds >= 0 {
delay := time.Duration(seconds) * time.Second
if delay > maxDelay {
delay = maxDelay
}
return now.Add(delay), nil
}
when, err := http.ParseTime(raw)
if err != nil || when.Before(now) {
return time.Time{}, errors.New("invalid Retry-After")
}
if when.Sub(now) > maxDelay {
return now.Add(maxDelay), nil
}
return when, nil
}
func Defer429(ctx context.Context, store Store, m Message, response *http.Response, now time.Time) error {
if response.StatusCode != http.StatusTooManyRequests {
return errors.New("expected HTTP 429")
}
eligibleAt, err := RetryAt(response.Header, now, 24*time.Hour)
if err != nil {
return err
}
return store.SaveDeferral(ctx, m, eligibleAt)
}
The caller acknowledges only after SaveDeferral succeeds. An absent or invalid Retry-After needs an explicit local policy, such as a bounded backoff schedule, rather than an accidental default from a queue library. Test both header forms, past dates, maximum delays, duplicate queue deliveries, commit-versus-acknowledgement interruption, and lease expiry with a frozen clock. A property test can assert that no computed eligibility time precedes now; an integration test can abandon leases on purpose and prove that reconciliation returns the work to a reviewable path.
Choose the timer mechanism by the failure you must explain
The comparison is not about which component has the shortest API. It is about where the obligation survives and what evidence remains after a disputed or repeated delivery.
| Mechanism | Appropriate constraint | Trade-off |
|---|---|---|
| Worker sleep | Disposable, very small experiments | Occupies concurrency and loses its timer when the process exits |
| Queue delay | The queue's delay range and regional placement meet the contract | Delivery history can be separate from the business record |
| Database schedule plus republish | Auditability and reconciliation are primary | Adds leases, indexes, polling, and database write load |
| Cron scan | A coarse delay floor is acceptable | Scan cadence sets the earliest practical retry time |
For webhook obligations associated with financial state, the database schedule is often the clearest model because the audit trail can join to the originating event. The catch is real: it is not suitable when a large population of long-lived timers would compete with latency-sensitive transactional queries. In that situation, use a durable scheduling system with timer semantics, regional placement, and observability that satisfy the same idempotency and audit requirements. A queue-delayed message is also a better fit when its documented delay range provides the precision the service needs.
US and EU deployment adds a boundary that is easy to blur in diagrams. Keep each region's delivery rows, queues, payload references, credentials, and workers local unless the legal and security model explicitly permits a transfer. Aggregate operational measures such as queue age, 429 counts, expired leases, and reconciliation gaps only after deciding which fields are permitted to leave the region. Compliance retention periods are organization-specific; counsel and security teams should review retention, payload deletion, and access controls instead of assuming that an audit log may live forever.
Roll out the scheduling path without changing the obligation
Start with one region and a low-risk event family. Run the scheduler in observation mode, calculating eligibility and comparing its candidate set with the current path while it claims nothing. Once the timestamps and regional routing agree, enable small lease batches and destination-level concurrency limits. Track queue age, eligible-row age, attempt distributions, receiver 429 rates, expired leases, and the count of events that reconciliation cannot classify.
Keep the rollout reversible at the routing layer, but preserve the same delivery identifier and idempotency key while switching paths. That detail is easy to miss — it is the difference between a controlled migration and a new set of side effects. Finally, rehearse recovery: stop publishing, let leases expire, resume the scheduler, and verify that repeated queue transport leads to one durable business outcome and a complete audit trail. Your mileage may vary on scan intervals, but the invariant should not vary: delayed work is durable state.
Top comments (0)