The hard part of a delayed webhook is not waiting thirty seconds. It is deciding what a cron job or message queue retry is allowed to do after the first delivery may already have reached a subscriber in a Node.js SaaS.
Short answer: use a message queue for per-delivery delays and retries, then enforce idempotency in the application; use cron for periodic reconciliation, not as the delivery clock. For a small Node.js SaaS, this is usually the clearest boundary between scheduling and business state.
That answer assumes a fan-out workflow: one order changes to shipped, and the SaaS must deliver that update to many subscriber endpoints. The queue can retry work. It cannot know whether a subscriber already applied the shipment update. Your receipt ledger has to answer that question.
Should a Node.js SaaS use cron or a message queue for delayed webhook retries?
Cron is a good fit when the question is calendar-shaped: “run the reconciliation job every five minutes.” A message queue is a better fit when the question is event-shaped: “retry this delivery in thirty seconds, then retry it again after a bounded delay.” Those are different ownership models, even if both start with a timer.
With cron, the database becomes the scheduler. A worker scans for due rows, claims some of them, sends requests, and writes the result. The design looks small until two scans overlap or a process dies after the request leaves the machine but before the row is updated. A lease can recover an abandoned claim. It cannot prove that the remote subscriber did not receive the request. The next scan may send it again.
That is not automatically wrong. Webhook delivery is commonly designed around at-least-once attempts. The important decision is to make duplicate delivery harmless. A queue makes the unit of retry explicit, while an application-owned idempotency record makes the business effect explicit.
The simplest split is:
| Concern | Best owner | Reason |
|---|---|---|
| A delivery becomes due | Queue or delayed-work mechanism | Each event can have its own retry time |
| A periodic sweep finds missing work | Cron | Calendar recurrence is its natural shape |
| A subscriber accepts an event | Delivery worker | It sees the response and timeout |
| The shipment change happens once | Subscriber-side idempotency key | The sender cannot control the receiver's commit |
| A stuck attempt is revisited | Lease or visibility timeout | Crashed workers must not own work forever |
The queue is not the source of truth. The shipment event and its delivery ledger are.
The integration boundary is where duplicates start
Consider order ord_4821 with three subscribers: the storefront, a warehouse dashboard, and a customer notification service. The application creates three delivery records, each with the same logical shipment event and a different destination. The first two return success. The third times out after the remote service has accepted the request.
If the worker treats a timeout as “definitely not delivered,” it retries. If it treats a timeout as “definitely delivered,” it may lose the update. The correct state is unknown, which is why the next attempt must carry the same event identity and why the receiver must deduplicate it before applying the business mutation.
There are several less dramatic failure modes:
- A producer publishes a task and crashes before recording the delivery row, or records the row and crashes before publishing it.
- Two workers receive the same task during a visibility timeout and both call the endpoint.
- A retry counter is reset when a new queue message is created, so a permanently failing destination never reaches a terminal state.
- A fan-out loop uses one mutable payload and accidentally changes the destination or signature between subscribers.
- A receiver returns
429, but the worker treats every non-2xx response as the same permanent failure.
The outbox pattern addresses the first gap: commit the event and an outbox record in one database transaction, then publish the outbox record from a separate process. It still does not make the network call exactly once. That promise is too strong across independent systems.
The useful promise is narrower: each logical delivery has a stable identity, and repeating that identity does not repeat its business effect.
The code example that reserves the receipt
Give every logical delivery an immutable event_id, and give every destination attempt enough context to reconstruct the same signed request. A receiver should persist the identity before or as part of applying the shipment change, under a unique constraint. If the identity is already present, it can return the prior outcome or a safe duplicate response.
HMAC is a practical way to authenticate the body when both sides share a secret. RFC 2104 defines the keyed-hashing construction; the signature should cover the exact bytes sent, not a re-serialized object that might differ across retries. Compare signatures using a constant-time operation, and include a timestamp or expiry policy if replay outside the intended window matters.
Here is a small Python example of the ledger boundary. It is intentionally independent of a queue library: the queue decides when to call the handler, while the database decides whether the event identity is new.
import hashlib
import hmac
import sqlite3
def sign(secret: bytes, body: bytes) -> str:
return hmac.new(secret, body, hashlib.sha256).hexdigest()
def reserve_delivery(db: sqlite3.Connection, event_id: str, body: bytes, secret: bytes) -> str:
db.execute(
"CREATE TABLE IF NOT EXISTS webhook_receipts ("
"event_id TEXT PRIMARY KEY, signature TEXT NOT NULL)"
)
digest = sign(secret, body)
try:
with db:
db.execute(
"INSERT INTO webhook_receipts(event_id, signature) VALUES (?, ?)",
(event_id, digest),
)
return "new_delivery"
except sqlite3.IntegrityError:
return "duplicate_delivery"
if __name__ == "__main__":
connection = sqlite3.connect(":memory:")
payload = b'{"order_id":"ord_4821","status":"shipped"}'
assert reserve_delivery(connection, "ship_4821_storefront", payload, b"secret") == "new_delivery"
assert reserve_delivery(connection, "ship_4821_storefront", payload, b"secret") == "duplicate_delivery"
print("idempotency boundary passed")
In production, the receiver still needs to define the transaction boundary. If inserting the receipt and changing shipment state happen in separate transactions, a crash between them can leave a receipt that suppresses the actual business change. If the remote API has its own idempotency contract, use it. If it does not, make the receiver's unique key and mutation atomic where possible.
The code also leaves one policy open on purpose: should a duplicate return the original 2xx result, or should it expose a distinct duplicate status? Either can work. The contract must be stable enough that the sender does not mistake a harmless duplicate for a new failure.
An eval harness before production
The happy path is a weak test. An eval harness for this workflow should inject a duplicate task, a timeout after remote acceptance, a 429 response, a worker termination before acknowledgement, and a database failure during receipt creation. Then run the same shipment event through all subscribers, because fan-out creates a separate correctness surface for every destination.
Measure business effects, not just queue depth:
- duplicate shipment mutations per event;
- age of the oldest ready delivery;
- attempts and terminal outcomes by destination;
- time from event creation to subscriber acceptance;
- signatures rejected by the receiver;
- work abandoned after a worker lease expires.
Backoff should be bounded and should honor a receiver's explicit retry guidance when one is provided. A maximum attempt count and a terminal state need to be visible to operators; “the queue still contains a message” is not a failure policy.
The cost model is broader than the scheduler's invoice. A database scan every minute may look cheap while creating contention, duplicate sends, and a large operational surface. A queue may cost more in infrastructure or introduce a new operational dependency, but it can make per-event timing and worker ownership explicit. I’m not sure which side wins for your traffic without measuring delivery volume, subscriber latency, and the number of destinations per event. Your mileage may vary.
For notebook-to-prod work, put these cases in the eval harness before tuning prompts or shaving tokens elsewhere. A green queue metric with two warehouse updates applied is still a failed system.
Rollout gates for delivery records
Cron remains the right choice for reconciliation. A periodic job can find shipment events whose expected delivery records are missing, whose leases expired, or whose retry state needs operator attention. It can then enqueue small, bounded units of work. This gives the system a repair path without making cron responsible for every individual delay.
Cron is also reasonable for low-volume internal notifications where a few minutes of timing variation is acceptable and the database already owns the state. The catch is that the scan must still use claims, leases, unique event identities, and an explicit terminal policy. “It runs every minute” does not make the send idempotent.
Choose a queue when subscribers need independent retry schedules, when a slow endpoint must not hold up other destinations, or when worker concurrency needs a clear limit. Stick with cron when the work is a recurring sweep and the sweep can safely enqueue or repair event-level records. Avoid using either mechanism as a substitute for a receiver-side idempotency contract.
That is the decision rule I would carry into a Node.js SaaS: queue the delayed shipment deliveries, reconcile with cron, and make event_id the stable key across every retry. The timer controls when to try. The ledger controls what counts as done.
Top comments (0)