Short answer: for a webhook follow-up that is due one hour later, record a durable obligation keyed by the source event, let workers claim due work, and reserve cron for recurring reconciliation rather than creating a cron entry for each event.
The decision is less about a Node.js library than about what remains true after the request handler has exited, a process has been redeployed, and the upstream sender has retried the same webhook. A delayed message, a database-backed queue, and a scheduled scan can all initiate work; none removes the need to identify the logical action once, preserve its history, and verify its outcome. For payment- or ledger-adjacent systems, that distinction is consequential: a timer is not evidence that an obligation was accepted.
One durable record changes the question from "did a timer fire?" to "which accepted event still lacks an explained terminal state?" That is a question an operator can query, audit, and reconcile.
How should a Node.js service schedule a webhook follow-up task one hour later?
Treat the incoming webhook as an input to a transaction, not as permission to start a process-local clock. Authenticate and validate it, derive a stable identity such as (source_event_id, action), and create a row whose run_at is the accepted timestamp plus one elapsed hour. A unique constraint makes a retried webhook point back to the same obligation. The HTTP response can then mean something precise: the service has durably accepted the future work. It does not claim that the later side effect has occurred. For a financial workflow, retain the minimum useful evidence: source event ID, action, payload digest where appropriate, scheduled time, state, attempt count, idempotency key, and timestamps for each state change. Keep sensitive payload data out of the work record unless the system's retention, access, and compliance rules permit it. An audit trail and a current-state record serve different purposes. The former explains how the state changed; the latter makes overdue work visible without replaying a log. The hard boundary is exactly-once semantics. Most execution transports can redeliver, and a worker can finish a downstream call just before it loses its lease. The practical target is exactly one logical business effect, obtained with an idempotency key that remains stable across attempts and with downstream confirmation when that confirmation exists. Don't discard the record merely because a connection received an acknowledgment; an acknowledgment can establish receipt without establishing the intended business result. Keep the state machine small: scheduled, leased, succeeded, retryable, cancelled, and a reviewed terminal failure state. Cancellation is a transition, not a deletion, because an absence of delivery may later need an explanation. Rescheduling needs an auditable revision or a version-checked update, otherwise an old worker can execute a superseded deadline. This seems fussy until a cancellation and a claimed job cross paths: without a version or an ordered transition record, neither the worker nor the auditor can establish which instruction governed the effect, and retrying the task can turn a scheduling ambiguity into a duplicated customer-facing action. The record is the decision point; the queue is only an execution aid.
The constraint comes before the mechanism
A delayed-message system fits when independent workers must absorb bursty follow-ups and the operating team can own broker behavior. A database work table is attractive when the webhook mutation and the obligation belong in the same transactional database: an outbox-style write gives both changes one commit point, avoiding an invoice update without its required follow-up, or a follow-up for a business change that rolled back. A fixed cron schedule fits a different identity: a recurring task such as finding all obligations overdue beyond a threshold.
Per-event cron entries are usually the odd one out. They distribute business state through machine or scheduler configuration, make cancellation and rescheduling harder to inspect, and leave reconciliation dependent on infrastructure inventory rather than a searchable business ledger. A single periodic scan, by contrast, can recover overdue records and alert on gaps while the actual deadline remains stored with the event.
| Mechanism | Good fit | Operational boundary |
|---|---|---|
| Delayed message | Independent workers and uneven event volume | Keep deduplication, cancellation, and business evidence outside the transport |
| Database work table | Moderate volume near transactional data | Polling and row contention consume capacity on the database |
| Fixed cron sweep | Periodic discovery and recovery | It must rediscover durable work; it is not the per-event record |
| Cron per event | Small host-bound administration | Poor visibility and lifecycle control for business obligations |
The catch is isolation. A database queue is not suitable when sustained polling or locking threatens the latency budget of the primary transaction workload; isolate execution onto a dedicated broker or separate store while retaining a business-facing status record. Conversely, a separate messaging dependency is a poor bargain for a small service with low volume and no operational capacity for another stateful component. Use a fixed sweep when the requirement is really a coarse scan, not a deadline attached to each webhook event.
Claim due work with a lease and an idempotency key
PostgreSQL can support a simple work-claim loop. FOR UPDATE SKIP LOCKED allows concurrent consumers to skip rows already locked by another transaction, which makes it useful for a database-backed queue; the lease, status update, idempotency key, and reconciliation process provide the correctness model around it. The query must be indexed around due time and state, and the transaction must change the record to a leased state before the worker performs the external action.
The following Go sketch expresses the storage contract without tying the design to a particular Node.js queue package. A Node.js handler should enforce the same constraint: InsertScheduled is unique by source event and action, while ClaimDue atomically assigns a lease.
package followup
import (
"context"
"time"
)
type Job struct {
ID string
Action string
RunAt time.Time
IdempotencyKey string
}
type Store interface {
InsertScheduled(context.Context, string, string, time.Time) (Job, error)
ClaimDue(context.Context, time.Time, time.Time) (Job, error)
MarkSucceeded(context.Context, string, time.Time) error
MarkRetryable(context.Context, string, time.Time, string) error
}
type Executor interface {
Apply(context.Context, string, string) error
}
func Schedule(ctx context.Context, store Store, eventID string, acceptedAt time.Time) (Job, error) {
return store.InsertScheduled(ctx, eventID, "webhook-follow-up", acceptedAt.Add(time.Hour))
}
func RunOne(ctx context.Context, store Store, executor Executor, now time.Time) error {
job, err := store.ClaimDue(ctx, now, now.Add(30*time.Second))
if err != nil {
return err
}
if err := executor.Apply(ctx, job.Action, job.IdempotencyKey); err != nil {
return store.MarkRetryable(ctx, job.ID, now.Add(time.Minute), err.Error())
}
return store.MarkSucceeded(ctx, job.ID, time.Now().UTC())
}
Retries are normal. Classify validation and authorization failures separately from temporarily unavailable downstream work, bound the retry policy, add jitter, and preserve the idempotency key across every attempt. The correct retry interval depends on the downstream contract, deadline, and rate limits; no generic interval is defensible without those inputs. A lease expiry is also a recovery mechanism, so monitoring must surface jobs whose lease age or overdue age exceeds the promised service window.
Make reconciliation a deployment requirement
Before enabling the downstream effect, run the scheduling path in shadow mode. Persist accepted webhook metadata, create obligations, claim them, and record what would have happened without emitting the effect. Compare counts over a defined interval: unique accepted events, scheduled records, due records, completed records, cancellations, and reviewed failures. Every accepted unique event should map to one explained state. Small discrepancy. Big problem.
Then enable a deterministic cohort and measure two latencies separately: acceptance-to-due and due-to-completion. The first points to clocking and scheduler behavior; the second points to worker capacity or downstream dependencies. Test duplicate webhook delivery, termination after an external effect but before the local acknowledgment, cancellation racing a claim, a leased-work deployment, and clock skew. Those tests turn an exactly-once aspiration into a set of observable invariants.
The rollout should also define ownership. Someone must review terminal failures, decide when a cancellation wins, and approve retention for the audit records. Cost is part of that decision, but it is wider than an invoice: database I/O, idle workers, retention, on-call load, and reconciliation time all matter. Your operating constraints may favor a different mechanism, yet the durable obligation and its evidence should remain the part that does not move.
Top comments (0)