Short answer: let a nightly trigger start a bounded scan, let the scan enqueue references to pending webhook work, and let idempotent workers perform delivery while durable application state remains the record used for recovery and reconciliation.
For an edtech payment system, this is the least complicated design that keeps an interrupted run explainable. The scheduler answers when to look. A durable outbox-style record answers what must be done. The queue moves a bounded piece of work. The worker owns the remote call. The reconciliation query finds anything that was missed, duplicated, or left behind by an expired claim.
That distinction matters more than the choice of timer. A payment provider may accept a request while the worker loses its connection before recording the response; a queue may deliver one message twice; and a scheduler may start late or fail to run. The desired property is exactly-once business effect, not an imaginary exactly-once transport. Stable delivery identities, idempotent state transitions, and an audit trail are the mechanisms that make that property attainable.
Short runs win.
The failure starts after the timer fires
The most dangerous assumption is that a successful scheduler invocation means useful work happened. Imagine the nightly run for an online course platform: the scanner claims 100 payment-provider notifications, publishes the references, and loses its database connection while marking them queued. The next run sees the lease as expired. If the system treats “published” as a permanent fact, 100 students may never receive a status update; if it blindly sends every reference again, the provider may observe duplicate effects. The correct response is neither to trust the timer nor to fear every duplicate. The durable delivery row must say what the application knew, the attempt record must say what it tried, and the receiver-facing identity must let a repeated attempt become an auditable no-op when the business operation already took effect. That same record also gives an operator a way to distinguish a late scan from a rejected provider request, which is why recovery belongs in the application state rather than in a cron log.
That is the trap.
What should a nightly Node.js worker do with pending webhook records?
Start with the business transaction. When an enrollment payment or refund creates a fact that another system must learn about, write the business change and an outbox record in the same database transaction. The delivery record should contain a stable delivery_id, destination, payload or payload reference, current status, attempt count, and next_attempt_at. It should also have enough information to explain why a delivery is pending without depending on a queue message that may later be acknowledged.
The nightly trigger calls a public HTTPS endpoint owned by the application. That endpoint selects a bounded page of records whose next_attempt_at is due, claims them with a lease or conditional update, and publishes references rather than large payloads. It returns after the handoff. A worker loads each reference, records an attempt, sends the webhook, persists the outcome, and acknowledges the queue message only after the durable transition succeeds.
The surrounding service may be written in Node.js, but the invariant is language-independent. This Go example shows the boundary without pretending that an in-memory map is a production database.
package main
import (
"encoding/json"
"net/http"
)
type Delivery struct {
ID string `json:"delivery_id"`
}
type Store interface {
ClaimDue(limit int) ([]Delivery, error)
MarkQueued(ids []string) error
}
type Publisher interface {
PublishBatch([]Delivery) error
}
func scheduledScan(store Store, publisher Publisher) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
batch, err := store.ClaimDue(100)
if err != nil {
http.Error(w, "claim failed", http.StatusConflict)
return
}
if err := publisher.PublishBatch(batch); err != nil {
http.Error(w, "publish failed", http.StatusBadGateway)
return
}
ids := make([]string, 0, len(batch))
for _, delivery := range batch {
ids = append(ids, delivery.ID)
}
if err := store.MarkQueued(ids); err != nil {
http.Error(w, "state update failed", http.StatusConflict)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]int{"claimed": len(batch)})
}
}
The ordering has an awkward edge. If publication succeeds and MarkQueued does not, the next scan can publish the same reference again after the lease expires. That is acceptable only if the worker treats delivery_id as an idempotency key and the receiver has an equivalent deduplication rule where the business effect requires one. It is better to see two attempts in an audit trail than to hide a possible duplicate behind a counter.
The audit trail is the recovery mechanism
Recovery begins with a state machine, not with a longer retry loop. A useful set of states is pending, claimed, queued, delivered, and dead_lettered, with explicit timestamps for claim, attempt, and terminal outcome. The exact names can differ, but every transition needs an owner and a condition that makes it safe to repeat.
For example, a worker can load a queued reference and atomically create an attempt record keyed by (delivery_id, attempt_number). It then sends the request with a delivery identifier, verifies the provider response, and commits the result. If the process dies between the remote response and the commit, a later attempt must be able to recognize the same business operation or ask the receiver to do so. A timeout is not proof that the provider rejected the request.
This is where payment reconciliation changes the design. A webhook log that says “sent” is not enough evidence that the provider and the local ledger agree. Store the provider's response metadata, request hash, response classification, and the correlation identifier used to find the related payment. Keep secrets out of logs, and make operator-visible records sufficient to trace a decision without exposing the payload itself.
The scanner should order by due time and a stable identifier, claim only a bounded page, and make claims expire. It should find due rows from durable state on every run. A missed nightly invocation then increases delay without deleting the work list, and an overlapping invocation encounters a claim rule rather than silently assuming that the first invocation completed.
One query is often more valuable than another timer: “Which payment-related deliveries have been due longer than the agreed recovery window, and which provider result is missing?” That query turns an incident into a finite queue of decisions.
When should a team reject this queue boundary?
The queue is a transport boundary. It is not the system of record for compliance evidence, customer support, or ledger reconciliation. A message should carry a compact reference such as delivery_id; the canonical payload, retention policy, attempt history, and terminal result belong in application storage.
This separation also limits damage when a batch is malformed or a destination is slow. The scanner can stop publishing new references while operators inspect pending rows. Workers can apply per-destination concurrency limits without holding the scheduler open. A dead-letter path can preserve the reference and reason for review, while the original outbox record remains queryable.
The design has limits. A basic queue is a poor fit when the requirement is a long-lived workflow with joins, compensation, or a replayable history for multiple consumer groups. It is also unsuitable as the only archive when regulatory evidence must outlive the queue's retention policy. Stick with a workflow engine or an append-only event platform when those properties are first-order requirements; use the bounded queue pattern when the actual job is dispatching independently recoverable deliveries.
There is no honest shortcut around destination behavior. A provider that rate-limits requests needs backoff and a per-destination budget. A provider that returns an ambiguous timeout needs reconciliation, not immediate blind replay. Payload signatures need a stable canonical representation, and clock-based due dates need a stated timezone policy. These are ordinary engineering constraints, but they are exactly where a nightly batch becomes a source of duplicate effects if they remain implicit.
How can a team roll out nightly webhook recovery safely?
Choose the simplest boundary that preserves the following facts after a process restart:
| Question | Required answer |
|---|---|
| What must be delivered? | A durable row with a stable delivery identity |
| What is ready now? | A query over next_attempt_at and claim state |
| What did the worker try? | An append-only attempt or equivalent audit record |
| What happened remotely? | A stored response classification and correlation ID |
| What happens after a duplicate? | An idempotent no-op or a receiver-side deduplication decision |
| How is a missed schedule repaired? | The next scan rediscovers due durable rows |
If one answer depends on the scheduler's memory, the design is not recoverable yet. If one answer depends only on a queue message, the design is not auditable enough for a payment-facing system.
The practical rollout is incremental. First write outbox rows and expose the due-work query in read-only mode. Next enable claims for a very small batch and verify that an expired lease is rediscovered. Then enable workers against a test receiver that deliberately returns timeouts, duplicate acknowledgements, and transient failures. Finally compare provider-side results with the local ledger and inspect the audit trail, including the case where publication succeeds just before the scanner process exits.
I'm not sure what batch size suits a particular school-payment population without its provider rate limits, payload sizes, and delivery latency; those measurements should set worker concurrency and page size. The decision rule remains stable: schedule a scan, publish references, perform delivery in workers, and reconcile durable pending state until every effect has an auditable terminal result.
Top comments (0)