Short answer: schedule a small, lease-protected scan that publishes stable outbox IDs, then let idempotent workers deliver delayed webhooks; do not make the nightly trigger perform network delivery itself.
The deciding constraint is recovery, not the syntax used to schedule a job. A batch that discovers due records, writes to a queue, calls arbitrary customer endpoints, retries failures, and declares success in one process has no useful failure boundary. It has one long timeout and an optimistic log line. For an SRE, that is an invitation to discover a growing backlog after the promised delivery window has already passed.
Keep the clock boring.
This runbook treats a scheduled scan as a producer with a short time budget. It claims a bounded set of due rows, submits only their immutable identifiers, and exits. A separate worker reloads the authoritative record, performs the outbound request, and records a state transition. The split gives each step a measurable SLO: discovery lag, accepted publications, oldest queued age, delivery attempts, and completed outcomes. A green trigger heartbeat alone proves almost nothing.
The signal that says a scheduled delivery path is unsafe
The risky design usually begins as an innocent loop: at a fixed time, select all pending webhook rows, POST each payload, and mark the batch finished. The loop couples database pagination to remote latency. One slow receiver consumes execution time that should have been available to find unrelated rows; a deployment interruption leaves the operator guessing which external effects occurred; and overlapping invocations can claim the same work unless ownership is explicit.
The worst dashboard for this system reports only queue depth. A shallow queue can coexist with a stalled scanner that is no longer finding due work. A deep queue can be acceptable during a controlled recovery if its oldest-item age is falling and the drain rate exceeds the arrival rate. Alert on the age of work that has not reached its promised business state, then use counts at each handoff to locate the missing transition.
There is no global exactly-once guarantee across a database, a message transport, and a remote HTTP endpoint. A process can successfully submit an ID and stop before it records that submission. Repeating that ID is the correct recovery behavior, provided the delivery path recognizes it. Design for at-least-once transfer and idempotent business effects instead of hiding this ambiguity behind a batch-level success field.
How should a cron trigger move delayed pending webhooks from a batch queue to Node.js workers?
Put the business change and an outbox row in the same database transaction. The transactional outbox pattern addresses the gap where a database update commits but the corresponding message is never published. Each row needs a stable ID, due time, delivery destination or subscription key, payload or payload reference, and an explicit state. The trigger claims only rows whose due time has arrived, using a lease or conditional update, so two schedules cannot own the same row at once.
After acceptance by the queue, record the publication state if possible, but do not treat that record as an atomic cross-system transaction. If the process stops between those two operations, the lease eventually expires and the row may be submitted again. The worker must therefore make duplicate IDs harmless. It can keep an idempotency record before invoking the receiver, or pass the stable ID as an idempotency key when the receiver supports one. Where the receiver cannot express that contract, ambiguous network outcomes require a reconciliation process; pretending otherwise shifts uncertainty onto users.
Ordering should be scoped to the actual business promise. Global FIFO ordering can let one blocked destination hold every tenant hostage. Per-subscription ordering, with a sequence number and a rule that holds a successor until its predecessor reaches a terminal state, is often enough. Some events do not need it. Cache invalidations that converge to a final state may tolerate reordering, while financial or lifecycle events frequently do not. Your mileage may vary because the receiver's semantics determine the contract.
The code below is intentionally transport-neutral. It shows the application boundary rather than a setup guide for a particular service. ClaimDue should be a short transaction that either leases eligible records or uses a conditional state update; its implementation must make a concurrent claim return no row.
package delivery
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
)
var ErrMissing = errors.New("outbox record is missing")
type OutboxRow struct {
ID string
}
type Store interface {
ClaimDue(ctx context.Context, limit int, lease time.Duration) ([]OutboxRow, error)
MarkPublished(ctx context.Context, id string) error
Load(ctx context.Context, id string) ([]byte, error)
MarkDelivered(ctx context.Context, id string) error
}
type Publisher interface {
Publish(ctx context.Context, body []byte) error
}
type Sender interface {
Send(ctx context.Context, id string, payload []byte) error
}
type Service struct {
Store Store
Publisher Publisher
Sender Sender
}
func (s Service) PublishDue(ctx context.Context, limit int) error {
rows, err := s.Store.ClaimDue(ctx, limit, 30*time.Second)
if err != nil {
return fmt.Errorf("claim due records: %w", err)
}
for _, row := range rows {
body, err := json.Marshal(struct {
OutboxID string `json:"outbox_id"`
}{OutboxID: row.ID})
if err != nil {
return fmt.Errorf("encode record ID: %w", err)
}
if err := s.Publisher.Publish(ctx, body); err != nil {
return fmt.Errorf("publish record ID %s: %w", row.ID, err)
}
if err := s.Store.MarkPublished(ctx, row.ID); err != nil {
return fmt.Errorf("record publication for %s: %w", row.ID, err)
}
}
return nil
}
func (s Service) Work(ctx context.Context, outboxID string) error {
payload, err := s.Store.Load(ctx, outboxID)
if errors.Is(err, ErrMissing) {
return nil
}
if err != nil {
return fmt.Errorf("load record %s: %w", outboxID, err)
}
if err := s.Sender.Send(ctx, outboxID, payload); err != nil {
return fmt.Errorf("send record %s: %w", outboxID, err)
}
return s.Store.MarkDelivered(ctx, outboxID)
}
The Node.js process shape does not change this design. Its scheduler can invoke the publisher, and its consumer can invoke the worker; the durability and duplicate-handling rules remain outside the runtime. Avoid placing credentials or full customer payloads in a task body. An ID plus an authoritative reload reduces accidental disclosure in queue inspection tools and makes payload updates easier to audit.
Choose capacity limits before the backlog chooses them for you
A bounded scan is a protection for the primary database, not a claim that the system will catch up. Start capacity planning with the sustained rate of newly due records, the sustained successful worker rate, and retry amplification. Recovery headroom is successful throughput minus new arrivals after retries. If the result is near zero, a backlog will never drain inside its SLO, even if each individual worker appears healthy.
| Approach | Fits when | Do not choose it when | Operational question |
|---|---|---|---|
| Managed task service | The team wants a durable HTTP task boundary without operating queue storage | Portability or broker-level control is a hard requirement | Does dispatch capacity hold oldest-item age within the delivery SLO? |
| Self-hosted queue | The team already funds storage, upgrades, backup, and recovery ownership | Those duties cannot join the on-call rotation | Can the broker and worker fleet absorb a retry surge? |
| Database dispatcher | Volume is modest and the database is the natural source of truth | Claim traffic would compete with critical transactional load | What scan and update budget remains at peak? |
| Workflow engine | Delivery is one step in a durable multi-step business process | A single outbound side effect is the entire job | How many active steps can the fleet sustain? |
The catch is organizational. A managed service can reduce control-plane work while constraining placement and transport choices; a self-hosted queue retains those choices while making persistence and recovery part of the team's pager duty. Stick with a database dispatcher when the rate is small, the schema can support indexed due-time claims, and adding another durable system creates more operational risk than it removes. None of these choices is suitable without an owner for backlog recovery.
Set per-destination concurrency as well as global concurrency. A receiver that slows down should consume only its own share of capacity. Batch size should be chosen from measured query cost and lease duration, then revised after a representative load test; a larger value is not automatically faster if it increases lock contention or leaves too much work stranded behind a stopped process.
Don't guess capacity.
Verify the handoff, then rehearse rollback
Before rollout, seed a non-production environment with duplicate IDs, records that become due at a page boundary, and a receiver that responds more slowly than the worker deadline. Verify that a duplicate converges on one business effect, a claimed row always reaches an explicit state, and the oldest pending age returns toward baseline after worker capacity resumes. These are handoff tests, not merely happy-path tests.
Run one targeted interruption test: submit an ID, prevent the publisher from recording its publication, then allow its lease to expire. The next scan should submit the ID again, and the worker should make that repeat safe. This exercises the exact place where an atomic transaction is unavailable.
Treat that test as a timeline, because the timestamps expose gaps that a final state can conceal. First, create one due record and capture its stable ID. Next, confirm that the scanner's claim makes a second scanner unable to claim it during the lease. Then accept the ID into the queue while intentionally withholding the local publication update, and wait for the claim to become eligible again. The later scan is allowed to submit the same ID; the critical assertion is that the receiver's business effect remains singular while the audit trail shows both transfer attempts. Repeat the exercise with a worker that finishes the remote action before its own final state write, because the symmetric ambiguity belongs in the same operating model. Finally, restore normal processing and verify that the record settles into a terminal state without manual edits. This sequence is deliberately less glamorous than a throughput benchmark, but it proves that the scheduled handoff degrades into an auditable duplicate rather than an untraceable loss. It also tells the on-call engineer what evidence to collect: the row's claim history, the queue message identifiers, the delivery attempts, and the receiver's idempotency record. Without those links, a retry policy is just a hope dressed up as automation.
The production view needs a scan heartbeat, discovered-row count, accepted-publication count, queue age, attempt rate, delivery outcomes, worker saturation, and a drain-time estimate. Stable outbox IDs belong in traces and structured logs at every boundary. Payloads and credentials do not.
For rollback, stop new claims before touching already accepted tasks. Preserve the rows and their IDs, pause consumers according to the transport's documented semantics, and reconcile pending, published, attempting, and delivered states before reopening the trigger. Define the rollback threshold in delivery-lag terms before the first cohort is enabled. A timestamp guess during pressure is not a runbook.
Top comments (0)