Short answer: use cron only to discover due media webhooks, commit each discovery to a PostgreSQL transactional outbox, and let a rate-limited worker pool deliver from the queue with a stable idempotency key. The schedule is a recovery clock, not proof that a webhook ran.
That distinction prevents the awkward failure in which a nightly scan marks 8,000 asset notifications as queued, crashes between the database update and queue publication, and silently strands part of the batch. It also accepts the opposite failure: a relay can publish the same item twice. The design must make duplicate execution harmless because it cannot make every boundary atomic.
This architecture decision record covers a media pipeline in which transcoding and rights-processing events accumulate as pending deliveries while downstream publishers enforce rate limits. The primary decision is not which cron library to install. It is where retry ownership lives, which identifier survives every handoff, and what evidence allows an operator to distinguish late work from lost work.
What must a cron trigger, queue, delayed webhook batch, and Node.js worker guarantee?
Start with invariants. A pending delivery has one durable identity, such as delivery_id, from discovery through the final HTTP attempt. Its payload is immutable or versioned. A successful terminal transition is conditional, so two workers cannot both turn the same logical delivery into two independent successes. A retry changes scheduling metadata, not identity.
Duplicates happen.
The guarantee should be stated narrowly: every eligible delivery remains discoverable until it reaches a terminal state, and repeated handling does not create a second logical effect. "Exactly once" is too broad across a database, a queue, HTTP, and a receiver you do not control. The transactional outbox pattern instead couples the local state change and the intent to publish in one database transaction; its documented consequence is that the relay may publish more than once, so the consumer must be idempotent.
Four failure boundaries matter:
- The cron process can stop during a scan. Unclaimed rows must still be eligible on the next pass.
- The transaction can commit while the relay has not yet published. The committed outbox row is the recovery record.
- The worker can deliver successfully but lose its acknowledgement. Redelivery must reuse the same idempotency key.
- The destination can answer with a retryable signal such as HTTP 429 or time out with no conclusive application result. Backoff must not create a new logical event.
HTTP 400 usually indicates that repeating an unchanged request won't help, while HTTP 429 explicitly signals rate limiting. Receiver contracts vary, though, and I'm not sure a timeout means failure for any particular publisher; only its API contract can resolve whether a later status lookup is available. In the absence of that lookup, retry with the same key and retain the ambiguous attempt in the audit trail.
Keep the states boring: pending, enqueued, leased, succeeded, and dead. Store next_attempt_at, attempt_count, lease_until, and the last classified outcome. An expired lease makes abandoned work visible again. A queue's acknowledgement timeout is transport behavior, whereas the database record is the business truth.
Name the failure boundary before choosing the scheduler
The nightly trigger should be deliberately weak. It wakes a reconciler, records a run identifier, and repeatedly claims bounded pages whose next_attempt_at is due. It does not hold a database transaction open while calling a queue or webhook. It doesn't load the entire backlog into memory either — a large media release can turn that shortcut into a memory spike and a burst that immediately collides with the publisher's quota.
Run overlap is normal. Row locking with skip-locked behavior, or an equivalent compare-and-set claim, lets two reconcilers divide work without treating overlap as an incident. A unique constraint on the outbox's logical message key closes the second gap: if a row is rediscovered, the insert conflicts rather than creating another logical dispatch.
The queue is a pressure buffer. It can spread work over time and support retry delivery, but it should not decide whether a rights-update webhook is still required. That decision belongs beside the media record and its delivery state. Managed task queues are explicitly designed for asynchronous work and expose controls for dispatch rate and retries; those controls help protect a limited worker pool, yet they do not replace application-level deduplication.
Use both a concurrency ceiling and a start-rate ceiling. Concurrency limits protect file descriptors, memory, and downstream in-flight capacity. Start-rate limits protect a destination that allows, for example, a bounded number of requests per interval. They solve different overload modes.
The retry clock also needs one owner. A clean arrangement is for the queue to handle short transport redeliveries while the delivery table owns longer application backoff via next_attempt_at. If both layers independently run long exponential schedules, the real next attempt becomes difficult to predict and an operator cannot answer a basic question: is this delivery waiting by policy, invisible in transport, or lost?
Compare the control-plane options
The useful comparison is about recovery evidence, not setup time.
| Control plane | Durable recovery evidence | Duplicate boundary | Best fit | Main limitation |
|---|---|---|---|---|
| Cron scans and calls webhooks directly | Delivery rows only | Process death around the remote call | Small, low-value batches with a receiver-side idempotency contract | A slow destination occupies the scanner and mixes discovery with execution |
| Cron scans and publishes directly | Delivery rows plus queue state | Database commit versus publish | Rebuildable notifications where periodic rescans are acceptable | A crash between state change and publish needs careful reconciliation |
| Cron writes a transactional outbox; relay publishes | Delivery rows and committed outbox rows | Relay acknowledgement versus publish | Auditable media events with strict retry ownership | More schema, retention, and relay operations |
| Per-item scheduled tasks | Task records managed by the scheduling system | Task execution versus worker acknowledgement | Precisely timed items with no need for set-based database discovery | Cancellation and bulk reconciliation span two control planes |
The third option is the decision here. The catch is operational weight: an outbox needs indexes, relay lag monitoring, retention, and capacity planning. It is not suitable when notifications are disposable, the dataset is tiny, and a periodic source-of-truth rescan is already cheap. In that case, stick with a direct scan-and-publish loop, but leave rows pending until publication is confirmed and make the scan repeatable.
Per-item scheduling is valid when each item has an independent future execution time and the external scheduler is intended to own that clock. It becomes less attractive for nightly set reconciliation, where eligibility depends on current database state and operators need to re-evaluate a whole cohort after a policy change.
No option removes receiver cooperation. Consider the ugly boundary in slow motion: worker A sends delivery media-7842; the receiver commits the rights update; the connection closes before A reads the response; A's 90-second lease expires; and worker B claims the same row. Marking the first attempt successful would be a guess, while creating a fresh key would invite a second logical update. Reusing media-7842 gives a cooperating receiver the evidence it needs to return the prior result or suppress the duplicate, and the sender's attempt log still records why the retry occurred. If the destination ignores idempotency keys and a POST has a non-idempotent effect, the lost response remains irreducibly ambiguous. A sender can suppress its own known duplicates; it cannot prove that an opaque remote side effect happened exactly once.
Put the critical path in one transaction
The following Python is intentionally built around generic database and queue interfaces. A Node.js service should preserve the same transaction boundaries and unique constraints; changing the runtime must not change the state machine.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from hashlib import sha256
import json
from typing import Protocol
class Database(Protocol):
def transaction(self): ...
class Publisher(Protocol):
def publish(self, message_id: str, body: bytes): ...
@dataclass(frozen=True)
class Delivery:
delivery_id: str
asset_id: str
event_type: str
payload: dict
def enqueue_due_page(db: Database, run_id: str, page_size: int = 250) -> int:
"""Atomically claim due deliveries and record publication intent."""
with db.transaction() as tx:
rows = tx.query(
"""
SELECT delivery_id, asset_id, event_type, payload
FROM webhook_delivery
WHERE state = 'pending' AND next_attempt_at <= now()
ORDER BY next_attempt_at, delivery_id
FOR UPDATE SKIP LOCKED
LIMIT %s
""",
[page_size],
)
for row in rows:
body = json.dumps(row, sort_keys=True, separators=(",", ":"))
message_id = sha256(
f"webhook:{row['delivery_id']}".encode()
).hexdigest()
tx.execute(
"""
INSERT INTO dispatch_outbox
(message_id, delivery_id, run_id, body, created_at)
VALUES (%s, %s, %s, %s, now())
ON CONFLICT (message_id) DO NOTHING
""",
[message_id, row["delivery_id"], run_id, body],
)
tx.execute(
"""
UPDATE webhook_delivery
SET state = 'enqueued', updated_at = now()
WHERE delivery_id = %s AND state = 'pending'
""",
[row["delivery_id"]],
)
return len(rows)
The outbox relay reads unpublished rows in bounded pages, publishes message_id with the body, and records publication after the queue accepts it. A stop after publish but before that record causes another publish. That's expected. The worker therefore claims the delivery with a lease and checks terminal state before sending.
def claim_delivery(db: Database, delivery_id: str, lease_seconds: int = 90):
now = datetime.now(timezone.utc)
with db.transaction() as tx:
return tx.query_one(
"""
UPDATE webhook_delivery
SET state = 'leased',
lease_until = %s,
attempt_count = attempt_count + 1,
updated_at = %s
WHERE delivery_id = %s
AND state <> 'succeeded'
AND (state <> 'leased' OR lease_until < %s)
RETURNING delivery_id, destination, payload, attempt_count
""",
[now + timedelta(seconds=lease_seconds), now, delivery_id, now],
)
def record_success(db: Database, delivery_id: str) -> None:
with db.transaction() as tx:
tx.execute(
"""
UPDATE webhook_delivery
SET state = 'succeeded', lease_until = NULL, updated_at = now()
WHERE delivery_id = %s AND state = 'leased'
""",
[delivery_id],
)
Send delivery_id as the receiver's idempotency key. Do not derive it from an attempt number, timestamp, or queue receipt, because each retry would then look new. If payload revisions represent new business events, allocate a new delivery identity explicitly rather than quietly mutating bytes beneath an old key.
The sample's 90-second lease and 250-row page are illustrative controls, not universal tuning advice. Set a lease above the measured high-percentile request duration plus acknowledgement margin, then verify expiry behavior under process termination. Set page size from transaction duration, relay throughput, and database contention. Your mileage may vary.
Operate for evidence, not optimism
Test the boundaries by stopping processes at named points: before commit, after outbox commit, after publish, after the destination accepts, and before success is recorded. Each test should finish with either one terminal logical delivery or a still-discoverable retry. Also run two cron instances concurrently, expire a worker lease, replay the same queue message, and return HTTP 429 with a Retry-After value from a controlled receiver.
Measure age.
Watch the oldest due next_attempt_at, outbox relay lag, lease-expiry count, attempts by outcome class, dead-letter count, and destination-specific dispatch rate because they explain different failure modes. Queue depth can fall while the oldest high-value webhook remains stuck behind repeated bad payloads.
Deployment deserves the same skepticism. Add the outbox and nullable state fields first, deploy code that can read both old and new rows, enable the relay at a low dispatch rate, and only then turn on the reconciler. Rollback must preserve committed outbox rows; deleting them because an application version changed destroys the audit trail.
Retention has a cost. Keep enough terminal delivery and outbox metadata to cover the business replay window and investigations, but move bulky payloads out of hot indexes and apply the media system's data-retention rules. Partitioning by creation time can make expiration predictable, provided the idempotency identity remains protected for the entire duplicate-delivery window.
The rejected design is a single nightly process that selects every pending row, calls each destination, and marks success inline. It remains a reasonable choice for a few low-impact callbacks, short runtimes, and a receiver that enforces idempotency. For a rate-limited media pool with a backlog large enough to outlive one process, its coupled scan-and-send loop gives too little evidence at precisely the moments when retry decisions matter.
Keep cron replaceable. The durable protocol is the row state, outbox record, stable identity, lease, and conditional terminal update. Once those invariants are explicit, a platform scheduler, a host timer, or a manually initiated recovery run can trigger the same reconciler without changing delivery semantics.
Top comments (0)