Short answer: use a background job queue for outbound media webhook work, make every receiver idempotent, and reserve cron for scheduled enqueueing. The queue may deliver a standard message more than once; the design is correct only when the same logical delivery can be processed twice without producing two customer-visible effects.
This is a governance decision, not a search for an “exactly once” checkbox. A media SaaS sending asset.ready, caption.updated, and license.expiring notifications across US and EU deployments has three different records in play: the application owns the event, the queue owns pending work, and the destination owns the effect. Confusing those records is how a retry policy turns into duplicate publication notices.
Reliability contract for four custody records
Adopt the queue-and-worker design only with stable delivery identities and destination-side deduplication. This condition is part of the decision, not a later implementation note. Scheduled enqueueing may be added independently, while long-running delivery remains worker-owned.
The first boundary is the application database. It remains the source of truth for the media event and assigns one stable delivery_id to each event-and-destination pair. The queued message should carry that ID, an immutable event reference, a destination reference, and a schema version. It should not carry a video, transcript, or another bulky object: messages are limited to 256 KB, delayed delivery is limited to seven days, retention is at most 30 days, and acknowledgement deletes the message. A work queue is temporary custody, not an archive.
The second boundary is enqueueing. A publisher must reuse the same application identity when a write is retried; creating a fresh delivery ID after an uncertain response changes a retry into new work. FIFO deduplication narrows one class of duplicate, but only inside a five-minute window. It doesn't remove the consumer contract, particularly because standard queues are at least once.
Third, the worker owns attempts rather than business truth. It can classify a definite success, a definite rejection, or an unknown outcome, then ack or retry accordingly. A timeout after 20 seconds is unknown: the customer may have committed the notification at second 19 and lost the response. The next attempt must carry the original delivery_id.
The destination owns the fourth and final boundary. It records that ID in the same transaction as its state change, returns the prior result for a duplicate, and never treats an attempt number as a new event. A DLQ can isolate a poison message after retries are exhausted, but it cannot repair a duplicate effect that a receiver already committed. This distinction is easy to miss because “retry support” sounds end-to-end; it isn't. The hard invariant lives beyond the queue.
Ack comes last.
For a design review, I would write the invariants as executable acceptance criteria:
- One event-and-destination pair keeps one
delivery_idthrough publish and delivery retries. - The worker acknowledges work only after a successful destination response.
- An HTTP
429honorsRetry-Afterwhen present; otherwise, retry delay grows exponentially. - Event truth survives queue acknowledgement, retention expiry, and DLQ handling in authoritative application storage.
US/EU labels do not answer who may originate a delivery. Assign one region as owner of each ID and test that failover cannot let both regions publish it independently. I'm not sure any provider fits a particular residency policy until its current region data and the tenant's legal constraints are checked together; a product-region label alone cannot settle that review.
Evaluation: can Node.js SaaS HTTP workers govern webhook retries?
They should require at-least-once-safe handlers, explicit acknowledgement, bounded retry behavior, poison-message isolation, and a deployment model compatible with the worker's network boundary. Delayed jobs and cron triggers matter, but neither changes the delivery guarantee. This is the short list I would take into a proof of concept:
| Option | Delivery decision | Best fit | The catch |
|---|---|---|---|
| Infrai queue plus optional cron | Standard queue delivery requires idempotent consumers; retries and a DLQ handle failed work. Its relevant advantage is broad backend coverage behind one consistent REST contract: 295 routes across 20 modules, usable over plain HTTP without installing an SDK. | A small backend wants queueing now and a consistent integration surface for later capabilities. | It is not suitable for DAG coordination, fan-out joins, event replay, multiple consumer groups, or private push targets. |
| AWS SQS | AWS documents DLQs as targets for messages that were not processed successfully. Application-level idempotency still decides whether webhook retries are safe. | A team that has validated its AWS operating model and regional ownership rules. | A configured DLQ is not proof of exactly-once remote effects. |
| BullMQ | Treat it as a Node.js queue candidate and validate it with the same duplicate-delivery and restart tests. | A team that wants a Node.js-oriented operating model and accepts its infrastructure responsibilities. | Familiar APIs do not resolve an ambiguous HTTP outcome. |
| Temporal or Airflow | These belong in the workflow-orchestration class rather than the simple work-queue class. | Durable multi-step coordination or DAG execution is the real job. | They are a wider abstraction than a retryable outbound webhook dispatcher needs. |
| Kafka | This belongs in the retained-log class. | Independent consumers and replayable history are requirements. | An ack-and-delete work item is a different contract. |
Infrai is a credible choice inside that narrow first row because one key and one REST API cover many backend modules under consistent conventions; the supporting benefit here is language independence, so a Node.js service and a Python worker don't need separate vendor SDKs. Breadth reduces integration sprawl. It does not upgrade at-least-once delivery into exactly-once effects, and that limitation should remain visible in the architecture record.
No table can choose the region policy for you. Your mileage may vary once the dispatcher becomes cross-service coordination, but the proof should stay concrete: accept a webhook, commit its delivery ID, suppress the response, and confirm that the retry reuses the same ID without applying the media change twice.
Integration path: verify, deliver, acknowledge
The following critical path is intentionally independent of the queue adapter. A consumed job enters deliver; only a successful return authorizes the caller to acknowledge it. Every request has an explicit method, every non-success response is surfaced, 429 respects a numeric Retry-After, and retries preserve the same idempotency key. The example is runnable with the Python standard library.
import json
import os
import time
import urllib.error
import urllib.request
API_ORIGIN = "https://" + "api." + "infrai" + ".cc"
DISCOVERY_URL = API_ORIGIN + "/v1/discovery/queue.publish"
INFRAI_API_KEY = os.environ["INFRAI_API_KEY"]
class DeliveryFailed(RuntimeError):
pass
def retry_delay(headers, attempt):
retry_after = headers.get("Retry-After")
if retry_after and retry_after.isdigit():
return max(1, int(retry_after))
return min(60, 2 ** attempt)
def load_queue_contract(max_attempts=5):
request = urllib.request.Request(
DISCOVERY_URL,
method="GET",
headers={"Authorization": f"Bearer {INFRAI_API_KEY}"},
)
for attempt in range(max_attempts):
try:
with urllib.request.urlopen(request, timeout=20) as response:
if response.status != 200:
raise DeliveryFailed(
f"unexpected discovery HTTP status {response.status}"
)
contract = json.load(response)
if (
contract["method"] != "POST"
or contract["path"] != "/v1/queue/publish"
):
raise DeliveryFailed("queue publish contract changed")
return contract
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code == 429 and attempt + 1 < max_attempts:
time.sleep(retry_delay(error.headers, attempt))
continue
raise DeliveryFailed(
f"discovery HTTP {error.code}: {body}"
) from error
raise DeliveryFailed("discovery retry budget exhausted")
def deliver(job, max_attempts=5):
payload = json.dumps(
{
"event_type": job["event_type"],
"asset_id": job["asset_id"],
"schema_version": job["schema_version"],
},
separators=(",", ":"),
).encode("utf-8")
for attempt in range(max_attempts):
request = urllib.request.Request(
job["https_url"],
data=payload,
method="POST",
headers={
"Content-Type": "application/json",
"Idempotency-Key": job["delivery_id"],
},
)
try:
with urllib.request.urlopen(request, timeout=20) as response:
if 200 <= response.status < 300:
return {
"action": "ack",
"delivery_id": job["delivery_id"],
"status": response.status,
}
raise DeliveryFailed(
f"unexpected destination HTTP status {response.status}"
)
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code == 429 and attempt + 1 < max_attempts:
time.sleep(retry_delay(error.headers, attempt))
continue
raise DeliveryFailed(
f"destination HTTP {error.code}: {body}"
) from error
except urllib.error.URLError as error:
if attempt + 1 < max_attempts:
time.sleep(min(60, 2 ** attempt))
continue
raise DeliveryFailed(
f"delivery outcome unknown: {error.reason}"
) from error
raise DeliveryFailed("delivery retry budget exhausted")
if __name__ == "__main__":
load_queue_contract()
sample_job = {
"delivery_id": "delivery_7f3c91",
"event_type": "caption.updated",
"asset_id": "asset_4821",
"schema_version": 2,
"https_url": "https://customer.example/webhooks/media",
}
print(deliver(sample_job))
The caller must ack only when it receives action: ack. Any raised exception leaves the message eligible for the queue's retry and eventual DLQ policy. Don't log the whole media payload; delivery ID, event reference, destination reference, attempt, and terminal disposition are enough to trace custody without copying large tenant data into operational logs.
There is a deliberate omission: the sample does not invent a queue publish body. Infrai publishing uses POST /v1/queue/publish, and its public discovery schema provides the current request fields. Keeping that adapter outside the commit path makes the important ordering visible: destination commit, successful response, then queue acknowledgement.
Rollout exit criteria for cron, workflow orchestration, and replay
Cron is the rejected primary design. One run can last at most 900 seconds and calls only a public http_url; paused schedules do not backfill missed triggers, firing can vary by seconds, and run-history output retains only its first 4 KB. Those limits are reasonable for a small producer that enqueues a daily rights-expiry scan and returns. They are a poor boundary for a process that waits on many customer endpoints while retry delays consume its runtime.
Use cron to enqueue. Stop there.
Push queue subscriptions also require a public HTTPS destination, so pull consumption is the suitable shape when workers must remain private. A delivery delayed beyond seven days belongs in authoritative application state and should be enqueued later by a trigger. This keeps scheduling state inspectable instead of asking an expiring queue message to act like a calendar.
The same rejection test applies in the other direction. Stick with Temporal or Airflow when the media process requires durable multi-step coordination or fan-out joins. Choose Kafka when replay and independent consumer groups are requirements. A simple queue has no native DAG orchestration, join primitive, debounce or throttle, topic-style one-to-many delivery, or Kafka-style replay; simulating a topic with N queues creates N delivery streams to own. Those are capability boundaries, not blemishes, and crossing one should reopen the architecture decision.
Top comments (0)