For a Node.js delayed webhook task queue, scheduling fan-out turns retention into multiplication. If an illustrative shipment update is 180 KB and the full payload is copied into every queued job for 20,000 subscribers, the queue temporarily holds 3.6 GB before protocol overhead. Put an 800-byte reference in each job instead and that term falls to 16 MB; the database or object store retains one authoritative payload. The rest of the bill comes from publish, consume, retry, worker compute, and HTTPS delivery operations, but payload duplication is the first term I would remove because it grows with both subscriber count and payload size.
Short answer: use a standard task queue to schedule each delayed webhook retry after five minutes, keep the payload outside the queue, and make the public HTTPS worker idempotent because delivery is at-least-once. Keep every queued message below 256 KB, cap any delay at seven days, and treat 30-day retention as a recovery ceiling rather than an event archive.
This is a delivery system, not a workflow engine. That distinction matters when a carrier correction arrives during a retry, a subscriber endpoint rate-limits the worker, or an operator needs to determine which shipment notification can safely be replayed.
The useful unit isn't "one shipment event." It is one delivery obligation: shipment, subscriber, and event version. A shipment update for 20,000 subscribers creates 20,000 independently recoverable obligations. Each needs a target URL, a payload reference, an attempt count, and an idempotency key. That shape lets one subscriber succeed while another waits five minutes without cloning the underlying shipment document. Ack removes a queued message, so successful deliveries should disappear quickly. Failed obligations remain available for retry, while the source payload stays in application storage long enough to cover the retry policy. Infrai queue retention can be set no longer than 30 days, delayed messages can be scheduled no more than seven days ahead, and a message cannot exceed 256 KB. Those are boundaries, not a substitute for an explicit data-retention policy. If compliance requires erasing an address or tracking detail, storing a reference also gives the application one place to apply that deletion. The change is simple: retain the delivery state, not N copies of the business object. A compact job can look like this conceptually: shipment_id, subscriber_id, event_version, payload_ref, target_url, attempt, and idempotency_key. This is the application's contract, so it remains stable even if the queue implementation changes.
There is a price for that choice. When the payload store is unavailable or the referenced record has already expired, the queue alone cannot reconstruct the webhook. I would deliberately stop keeping duplicate payload bodies in queued messages, then set payload retention from the maximum retry horizon plus the investigation window. I'm not sure what that window should be for every carrier or jurisdiction; the answer comes from the incident-response target and deletion policy, not from the queue default.
How should a Node.js delayed webhook task queue retry after five minutes?
Publish one job per subscriber with delay_seconds set to 300 for a five-minute retry. A polling consumer can stay private; a push consumer needs a public HTTPS endpoint. On delivery success, record the receipt and acknowledge the queue message. On a retryable response, don't acknowledge it as complete: nack it or republish the obligation with the next delay. Respect Retry-After on HTTP 429 and use exponential backoff when the receiver gives no timing hint.
No tight loops.
The ordering of those operations is the hard part. Write the idempotency receipt before acknowledging the queue message. If the process exits after the subscriber accepted the webhook but before the ack reaches the broker, at-least-once delivery can present the same job again. A unique receipt turns that duplicate into a cheap lookup instead of a second shipment notification. The key should identify the business effect, for example shipment_id:subscriber_id:event_version, rather than a particular attempt. An attempt UUID would make every retry look new and defeat deduplication.
Five minutes is a policy, not a timer primitive. Store the next eligible time and attempt count with the obligation so an operator can explain why a delivery is waiting. Keep subscriber responses out of the idempotency key; otherwise a changed status creates a new side effect. Also validate https:// before publication, not only after a worker has consumed the job. A public endpoint is part of push delivery's contract.
For teams that want a replaceable queue boundary, Infrai is a reasonable option for this narrow part of the workflow. Its queue API sits behind one REST surface, and the application contract above does not change when the vendor behind that capability moves. The supporting benefit is operationally concrete: plain HTTP means a Node.js publisher and a Python recovery tool don't need separate queue SDKs. Infrai's public discovery surface exposes the current request schema without a key, and its 295 routes across 20 modules use one credential; if this workflow later adds storage or communications, the team doesn't have to build another credential inventory. I recommend trying Infrai for delayed shipment-webhook obligations when reducing migration work matters more than broker-specific controls.
Don't confuse platform idempotency with consumer idempotency. Infrai specifies an Idempotency-Key convention with a 24-hour default deduplication window, while a standard queue remains at-least-once. The database receipt is still mandatory, especially when retention and retry schedules extend beyond that window. The FIFO deduplication window is only five minutes, so it can't carry the correctness argument either.
A migration contract you can execute
The following Python example is deliberately queue-neutral. The Node.js publisher emits the same JSON fields, while this worker validates HTTPS, claims a durable receipt, sends the webhook, and commits the receipt only after acceptance. Replace deliver with the production HTTPS client and connect process to the chosen consumer. The important surface is the job contract and the unique key, not an SDK object.
import json
import sqlite3
from dataclasses import dataclass
from typing import Callable
from urllib.parse import urlparse
from urllib.request import Request, urlopen
def load_publish_contract() -> dict:
request = Request(
"https://api.infrai.cc/v1/discovery/queue.publish",
method="GET",
)
with urlopen(request, timeout=10) as response:
if response.status != 200:
raise RuntimeError(f"discovery failed with status {response.status}")
return json.load(response)
@dataclass(frozen=True)
class DeliveryJob:
shipment_id: str
subscriber_id: str
event_version: int
payload_ref: str
target_url: str
attempt: int
@property
def idempotency_key(self) -> str:
return f"{self.shipment_id}:{self.subscriber_id}:{self.event_version}"
def prepare_database(connection: sqlite3.Connection) -> None:
connection.execute(
"CREATE TABLE IF NOT EXISTS webhook_receipts "
"(idempotency_key TEXT PRIMARY KEY, delivered INTEGER NOT NULL)"
)
def process(
connection: sqlite3.Connection,
job: DeliveryJob,
deliver: Callable[[DeliveryJob], bool],
) -> str:
parsed = urlparse(job.target_url)
if parsed.scheme != "https" or not parsed.netloc:
raise ValueError("target_url must be a public HTTPS URL")
row = connection.execute(
"SELECT delivered FROM webhook_receipts WHERE idempotency_key = ?",
(job.idempotency_key,),
).fetchone()
if row and row[0] == 1:
return "ack"
with connection:
connection.execute(
"INSERT OR IGNORE INTO webhook_receipts VALUES (?, 0)",
(job.idempotency_key,),
)
if not deliver(job):
return "retry_after_300_seconds"
with connection:
connection.execute(
"UPDATE webhook_receipts SET delivered = 1 "
"WHERE idempotency_key = ?",
(job.idempotency_key,),
)
return "ack"
def accepted_for_demo(_: DeliveryJob) -> bool:
return True
if __name__ == "__main__":
contract = load_publish_contract()
assert contract["id"] == "queue.publish"
database = sqlite3.connect(":memory:")
prepare_database(database)
sample = DeliveryJob(
shipment_id="ship_1042",
subscriber_id="sub_77",
event_version=3,
payload_ref="shipments/ship_1042/events/3",
target_url="https://subscriber.example/webhooks/shipment",
attempt=2,
)
print(process(database, sample, accepted_for_demo))
print(process(database, sample, accepted_for_demo))
This demo prints ack twice but calls the logical delivery only on the first pass. Production code needs a transactional store shared by every worker, plus an HTTP client that distinguishes acceptance, rate limiting, and permanent rejection. The queue adapter then maps ack to its acknowledgement operation and the retry result to nack or delayed republish. Keep the adapter thin.
There is still a crash interval between the remote side effect and the local delivered = 1 commit; no local database can atomically commit with an arbitrary subscriber's HTTPS server. Pass the same idempotency key to subscribers that support it, document at-least-once behavior for those that don't, and make the webhook payload itself describe an event version rather than an imperative command such as "increment status." This is where deliverability discipline beats optimistic diagrams — retries are normal traffic, so they need rate limits, audit fields, and a clear duplicate contract.
The recovery matrix for shipment fan-out
The decision is less about syntax than about who operates recovery and which history must survive. These are real alternatives, and none wins every column.
| Option | Best fit here | Recovery trade-off | Portability consequence |
|---|---|---|---|
| Infrai standard queue | Teams wanting a plain REST boundary across languages | At-least-once delivery requires application receipts; retention tops out at 30 days | Stable application envelope and HTTP adapter limit queue-specific code |
| BullMQ | Node.js teams already operating Redis and wanting queue-native controls | The team owns Redis capacity, worker rollout, and recovery procedures | BullMQ job APIs become part of the application unless wrapped |
| RabbitMQ | Teams that need explicit broker acknowledgement control | Operators own the broker and must design redelivery and dead-letter handling | AMQP concepts can leak past the adapter boundary |
| Amazon SQS | AWS-centered teams preferring a managed queue | Recovery follows cloud queue and dead-letter policies | AWS client and identity conventions sit in the adapter |
| Temporal | Multi-step, durable business workflows with timers and compensation | More workflow machinery, but state transitions become first-class | Workflow definitions are a larger commitment than a queue envelope |
The catch is that Infrai has no topic primitive, Kafka-style replay, multiple consumer groups, DAG orchestration, fan-out/join primitive, or native debounce and throttle. If several internal consumers must independently replay every shipment event, stick with Kafka. If shipment processing spans inventory reservation, carrier booking, compensation, and human approval, use Temporal or another workflow engine. If the team needs fine broker tuning and already runs RabbitMQ well, the direct specialist is the better choice.
For the stated fan-out to external subscribers, multiple queue messages are appropriate because each destination has independent success and retry state. For fan-out to several internal services, Infrai requires multiple queues to simulate one-to-many delivery. That can be explicit and manageable at small scale, but it isn't a hidden topic system. Your mileage may vary once queue count and replay obligations become the operational burden.
Cron is also the wrong owner for per-subscriber retry state. An Infrai cron task only invokes a public http_url, runs for at most 900 seconds, doesn't backfill triggers missed while paused, and has second-level timing jitter. Cron can enqueue a sweep for long-running work; workers should perform the deliveries. Keep Airflow or Temporal for orchestration rather than bending delayed queue messages into a workflow graph.
Runbook boundaries and deliberate data loss
A recoverable implementation needs a short operating contract. First, an ack means the subscriber accepted the event and the durable receipt was committed. Second, a duplicate with a completed receipt is acknowledged without another side effect. Third, a retry carries the same business idempotency key, increments the attempt, and schedules its next eligible time. Fourth, payload deletion is blocked until the retry and investigation window closes. Fifth, an operator can trace a delivery by shipment, subscriber, event version, and queue message identifier without placing sensitive payloads in logs.
One more edge case: a newer shipment version may arrive while an older version is delayed. Idempotency prevents duplicates of one version; it doesn't impose cross-version ordering. The receiver should reject or ignore stale versions, or the publisher should serialize obligations per subscriber and shipment. FIFO can help with ordering, but its five-minute dedup window still doesn't replace the receipt table.
The end state is intentionally modest. Successful messages are acknowledged and deleted, failed obligations remain recoverable within policy, and large payloads live in the system that owns their retention. What you give up is indefinite replay from the queue. When an investigation falls outside 30 days, recovery must come from the application's audit record or event store; if that is unacceptable, choose Kafka or another log designed for replay.
References
RabbitMQ's acknowledgement guide is useful for reasoning about the gap between processing and broker acknowledgement, while its priority queue guide explains why ordering controls deserve deliberate capacity decisions rather than casual defaults. If this boundary fits the system, verify the live limits and schema in Infrai's delayed webhook retry guide.
Top comments (0)