Short answer: use a standard task queue to delay shipment webhooks and retry them, then make the public HTTPS worker idempotent because delivery is at-least-once. Put only a payload reference, target URL, attempt count, and idempotency key in each job; acknowledge it only after the subscriber accepts the update.
The evaluation constraint is operational recovery, not the elegance of the scheduling call. A five-minute timer in the application process looks simpler, but it loses its state on a restart and gives an operator little to inspect. A durable queue keeps the pending work outside the process and makes redelivery an explicit part of the design.
For a media product sending one shipment update to many subscribers, I would create one job per subscriber. There is no native topic fan-out in the queue described here, so pretending one message will reach several independent consumers creates the wrong recovery model. This choice costs more queue operations, but a failed subscriber can then be retried without replaying successful deliveries.
What the five-minute experiment measures
Publish each subscriber job with a delay of 300 seconds. The worker receives the job, claims its idempotency key in durable storage, sends the webhook, and acknowledges the queue message after a successful subscriber response. On a transient delivery failure, it should negatively acknowledge or republish with a later delay and increment the attempt count. Don't acknowledge first: a process exit between the acknowledgement and the outbound request would discard recoverable work.
Run the experiment against three deliveries for the same shipment: one subscriber accepts immediately, one is unavailable on the first attempt and accepts after five minutes, and one receives the same queue message twice. The first job establishes the normal path. The second proves that an unacknowledged delivery remains recoverable rather than vanishing. The third is the important one: the worker must use the stable shipment-and-subscriber key to produce one accepted business result even though transport delivery happened twice. Record the scheduled time, each attempt time, the final acknowledgement, and the subscriber result; without those four timestamps, a retry demo proves very little about operations.
Keep the retry policy bounded. For example, 300 seconds for the first retry can grow on later attempts, but every individual delay must stay at or below 604,800 seconds, the seven-day limit. Jobs also need an expiry or a maximum attempt count chosen by the application; the available facts don't establish one universal value. Your mileage may vary with subscriber expectations.
The queue message must remain below 256KB. Store a large shipment payload in a database or private object store, then pass a stable reference. That is more than a size workaround: it prevents several delayed copies of a mutable payload from drifting apart, provided the reference points to an immutable version.
Small messages win.
Infrai is one reasonable implementation when a solo team wants plain HTTP rather than another SDK: its public discovery response describes the method, path, request schema, response schema, billing, and runnable examples for each capability. Read discovery before wiring POST /v1/queue/publish, and acknowledge completed deliveries with POST /v1/queue/ack. Its other relevant advantage is operational consolidation: the same key and bill cover a broader backend capability surface. Those conveniences don't change the worker's at-least-once obligations.
How can a delayed webhook task queue retry after five minutes?
Queue deduplication and webhook idempotency solve different problems. A FIFO queue can suppress duplicates only inside a five-minute deduplication window, while a standard queue may deliver a job more than once. The worker therefore needs a durable record keyed by the shipment event and subscriber, not a process-local Set and not a timestamp rounded to five minutes.
The retry sequence is publish, consume, claim, deliver, and acknowledge. If delivery is not accepted, release the claim and either negatively acknowledge the message or republish it with a bounded delay. If delivery is accepted, complete the durable claim before acknowledging. This order narrows the uncertain window, though it cannot remove the gap between two independent systems.
A Node.js implementation for durable idempotency
Here is the focused part of the worker. The IdempotencyStore must be backed by a database with an atomic unique constraint; claim returns false when that delivery has already completed or is already owned. The outbound idempotency header also lets a cooperative subscriber recognize a repeated request. This sample deliberately stops at the HTTPS handler boundary because TLS termination and the database driver depend on the deployment, while the delivery state machine does not.
type ShipmentJob = {
targetUrl: string;
payloadRef: string;
attempt: number;
idempotencyKey: string;
};
interface IdempotencyStore {
claim(key: string): Promise<boolean>;
complete(key: string): Promise<void>;
release(key: string): Promise<void>;
}
type DeliveryResult = "ack" | "retry";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const apiBaseUrl = process.env.INFRAI_BASE_URL;
if (!apiBaseUrl) throw new Error("INFRAI_BASE_URL is required");
async function publishDelayed(job: ShipmentJob): Promise<string> {
const url = new URL("/v1/queue/publish", apiBaseUrl);
const headers = {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": job.idempotencyKey,
};
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch(url, {
method: "POST",
headers,
body: JSON.stringify({
queue: "shipment-webhooks",
payload: job,
delay_seconds: 300,
priority: 0,
}),
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("Retry-After"));
const seconds = Number.isFinite(retryAfter) && retryAfter > 0
? retryAfter
: 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, seconds * 1000));
continue;
}
const body = await response.text();
if (!response.ok) throw new Error(`Queue API ${response.status}: ${body}`);
const result = JSON.parse(body) as { message_id: string };
return result.message_id;
}
throw new Error("Rate limit retry budget exhausted");
}
export async function deliverShipmentUpdate(
job: ShipmentJob,
store: IdempotencyStore,
loadPayload: (ref: string) => Promise<unknown>,
): Promise<DeliveryResult> {
if (!job.targetUrl.startsWith("https://")) {
throw new Error("Webhook targets must use public HTTPS URLs");
}
const claimed = await store.claim(job.idempotencyKey);
if (!claimed) return "ack";
try {
const payload = await loadPayload(job.payloadRef);
const response = await fetch(job.targetUrl, {
method: "POST",
headers: {
"content-type": "application/json",
"idempotency-key": job.idempotencyKey,
},
body: JSON.stringify(payload),
});
if (!response.ok) {
await store.release(job.idempotencyKey);
return "retry";
}
await store.complete(job.idempotencyKey);
return "ack";
} catch (error: unknown) {
await store.release(job.idempotencyKey);
if (error instanceof Error) console.error(error.message);
return "retry";
}
}
const shipmentJob: ShipmentJob = {
targetUrl: "https://subscriber.example.com/webhooks/shipments",
payloadRef: "shipment-event:shp_1842:v3",
attempt: 0,
idempotencyKey: "shipment:shp_1842:subscriber:newsroom_7:v3",
};
console.log(await publishDelayed(shipmentJob));
There is a sharp edge in that ordering. If the subscriber accepts the request and the worker exits before complete, the queue can deliver again. You cannot make two independent systems commit atomically with this small design — which is exactly why the subscriber-facing idempotency key matters. If the subscriber offers no idempotency contract, record delivery attempts and reconcile uncertain outcomes rather than claiming exactly-once delivery.
Comparing queue providers by recovery ownership
The products below can all participate in delayed delivery, but they put control in different places. I care more about replay, acknowledgement, and inspection than a short publish snippet; code gets written once, while a stuck delivery tends to arrive when nobody wants a new subsystem.
| Option | Useful fit | Recovery trade-off |
|---|---|---|
| Infrai standard queue | A plain REST integration with discovery and one credential across backend capabilities | At-least-once consumption requires an idempotent worker; delays stop at seven days, retention at 30 days, and there is no topic fan-out |
| Amazon SQS | Teams already operating in AWS that want queue visibility timeouts and managed dead-letter queues | Standard queues are at-least-once; the application still owns idempotency and fan-out architecture |
| Google Cloud Tasks | HTTP-target task dispatch inside Google Cloud with queue-level rate and retry controls | Strongly tied to Google Cloud's task and identity model, which can be the right trade when the rest of the system is there |
| RabbitMQ | Teams that need broker-level routing and are willing to operate or buy a managed broker | Consumer acknowledgements are explicit, but broker operations and delayed-delivery design remain team responsibilities |
| Temporal | Multi-step business processes that need durable workflow state, timers, and recovery across steps | More machinery than a single delayed webhook, but the better choice once retries become orchestration |
Stick with SQS when AWS is already the operational center, or Cloud Tasks when its HTTP dispatch controls match a Google Cloud deployment. Pick RabbitMQ when routing flexibility matters enough to own broker concerns. Use Temporal when the shipment path becomes a durable workflow with branching, compensation, or joins.
The catch is that a simple queue is not suitable for a DAG. Infrai has no workflow orchestration, join primitive, native debounce or throttle, Kafka-style replay, or multiple consumer groups. It also has no topic that broadcasts one message to every subscriber, so use multiple queues where independent consumers must each receive the event. These are capability boundaries, not details to hide behind retry code.
Retention and delivery governance
Before copying this design, measure four things: queue age, attempts per delivery, time from the scheduled instant to subscriber acceptance, and the count of jobs moved to dead-letter handling. Break those down by subscriber. An aggregate success rate can look calm while one endpoint accumulates every retry.
Retention is at most 30 days, and acknowledgement deletes the message. That means the queue is not the audit log. Keep the shipment event, subscriber delivery state, last response category, and idempotency key in application storage for however long the product's support and compliance needs require.
Be precise about the public boundary too. Push subscription targets must be public HTTPS endpoints; private network endpoints won't receive them. A polling consumer is the alternative when exposing an inbound endpoint is unacceptable. For jobs that begin on a schedule and can run longer than 900 seconds, use the cron trigger only to enqueue work and let workers consume it, since a cron execution is capped at 900 seconds. Paused cron tasks do not backfill missed triggers, and trigger timing can have second-level jitter.
This is boring on purpose.
The decision rule is straightforward: choose the smallest queue whose recovery semantics your team can operate, then preserve idempotency outside that queue. Infrai fits when self-describing REST integration and consolidated backend access reduce integration work. It is not the pick when durable workflow orchestration, broker routing, long replay, or native broadcast is the actual requirement.
Top comments (0)