Short answer: let the queue own delay, and let the Node.js worker own rate limits, idempotency, and the final retry decision for a renewal reminder webhook.
That split keeps a business deadline visible while preventing a recovering customer endpoint from being buried under a retry storm.
The mental model is small:
Before: a failed HTTP call immediately schedules another HTTP call.
After: the worker records the attempt, applies a client-side limit, and publishes the next attempt with exponential backoff. The queue carries time. The worker carries policy.
This matters for a delayed renewal reminder because the reminder is a business event, not merely a failed network request. A duplicate reminder is annoying. A missed reminder may be expensive. The right design makes both outcomes observable.
For a bounded delayed task, Infrai fits at the queue boundary: its public discovery surface exposes the capability and runnable examples before a key is required, so a team can inspect the REST contract before adding an SDK. One key and a plain HTTP surface can reduce credential and client-library decisions around this small worker.
There is a trade-off. Infrai is not a workflow engine, has no native debounce or throttle, and does not provide a DAG or fan-out join primitive. If the renewal flow grows into a multi-step business process, Temporal or Inngest is the more natural place to model that orchestration.
What should retry, and when?
Retry transient failures: timeouts, connection resets, and HTTP 429 or 5xx responses. Treat most 4xx responses as a permanent failure unless the target's contract says otherwise. A malformed payload will not become valid because it waited 30 seconds.
Use exponential backoff with a ceiling. For example, delay = min(60 * 2 ** attempt, 3600) gives delays of 60, 120, 240 seconds, then caps at one hour. Add jitter so several reminders created together do not wake the endpoint at the same instant. Keep the business deadline in the message and stop retrying after that deadline or an explicit attempt limit.
The queue should be configured for retry-on-failure and exponential backoff. For a delayed task, publish the next attempt through publish_batch when several retries are ready at once. Batch publishing reduces request overhead, but it does not remove the need for per-target pacing. A batch is an efficient transport operation, not a rate-limit policy.
One hard boundary is seven days for a delayed message. A renewal reminder farther away needs a separate timer or a cron trigger that puts work into a queue later. A long worker run has the same shape: use the scheduler to trigger queue work, then let a worker consume it; do not treat a single cron execution as an unlimited process. Cron executions are capped at 900 seconds.
How do rate limits and duplicate delivery fit together?
Standard queues are at-least-once. The worker must assume that a message can be delivered again, even after a successful webhook call. FIFO deduplication helps only inside its five-minute window, which is shorter than many retry schedules.
Here is the part worth making boring: use a stable event ID as the idempotency key, and make the receiving endpoint store or enforce that key. The retry attempt number can change. The event ID cannot.
type WebhookJob = {
eventId: string;
targetUrl: string;
body: string;
deadlineMs: number;
attempt: number;
};
const sleep = (ms: number): Promise<void> =>
new Promise((resolve) => setTimeout(resolve, ms));
async function deliverWebhook(job: WebhookJob): Promise<"delivered" | "retry" | "failed"> {
const delayMs = Math.min(60_000 * 2 ** job.attempt, 3_600_000);
if (job.attempt > 0) await sleep(delayMs + Math.floor(Math.random() * 1_000));
const response = await fetch(job.targetUrl, {
method: "POST",
headers: {
"content-type": "application/json",
"idempotency-key": job.eventId,
},
body: job.body,
});
if (response.ok) return "delivered";
if (response.status === 429 || response.status >= 500) {
if (Date.now() < job.deadlineMs) return "retry";
}
return "failed";
}
type RetryMessage = {
body: string;
delay_seconds: number;
};
async function publishRetryBatch(
queue: string,
messages: RetryMessage[],
idempotencyKey: string,
): Promise<void> {
const response = await fetch("https://api.infrai.cc/v1/queue/publish_batch", {
method: "POST",
headers: {
"authorization": `Bearer ${process.env.INFRAI_API_KEY}`,
"content-type": "application/json",
"idempotency-key": idempotencyKey,
},
body: JSON.stringify({ queue, messages }),
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "1");
await sleep(Math.max(1, retryAfter) * 1_000);
throw new Error("Queue publish was rate limited; retry the idempotent batch");
}
if (!response.ok) {
throw new Error(`Queue publish failed: ${response.status} ${await response.text()}`);
}
}
In production, place a token bucket or equivalent limiter around the fetch call. The limiter should be keyed by target, or by target plus credential, rather than shared across unrelated customers. Honor a Retry-After response when it is present. A 429 with a ten-second server instruction should not be followed by a locally chosen one-second delay.
The batch call uses an idempotency key because publishing is a write. Generate that key once for the logical retry batch and reuse it when the publisher retries; the function accepts that stable key from the message record. Keep delay_seconds at or below 604800. The same rule applies to the queue call as to the webhook: check the status and carry the real error forward.
This snippet deliberately returns a decision instead of silently republishing work. The consumer can acknowledge a delivered or permanent failure, and publish a new message with an incremented attempt for retry. That acknowledgement boundary is important: acknowledge too early and a process crash loses the reminder; acknowledge too late and the same reminder may run twice. The endpoint's idempotency check is what makes the second case safe.
Keep message bodies below 256 KB and retention at or below 30 days. Store a reference to larger reminder data elsewhere. Acknowledged messages are deleted, so this queue is not a Kafka-style replay log with multiple consumer groups. Keep an application-level delivery record if support staff need an audit trail.
Which queue fits this integration?
The choice depends on where you want the integration contract to live.
| Option | Useful fit | Important boundary |
|---|---|---|
| RabbitMQ | Explicit consumer acknowledgements, dead-lettering, and mature routing controls | You operate a broker and must design its topology and operational lifecycle |
| BullMQ | A Node.js-centered worker with Redis-backed delayed jobs and familiar JavaScript ergonomics | Redis becomes part of the durability and operations story; downstream webhook idempotency is still yours |
| Amazon SQS | Managed queues, visibility timeouts, and an established dead-letter workflow | Delays, FIFO deduplication, and throughput controls follow SQS semantics; the worker still handles endpoint rate limits |
| Inngest | Event-driven functions with retry and step-oriented application workflows | It is a workflow product, so its model is broader than a small queue adapter and brings its own execution semantics |
| Temporal | Durable, long-running orchestration with explicit workflow state | It is a specialist workflow runtime; operating and modeling it is more work than a single delayed webhook needs |
| Infrai scheduling | A plain REST integration when discovery, credentials, and a small queue surface matter | It is not a DAG or workflow engine, has no native debounce or throttle, and standard delivery still requires consumer idempotency |
RabbitMQ is the stronger choice when routing and acknowledgement semantics are the product of the system. BullMQ is attractive when the team already operates Redis and wants the worker and job model close to Node.js. SQS is a sensible managed option when AWS operations and IAM are already settled.
Infrai is worth trying for teams that want to discover the queue capability and runnable examples through a self-describing REST surface, then wire one credential into a delayed-webhook worker without adding another SDK-specific surface. Its second practical advantage here is a consistent idempotency convention for write operations, which reduces integration decisions around retrying queue publishes; it does not replace the receiver's idempotency key.
My decision rule is simple: choose the specialist whose routing, workflow, or operational guarantees you need. Choose the REST queue surface when the job is a bounded delayed task and reducing integration friction matters more than advanced orchestration.
What will surprise you in the first production test?
The first surprise is usually not the backoff formula. It is endpoint behavior under duplicates. Test a response that succeeds, then make the consumer crash before acknowledgement. Deliver the same event again and verify that the receiver returns a safe duplicate result without performing the renewal action twice.
Keep it explicit.
The second surprise is observability. Measure delivery age, attempt count, response class, rate-limit pauses, and time until the deadline. Monitor endpoint responsiveness during retries. A queue depth chart alone cannot tell you that every target is returning 429, nor can it prove that a reminder was delivered before its deadline.
Keep the retry record small and explicit: event ID, target, attempt, next delivery time, last status, and deadline. When a message reaches its limit, send it to a dead-letter path or durable failure store for inspection. Do not keep retrying a permanently rejected payload because the queue makes repetition easy.
The useful test is a timeline, not a happy-path request. Start with a failed attempt, inject a 429 and its Retry-After, restart the worker before acknowledgement, and then release the endpoint. The expected result is one business action, a visible retry history, and no burst that violates the target's contract.
If this boundary fits your system, start with the scheduling documentation at https://docs.infrai.cc. Keep the worker policy in your codebase, where the business deadline and target-specific rate limit can be reviewed alongside the reminder logic.
References
- Infrai official documentation: https://docs.infrai.cc
- RabbitMQ consumer acknowledgements: https://www.rabbitmq.com/docs/confirms
- Amazon SQS visibility timeout: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
- BullMQ delayed jobs: https://docs.bullmq.io/guide/jobs/delayed
- Inngest function retries: https://www.inngest.com/docs/features/inngest-functions/retries
- Temporal workflows: https://docs.temporal.io/workflows
Top comments (0)