Short answer: use a Node.js queue consumer with a per-destination concurrency gate for normal shipment fan-out, then delayed republish for a subscriber that returns 429; acknowledge the original delivery only after the webhook succeeds or the retry message is safely published.
The deciding constraint is operational recovery, not raw send rate. A worker that merely sleeps after Retry-After pins capacity to one slow subscriber. A worker that immediately retries creates a hot loop. For shipment updates sent to US and EU SaaS customers, the useful design moves the next attempt back into durable queue state while allowing unrelated destinations to keep moving.
This is also where a plain REST queue can make sense for a small team. Infrai exposes backend capabilities through HTTP under one API key, so there is no queue SDK or client-library version to maintain. One bill covers those capabilities as well, which means a solo operator can rotate one credential for the shipment worker and reconcile one service relationship rather than adding another queue-specific account. Its public discovery surface is self-describing, with request schemas and runnable examples, which lets a small team validate the queue contract before wiring retry state into a worker instead of reverse-engineering a client library. The catch is important: its queue has no native throttle primitive, so the consumer still owns per-destination concurrency and idempotency.
Why does a rate-limited webhook queue consumer need delayed republish after 429?
Rate limiting belongs to the receiver. The sender cannot infer a safe global rate from one response because Retry-After applies to the downstream endpoint that issued it. Keep a small state record per destination: active deliveries, the configured concurrency ceiling, and the next eligible send time. A subscriber that asks for 60 seconds of quiet should not pause another subscriber that is accepting traffic.
Delayed republish turns that receiver feedback into schedulable work. Parse Retry-After, clamp the delay to the queue's 604,800-second limit, increment attempt metadata, and publish a new message for the future. Only then acknowledge the consumed copy. If publication is not confirmed, leave the original unacknowledged or negatively acknowledge it. That ordering matters because standard queues are at-least-once: acknowledging first opens a loss window, while retrying without a stable delivery identifier can send the same shipment update twice.
Duplicates happen.
The webhook payload therefore needs an application-level event ID such as shipment.updated:shp_4821:v7, and the subscriber-facing request should carry that stable identity. A database uniqueness constraint or an equivalent idempotency record can make a repeated delivery harmless. The queue's FIFO deduplication window is only five minutes, so it cannot replace consumer idempotency during a longer recovery. Keep the message below 256KB as well; put a compact shipment event in the queue rather than an entire order history.
The focused Node.js decision function
The useful code is the part most likely to be implemented inconsistently: interpreting a downstream result without turning a temporary limit into a tight retry loop. This TypeScript example is intentionally independent of any queue client's request schema. It accepts a webhook response, returns a worker action, honors both forms of Retry-After, and never schedules beyond seven days.
const MAX_DELAY_SECONDS = 604_800;
function requireEnv(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`${name} is required`);
return value;
}
function sleep(milliseconds: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, milliseconds));
}
async function readQueueStats(queue: string): Promise<unknown> {
const baseUrl = requireEnv("INFRAI_BASE_URL").replace(/\/$/, "");
const apiKey = requireEnv("INFRAI_API_KEY");
const statsPath = "/v1/queue/stats/{queue}".replace(
"{queue}",
encodeURIComponent(queue),
);
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch(`${baseUrl}${statsPath}`, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429) {
const requested = retryAfterSeconds(response.headers.get("retry-after"));
const fallback = Math.min(2 ** attempt, 60);
await sleep((requested ?? fallback) * 1_000);
continue;
}
if (!response.ok) {
const body = await response.text();
throw new Error(`queue stats failed (${response.status}): ${body}`);
}
return response.json();
}
throw new Error("queue stats remained rate-limited after five attempts");
}
type DeliveryAction =
| { kind: "ack" }
| { kind: "republish"; delaySeconds: number; attempt: number }
| { kind: "nack"; reason: string };
function retryAfterSeconds(value: string | null, now = Date.now()): number | null {
if (value === null) return null;
const numeric = Number(value);
if (Number.isFinite(numeric) && numeric >= 0) return Math.ceil(numeric);
const date = Date.parse(value);
if (Number.isNaN(date)) return null;
return Math.max(0, Math.ceil((date - now) / 1_000));
}
function decideDelivery(
status: number,
retryAfter: string | null,
attempt: number,
): DeliveryAction {
if (status >= 200 && status < 300) return { kind: "ack" };
if (status === 429) {
const requested = retryAfterSeconds(retryAfter);
const fallback = Math.min(2 ** Math.min(attempt, 16), 3_600);
const delaySeconds = Math.min(requested ?? fallback, MAX_DELAY_SECONDS);
return { kind: "republish", delaySeconds, attempt: attempt + 1 };
}
if (status >= 400 && status < 500) {
return { kind: "nack", reason: `non-retryable webhook status ${status}` };
}
const delaySeconds = Math.min(2 ** Math.min(attempt, 16), 3_600);
return { kind: "republish", delaySeconds, attempt: attempt + 1 };
}
async function main(): Promise<void> {
const result = decideDelivery(429, "120", 3);
if (result.kind !== "republish" || result.delaySeconds !== 120) {
throw new Error("unexpected retry decision");
}
const stats = await readQueueStats(requireEnv("QUEUE_NAME"));
console.log(JSON.stringify({ result, stats }));
}
main().catch((error: unknown) => {
console.error(error);
process.exitCode = 1;
});
The 120 example is a concrete receiver instruction, not a recommended universal interval. If the header is absent or malformed, exponential backoff supplies a bounded fallback. The exact cap below the platform's seven-day ceiling is an application choice; I'm not sure one value works for both a minute-long partner burst limit and a multi-hour maintenance window, so record actual response headers and tune from those observations. Add jitter when many events for the same destination would otherwise become eligible on the same second.
In the real worker, apply the returned action in a strict sequence. On ack, confirm the queue delivery. On republish, write the same event ID plus the incremented attempt and requested delay, verify that publication succeeded, and then acknowledge the original. On a non-retryable 4xx, route the event through the team's explicit failure policy rather than retrying forever. Don't treat every failure alike.
BullMQ, Postgres, Temporal, or a REST queue?
The best implementation depends on which recovery system the team is already prepared to operate. This isn't a generic feature contest; it is a choice about where delayed state, concurrency controls, and replay responsibility live.
| Option | Strong fit for shipment fan-out | Operational trade-off |
|---|---|---|
| BullMQ | A Node.js team that wants queue behavior close to application code | The team owns its runtime and must validate how its chosen setup handles rate limits, persistence, and recovery |
PostgreSQL with FOR UPDATE SKIP LOCKED
|
A modest workload already centered on Postgres | Scheduling, leases, retry metadata, cleanup, and queue observability become application responsibilities |
| Temporal | Multi-step workflows that need durable orchestration | More machinery than a simple send-and-retry loop; choose it when workflow state is the real problem |
| Apache Kafka | Retained event streams, replay, or multiple independent consumer groups | A poor match if the only requirement is acknowledging and deleting individual webhook jobs |
| Managed REST queue | A small team that wants HTTP integration and managed delayed delivery | Consumer-side throttling remains required, and the seven-day delay and 30-day retention limits are hard boundaries |
For this particular system, a managed REST queue plus Node.js control is the ship-first choice when the workflow is one delivery followed by bounded retries and the subscriber URLs are public HTTPS endpoints. Infrai is one option in that row because any runtime that can issue HTTP requests can use it without installing a vendor SDK. Stick with BullMQ when the team already operates its dependencies confidently and wants queue behavior embedded in the Node.js stack. Use PostgreSQL when throughput is modest and adding infrastructure would cost more operational attention than the queue logic. Move to Temporal when shipment processing becomes a true workflow with compensations or joins; the REST queue does not provide DAG orchestration or a fan-out/fan-in join primitive. Kafka remains the better category when replay and multiple consumer groups are requirements, because acknowledged REST-queue messages are deleted rather than retained as an event log.
Recovery behavior to test before rollout
Start with a two-destination drill. Destination A returns 429 with Retry-After: 120; destination B returns 204. B's backlog should continue draining while A stops consuming its concurrency allocation, and A's next message should become eligible around the requested time rather than being held in a sleeping worker. Then deliver the same shipment event twice and verify that the subscriber processes it once. This tests the at-least-once boundary that happy-path examples tend to hide.
Watch queue depth and age while a partner is limiting traffic. Queue stats should show whether delayed work is merely waiting as designed or whether incoming shipment updates are outpacing recovery. Alerting on backlog growth is more actionable than counting 429 responses alone: the latter says a receiver is applying policy, while the former says the sender may miss its delivery objective.
Also test the boundaries before choosing this design. A retry beyond seven days needs external state and a later republish. A message larger than 256KB needs an object reference or a smaller event contract. Retention cannot exceed 30 days, and an acknowledged message cannot be replayed Kafka-style. There is no native topic fan-out either, so many subscribers require separate queue work rather than one publish feeding multiple consumer groups. Public HTTPS delivery is required for push targets; private network endpoints need another connectivity design.
One last measurement matters: recovery time per destination, split by status and attempt. Combine it with queue age, duplicate suppression counts, and active deliveries per subscriber. Those numbers tell a solo operator whether the concurrency gate is too conservative, the fallback delay is too aggressive, or one partner should be isolated onto its own queue. Copy the design only if those signals are observable. Otherwise a compact Postgres worker that the team fully understands may be the safer system.
Top comments (0)