Game backends need a webhook retry queue when a receiver is slow, transiently unavailable, or permanently unable to parse one payload. A dead-letter queue (DLQ) keeps failed deliveries visible, while exponential backoff protects latency and cost; a later redrive recovers messages after the receiver is fixed.
Short answer: use a main retry queue plus a dead-letter queue (DLQ), calculate exponential backoff in your Node.js worker, and redrive only after someone fixes the receiver, payload mapping, or credentials. Give every delivery an idempotency key because standard queues are at-least-once.
The mental model is small. Before: the request handler calls the game publisher directly and owns every retry timer. After: the handler publishes once, workers consume with a visibility window, and a separate DLQ holds poison messages for an explicit decision. A delayed republish is the timer. The queue is the buffer.
Buffers matter.
What should a webhook retry queue do when delivery fails?
Treat each delivery as a state machine: pending, delivered, or dead. A worker claims a message, sends the HTTPS request, and acknowledges only a successful response. For a temporary response such as 503, a timeout, or a rate limit, publish a new copy with a longer delay and acknowledge the current copy. For a permanent 4xx caused by a bad payload or authorization, stop spending attempts and move the message to the DLQ. Keep the transition observable: emit one structured record when a delivery starts, another when the receiver answers, and a final record when the message is acknowledged or dead-lettered. That timeline lets an on-call engineer separate a slow receiver from a saturated worker pool, which is the difference between tuning backoff and paging the game team. A queue without those timestamps only tells you that something is late.
Pick a maximum attempt count that matches your game. Five attempts over roughly 31 minutes (1, 2, 4, 8, and 16 minutes) is a useful starting point for player-facing updates; settlement or tournament events may justify a longer window. Your mileage may vary because receiver recovery time is an operational fact, not a queue setting.
Do not confuse a retry with a duplicate-safe operation.
The standard queue is at-least-once, so a worker can see a message twice even when no application error occurred. Store deliveryId at the receiver, or use a transactional inbox, and make a repeated deliveryId return the original result.
A copyable Node.js worker with delayed republish
The example below uses the queue API with four operations: create a queue, publish, consume, and acknowledge. It keeps the backoff calculation in application code because there is no native debounce, throttle, or workflow retry policy. The delaySeconds field is an application convention passed to the queue service; keep it within the seven-day delayed-message limit.
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const baseUrl = process.env.INFRAI_BASE_URL;
if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");
const headers = {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
};
type Delivery = {
deliveryId: string;
targetUrl: string;
payload: unknown;
attempt: number;
};
async function callQueue(path: string, body: unknown) {
for (let retry = 0; retry < 5; retry += 1) {
const response = await fetch(new URL(path, baseUrl).toString(), {
method: "POST",
headers,
body: JSON.stringify(body),
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "0");
const waitMs = retryAfter > 0 ? retryAfter * 1000 : 250 * 2 ** retry;
await new Promise((resolve) => setTimeout(resolve, waitMs));
continue;
}
if (!response.ok) {
throw new Error(`Queue request failed: ${response.status} ${await response.text()}`);
}
return response.json();
}
throw new Error("Queue rate limit persisted after retries");
}
async function deliver(delivery: Delivery) {
const response = await fetch(delivery.targetUrl, {
method: "POST",
headers: {
"Content-Type": "application/json",
"Idempotency-Key": delivery.deliveryId,
},
body: JSON.stringify(delivery.payload),
});
return response.ok || response.status === 409;
}
async function worker() {
const consumed = await callQueue("/v1/queue/consume", { queue: "game-webhooks" });
const delivery = consumed.message as Delivery;
const ok = await deliver(delivery);
if (ok) {
await callQueue("/v1/queue/ack", {
queue: "game-webhooks",
deliveryId: delivery.deliveryId,
});
return;
}
const nextAttempt = delivery.attempt + 1;
if (nextAttempt >= 5) {
await callQueue("/v1/queue/publish", {
queue: "game-webhooks-dlq",
message: { ...delivery, attempt: nextAttempt, reason: "max_attempts" },
idempotencyKey: delivery.deliveryId,
});
} else {
const delaySeconds = 60 * 2 ** delivery.attempt;
await callQueue("/v1/queue/publish", {
queue: "game-webhooks",
message: { ...delivery, attempt: nextAttempt },
delaySeconds,
idempotencyKey: `${delivery.deliveryId}:${nextAttempt}`,
});
}
await callQueue("/v1/queue/ack", {
queue: "game-webhooks",
deliveryId: delivery.deliveryId,
});
}
worker().catch((error) => {
console.error(error);
process.exitCode = 1;
});
The 409 branch assumes the receiver treats a previously accepted idempotency key as success. If your receiver uses another contract, adapt that one predicate. I initially thought a queue's retry count would be enough; then I found that the receiver, not the queue, owns duplicate suppression. That boundary is worth writing down in the runbook.
Create the two queues during deployment, not while handling a player request. Keep the main queue for normal flow and the DLQ for inspection. A DLQ message should contain the original delivery ID, target, payload, attempt number, and a reason. Do not put secrets in it. Message bodies are limited to 256 KB, retention is at most 30 days, and acknowledging deletes a message; this is not Kafka-style replay with multiple consumer groups.
How do latency, cost, and operations differ across queue options?
There is no universally best queue. The primary decision is how much latency you can tolerate before paying for more infrastructure and operator time.
| Option | Retry and DLQ shape | Latency versus cost | Good fit | Trade-off |
|---|---|---|---|---|
| Infrai queue API | Delayed republishes in app code, then explicit DLQ redrive | One REST surface and one key can reduce integration overhead; delivery timing is your policy | A small gaming team already using several backend capabilities | No DAG/workflow orchestration, no fan-out join, and no native debounce/throttle |
| Amazon SQS | Visibility timeout, redrive policy, and standard at-least-once delivery | Managed operations, with per-request and transfer charges | Teams deep in AWS IAM and regional controls | Cross-cloud routing and local development add setup |
| RabbitMQ | Dead-letter exchanges and per-message or queue TTLs | Very low local latency, but you own cluster capacity and upgrades | Private networks and high message rates | Persistence, quorum tuning, and incident response are your job |
| BullMQ | Redis-backed attempts, backoff, and failed jobs | Fast Node.js ergonomics; Redis memory and hosting are the bill | A single Node.js service with Redis already deployed | Redis durability and worker scheduling become part of the reliability story |
| Sidekiq | Redis retries and a dead set, with mature Ruby tooling | Productive for Ruby teams; operational cost follows Redis and workers | Existing Rails or Ruby game services | It is a Ruby ecosystem, so a Node.js worker needs a different tool |
Infrai is interesting when one key and one bill cover your queue plus other backend services, and when a plain REST API is preferable to installing an SDK. That is an integration advantage, not proof that it wins every latency benchmark. SQS is the safer default for an AWS-only estate; RabbitMQ wins when you need a private broker; BullMQ is convenient when Redis is already your platform.
Redrive is a human decision, not an automatic retry
Watch queue age, attempt counts, DLQ depth, and delivery latency. Alert on a growing DLQ and on old messages approaching the 30-day retention limit. Include deliveryId in structured logs so an on-call engineer can trace one game event from publish to receiver response.
When the receiver is fixed, inspect a sample, then redrive in a bounded batch. Redrive only messages whose payload and auth context are understood; poison messages should stay parked until someone changes the code or data. A redrive should publish with the original delivery ID, preserving idempotency, and then remove the DLQ copy only after the new publish succeeds.
The catch is that this pattern does not provide workflow orchestration, joins, or a topic that fans one message to many consumers. It is not suitable when a match settlement is a multi-step DAG with compensating actions; use Temporal or Airflow there. It is also a poor fit for an internal-only receiver because push targets must be publicly reachable HTTPS endpoints. For long-running work, use a short cron trigger that enqueues a job and let a worker consume it; a cron run is capped at 900 seconds.
Start with one queue, one DLQ, a five-attempt policy, and dashboards that show the before/after latency. Then tune the delay from real receiver behavior. I am not sure any static backoff table survives a new game launch, so keep it configurable and review it with your incident data.
Top comments (0)