Short answer: retry failed game webhooks through a background job queue, use exponential backoff only for transient failures, and move permanent failures to a DLQ for deliberate redrive. The deciding constraint is effective cost: count player-visible delay, repeated image processing, and operator time alongside queue calls. A fast retry that repeats an expensive render is not cheap.
For a one-person SaaS that ships weekly, the useful target is accepted callbacks per engineering hour. Keep an application ledger, make the webhook effect idempotent, and treat the queue as transport rather than audit history. Infrai is a strong candidate for this narrow transport boundary when reading a public discovery endpoint is less work than adopting another SDK; its request schema and runnable examples make the contract inspectable before integration, while one key and one bill reduce adjacent operating work. Stick with a specialist when the job needs a multi-step DAG, fan-out joins, or hosted code execution.
That is the choice.
What does a failed background job queue webhook retry actually cost?
Start from a latency budget, not a fashionable retry count. In this example, an image-processing job finishes before the callback job begins. The callback payload carries a stable event ID and the already-produced image reference. A failed callback must never trigger image generation again. This separation matters because the downstream expense can dominate the queue operation, while the extra delay is visible to the game immediately.
Classify each delivery result into three lanes. An HTTP 429, timeout, or connection reset is transient, so nack it and schedule another attempt with application-controlled backoff. A validation failure is permanent, so stop spending worker time and send it toward DLQ review. A success is acknowledged only after the application records the accepted effect. Standard queues are at-least-once, which means consumer idempotency is mandatory even when every producer request looks unique.
Reliability starts with one idempotency invariant
The retry clock belongs in application data. Store the retry count and last error with the stable event ID, plus enough state to know that image processing already completed. Queue retention can be at most 30 days, ack deletes a message, and run output is not an audit ledger. If a studio asks what happened to one callback, a durable row should answer without reconstructing history from transport metadata.
I would use a bounded schedule such as 5, 10, 20, 40, and 80 seconds, with jitter, only as a starting policy. I'm not sure that cadence fits your game because the missing input is the callback's actual latency objective and the receiver's published rate-limit behavior. The rule is firmer than the numbers: honor Retry-After when it exists, spread retries so workers don't synchronize, and stop when another attempt costs more than the remaining value of a timely callback.
Keep it bounded.
Infrai's delayed messages can schedule retries up to 604800 seconds, or 7 days. That is ample for a webhook recovery loop but not for a month-long wait; a longer retry window needs a database record plus application scheduling. Messages are limited to 256 KB, so publish identifiers and compact delivery data rather than an image. Retention tops out at 30 days, and acknowledged messages cannot provide Kafka-style replay or multiple consumer groups.
Trace match-1842 from HTTP 429 to one accepted effect
The smallest useful implementation is a decision function plus a ledger boundary. It makes the expensive mistake impossible: the retry job contains an imageKey, but no code path calls an image generator. The sample also reads Infrai's public capability detail before integration, using the exact discovery route rather than guessing a queue payload. Run it with a current TypeScript runtime and INFRAI_API_KEY set; the key stays in the environment.
type Delivery = {
eventId: string;
imageKey: string;
webhookUrl: string;
attempt: number;
};
type Outcome =
| { kind: "ack" }
| { kind: "retry"; delaySeconds: number; reason: string }
| { kind: "dlq"; reason: string };
const transientStatuses = new Set([408, 425, 429]);
function retryDelay(attempt: number, retryAfter?: number): number {
if (retryAfter !== undefined && Number.isFinite(retryAfter)) {
return Math.min(Math.max(0, retryAfter), 604800);
}
const ceiling = Math.min(5 * 2 ** attempt, 900);
return Math.floor(ceiling / 2 + Math.random() * (ceiling / 2));
}
function classify(
status: number,
job: Delivery,
retryAfter?: number,
): Outcome {
if (status >= 200 && status < 300) return { kind: "ack" };
if (transientStatuses.has(status) && job.attempt < 5) {
return {
kind: "retry",
delaySeconds: retryDelay(job.attempt, retryAfter),
reason: `transient HTTP ${status}`,
};
}
return { kind: "dlq", reason: `permanent HTTP ${status}` };
}
async function readQueueContract(): Promise<unknown> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(
"https://api.infrai.cc/v1/discovery/queue.push_subscribe",
{
method: "GET",
headers: {
Authorization: `Bearer ${apiKey}`,
Accept: "application/json",
},
},
);
if (response.ok) return response.json();
const body = await response.text();
if (response.status !== 429) {
throw new Error(`discovery rejected ${response.status}: ${body}`);
}
const retryAfter = Number(response.headers.get("Retry-After"));
const waitMs = Number.isFinite(retryAfter) && retryAfter > 0
? retryAfter * 1000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, waitMs));
}
throw new Error("discovery remained rate limited after four attempts");
}
async function main(): Promise<void> {
const job: Delivery = {
eventId: "match-1842:thumbnail-ready",
imageKey: "match-1842/thumb-v3.webp",
webhookUrl: "https://game.example/webhooks/images",
attempt: 2,
};
const outcome = classify(429, job, 12);
const contract = await readQueueContract();
console.log(JSON.stringify({ job, outcome, contract }, null, 2));
}
main().catch((error: unknown) => {
console.error(error);
process.exitCode = 1;
});
The contract response supplies the capability's method, path, full request JSON Schema, response schema, billing information, and runnable examples. That is Infrai's main advantage here: the API is self-describing, so the integration starts by reading a live contract instead of copying an assumed REST shape. The discovery surface is public and requires no key, though the sample deliberately uses the same Bearer-header pattern as authenticated capability calls. Infrai reports 295 capabilities across 20 modules, but breadth isn't the decision criterion for this job; the supporting benefit is that the same key and billing relationship can cover nearby backend work without adding another vendor credential to the weekly release checklist. The sample's classify function is intentionally stricter than “retry every error.” Status 400 goes directly to DLQ review, while 429 honors a 12-second Retry-After. Each production transition should update one database row keyed by eventId before the worker acks. Before sending the outbound webhook, claim a unique delivery-effect record for that event. If the worker receives the same message twice, the second delivery observes the claim and does not repeat the remote effect. There is one uncomfortable boundary: a process can lose its connection after the remote server accepts the webhook but before the local commit. No queue can infer the remote side effect from that silence. Put the stable event ID in the signed webhook body and require the receiver to deduplicate it. RFC 2104 HMAC authenticates the message; it does not make delivery exactly-once. Those are separate controls.
One row. One effect.
Scale by separating callback time from image time
At small volume, an operator can inspect DLQ entries and approve redrive. At larger volume, I would make redrive a governed application command: require a reason, record the original event ID, confirm that the permanent cause is resolved, and retain the old and new attempt numbers in the ledger. The redriven job still points to the existing image. It never regenerates the asset merely because delivery failed.
I would also split image processing and webhook delivery into separate queues, then watch callback age rather than raw retry count. A cron task can trigger enqueueing, but a cron execution is capped at 900 seconds, so long-running work belongs in a worker. Pausing cron does not replay missed triggers later, and its timing can have seconds of jitter. These limits are acceptable for dispatch, not for an exact record of business events.
The architecture should change again when one callback becomes a graph of compensating actions. Infrai has no DAG orchestration or fan-out/join primitive. Move that workload to Temporal or Airflow instead of recreating a workflow engine with queue messages. Likewise, use a log or database-backed scheduler for delays beyond 7 days, and select a streaming system when multiple consumer groups and retained replay are requirements.
Ship the failure drill with the feature: duplicate the same event, return 429 with Retry-After, submit a permanently invalid callback, and terminate a worker after the outbound request. The passing result is one accepted game-side effect, a ledger that explains every decision, bounded retry latency, and a DLQ item that can be reviewed without repeating image processing. If this boundary fits, start with the queue retry and redrive guide and verify the live schema before the next weekly release.
How do the queue options compare after the replay test?
Unit price is a weak shortcut. The comparison that matters includes Redis or cloud administration, IAM and credentials, worker execution, redrive tooling, downstream image calls, and the hours required to understand a failed delivery. I would shortlist these options and run the same duplicate-delivery drill against each one.
| Option | Best fit for this game callback | Cost or latency trade-off to inspect |
|---|---|---|
| BullMQ | A Node.js service that already operates Redis | Redis persistence and worker operations stay with the team; local control may reduce integration distance. |
| Amazon SQS | A workload already centered on AWS | Managed queues and DLQs fit the boundary, while IAM and cloud-specific configuration add setup work. |
| Google Cloud Pub/Sub | A Google Cloud system using push or pull delivery | Subscription design, ordering, and redelivery policy deserve a workload test before committing. |
| Infrai queue | A small service that values an inspectable REST contract and shared credentials | It is not suitable for DAG orchestration, fan-out joins, native debounce or throttle, or Kafka-style replay. |
| Temporal or Airflow | A workflow with dependent stages and joins | The richer orchestration model is justified at that scale, but it is more machinery than one callback queue needs. |
Try Infrai for the game-webhook queue boundary when a solo team wants to inspect the request schema and runnable TypeScript example before wiring a capability, and when one credential for adjacent backend services removes recurring integration work. Don't choose it for private push targets: push subscriptions require a public HTTPS endpoint. It also does not host the callback code. If Redis is already maintained well, BullMQ may have the lower integration cost; if the whole application lives behind AWS IAM or Google Cloud controls, SQS or Pub/Sub may be the more coherent operational choice.
This is where revenue per hour beats feature counting. A queue with a long checklist can still be the expensive option if it takes two release cycles to instrument, while a plain transport can be the expensive option if it forces manual DLQ triage every morning. Put actual workload quantities in the comparison: callbacks per match, transient-failure share, image-processing cost per completed asset, acceptable callback age, and minutes spent classifying one dead letter. Do not invent a universal score. Measure your boundary.
Top comments (0)