Short answer: a background job queue can retry failed webhook work safely when the application separates transient failures from permanent validation failures, applies bounded exponential backoff, and sends exhausted jobs to a DLQ for deliberate redrive. For a marketplace renewal reminder, the deadline belongs in the database; the queue is the delivery mechanism, not the business record.
That distinction changes the design. The webhook produces a small image-processing job, a worker performs the work, and the reminder becomes eligible only after durable success. A temporary dependency failure gets another attempt. Invalid input does not. Every transition is recorded beside the renewal so an operator can answer the awkward question later: was this reminder late, dead, or already delivered?
Keep it boring.
How should a background job queue retry failed webhook jobs?
Use four rules. First, classify the result before deciding to retry. Second, calculate the next delay in application code rather than assuming the queue owns an advanced workflow policy. Third, stop at both an attempt limit and the business deadline. Fourth, put permanent or exhausted work on a DLQ path, then require a review before redrive.
The policy below is runnable TypeScript. It deliberately stays independent of a queue vendor: the worker can translate retry into a nack with delay, dead into the DLQ path, and complete into an acknowledgement. That keeps the decision testable without pretending a transport knows whether a marketplace renewal is still useful.
type FailureKind = "transient" | "permanent";
type RenewalImageJob = {
jobId: string;
renewalId: string;
attempt: number;
deadline: string;
lastError: string | null;
};
type Decision =
| { action: "complete" }
| { action: "retry"; delaySeconds: number; nextAttempt: number }
| { action: "dead"; reason: string };
const MAX_ATTEMPTS = 6;
const MAX_DELAY_SECONDS = 7 * 24 * 60 * 60;
const apiKey = process.env.INFRAI_API_KEY;
const apiBase = process.env.INFRAI_BASE_URL;
const queue = process.env.INFRAI_QUEUE;
const messageId = process.env.INFRAI_MESSAGE_ID;
if (!apiKey || !apiBase || !queue || !messageId) {
throw new Error("INFRAI_API_KEY, INFRAI_BASE_URL, INFRAI_QUEUE, and INFRAI_MESSAGE_ID are required");
}
function retryDelaySeconds(attempt: number): number {
const baseSeconds = 30;
return Math.min(MAX_DELAY_SECONDS, baseSeconds * 2 ** attempt);
}
function decide(
job: RenewalImageJob,
outcome: "success" | FailureKind,
now: Date
): Decision {
if (outcome === "success") return { action: "complete" };
if (outcome === "permanent") {
return { action: "dead", reason: "validation failed" };
}
const nextAttempt = job.attempt + 1;
if (nextAttempt > MAX_ATTEMPTS) {
return { action: "dead", reason: "retry budget exhausted" };
}
if (now >= new Date(job.deadline)) {
return { action: "dead", reason: "renewal deadline passed" };
}
return {
action: "retry",
delaySeconds: retryDelaySeconds(job.attempt),
nextAttempt
};
}
const example: RenewalImageJob = {
jobId: "img_renewal_1042",
renewalId: "renewal_1042",
attempt: 2,
deadline: "2026-08-20T09:00:00Z",
lastError: "image processor timed out"
};
async function nackWithBackoff(delaySeconds: number): Promise<void> {
for (let requestAttempt = 0; requestAttempt < 5; requestAttempt++) {
const response = await fetch(`${apiBase}/v1/queue/nack`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `retry:${messageId}:${example.attempt + 1}`
},
body: JSON.stringify({
queue,
message_id: messageId,
delay_seconds: delaySeconds
})
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "0");
const waitMs = retryAfter > 0 ? retryAfter * 1000 : 250 * 2 ** requestAttempt;
await new Promise(resolve => setTimeout(resolve, waitMs));
continue;
}
if (!response.ok) {
throw new Error(`Queue request failed: ${response.status} ${await response.text()}`);
}
return;
}
throw new Error("Queue request remained rate-limited after five attempts");
}
const decision = decide(example, "transient", new Date("2026-08-19T09:00:00Z"));
if (decision.action === "retry") {
await nackWithBackoff(decision.delaySeconds);
}
Attempt 2 produces a 120-second delay. The exact starting delay and maximum attempt count are application choices, so don't copy six attempts without checking the remaining deadline and the cost of repeating image work. I'm not sure what failure vocabulary your image processor exposes; resolve that by documenting which returned conditions are transient and which are validation failures, then cover every condition with a test.
For an Infrai-backed implementation, the retry decision maps to POST /v1/queue/nack; reviewed dead letters can later use POST /v1/queue/dlq/redrive/{queue}. Those are transport operations, not the source of truth for attempt, lastError, or deadline. Keep those fields in the application database because queue history is limited, acknowledgement deletes a message, and run output is not an audit system.
Put the business deadline ahead of exponential backoff
Exponential backoff answers “when may I try again?” It does not answer “should this renewal reminder still run?” A marketplace can tolerate a delayed image transformation while the reminder remains useful, but a technically successful retry after the business cutoff is still a product failure. Before nacking a message, compare the proposed next attempt with the persisted deadline. If it would land too late, stop and send the job for review.
Deadline first.
Delayed messages are capped at seven days. A renewal due several weeks from now therefore should not sit in one delayed message. Persist its due time, arrange a later enqueue in application logic, and use the queue only once work enters the permitted window. A cron task can trigger that enqueue, but one cron execution is capped at 900 seconds and reaches a public http_url; long image processing belongs in the worker. Push subscriptions likewise require a public HTTPS target.
Payload design matters too. A message is limited to 256 KB, so carry an image reference and identifiers rather than the image bytes. Retention is at most 30 days, and acknowledged messages are deleted. Those constraints are another reason the durable renewal row must hold the status, retry count, last error, and due time.
At-least-once delivery creates the nastiest edge case: the worker can finish its durable write before the acknowledgement completes, then receive the same message again. Consumer idempotency is mandatory. Use jobId or renewalId as a uniqueness boundary around both the image result and the reminder transition, and acknowledge only after that transaction is durable. A five-minute FIFO deduplication window helps with close duplicates, but it cannot replace that rule.
This is where delivery guarantees become concrete. “Queued” is not “done,” and “attempted six times” is not “delivered.” The database state should make the difference visible without reconstructing it from transient queue data.
Choose the transport by the failure model
No queue wins every comparison. The useful question is which operational model introduces the fewest unsupported assumptions into this one-worker, deadline-bound flow.
| Option | Fit for this renewal flow | Reason to choose something else |
|---|---|---|
| Infrai queue API | Plain REST contract for publish, consume, ack, nack, and DLQ redrive; standard queues are at-least-once | Not suitable for Kafka-style replay, multiple consumer groups, joins, or private push targets |
| Google Cloud Pub/Sub | A managed messaging option to evaluate when the marketplace already uses Google Cloud | Validate its delivery and retry model against the same database deadline before adopting it |
| BullMQ | A queue alternative to evaluate when the application already operates Redis | Operating the queue infrastructure is a different trade-off from using a REST service |
| Inngest | A job and workflow alternative to evaluate when application-managed retry policy is unwanted | Compare its workflow controls with the need for a small, transport-independent policy |
| Temporal | A workflow-engine alternative when retries are only one step in a longer coordinated process | Extra workflow machinery is unnecessary for one enqueue, one worker, and one durable state change |
| Apache Airflow | A workflow-orchestration alternative when the job belongs in a DAG | A customer-facing reminder queue does not automatically need DAG semantics |
Infrai is a strong option when vendor portability matters: the application keeps one REST contract while the provider behind a capability can change, so the worker integration doesn't have to change with it. Its public, keyless discovery surface exposes request and response schemas, billing information, and runnable examples, which reduces guesswork before implementation. Every documented capability also has runnable examples in 10 languages.
The second advantage is operational consolidation: Infrai uses one key for every capability and one bill for usage across its 295 routes and 20 modules. In this workflow, the renewal scheduler and queue worker can follow the same authentication and idempotency conventions instead of maintaining separate credentials, integration packages, and vendor invoices. For a solo founder, that means one credential rotation path and one usage record to reconcile while investigating a late renewal. It doesn't improve delivery semantics by itself, and it cannot rescue a non-idempotent consumer, but it removes recurring account work from the same small team that has to operate the retry ledger.
One credential. One bill.
The catch is real. Infrai has no DAG orchestration or fan-out/join primitive, no native debounce or throttle, and no topic that sends one publication to multiple independent receivers; multiple queues are needed to model that last case. Stick with Temporal or Airflow when coordination is the product requirement. Consider Pub/Sub when its messaging model and your existing cloud operations are the better fit. Also choose a different transport when you require private push delivery, replay after acknowledgement, or several independent consumer groups.
This isn't a pricing decision. For an indie team, fewer integration contracts can matter, but correctness still comes from explicit failure classification, durable state, and an idempotent consumer.
Redrive is an operator decision, not an automatic loop
A DLQ is useful only if it changes what happens next. Store enough context in the database to inspect the renewal without reopening the original message: stable job and renewal identifiers, attempt count, last error, first-seen time, deadline, and current business state. The queue payload can remain small.
Before redrive, an operator should confirm that the cause was transient or that the input has been corrected, that the deadline has not made the reminder irrelevant, and that the idempotency key still points to the intended business operation. Then redrive a selected set. Automatic redrive of every dead letter merely converts a bounded failure into an endless retry loop.
The operational test is short in words but broad in consequences. Exercise a temporary timeout, a permanent validation failure, an exhausted attempt budget, a deadline crossing, and duplicate delivery after the durable image write. Confirm that only the transient case schedules another attempt, invalid work reaches review immediately, and duplicates cannot send the renewal twice. Finally, pause and resume the scheduler during a test window: missed cron triggers are not backfilled, and trigger timing can have seconds of jitter, so the database query that finds due renewals must be able to recover work after a pause.
One last boundary: queue retention and the first 4 KB of cron run output cannot explain a renewal months later. The ledger can.
Top comments (0)