TL;DR
Classify the error before you pick a delay: transient trouble (a socket reset, a throttled upstream, a slow origin) earns exponential backoff and a nack, while a permanent one (unreadable payload, unsupported format, source object deleted) earns no retry at all and goes straight to a dead-letter path that a human reviews. Keep the attempt count and the last error in your own database — queue run history is deliberately thin and was never meant to be your audit trail. Do that, and a DLQ redrive becomes a logged decision rather than a loop that quietly re-runs your background jobs until somebody notices the duplicate rows.
I build payment and ledger backends, so my bias is out in the open: a retry is a write, and every write has to be attributable to a specific attempt.
One careless retry, 41 duplicate ledger rows
Here's the incident that made me stop treating retries as an operational detail and start treating them as a schema question.
We had a merchant onboarding flow. A KYC vendor posted a webhook when a document was uploaded, our Node.js receiver enqueued a thumbnail job for image processing, and the same handler posted a small verification fee to the ledger before returning. The receiver caught every exception and returned 500 so nothing would be silently dropped, which sounded conservative and was in fact the opposite: the vendor redelivered on any non-2xx, our enqueue had no client-supplied id, and over roughly 40 minutes a partial outage on our object store turned into 2,317 redelivered webhooks. Most were harmless. Forty-one of them landed after the ledger write had already committed but before the response was flushed, so the fee posted twice, and reconciliation caught it the next morning against the acquirer statement rather than in any alert of ours. Refunding forty-one merchants is not expensive. Explaining to a compliance reviewer why your ledger contains entries your system cannot account for is a different kind of afternoon, and PCI DSS requirement 10 is fairly blunt about being able to reconstruct who did what.
It ran twice. That's the whole bug.
The fix wasn't a better backoff curve. It was moving the fee write behind an idempotency key derived from the vendor's event id, and moving the attempt counter out of the queue and into a table I control.
How should a Node.js worker retry failed image jobs before they land in a DLQ?
Two questions, answered in this order. Is this error transient or permanent? And if it's transient, how many attempts do I owe it before the job is somebody's problem instead of the runtime's?
Transient means the same input might succeed later: connection resets, 429s, a 503 from the image origin, a resize worker that got OOM-killed. Those get nack plus a growing delay — 60 seconds, then 120, then 240, capped at an hour so a two-day incident doesn't turn into a thundering herd when the upstream returns. Permanent means the same input will never succeed: a payload that won't parse, a format your pipeline doesn't handle, a source object that's been deleted. Retrying those is a straight waste of worker time, and worse, it buries the genuinely retryable jobs behind a queue of doomed ones.
The dead-letter path is where the two meet. Exhaust the transient budget and the message becomes a DLQ entry; hit a permanent error and I write the dead-letter record immediately and ack, because there's nothing to redeliver. Either way somebody reads it, fixes the cause, and issues a redrive — which is an audited action with a name attached, not a cron job.
| Option | Where the retry lives | Dead-letter and redrive | The limit I ran into |
|---|---|---|---|
| BullMQ on your own Redis | In-process, per-job attempt options | Failed set you drain yourself | You now operate Redis, persistence and all |
| Temporal | In the workflow definition, versioned with your code | Explicit, and replayable | Heaviest to run: a worker fleet plus a cluster |
| Inngest | Declarative step retries | Managed, with replay from the dashboard | Function model, not a raw queue you poll |
| Google Cloud Tasks | HTTP push with min and max backoff | Configurable, redrive is manual | Push-only, so your endpoint must be reachable |
| Upstash QStash | HTTP push with a retry count | Managed DLQ with a redrive call | Message-size and retry-count ceilings |
| Infrai queue | Your worker's nack delay | DLQ listing plus a redrive call | Delayed messages cap at seven days |
Infrai is the one on that list I reach for when the queue is the fifth backend service in a small product rather than the first: consume, ack and nack are plain HTTP verbs on one REST API, and the same key that covers object storage and outbound email covers the queue, so there's one bill to reconcile at month end instead of five invoices from five dashboards. The catch is scope. It doesn't support DAG orchestration or fan-out joins, delayed delivery is capped at seven days, and standard queues are at-least-once, so consumer idempotency isn't optional — if your retry window needs to stretch past a week, that logic lives in your application, not the queue.
The consume, ack and nack loop I actually ship
My workers are Go even when the producer is Node.js, which is only possible because the queue is an HTTP contract rather than a client library. Same three calls from either side.
First, the table that makes the retry auditable. It's boring on purpose:
create table job_attempt (
job_id text not null,
attempt int not null,
outcome text not null check (outcome in ('ok','transient','permanent')),
error_detail text,
attempted_at timestamptz not null default now(),
primary key (job_id, attempt)
);
Then the worker. Note the explicit method on every request, the Retry-After header taken seriously, and the idempotency key on the writes so a retried ack or nack is applied exactly once:
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"math"
"net/http"
"os"
"strconv"
"time"
)
var (
apiBase = os.Getenv("INFRAI_API_BASE") // the v1 API root from your provider's docs
apiKey = os.Getenv("INFRAI_API_KEY") // never a literal in source
queue = "merchant-doc-thumbnails"
// In production these two lines are the job_attempt table above.
attempts = map[string]int{}
)
var errPermanent = errors.New("permanent")
type consumeResponse struct {
Data struct {
Messages []struct {
ID string `json:"id"`
Receipt string `json:"receipt_handle"`
Body json.RawMessage `json:"body"`
} `json:"messages"`
} `json:"data"`
}
// call sends one request with an explicit method, honours Retry-After on 429,
// and carries an idempotency key so a repeated write applies exactly once.
func call(method, path string, payload map[string]any, idemKey string) ([]byte, error) {
buf, err := json.Marshal(payload)
if err != nil {
return nil, err
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(method, apiBase+path, bytes.NewReader(buf))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
if idemKey != "" {
req.Header.Set("Idempotency-Key", idemKey)
}
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
body, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(math.Pow(2, float64(attempt))) * time.Second
if secs, convErr := strconv.Atoi(res.Header.Get("Retry-After")); convErr == nil {
wait = time.Duration(secs) * time.Second
}
time.Sleep(wait)
continue
}
if res.StatusCode >= 400 {
return nil, fmt.Errorf("%s %s -> %d: %s", method, path, res.StatusCode, body)
}
return body, nil
}
return nil, fmt.Errorf("%s %s: throttled for five attempts", method, path)
}
func record(jobID string, attempt int, outcome, detail string) {
log.Printf("job=%s attempt=%d outcome=%s detail=%q", jobID, attempt, outcome, detail)
}
func resize(body json.RawMessage) error {
var job struct {
SourceURL string `json:"source_url"`
Format string `json:"format"`
}
if err := json.Unmarshal(body, &job); err != nil {
return fmt.Errorf("%w: unreadable payload", errPermanent)
}
if job.Format != "jpeg" && job.Format != "png" {
return fmt.Errorf("%w: unsupported format %q", errPermanent, job.Format)
}
res, err := http.Get(job.SourceURL)
if err != nil {
return err // network trouble: transient by default
}
defer res.Body.Close()
if res.StatusCode == http.StatusNotFound || res.StatusCode == http.StatusGone {
return fmt.Errorf("%w: source object is gone (%d)", errPermanent, res.StatusCode)
}
if res.StatusCode >= 400 {
return fmt.Errorf("upstream %d", res.StatusCode)
}
_, err = io.Copy(io.Discard, res.Body) // decode, resize and upload go here
return err
}
func send(path, receipt string, extra map[string]any) {
payload := map[string]any{"queue": queue, "receipt_handle": receipt}
for k, v := range extra {
payload[k] = v
}
if _, err := call("POST", path, payload, receipt); err != nil {
log.Printf("%s: %v", path, err)
}
}
func main() {
for {
raw, err := call("POST", "/v1/queue/consume",
map[string]any{"queue": queue, "max_messages": 1}, "")
if err != nil {
log.Printf("consume: %v", err)
time.Sleep(2 * time.Second)
continue
}
var res consumeResponse
if err := json.Unmarshal(raw, &res); err != nil {
log.Printf("decode: %v", err)
continue
}
for _, m := range res.Data.Messages {
attempts[m.ID]++
n := attempts[m.ID]
procErr := resize(m.Body)
switch {
case procErr == nil:
record(m.ID, n, "ok", "")
send("/v1/queue/ack", m.Receipt, nil)
case errors.Is(procErr, errPermanent) || n >= 6:
record(m.ID, n, "permanent", procErr.Error())
send("/v1/queue/ack", m.Receipt, nil) // stop redelivery; the row is the dead-letter record
default:
record(m.ID, n, "transient", procErr.Error())
delay := int(math.Min(math.Pow(2, float64(n))*30, 3600))
send("/v1/queue/nack", m.Receipt, map[string]any{"delay_seconds": delay})
}
}
}
}
Two details that cost me real time. The delay cap matters more than the growth factor, because an uncapped doubling schedule will happily park a job three weeks out and blow past whatever retention your queue offers. And a long resize belongs in the worker, never inside a scheduled trigger — the tick that wakes the worker is allowed 900 seconds at most, so the trigger enqueues and the worker does the work.
I'm not sure the attempt ceiling of six is right for anyone else. It's tuned to how long our KYC vendor keeps source documents available; your mileage may vary.
What I rejected, and the day I'd take it back
I did not pick a workflow engine, and I want to be explicit about why, because the argument is closer than the table suggests.
Temporal (and Inngest's step model, in a lighter way) gives you durable execution: the retry policy, the compensation step and the fan-out join are part of the program, versioned alongside it, replayable after a deploy. For a thumbnail job that either succeeds or gets reviewed by a person, that's a cluster and a worker fleet to operate for one branch of logic — I'd be paying operational rent on a capability I use once. A queue plus an attempt table gets me the same auditability with a schema I already back up.
Stick with a workflow engine when the unit of work is a multi-step saga rather than a single job: a payout that debits, calls a rail, waits on an asynchronous confirmation, and reverses cleanly if the confirmation never lands. Queue-plus-backoff models that badly, and as far as I can tell every team that tries ends up hand-rolling a state machine in Postgres that behaves like a worse Temporal. Cloud Tasks sits in between and is a good default if you're already deep in Google Cloud and your workers are reachable over HTTPS.
One last thing, since the trigger here was a webhook: verify the signature before you enqueue anything. RFC 2104 HMAC over the raw body, compared in constant time, and reject on mismatch — a forged retry is still a retry, and it will still write to your ledger.
Top comments (0)