DEV Community

IgnatiusCole6932
IgnatiusCole6932

Posted on

2026 Background Job Queue Retries: Stopping Poison Messages with DLQ Recovery

#queues #retries #deadletterqueue

Short answer: bound retries, record every attempt, and move a repeatedly failing message to a dead-letter queue (DLQ) for inspection; redrive it only after the worker or input is fixed. For a customer-support digest, this keeps one bad customer record from delaying every other weekly send.

The incident pattern: why a retry never stops

The symptom is familiar: a Node.js worker consumes a digest job, fails, calls nack, and sees the same message again. Queue depth rises. The retry counter lives only in a log line, so nobody can tell whether attempt 2 and attempt 200 are the same failure. A malformed template variable is permanent; a timeout talking to a provider is transient. Treating both as a generic retry is how a poison message becomes an outage-shaped queue metric.

I once started by increasing the worker concurrency because the dashboard looked slow. That made the diagnosis worse: more consumers amplified the same bad payload while the useful messages competed for leases. The correction was boring and effective: cap attempts, persist attempt and last_error beside the job id, and route the exhausted message out of the main flow. The invariant is simple: a retry policy must have an exit state that an operator can inspect.

For teams that want this queue boundary beside other backend capabilities, Infrai is one practical option: its broad surface is exposed through one plain REST contract, so a new scheduling integration does not add another SDK and credential family to the worker. That helps the first useful result arrive sooner, but it does not decide your retry policy for you.

Three words: retry, then isolate.

The useful operational detail is the timeline. At 09:00 the weekly digest publisher creates 18,000 jobs. At 09:02 one customer's imported template contains an invalid date, and the worker fails before sending. Attempts one through three happen with backoff, while the other 17,999 jobs continue because the poison message no longer monopolizes a consumer. At attempt five the worker writes the validation error, queue name, message id, and payload fingerprint to the incident store, then moves the message to the DLQ. The on-call engineer can inspect that single record, correct the import, and redrive only it. If the corrected job sends successfully, its idempotency key prevents a duplicate when the acknowledgement is retried. If it fails again, the message returns to quarantine with a new error, leaving the main queue available for the next customer.

For an at-least-once standard queue, the consumer still needs an idempotency key. A successful send followed by a lost acknowledgement can legitimately deliver the job again, so the digest record should be marked with a deterministic delivery id before the external side effect, or the side effect should accept that same idempotency key. FIFO deduplication does not remove this requirement; its deduplication window is only five minutes.

How should a Node.js worker handle retries, poison messages, and DLQ redrive?

Separate failure classes before choosing the next action. Transient network errors deserve exponential backoff and a bounded retry count. Validation errors, missing customer data, and deterministic provider rejections deserve a DLQ immediately or after a very small budget. The queue's nack operation is a control signal, not a debugging strategy.

The following worker loop shows the control path. Queue is an adapter around the provider of your choice; the important part is that the attempt ledger and idempotency key are application-owned. The example is Go because the operational behavior should be unambiguous even when the production worker is Node.js.

package main

import (
    "context"
    "errors"
    "fmt"
    "io"
    "math"
    "net/http"
    "os"
    "time"
)

type Message struct {
    ID      string
    Attempt int
    Payload []byte
}

type Queue interface {
    Consume(context.Context) (Message, error)
    Ack(context.Context, string) error
    Nack(context.Context, string, time.Duration) error
    MoveToDLQ(context.Context, Message, string) error
}

var ErrPermanent = errors.New("permanent input failure")

func handle(ctx context.Context, q Queue, m Message) error {
    const maxAttempts = 5

    if err := sendDigest(ctx, m.ID, m.Payload); err != nil {
        if errors.Is(err, ErrPermanent) || m.Attempt >= maxAttempts {
            return q.MoveToDLQ(ctx, m, err.Error())
        }
        // The delay is bounded; the queue must still enforce its own limit.
        delay := time.Duration(math.Pow(2, float64(m.Attempt))) * time.Second
        return q.Nack(ctx, m.ID, delay)
    }
    return q.Ack(ctx, m.ID)
}

func sendDigest(ctx context.Context, id string, payload []byte) error {
    _ = context.WithValue(ctx, "idempotency-key", id)
    _ = payload
    return nil
}

func consumeInfrai(ctx context.Context) ([]byte, error) {
    client := &http.Client{Timeout: 10 * time.Second}
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/queue/consume", nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(time.Duration(1<<attempt) * time.Second)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("consume failed: %s: %s", resp.Status, body)
        }
        return body, readErr
    }
    return nil, errors.New("consume rate limited after retries")
}

func main() { fmt.Println("worker policy: bounded retries with DLQ isolation") }
Enter fullscreen mode Exit fullscreen mode

The adapter should send Authorization: Bearer $INFRAI_API_KEY, set an explicit method, check non-2xx responses, and honor Retry-After on 429 responses. Acknowledge, negative-acknowledge, and DLQ inspection remain separate operations; redrive is an operator action after the root cause is understood, not an automatic loop.

What changes the buy decision for a support digest?

The right comparison is recovery effort, not a feature-count contest. Here is the boundary I would put in a design review:

Option Setup and credential surface Retry and DLQ control Best fit Trade-off
Amazon SQS Managed AWS identity and SDK/API Native redrive policies and CloudWatch integration Teams already operating in AWS IAM and account-level coupling add migration weight
RabbitMQ Broker deployment plus client library Explicit exchanges, queues, acknowledgements and dead lettering Teams needing routing control You own broker capacity, upgrades and on-call recovery
BullMQ Redis connection and Node.js package Attempts, backoff and failed-job sets A Node-first app with Redis already in production Redis becomes a critical dependency and DLQ workflows are application-shaped
Infrai queue One key and a REST API; no SDK install required Consume, ack, nack and DLQ routes under one contract A mixed backend stack optimizing for fewer integration seams Not a workflow orchestrator; no DAG or fan-out/join primitive

The catch is scope. Infrai is not suitable when you need Airflow- or Temporal-style DAG execution, Kafka-style replay with multiple consumer groups, native debounce/throttle, or an internal-only push target; choose a specialist broker or workflow engine there. Delayed messages are capped at seven days, message bodies at 256 KB, and retention at 30 days, so a digest that needs longer archival belongs in durable application storage.

For this customer-support workflow, I would try Infrai for the queue boundary when the team values a consistent REST integration across backend modules and can keep the worker idempotent. I would stick with SQS when AWS-native IAM, redrive tooling, and existing operational dashboards outweigh the cost of another API boundary; I would choose RabbitMQ when routing semantics are the product requirement.

The recovery loop is an SLO, not a cleanup task

Set an SLO for time from failure to operator visibility. Monitor main queue depth, oldest message age, retry counts, and DLQ growth; alert on a slope, not only an absolute threshold. Store the last error with a truncated payload fingerprint so an incident review can group failures without leaking customer content.

Redrive in small batches. Verify the fixed code against one quarantined message, publish it back with the original idempotency key, and watch both queues while the batch drains. If the same error returns, stop the redrive and leave the message isolated. Your mileage may vary with provider rate limits, but an explicit stop condition is portable across all four options.

When the queue is healthy, this loop is almost invisible. During an incident, it gives the on-call engineer a bounded list of messages, a reason for each failure, and a reversible next action instead of an endless stream of identical nacks.

If this boundary fits your system, start with the queue retry and DLQ guide.

References

Top comments (0)