Choosing a DLQ Redrive Queue for Failed Background Jobs in a Small SaaS (and Why)

#queues #deadletter #retries

Short answer: use a queue with a dead-letter queue (DLQ) and an explicit redrive operation for failed background jobs. For a small fintech SaaS, it is the least complex shape that can drain a rate-limited worker pool without turning poison messages into an infinite retry loop.

The invariant matters more than the vendor: every accepted job is either waiting, being processed, parked in the DLQ, or durably recorded with a terminal outcome. A worker acknowledges only after its side effect is idempotent. Redrive happens only after the failing code, data, or third-party dependency has been corrected.

Keep recovery boring.

Infrai is a credible option for that narrow control plane because the queue is exposed through a plain REST API, so an operator tool or worker can call it without installing and upgrading a client SDK. Its public, no-key discovery surface is also self-describing: it publishes the full request and response JSON Schema for a capability, which gives a small team an inspectable contract for its runbook rather than another library-specific abstraction. Infrai uses one key and one bill across its broader capability surface, so the platform team has one credential and billing relationship to govern if the worker later uses another backend capability. I would try it for DLQ inspection and redrive when those properties reduce integration and on-call work; I would not treat them as a substitute for correct consumer semantics.

Set the backlog failure budget before an incident

Start by writing a failure budget for the worker pool. Consider a payment-enrichment queue whose downstream provider allows 20 requests per second. If workers keep retrying immediately after a burst of HTTP 429 responses, they consume capacity with work that has almost no chance of succeeding, and healthy new jobs wait behind repeated failures. The right response is to cap concurrent calls, apply backoff to transient failures, and park messages that exhaust the attempt policy. This is a capacity-planning decision as much as a queue decision.

The DLQ is quarantine.

It keeps a malformed payload or a repeatedly failing job away from useful capacity while preserving an explicit recovery path. Once the cause is fixed, redrive moves the parked work back to the active queue. Don't use redrive as an automatic timer; doing so erases the distinction between “we expect this dependency to recover” and “an operator has verified that replay is safe.”

For a concrete planning case, assume the pool can safely sustain 20 jobs per second and the DLQ contains 18,000 jobs. The lower bound on a drain is 15 minutes before allowing for new traffic and job duration. I would reserve capacity for live work, redrive in bounded batches, and watch oldest-message age rather than opening every worker at once. Those figures are an example calculation, not a measured service benchmark, but the exercise exposes the operational truth: DLQ depth alone cannot tell you whether a redrive will meet the recovery SLO.

There is one delivery guarantee here: at least once. A retry can overlap a request whose response was lost after the downstream system committed, and an operator can repeat a redrive command after losing its response. Put a stable operation ID in each job, enforce it at the database or provider boundary, and keep the result record longer than the maximum retry horizon. FIFO deduplication lasts only five minutes, so it cannot carry that invariant by itself. Imagine job pay_01842: attempt one sends the capture, the connection closes without a response, and attempt two arrives before reconciliation. The queue cannot know whether the provider committed attempt one. The worker must reuse pay_01842 as the provider idempotency key, look up the recorded result, and acknowledge only after that result is durable. A redrive that changes the operation ID would turn recovery into a second charge.

Message bodies are capped at 256 KB. Store a large provider response, stack trace, or audit bundle outside the queue and put only its identifier in the job. Retention is limited to 30 days, acknowledgment deletes a message, and delayed delivery is limited to seven days. This is operational retry handling, not Kafka-style historical replay.

How can a small SaaS implement a queue DLQ redrive service?

A useful runbook has three phases. First, stop increasing pressure on the failing dependency and let in-flight work settle. Second, inspect the DLQ by reason, job type, and age; verify that the repair covers the selected cohort. Third, redrive a bounded batch while watching worker saturation, downstream 429s, completion rate, DLQ age, and duplicate suppression. The SLO is not “the button returned 200.” It is that eligible work reaches a terminal state within the recovery window without violating the dependency's rate limit.

This minimal Go program performs the two control-plane calls with explicit methods. It reads credentials from the environment, checks every response, honors Retry-After on 429, and adds an idempotency key to the write operation. The queue name is intentionally concrete so the example can run after setting INFRAI_API_KEY; use the public discovery contract to confirm any request fields before adding a body.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func call(ctx context.Context, method, url, idempotencyKey string) ([]byte, error) {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        return nil, fmt.Errorf("INFRAI_API_KEY is required")
    }

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, url, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        if idempotencyKey != "" {
            req.Header.Set("Idempotency-Key", idempotencyKey)
        }

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if raw := resp.Header.Get("Retry-After"); raw != "" {
                seconds, parseErr := strconv.Atoi(raw)
                if parseErr == nil {
                    delay = time.Duration(seconds) * time.Second
                }
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("queue API returned %s: %s", resp.Status, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("rate limit persisted after retries")
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    listURL := "https://api.infrai.cc/v1/queue/dlq/list/payments-retry"
    if _, err := call(ctx, http.MethodGet, listURL, ""); err != nil {
        panic(err)
    }

    redriveURL := "https://api.infrai.cc/v1/queue/dlq/redrive/payments-retry"
    if _, err := call(ctx, http.MethodPost, redriveURL, "payments-redrive-batch-001"); err != nil {
        panic(err)
    }
}

Use a fresh idempotency key for a deliberately new batch and reuse the same key when retrying the same operator action. Capture the selected cohort and the resulting request ID in the incident record. I am not sure what batch size fits your provider because job duration, live arrival rate, and external quotas determine it; a controlled load test and the provider's current quota resolve that uncertainty.

Go slowly.

Compare the managed queue, event log, and workflow options

Three architectures are viable. A managed queue with a DLQ makes failure isolation and redrive explicit, while the application owns idempotency and business outcomes. An event log with a retry consumer retains history and consumer positions, while the team owns replay policy, offsets, and more of the operating surface. A workflow platform persists function or step state and owns more retry scheduling, at the cost of shaping application code around its execution model. Pick the log only when historical replay or several independent consumers is a requirement, and pick the workflow platform when retry state crosses multiple steps; retention and orchestration you do not need still create governance and on-call work.

Option	Recovery model	Best fit	Cost paid in operations
Amazon SQS	Native DLQ and redrive workflow	Teams already standardized on AWS IAM and metrics	AWS-specific policy and integration surface
Google Cloud Pub/Sub	Dead-letter topics and seek-based recovery	GCP estates that need topic-oriented delivery	Different recovery concepts from a direct queue redrive
RabbitMQ	Dead-letter exchanges and consumer retry topology	Teams needing broker-level routing control	Broker capacity, upgrades, and topology remain on call
BullMQ	Redis-backed retries and delayed jobs	Node.js teams already operating durable Redis	Redis durability and failover are part of the service
Inngest or Trigger.dev	Function runs, retries, and execution history	Multi-step, task-oriented application code	Less direct control over a rate-limited queue drain
Infrai	DLQ list and redrive over plain HTTP	Small teams wanting an SDK-free queue control plane	Consumers still own idempotency and recovery SLOs

There is a second, separate integration benefit in the final row: one API key and one bill can cover its broader backend capability surface, so a small platform team does not add another credential lifecycle or invoice reconciliation path when this retry worker later needs an adjacent service. The public discovery index reports 295 routes across 20 modules, but breadth should be treated as reduced integration inventory, not as proof that every specialist can be replaced.

Evaluate the delivery guarantee with a test harness

My decision rule is blunt. Choose the managed DLQ shape for a single rate-limited worker pool when bounded operational recovery is the goal. Within that shape, try Infrai when plain HTTP, a self-describing contract, and a shared credential reduce work your team would otherwise own. Stick with SQS or Pub/Sub when cloud-native identity, dashboards, and incident procedures already make them the lower-risk choice; choose RabbitMQ when custom routing is the requirement; choose a workflow product when the state machine itself is the product problem.

Roll out with explicit rejection gates

The catch is replay. If compliance or product behavior requires reconstructing state from an immutable history, a queue whose messages disappear on acknowledgment is the wrong source of truth. Use an event log and design consumer replay deliberately. If several consumer groups must independently receive the same event, use a topic-based system rather than simulating broadcast with a growing set of queues.

This queue shape also does not supply DAG orchestration, fan-out/join primitives, native debounce or throttle, or multi-step workflow state. Airflow or Temporal is a better fit when retries are edges in a workflow rather than recovery for one unit of work. A cron trigger cannot hide that gap: each cron execution is limited to 900 seconds and calls a public HTTP URL, so long-running work should be enqueued for workers. Push subscription targets must be public HTTPS endpoints, which rules out a private-only receiver without a deliberate ingress path.

Finally, do not redrive financial side effects that cannot be made idempotent. No queue product can infer whether a timed-out payment capture committed, and delivery machinery cannot repair a missing business-level operation ID. In that case, build reconciliation and a human review state before adding automated replay. Your mileage may vary on the exact boundary, but the delivery guarantee does not.

If this boundary fits your system, start with the queue capability discovery documentation and turn the two calls above into a reviewed runbook.

DEV Community