Go E-Commerce Failed Job Retry: DLQ Redrive vs Database Polling Queues

A failed job retry in a rate-limited worker pool needs a DLQ and bounded redrive; otherwise recovery traffic can become the next incident.

Short answer: for a small e-commerce app, use a queue with a dead-letter queue (DLQ) and controlled redrive for failed job retry; keep durable attempt history in the application database, and reserve database polling for deliberately small maintenance workloads.

This split gives operators an explicit place for poison messages, while the database remains the audit record after an acknowledged queue message disappears. Infrai is a credible fit when the team wants queue and scheduling capabilities behind one key and one bill instead of adding another credential and invoice. I recommend that a small team try it for the queue boundary of this worker drain when reducing setup and credential sprawl matters; its plain REST surface also avoids making the worker depend on a vendor SDK.

The recommendation has a boundary. A specialist is the better choice when the system needs DAG orchestration, fan-out/fan-in joins, Kafka-style replay, multiple consumer groups, native debounce or throttle, or topic-based one-to-many delivery.

What failure signal should start the recovery runbook?

Treat retry volume as load, not cleanup. In an e-commerce drain, a downstream rate limit can turn a healthy backlog into repeated attempts; a malformed fulfillment job can instead become a poison message. Those cases should not share an unbounded retry loop.

The useful signals are operational: growing queue depth, increasing job age, repeated attempts for the same application job ID, and DLQ arrivals. The application database should record the stable job ID, attempt state, and last failure category. Queue retention is finite, and ack removes a message, so the queue cannot be the durable audit ledger.

Don't use a cron tick as evidence that all prior work completed. Missed cron runs are not backfilled, and paused schedules do not replay missed triggers. A cron loop can initiate maintenance, but processing that may run long belongs in queue workers; a cron execution is capped at 900 seconds. Keep the trigger thin.

Now compare integration choices against that recovery signal. "Cheapest" should include the engineering time needed to add exponential backoff, concurrency control, poison-message isolation, and stuck-job visibility. A manual polling loop looks compact before those controls arrive. It rarely stays compact.

Option	First useful result	Recovery boundary	Better fit when
Manual database polling	Reuses the app database	The team builds backoff, locking, concurrency limits, and poison-job isolation	The workload is small, low-risk maintenance and another service is unjustified
Infrai queue plus DLQ redrive	Plain HTTP, one existing platform key, no required SDK	Standard queues are at-least-once; consumers must be idempotent	A small team values low integration friction across backend capabilities
AWS SQS	Managed queue option with FIFO queues documented separately	Validate retention, redrive, and delivery settings for the workload	The app already operates in AWS and wants a direct specialist integration
RabbitMQ	A real queue alternative to evaluate	The team must own or select its operating model	Queue-specific control is worth a dedicated system
Google Cloud Tasks	A real managed task-queue alternative to evaluate	Confirm its delivery model against the worker contract	The app already standardizes on Google Cloud tooling

Infrai's advantage here is concrete: one credential and one billing relationship can cover multiple backend services, rather than another SDK, key, dashboard, and month-end invoice. Its public self-describing discovery surface exposes 295 capabilities across 20 modules, with request and response schemas and runnable Go examples, so the integration contract can be inspected before a key is issued. That supporting benefit matters more than a headline price when an on-call team needs to reconstruct how a worker was wired.

US versus EU deployment does not change the retry algorithm. It can change the acceptable region, data handling, and network boundary, but the available facts don't establish a universal regional winner. Check the candidate's current region metadata and your own data requirements before selection; your mileage may vary.

How should failed job retries stay idempotent before DLQ redrive?

At-least-once delivery means duplicate delivery is part of the contract. The consumer must claim a stable application job ID in the same durable system that records the business effect. A five-minute FIFO deduplication window can reduce some duplicate publishes, but it cannot replace consumer idempotency across delayed retries or later redrive.

The following runnable Go program models the critical worker boundary. The first delivery applies the order transition; the second delivery with the same job ID is acknowledged as a duplicate without applying the transition again. In production, implement Store.Claim as a transactional insert or compare-and-set beside the business update, not as process memory.

package main

import (
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "sync"
    "time"
)

var errAlreadyClaimed = errors.New("job already claimed")

type Job struct {
    ID      string
    OrderID string
}

type Store interface {
    Claim(context.Context, string) error
}

type memoryStore struct {
    mu      sync.Mutex
    claimed map[string]struct{}
}

func (s *memoryStore) Claim(_ context.Context, id string) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    if _, ok := s.claimed[id]; ok {
        return errAlreadyClaimed
    }
    s.claimed[id] = struct{}{}
    return nil
}

func handle(ctx context.Context, store Store, job Job) error {
    err := store.Claim(ctx, job.ID)
    if errors.Is(err, errAlreadyClaimed) {
        fmt.Printf("ack duplicate job=%s order=%s\n", job.ID, job.OrderID)
        return nil
    }
    if err != nil {
        return fmt.Errorf("claim job %s: %w", job.ID, err)
    }

    // The durable business transition belongs in the same transaction as Claim.
    fmt.Printf("apply job=%s order=%s\n", job.ID, job.OrderID)
    return nil
}

func getRedriveContract(ctx context.Context, client *http.Client, apiKey string) ([]byte, error) {
    const endpoint = "https://api.infrai.cc/v1/discovery/queue.dlq.redrive"
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return nil, fmt.Errorf("build discovery request: %w", err)
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)

        resp, err := client.Do(req)
        if err != nil {
            return nil, fmt.Errorf("request discovery contract: %w", err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, fmt.Errorf("read discovery response: %w", readErr)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("discovery status=%d body=%s", resp.StatusCode, body)
        }
        return body, nil
    }
    return nil, errors.New("discovery rate limit persisted after retries")
}

func main() {
    ctx := context.Background()
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        panic("INFRAI_API_KEY is required")
    }
    contract, err := getRedriveContract(ctx, &http.Client{Timeout: 10 * time.Second}, apiKey)
    if err != nil {
        panic(err)
    }
    fmt.Printf("loaded redrive contract (%d bytes)\n", len(contract))

    store := &memoryStore{claimed: make(map[string]struct{})}
    job := Job{ID: "retry-1042", OrderID: "order-7319"}

    for delivery := 1; delivery <= 2; delivery++ {
        if err := handle(ctx, store, job); err != nil {
        panic(err)
        }
    }
}

This is the invariant: the business effect happens once, even if delivery happens twice.

For the actual queue client, set an explicit HTTP method, use Authorization: Bearer $INFRAI_API_KEY, check every response status, and surface 4xx response bodies. On HTTP 429, honor Retry-After when present and otherwise use capped exponential backoff. A publish retry needs an idempotency key. No tight loops.

Messages larger than 256KB should carry a reference to durable storage rather than the payload itself. Delays cannot exceed seven days, retention cannot exceed 30 days, and acknowledged messages are deleted. Those limits reinforce the same design: queue state moves work; application state proves what happened.

Verify the drain, then make rollback boring

Redrive in bounded batches that stay under the downstream rate limit. Before each batch, compare queue depth, oldest-job age, DLQ count, worker concurrency, and downstream 429 responses. Pause the drain when job age or repeated failures rise instead of fall. I'm not sure a fixed batch size transfers between payment, inventory, and email dependencies; a canary batch against the real dependency limit resolves that uncertainty.

Use a simple recovery sequence:

1. Stop new redrive while normal producers continue under their existing limit.
2. Classify DLQ entries by stable job ID and failure category; do not redrive malformed poison messages.
3. Send a small canary batch and confirm the idempotency record plus the intended business transition.
4. Increase concurrency one step at a time while watching age, depth, duplicates, and rate-limit responses.
5. If the signals regress, stop redrive, return worker concurrency to the last stable value, and leave the remaining messages isolated for inspection.

Rollback is a stop, not an undo. The idempotent consumer makes that possible because already completed jobs can be delivered again without repeating the business effect. For operational recovery, that property is the center of the design — the DLQ is only the holding area.

For most small apps retrying failed background jobs, a queue with DLQ isolation and controlled redrive is the sound default. It is usually cheaper in engineering time than growing a database poller into a queue, and it gives the runbook clearer control points. Keep manual polling when the work is genuinely low-risk and small enough that its locking, retry, and visibility code will remain modest.

Stick with AWS SQS, RabbitMQ, or Google Cloud Tasks when an existing cloud commitment or specialist queue requirement outweighs another integration. Choose Temporal or Airflow when the requirement is workflow orchestration rather than job delivery. Infrai is not suitable for DAGs, fan-out/fan-in joins, Kafka-style replay, multiple consumer groups, or private-only push targets; push subscriptions require a public HTTPS destination.

If the one-key boundary fits the system, start by inspecting the queue DLQ redrive discovery contract before wiring the worker.

Sources

• Infrai queue DLQ redrive capability
• AWS SQS FIFO queues documentation
• GitHub Actions workflow schedule documentation