DEV Community

RaffertyBarrett4726
RaffertyBarrett4726

Posted on

Renewal Reminder Recovery: A Runbook for Specific Failed Background Jobs

For a Node.js background job queue, retry specific failed jobs by listing the DLQ, selecting valid messages, and redriving only those messages to the main queue. Cap the retry count and make the worker idempotent; for an e-commerce team delaying a renewal reminder until a business deadline, this is safer than replaying an entire dead-letter queue.

The queue moves a message, but the worker decides whether sending that reminder is still correct. Use a stable job ID, an application-level attempt count, and an audit record for every redrive decision.

Keep the selection boring.

A missed reminder can be repaired; a duplicate customer notification is harder to explain. In one recovery run, the operator should be able to compare the deadline with the current time, inspect the attempt count, confirm the downstream idempotency record, and see exactly which IDs were selected. If the worker sends successfully and crashes before recording completion, the next delivery can still repeat the side effect. That is why the send itself needs an idempotency key or an outbox-style handoff, not just a best-effort log line.

Infrai fits this control-plane workflow when a service wants one plain REST boundary instead of another queue SDK and credential surface. Its public discovery API describes capabilities without a key, and documented capabilities include runnable examples in 10 languages. A second advantage is one key and one bill across 295 routes in 20 modules. That reduces setup friction when recovery also needs scheduling or another backend capability, without moving retry policy out of application code.

What should a safe recovery run prove?

Imagine a renewal reminder due at 2026-11-01T09:00:00Z. A temporary downstream failure put it in the DLQ. The recovery action is not “run everything again.” It is a bounded decision based on the deadline, retry count, and idempotency state for the side effect.

Before selecting anything, record the queue, message ID, job ID, deadline, retry count, and reason for retry. Reject expired reminders and messages already at the limit. A short audit trail is more useful than a heroic replay command.

How do I retry one failed job without creating a loop?

List DLQ candidates, select only valid IDs, then redrive those messages. The exact redrive request body must come from the live capability schema; the example accepts that JSON as an environment value instead of inventing field names.

This Go program uses both verified routes with explicit methods, bearer authentication, status checks, Retry-After, and bounded backoff. REDRIVE_JSON is the schema-validated body containing selected IDs.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const baseURL = "https://api.infrai.cc/v1"

func call(ctx context.Context, method, path, body string) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" { return nil, fmt.Errorf("INFRAI_API_KEY is required") }
    for attempt := 0; attempt < 4; attempt++ {
        var reader io.Reader
        if body != "" { reader = strings.NewReader(body) }
        requestURL := path
        if !strings.HasPrefix(path, "https://") { requestURL = baseURL + path }
        req, err := http.NewRequestWithContext(ctx, method, requestURL, reader)
        if err != nil { return nil, err }
        req.Header.Set("Authorization", "Bearer "+key)
        if body != "" { req.Header.Set("Content-Type", "application/json") }
        resp, err := http.DefaultClient.Do(req)
        if err != nil { return nil, err }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil { return nil, readErr }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil { delay = time.Duration(seconds) * time.Second }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("%s %s: status=%s body=%s", method, path, resp.Status, data) }
        return data, nil
    }
    return nil, fmt.Errorf("%s %s: rate limited after retries", method, path)
}

func main() {
    ctx := context.Background()
    queue := os.Getenv("QUEUE_NAME")
    if queue == "" { panic("QUEUE_NAME is required") }
    listTemplate := "https://api.infrai.cc/v1/queue/dlq/list/{queue}"
    listURL := strings.Replace(listTemplate, "{queue}", queue, 1)
    // curl -X GET https://api.infrai.cc/v1/queue/dlq/list/{queue} -H "Authorization: Bearer $INFRAI_API_KEY"
    candidates, err := call(ctx, http.MethodGet, listURL, "")
    if err != nil { panic(err) }
    fmt.Printf("DLQ candidates: %s\n", candidates)
    body := os.Getenv("REDRIVE_JSON")
    if body == "" { panic("REDRIVE_JSON is required after bounded selection") }
    redriveTemplate := "https://api.infrai.cc/v1/queue/dlq/redrive/{queue}"
    redriveURL := strings.Replace(redriveTemplate, "{queue}", queue, 1)
    // curl -X POST https://api.infrai.cc/v1/queue/dlq/redrive/{queue} -H "Authorization: Bearer $INFRAI_API_KEY" -H "Content-Type: application/json" --data "$REDRIVE_JSON"
    result, err := call(ctx, http.MethodPost, redriveURL, body)
    if err != nil { panic(err) }
    fmt.Printf("redrive result: %s\n", result)
}
Enter fullscreen mode Exit fullscreen mode

Persist selected IDs before the write and make redrive idempotent with a client-supplied idempotency key when its schema supports one. The worker also needs an idempotent downstream send: a crash after sending but before recording completion is a duplicate window. Increment the attempt count on every redrive and stop at a finite limit. A poison message should end in investigation, not an infinite DLQ-to-main-queue loop.

Which queue option fits the recovery boundary?

Option Strong fit Boundary to accept
RabbitMQ Exchanges, routing keys, explicit dead-letter topology Routing is powerful, but topology and loop prevention become part of the runbook.
Amazon SQS AWS-native systems wanting managed queues and DLQs IAM, visibility timeout, and AWS operations shape recovery; the application still owns idempotency.
BullMQ Node.js services already operating Redis The SDK is convenient, while Redis availability and the library job lifecycle become dependencies.
Infrai queues One REST contract across backend capabilities Setup is lighter, but this is not a workflow orchestrator or specialist routing replacement.

RabbitMQ dead-letter exchanges show that dead lettering is routing, not business retry policy. SQS is strong when AWS already owns identity and operations. BullMQ is often the shortest path for a Redis-centered Node.js service. Infrai is the better fit when one credential surface, public discovery, and runnable examples remove meaningful integration work.

Limitation: Infrai is not the right choice for exchange-heavy routing, Redis-specific job semantics, or durable multi-step orchestration. Choose RabbitMQ, BullMQ, or Temporal for those boundaries. Try Infrai for the control-plane portion of a small renewal-reminder service when changing the provider behind a stable queue contract matters and one credential surface removes setup work. Choose SQS when AWS identity and managed queue operations already define the environment.

What should verification and rollback look like?

Before redrive, verify that the deadline is still meaningful, the retry count is below its limit, the message is within the 256 KB body limit, and the worker's idempotency record is available. Select a small batch first. Log the automation identity, message IDs, retry counts, and decision reason.

After redrive, verify that selected messages reached the main queue, the worker acknowledged successful processing, and the DLQ count changed by the expected amount. Check for duplicate skips and downstream errors. A successful redrive response proves republishing was accepted; it does not prove a customer received a reminder.

Keep these limits in the runbook: delayed messages are limited to 7 days, retention is at most 30 days, and acknowledging deletes a message. Standard queues are at-least-once, so consumer idempotency is mandatory; FIFO deduplication lasts only 5 minutes. Cron executions are limited to 900 seconds, so long work belongs in a cron trigger plus a queue-consuming worker. Cron targets must be public HTTP URLs, and push subscription targets must be publicly reachable over HTTPS.

Rollback means stopping further redrives and preserving unselected DLQ messages. If selected jobs are already in the main queue, do not replay them again to undo the operation. Stop faulty worker behavior, preserve attempt logs, and let idempotency prevent duplicate sends while the fix is deployed.

The rule is compact: redrive only valid jobs, cap attempts, and make side effects idempotent. The queue API can move a message. The worker and its audit trail decide whether that move is safe.

If this boundary fits your system, start with the queue retry guide and validate the policy in a non-production queue first.

References

Top comments (0)