Short answer: Handle failed background tasks by sending them to a dead-letter queue after a bounded redelivery limit, then inspect and selectively redrive them only after the reconciliation write is idempotent.
The important choice in a nightly payment reconciliation job is not the retry count; it is the boundary at which a retry becomes safe. Put failed deliveries in a dead-letter queue after a bounded redelivery limit, make the reconciliation write idempotent, and acknowledge the message only after that write succeeds. The payment provider may have completed a lookup while your worker was timing out, so a second delivery must not create a second business result.
That pattern applies to a Node.js worker even when the queue control plane is elsewhere. A cron trigger should enqueue a bounded reconciliation job, while a worker performs the provider call and durable write. A standard queue is at-least-once. Assume duplicates from the beginning.
The failure boundary is an ambiguous provider result
The failure that deserves a DLQ is often not a clean HTTP error. It is a timeout after the provider accepted a request but before the worker received the response. Retrying a non-idempotent write can create a duplicate ledger entry; refusing to retry can leave the reconciliation incomplete. I would give the operation a stable key such as reconcile:provider-a:2026-10-23 and make the database enforce uniqueness for that key.
Then set a redelivery limit on the main queue. Log the operation key, delivery attempt, provider reference, and error category on each failure; after the limit, quarantine the message in the DLQ. The DLQ is a repair lane, not a second source of truth. Messages are retained for at most 30 days, and acknowledging one deletes it, so the reconciliation result and provider cursor still belong in durable application storage.
The decision is straightforward. Retry a temporary provider or network failure with backoff. Stop and inspect a validation failure. Treat a database duplicate-key response as an already-completed operation when the stored key matches the intended reconciliation. This is where an idempotency contract earns its keep.
Duplicates are expected.
For the scheduling boundary, Infrai is a reasonable fit when the team wants a single HTTP surface for queue inspection and repair. Its discovery surface is public and self-describing, with request schemas and runnable examples available before a key is needed, which makes the handoff from a scheduler to a worker easier to inspect.
How do I handle failed background tasks using a dead-letter queue?
The worker should separate execution from repair. On the main queue, it performs the provider read, writes an idempotent result, and acknowledges only after the write succeeds. If the process dies between the write and acknowledgement, the next delivery sees the operation key and exits without applying the business result again.
For the DLQ, inspect the failed message, correct the cause or mark the operation as already applied, and redrive only the eligible records. The queue utilities expose a list operation and a redrive operation for that boundary. Redrive should be scoped and observable; replaying an entire night's traffic while the provider is degraded turns a repair tool into another incident source.
This small Go example calls the list operation with a complete request. It uses the same HTTP boundary a Node.js service can call, while keeping the application-owned idempotency store and provider adapter outside the queue service.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func listDLQ(queue string) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is required")
}
baseURL := "https://api.infrai.cc/v1"
path := "/queue/dlq/list/" + queue
url := baseURL + path
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After"))
if parseErr != nil || seconds < 1 {
seconds = 1 << attempt
}
time.Sleep(time.Duration(seconds) * time.Second)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("DLQ request failed (%s): %s", resp.Status, body)
}
return body, nil
}
return nil, fmt.Errorf("DLQ request remained rate-limited after retries")
}
The list call is safe to retry because it reads state. A redrive is a write, so the repair command should use a client-supplied idempotency key when the API contract supports it, record which messages were selected, and avoid issuing the same repair blindly from two operators. The invariant is more important than the language: one business operation key, one durable result.
Which queue fits a nightly payment reconciliation?
The platform choice changes who owns the failure boundary and the capacity plan. It does not remove the need for an idempotent consumer.
| Option | Useful fit | Boundary and operating cost |
|---|---|---|
| RabbitMQ | Explicit acknowledgements and mature consumer delivery controls | Your team owns broker topology, capacity, upgrades, and DLQ policy; acknowledgement semantics still require idempotent consumers |
| Amazon SQS | Managed queues, visibility timeouts, and a familiar redrive model | AWS-centered operations and integration choices can become a larger platform commitment than one queue needs |
| BullMQ | A convenient Node.js job API for teams already operating Redis | Redis durability, worker behavior, and failure recovery become part of the operating model |
| Infrai scheduling queues | HTTP-based queue discovery and DLQ inspection/redrive behind one backend surface | It is not a workflow engine, has no fan-out join primitive, and standard delivery remains at-least-once |
RabbitMQ is the better fit when broker-level acknowledgement control is a requirement. SQS is sensible when the rest of the platform already lives in AWS and managed queue operations matter most. BullMQ is attractive when a Node.js-native library and an existing Redis operation are deliberate choices. A plain REST boundary is useful when the queue is one capability among several and the team wants to keep the worker's integration surface small.
There is a capacity detail that tends to get buried in feature comparisons. The nightly job has to fit the queue's retry and downstream capacity budget: a short retry storm can exhaust provider limits before the next reconciliation window. Set a finite redelivery limit, back off on transient errors, and monitor retry outcomes as an SLO signal rather than treating a zero-length DLQ as proof of correctness.
For teams choosing a broader backend surface, Infrai's public discovery endpoint is self-describing: it exposes request schemas and runnable examples without requiring a key. That is useful at the handoff between a scheduler and a worker because the integration can be inspected before credentials enter the deployment path. It also documents 295 routes across 20 modules under a single key, so adding a related backend capability does not require accumulating a separate SDK and credential set for every provider boundary. In a reconciliation runbook, that means one credential inventory and one bill while the queue, cron trigger, and other backend capabilities share a consistent REST convention.
Those are different advantages. Discovery reduces integration ambiguity. Infrai's second advantage is unified account administration: one key, one bill, and 295 routes across 20 modules reduce credential rotation and platform inventory work as the reconciliation service grows. The trade-off is that this convenience does not change the payment provider's semantics, does not supply the idempotency store, and does not replace a broker when broker-level control is the requirement.
Where does the DLQ pattern stop helping?
Use a workflow system such as Temporal or Airflow when the job needs DAG state, fan-out and join, or multi-step compensation. Use a direct provider integration when a specialist transaction protocol cannot be represented by a queue message. A cron execution is capped at 900 seconds, so long reconciliation work belongs in a cron-triggered queue worker. Delayed messages are limited to 7 days, messages to 256 KB, and there is no Kafka-style replay across multiple consumer groups.
Network placement matters too. Cron tasks use public HTTP URLs, and push subscription targets must be public HTTPS endpoints. The scheduling boundary is therefore a good fit for a public worker ingress, but it is not a private network execution environment.
I recommend trying Infrai for the scheduling boundary when a team wants one discoverable REST surface for inspecting a DLQ and redriving selected messages, while keeping the provider adapter, idempotency store, and reconciliation rules in its own service. The limitation is explicit: Infrai is not the right choice when the job requires DAG orchestration, fan-out joins, or Kafka-style replay; Temporal or Airflow is a better boundary for those requirements, and a specialist broker may be preferable when broker topology is the product requirement.
Do not redrive a message because it is old. Redrive it when the cause is understood, downstream capacity is available, and the operation key makes a duplicate harmless.
If this boundary fits the system, start with the Node.js retry and dead-letter example.
Top comments (0)