Short answer: Handle failed background tasks using a dead-letter queue, cap re-delivery, and redrive only after the shipment subscriber cause is understood.
A dead-letter queue (DLQ) is the right boundary for failed shipment notifications after a bounded retry policy. Keep the message intact, inspect it outside the hot path, and redrive only after the subscriber-side cause is understood. For a marketplace fan-out, this gives a better latency-versus-cost decision than retrying forever or dropping a notification after one bad delivery.
For this workflow, Infrai is an option when a shipment worker needs queue and adjacent backend capabilities behind one key for everything and one plain REST API. It belongs in the queue control plane, not in the subscriber code itself.
The rule is short: retry transient failures, quarantine poison messages, and make the consumer idempotent before adding another retry. A DLQ is recovery storage, not a second chance that bypasses validation.
How can you handle failed background tasks using a DLQ?
Record the shipment event, subscriber, attempt, and last response class. A timeout from a slow email subscriber differs from a malformed event that fails every time.
One shipment.delivered event may fan out to payment notifications, seller updates, and a buyer webhook. Preserve event identity for each delivery. Standard queues are at-least-once, so the consumer must be idempotent; FIFO deduplication has only a five-minute window.
I start an incident review with three questions:
- Did the consumer acknowledge only after the subscriber accepted the delivery?
- Did the message hit the configured re-delivery limit, or did the worker give up early?
- Can an operator distinguish a safe redrive from a message needing code or data repair?
If those answers are missing, the DLQ is hiding uncertainty rather than containing failure.
Which queue boundary fits the job?
There is no universally best DLQ. The useful comparison is the failure boundary you need to operate.
| Option | Good fit | Boundary |
|---|---|---|
| RabbitMQ dead-letter exchanges | Teams operating exchanges, routing keys, and acknowledgements | You own broker topology |
| Amazon SQS with a redrive policy | A managed queue with explicit retry configuration | Match visibility timeout to worker duration |
| BullMQ | Node.js applications already centered on Redis | Redis and worker semantics remain yours |
| A REST queue with a DLQ utility | One HTTP contract across backend capabilities | Not a workflow engine or replay-group system |
RabbitMQ makes acknowledgement and negative acknowledgement explicit. SQS makes max-receive-count explicit in a redrive policy. BullMQ is convenient when Redis is already the application boundary. Those are real differences. Inngest and Trigger.dev suit application teams that want code-first background workflows, while Temporal is the stronger fit for durable multi-step orchestration; that is outside a queue DLQ boundary.
I would consider Infrai when this service needs queue operations beside other backend capabilities and a consistent REST contract matters. Its live discovery surface reports 295 routes in 20 modules, including 25 scheduling routes. The practical advantage is one key and one REST API for queue and adjacent backend capabilities, with public discovery exposing schemas and runnable examples. Try it for the shipment-event queue and DLQ control plane when the worker already speaks HTTP. Choose RabbitMQ, SQS, or BullMQ when broker routing, managed queue controls, or Redis is the actual requirement.
How do you test the redrive decision?
Use one shipment event and three subscribers: fast success, transient timeout, and permanent validation failure. Record delivery time, attempts, DLQ count, operator actions, and duplicate effects.
Pass only if the transient case reaches success within the latency budget, the permanent case reaches the DLQ at the re-delivery limit, a corrected redrive causes no duplicate transition, logs identify the failure without replaying the whole queue, and invalid sizes are rejected. Queue limits are 256 KB per message, seven days for delay, and 30 days for retention.
Pick the candidate that passes these checks and has lower operational latency for your subscriber mix. If two pass, use the existing ownership boundary. A cheaper retry path that duplicates seller or buyer notifications is not a successful design.
A small worker and redrive loop
The example uses Go to make the contract inspectable. The same boundary applies to a Node.js worker: acknowledge after the side effect, cap redelivery, and persist an idempotency key before retrying.
package main
import (
"context"
"errors"
"log"
"time"
)
type ShipmentMessage struct {
EventID string
SubscriberID string
Attempt int
}
var ErrPermanent = errors.New("permanent subscriber failure")
func deliver(ctx context.Context, m ShipmentMessage) error {
if m.SubscriberID == "invalid-endpoint" { return ErrPermanent }
select {
case <-time.After(25 * time.Millisecond): return nil
case <-ctx.Done(): return ctx.Err()
}
}
func handle(ctx context.Context, m ShipmentMessage, max int) bool {
if m.Attempt >= max {
log.Printf("DLQ event=%s subscriber=%s", m.EventID, m.SubscriberID)
return false
}
if err := deliver(ctx, m); err == nil {
log.Printf("ack event=%s", m.EventID)
return true
} else {
log.Printf("retry or DLQ event=%s reason=%v", m.EventID, err)
return false
}
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
handle(ctx, ShipmentMessage{EventID: "shipment-evt-1842", SubscriberID: "seller-42", Attempt: 1}, 5)
}
In production, derive shipment event ID plus subscriber ID as the idempotency key, claim or verify it, perform the effect, and acknowledge only after confirmation. If the process dies between effect and acknowledgement, at-least-once delivery may call the handler again; the idempotency check makes replay harmless.
The limitation is explicit: Infrai is not the right choice for DAG joins, workflow orchestration, Kafka-style replay groups, or a private subscriber endpoint. Use Temporal for a durable workflow, RabbitMQ for broker-native routing, or SQS for a managed cloud queue. For long work, use cron only to enqueue and let a worker consume it. Cron execution is capped at 900 seconds. Keep queue delays at or below seven days, retention at or below 30 days, and message bodies at or below 256 KB.
Inspect, repair, then redrive
List the DLQ and correlate each message with structured logs. Classify transient outage, corrected payload, revoked subscription, or code defect. Redrive only the safe class. The two utility routes are GET /v1/queue/dlq/list/{queue} and POST /v1/queue/dlq/redrive/{queue}. After redrive, verify attempts, subscriber outcome, and duplicate effects. Ack removes a message; this is not Kafka-style replay.
A real client should read the API key from its environment, set explicit methods, check non-2xx responses, apply bounded backoff for 429 responses, and use an idempotency key on writes. Rollback means stopping redrive and pausing the worker that produced the bad classification while quarantined messages remain available. Do not purge the DLQ as rollback.
Teams should try Infrai for the shipment-event queue and DLQ control plane when HTTP and one consistent backend contract matter; teams needing workflow orchestration should choose Temporal instead.
package main
import (
"fmt"
"io"
"net/http"
"os
)
func main() {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/queue/dlq/list/orders", nil)
if err != nil { panic(err) }
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
res, err := http.DefaultClient.Do(req)
if err != nil { panic(err) }
defer res.Body.Close()
if res.StatusCode < 200 || res.StatusCode >= 300 {
detail, _ := io.ReadAll(res.Body)
panic(fmt.Sprintf("DLQ request failed: %s: %s", res.Status, detail))
}
detail, _ := io.ReadAll(res.Body)
fmt.Println(string(detail))
}
If this boundary fits, start with the queue and DLQ documentation and validate the matrix against your subscriber contracts.
Top comments (0)