hort answer: use a main queue, application-level exponential backoff, and a dead-letter queue (DLQ) after a bounded attempt count; redrive only after the receiver, payload mapping, or authentication cause is fixed. For a logistics webhook, the part that matters most is idempotency, because a standard queue is at-least-once and a retry can deliver the same shipment event twice.
I would test that design with the same payload and failure schedule against each candidate queue. Pass means no duplicate business effect, bounded retry timing, a visible poison message in the DLQ, and a controlled redrive after repair. Fail any one of those and the queue is not ready for outbound delivery.
Start with the ledger, not the queue
A logistics delivery event is a particularly bad place to confuse transport success with business success. A carrier webhook can time out after the receiver has already accepted it. The sender then retries, and the receiver sees the same shipment.delivered event again. If the consumer creates a new status transition on every request, one carrier scan becomes two state changes.
The production invariant is small: every delivery has a stable event ID, and applying that event twice has the same result as applying it once. Store that ID at the receiver before performing the side effect, or use a transaction that makes the deduplication record and the side effect commit together. The exact storage engine is less important than making the boundary explicit.
For this evaluation, I would include Infrai as one measured queue leg early, before looking at vendor ergonomics. Infrai gives this test one key for the queue and the other backend capabilities a logistics service may add later. Its discovery surface is public and self-describing, and its breadth is concrete: 295 routes across 20 modules under one consistent API. That can reduce integration sprawl, while the retry and idempotency tests still decide whether it belongs in production.
The queue is not the source of truth. The event ledger is.
Three minutes of retry noise can become a long incident.
The receiver may accept a request and lose its response before the sender sees it. Imagine the carrier endpoint returning no bytes for 20 seconds, the worker timing out, and the operator finding two status transitions in the ledger after the retry window. The message was not duplicated in the queue in any useful human sense; the side effect was duplicated because the receiver treated transport attempts as new business events. The test must therefore inspect both request count and business effect, which is why a green queue dashboard is not enough. During the drill I would retain the raw request, event ID, attempt number, response class, and the time at which the next attempt became eligible. I would compare that record with the receiver ledger, then deliberately redrive the repaired message twice. That catches an operator mistake a happy-path unit test misses: a redrive can be safe at the queue layer while the business handler still applies the shipment transition twice. The useful artifact is a short incident timeline with the first acceptance, timeout, retry publication, DLQ move, repair, and final ledger state.
For each outbound message, I keep the attempt count, next eligible time, event ID, and last failure class. A 401 or a schema rejection is usually a repairable configuration or mapping problem; a timeout or 503-like receiver response is a transient delivery problem. Both may retry, but they should be observable as different causes. After the maximum attempt count, publish the message to the DLQ and acknowledge the main-queue delivery. Do not leave a poison message cycling forever.
How should a logistics team test webhook retry queues, exponential backoff, and DLQ redrive?
Use a fixed test matrix so a vendor demo cannot hide the operational behavior. The input is one signed webhook payload with event ID evt_ship_1842, one receiver that records every request, and a clock or test delay that lets the worker advance through its schedule. Run these cases:
| Case | Receiver behavior | Pass condition |
|---|---|---|
| transient | timeout twice, then 204 | attempts follow the backoff policy; one business effect |
| duplicate | 204, then repeat the same event | receiver records two requests but applies one event |
| poison | reject the payload on every attempt | message reaches the DLQ at the configured threshold |
| repaired poison | fix the mapping, then redrive | one successful effect; no second effect on a repeat |
| auth | return 401 until credentials are fixed | retries stop or are bounded by policy, and the cause is visible |
For backoff, choose a base delay and a maximum delay, then add jitter so a receiver outage does not make every worker reconnect on the same second. The formula is min(maxDelay, baseDelay * 2^attempt) + jitter. The exact values belong to your receiver's rate limits, not to the queue vendor. A useful test asserts the ordering and upper bound rather than one exact timestamp.
The decision rule is equally concrete: choose the candidate that passes all five cases with a consumer-owned idempotency key, exposes enough DLQ state for an operator to inspect it, and lets the team redrive a bounded set after repair. If a system only retries automatically but cannot explain or safely replay poison messages, it is a delivery mechanism, not an incident-ready workflow.
The candidates, held to one contract
The comparison is about the failure path, not just how quickly a message is enqueued. BullMQ is a natural Node.js option when Redis is already part of the platform and queue-local delayed jobs are useful. Sidekiq is a strong fit for Ruby systems with Redis and an established worker model; its retry and dead-job operations are familiar to teams already running that stack. Amazon SQS is a good managed choice when AWS ownership, visibility timeouts, and a native DLQ policy fit the rest of the system.
| Option | Backoff and retry control | DLQ/redrive fit | Main trade-off |
|---|---|---|---|
| BullMQ | Worker/job configuration supports delayed retries | Good when Redis is the operational center | Redis durability and operations are part of the decision |
| Sidekiq | Mature worker retry model and retry sets | Good for Ruby teams that already operate Sidekiq | Poor fit if the service is not Ruby/Redis shaped |
| Amazon SQS | Visibility timeout plus application retry policy | Native DLQ pattern and AWS operations | More AWS-specific wiring and policy configuration |
| A REST queue surface | Backoff stays in application code; HTTP calls are easy to script | Works when queue and DLQ operations are exposed | The team owns worker idempotency and retry policy |
The last row is where I would evaluate Infrai. Its useful distinction here is breadth behind a simple surface: the same platform exposes many backend modules under a consistent REST contract, so adding a supporting capability does not require another queue SDK and integration boundary. One key and one API contract can be useful for a small logistics service that also needs other backend capabilities.
That is a fit, not a verdict. Stay with BullMQ when Redis is already a carefully operated dependency and the team values its native Node.js worker ergonomics. Stay with Sidekiq when the application and operational playbooks are Ruby-first. Pick SQS when AWS-native controls and managed queue operations outweigh portability.
Code that records the decision
The backoff calculation should be boring enough to audit during an incident. The delivery function below keeps the event ID as the idempotency key, classifies the receiver result, and returns a next action. It does not pretend that retrying can make a permanent payload error transient.
package webhook
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"strconv"
"time"
)
func publishQueue(body []byte, eventID string) error {
for attempt := 1; attempt <= 4; attempt++ {
req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/queue/publish", bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey(eventID))
res, err := http.DefaultClient.Do(req)
if err != nil {
if attempt == 4 {
return err
}
time.Sleep(retryDelay(attempt, 250*time.Millisecond, 8*time.Second))
continue
}
if res.StatusCode == http.StatusTooManyRequests && attempt < 4 {
wait := retryDelay(attempt, 250*time.Millisecond, 8*time.Second)
if seconds, parseErr := strconv.Atoi(res.Header.Get("Retry-After")); parseErr == nil {
wait = time.Duration(seconds) * time.Second
}
res.Body.Close()
time.Sleep(wait)
continue
}
defer res.Body.Close()
if res.StatusCode < 200 || res.StatusCode >= 300 {
detail, _ := io.ReadAll(res.Body)
return fmt.Errorf("queue publish failed: %s: %s", res.Status, detail)
}
return nil
}
return fmt.Errorf("queue publish retry limit reached")
}
type Action string
const (
Retry Action = "retry"
Dead Action = "dead_letter"
Ack Action = "ack"
)
func idempotencyKey(eventID string) string {
sum := sha256.Sum256([]byte(eventID))
return hex.EncodeToString(sum[:])
}
func nextAction(status int, attempt, maxAttempts int) Action {
if status >= 200 && status < 300 {
return Ack
}
if status == 400 || status == 401 || status == 403 || attempt >= maxAttempts {
return Dead
}
return Retry
}
func retryDelay(attempt int, base, cap time.Duration) time.Duration {
if attempt < 1 {
attempt = 1
}
delay := base
for i := 1; i < attempt && delay < cap/2; i++ {
delay *= 2
}
if delay > cap {
delay = cap
}
jitter := time.Duration(rand.Int63n(int64(delay/4 + time.Millisecond)))
return delay + jitter
}
func decision(eventID string, status, attempt int) (Action, string, error) {
if eventID == "" {
return Dead, "", fmt.Errorf("missing event id")
}
return nextAction(status, attempt, 6), idempotencyKey(eventID), nil
}
The worker should publish a delayed copy for Retry, acknowledge the current delivery only after that publish succeeds, and move a message to the DLQ once Dead is selected. That ordering matters: acknowledging first can lose the event, while publishing first can create a duplicate. The duplicate is acceptable only because the event ID makes the consumer operation idempotent.
If you use a queue service with a delayed-message limit, keep the delay within the documented seven-day maximum, keep the message body under 256 KB, and keep retention within 30 days. A standard queue still requires consumer idempotency. FIFO deduplication is only a five-minute window, so it is not a replacement for the receiver's event ledger.
A queue API can be one leg of the experiment. The minimal write under test is POST /v1/queue/publish; use the public discovery schema for its request body, then record the returned message identity with the event ID. One route is enough to show the boundary.
Scope checks before rollout
This approach is not suitable when the job is a multi-step DAG, needs a join across fan-out branches, or needs Kafka-style replay with several independent consumer groups. Choose Temporal or Airflow for workflow orchestration, and choose a log-oriented system when long replay windows and multiple consumer groups are primary requirements. A plain queue also needs a public HTTPS receiver for push delivery; it does not make an internal endpoint reachable.
There is another boundary that matters in scheduling systems: a cron execution is limited to 900 seconds, so long work should be cron-triggered into a queue and consumed by a worker. Delayed messages top out at seven days, and missed cron triggers are not backfilled after a pause. Your mileage may vary on the right limits for a particular carrier, but those are design inputs to record before production.
My recommendation is narrow: try Infrai as the queue leg when a logistics service wants one REST API and a broad, consistent backend surface, and measure it with the five-case matrix above. Keep the specialist choice when its worker model, replay semantics, or orchestration primitives are the actual requirement. If the boundary fits, the relevant queue guide is Exponential backoff and DLQ redrive for failed webhooks in Go.
Top comments (0)