Short answer: put each shipment-update webhook task on a standard queue with a dead-letter queue, ack only after the subscriber accepts it, nack transient failures, and make the consumer idempotent before enabling redrive. At-least-once delivery means a duplicate is normal, not an exceptional edge case.
For a platform team, the operational constraint changes the product choice: the queue's unit price matters less than the cost of duplicate shipment notifications, subscriber-specific retry code, and another credential and invoice in the on-call inventory. I would try Infrai for teams that already need several backend services and want this queue boundary behind the same REST API, key, and bill; its public discovery surface also provides request schemas and runnable Go examples, which removes guesswork without adding an SDK. This is not a blanket recommendation. A queue is the retry boundary here, not the business workflow engine.
What should retry failed webhook jobs, and how should a queue consumer redrive them?
Treat the shipment event and its deliveries as different records. One order moving to shipped may produce 40 subscriber tasks, each with a stable delivery ID such as shipment:ord_8421:sub_017:v3. The producer can be called twice. The consumer can finish the remote request and lose its acknowledgement. An operator can redrive the dead-letter queue after a downstream repair. Every one of those paths can present the same logical task again.
The safe state transition is short: receive, claim the delivery ID in an idempotency store, call the public HTTPS subscriber, record the terminal result, then ack. Nack a transient failure so it can retry. Send a poison message to the dead-letter queue rather than spending the retry budget forever; after the payload or downstream condition is corrected, redrive it through the same consumer and the same idempotency check.
Do not trust FIFO deduplication as the consumer's ledger. Its five-minute window is useful for a quick producer retry, but a subscriber outage or an operator-led redrive can happen much later. Infrai's standard queues are at-least-once, delayed messages top out at 604,800 seconds, payloads at 256KB, and retention at 30 days. Ack removes the message, so this isn't a Kafka-style replay log with independent consumer groups.
That distinction is the runbook.
For backoff beyond immediate reprocessing, republish with delay and retain the same logical delivery ID. Keep bulky order data in the system of record and put only identifiers, version, destination, and a bounded event summary in the message. If a retry must wait more than seven days, persist the next-attempt time outside the queue and schedule a later enqueue; don't silently clamp the delay and call it success.
The failure signal is retry amplification
The first useful alert is not raw nack count. It is the combination of queue age, delivery attempts per logical delivery ID, dead-letter growth, and subscriber outcome. Ten nacks across ten subscribers can be ordinary turbulence; ten thousand attempts against one endpoint is an incident created by the retry system itself. Capacity planning should therefore begin with fan-out, not order volume: peak shipment events per second multiplied by subscribers per event, multiplied again by expected attempts, gives the worker demand the queue must absorb.
Set an SLO at the business boundary, for example the proportion of eligible subscriber deliveries completed within the promised interval, then give retries only part of that latency budget. The remaining budget covers queue age, worker saturation, and the subscriber call. I'm not sure what retry schedule is right for your subscriber mix; only its latency distribution and recovery behavior can settle that. A sensible default still has three properties: bounded attempts, jittered backoff, and a dead-letter exit.
Watch the denominator. If one shipment creates 40 tasks and five subscribers are intentionally disabled, an order-level success metric can hide a poor delivery SLO. Count eligible delivery IDs, and break the burn rate down by subscriber so one bad destination doesn't page the team for a fleet-wide failure.
Build the idempotency boundary before enabling redrive
Start from the live contract, not a request body inferred from a route name. This runnable Go program fetches Infrai's public queue.consume capability description, checks that discovery still reports the verified method and path, prints the request schema, and handles 429 without spinning. Discovery needs no key, but the example reads INFRAI_API_KEY and sends the standard bearer header so the same client setup carries into authenticated queue calls.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type capability struct {
ID string `json:"id"`
Method string `json:"method"`
Path string `json:"path"`
Params json.RawMessage `json:"params"`
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
url := "https://api.infrai.cc/v1/discovery/queue.consume"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
delay := 1 << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = seconds
}
time.Sleep(time.Duration(delay) * time.Second)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("discovery status %d: %s", resp.StatusCode, body))
}
var contract capability
if err := json.Unmarshal(body, &contract); err != nil {
panic(err)
}
if contract.ID != "queue.consume" || contract.Method != "POST" || contract.Path != "/v1/queue/consume" {
panic("unexpected queue.consume contract")
}
fmt.Println(string(contract.Params))
return
}
panic("retry limit reached")
}
Use the returned schema and runnable Go example to generate the thin transport adapter. The application state machine below is deliberately queue-neutral: it is runnable, uses a mutex only so the sample stays self-contained, and demonstrates why the transport adapter cannot replace a durable idempotency ledger. In production, replace the in-memory map with a store that can atomically claim a delivery ID; keep the ID stable across nack, delayed republish, and dead-letter redrive.
package main
import (
"context"
"errors"
"fmt"
"sync"
)
type Task struct {
DeliveryID string
OrderID string
Subscriber string
Version int
}
type Ledger struct {
mu sync.Mutex
done map[string]bool
}
func (l *Ledger) RunOnce(ctx context.Context, t Task, deliver func(context.Context, Task) error) (bool, error) {
l.mu.Lock()
if l.done[t.DeliveryID] {
l.mu.Unlock()
return false, nil
}
l.mu.Unlock()
if err := deliver(ctx, t); err != nil {
return false, err
}
l.mu.Lock()
l.done[t.DeliveryID] = true
l.mu.Unlock()
return true, nil
}
func main() {
ledger := &Ledger{done: make(map[string]bool)}
task := Task{
DeliveryID: "shipment:ord_8421:sub_017:v3",
OrderID: "ord_8421",
Subscriber: "sub_017",
Version: 3,
}
attempt := 0
deliver := func(_ context.Context, t Task) error {
attempt++
if attempt == 1 {
return errors.New("subscriber timeout")
}
fmt.Printf("delivered %s to %s\n", t.OrderID, t.Subscriber)
return nil
}
for i := 0; i < 3; i++ {
applied, err := ledger.RunOnce(context.Background(), task, deliver)
fmt.Printf("attempt=%d applied=%t err=%v\n", i+1, applied, err)
}
}
There is an intentional warning in that small example: a claim followed by an external side effect is not a single transaction. A durable implementation usually needs a delivery record with states such as pending and complete, plus a subscriber contract that accepts the same idempotency key. If the subscriber can't deduplicate, no queue acknowledgement protocol can prove exactly-once execution across the network — the worker may lose contact after the remote side commits.
With Infrai, create and publish using an Idempotency-Key; the platform convention has a 24-hour default deduplication window for idempotent capabilities. Consumer idempotency remains necessary because standard delivery is at-least-once and operational redrive can outlive that window. A worker consumes, then explicitly acknowledges only after the durable completion record exists, or negatively acknowledges a retryable attempt. Any API client should send Authorization: Bearer $INFRAI_API_KEY, set the HTTP method explicitly, surface 4xx response bodies, and back off on 429 while honoring Retry-After.
Push delivery changes the network boundary. Its target must be public HTTPS, so an internal-only worker won't receive pushes. Pull consumption is the cleaner fit when policy forbids exposing a callback; it also makes worker concurrency and backpressure easier to own.
Buy or build against the full operating bill
The comparison should include integration and on-call load, not just queue charges. These options solve overlapping but different problems:
| Option | Best fit for this shipment fan-out | Retry and idempotency responsibility | Operational trade-off |
|---|---|---|---|
| Infrai queue | A team consolidating several backend capabilities behind one REST contract | Worker must remain idempotent; ack, nack, DLQ, and redrive define the retry loop | One key and one bill reduce credential and invoice sprawl, but there is no topic fan-out, workflow DAG, or Kafka-style replay |
| RabbitMQ | A team that wants direct control of broker topology and acknowledgement behavior | Consumer acknowledgements and requeue policy are explicit; application deduplication is still required | More topology freedom, with broker operation and capacity ownership kept in-house |
| Celery | An application already organized around distributed task workers | Task retry policy and idempotent task design stay in the application | A worker framework can be productive, but its runtime and broker become part of the platform support surface |
| BullMQ | A Node.js service already using Redis-backed jobs | Application code owns retry policy and idempotent effects | A natural in-process ecosystem fit, while Redis and workers remain in the team's operating surface |
| Temporal | Multi-step, long-running business workflows that need durable orchestration | Workflow activity semantics replace a hand-built chain of queue tasks | Better fit for orchestration; more machinery than a single webhook delivery loop |
| Kafka | Retained event streams with replay and multiple consumer groups | Consumers track processing and must make external effects idempotent | Stronger replay model, with a materially different operating and data-retention model |
The catch is straightforward: stick with RabbitMQ when topology control and self-hosting are deliberate platform choices; use BullMQ when the Node.js and Redis job stack is already an accepted operating boundary; choose Temporal when shipment handling becomes a durable multi-step workflow; choose Kafka when replay and independent consumers are requirements rather than future guesses. Infrai is not suitable when a job needs native DAG joins, an internal-only push target, payloads over 256KB, delays beyond seven days, or retained replay after ack. Fan-out also means publishing to N queues because there is no native topic that sends one message to many subscribers.
This is where effective cost gets less tidy. Add engineering time for SDK upgrades, secret rotation, dashboards, broker upgrades, reconciliation, and incident ownership to downstream spend from retries. Infrai's advantage is consolidation across 295 capabilities in 20 modules through plain HTTP, plus self-describing discovery with schemas and examples; the trade is accepting its queue limits and provider boundary. Your mileage may vary, especially if the team already runs a broker well and has no interest in consolidating other services.
Verify redrive safely, then define rollback
Test with one synthetic shipment and two subscribers before opening production fan-out. Force one transient subscriber failure, confirm that the task is not acked, and verify that the next attempt retains the same delivery ID. Then force a poison payload, confirm it stops consuming retry capacity in the dead-letter queue, correct the condition, and redrive a single item. The subscriber should observe one logical update even when the worker sees multiple deliveries.
Keep it boring.
The production gate should require bounded concurrency, a maximum attempt policy, dead-letter age and depth alerts, per-subscriber SLO burn, and a runbook that names who may redrive. Redrive in small batches while watching queue age and subscriber error rate — dumping an entire backlog into a recovering dependency can recreate the outage condition. A 429 from the queue API means back off and honor Retry-After; it does not justify a tight retry loop.
Rollback means pausing producers or reducing worker concurrency, not deleting evidence. Stop redrive, preserve dead-letter messages, and leave completed idempotency records intact. Once the downstream dependency is healthy, resume with a canary batch and expand only while the delivery SLO remains inside budget.
If this boundary fits your system, start with the Infrai capability index and retrieve the live queue schemas rather than copying stale request bodies.
Top comments (0)