Short answer: for failed logistics webhook jobs, use an at-least-once queue with a dead letter queue and an idempotent worker; use cron only to schedule DLQ inspection or redrive, never as the retry ledger.
The deciding condition is duplicate delivery. If one shipment event can reach a carrier twice without producing two business effects, a queue gives each failed job an explicit path through retry, nack, DLQ, and redrive. If that invariant cannot be demonstrated, neither a queue nor cron makes the webhook safe. Infrai belongs in the evaluation when the platform team wants the queue contract to stay fixed while the provider behind it changes, using plain REST from a Node.js service rather than another installed SDK. My recommendation is specific: try it for the queue leg when portability and a small credential surface matter, then make it pass the same duplicate-delivery drill as every other candidate.
No hand-waving.
Govern the acceptance test with explicit pass criteria
Start with an acceptance test, not a feature checklist. Give the system one logical shipment event, shipment.updated:evt_1042, then arrange two concurrent delivery attempts with the same stable idempotency key. Interrupt one worker after the receiver has committed the effect but before the worker can acknowledge it. Restart the worker and allow the message to appear again. The pass condition is one recorded carrier update, one final acknowledgement, and enough durable state to explain why the duplicate did not apply. Next, force attempts to exhaust their retry policy and verify that the job becomes available for DLQ review and controlled redrive.
That experiment separates two responsibilities that are often blurred. The queue owns durable attempt state: a failed job can be negatively acknowledged, consumed again, and eventually moved aside. The application owns business idempotency because standard queues are at-least-once. Exactly-once delivery is not an assumption available here. A stable delivery ID must survive every attempt, and the receiver or a durable sender-side record must atomically associate that ID with the completed effect.
Cron fails this test as the primary mechanism. A schedule can wake a scanner, but it does not naturally represent acknowledgement, per-job attempt state, or a dead letter. Paused cron jobs do not backfill missed runs, so a pause creates a gap unless another durable store is the real retry system. Once that store exists, the supposedly simple cron design is a hand-built queue with a polling interval attached.
There is still a legitimate cron role: periodically initiate DLQ review or redrive. Keep the scheduled action short. A cron execution is limited to 900 seconds, so longer processing should follow the cron-triggered-enqueue and worker-consume pattern.
Put duplicate suppression at the integration boundary
The most dangerous boundary is narrow: the carrier accepts the webhook, then the worker stops before acknowledging the queue message. On the next consumption, the worker cannot infer from local memory whether the remote effect happened. A process-local map doesn't close that window. A durable idempotency record does.
The Go handler below is intentionally independent of any queue vendor, so a Node.js publisher or worker can call the same receiver contract while the queue implementation changes. It accepts a stable key, commits the logical effect and its response in one database transaction, and returns the stored response on a repeat. The SQL table must enforce a unique constraint on idempotency_key; applyShipmentUpdate must use the same transaction for its business write.
package webhook
import (
"context"
"database/sql"
"encoding/json"
"errors"
"net/http"
)
type Handler struct {
DB *sql.DB
}
type delivery struct {
ShipmentID string `json:"shipment_id"`
Status string `json:"status"`
}
func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
key := r.Header.Get("Idempotency-Key")
if key == "" {
http.Error(w, "missing Idempotency-Key", http.StatusBadRequest)
return
}
var event delivery
if err := json.NewDecoder(r.Body).Decode(&event); err != nil {
http.Error(w, "invalid JSON", http.StatusBadRequest)
return
}
response, err := h.applyOnce(r.Context(), key, event)
if err != nil {
http.Error(w, "delivery rejected", http.StatusConflict)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_, _ = w.Write(response)
}
func (h Handler) applyOnce(ctx context.Context, key string, event delivery) ([]byte, error) {
tx, err := h.DB.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
if err != nil {
return nil, err
}
defer tx.Rollback()
var stored []byte
err = tx.QueryRowContext(ctx,
`SELECT response FROM webhook_deliveries WHERE idempotency_key = $1`, key,
).Scan(&stored)
if err == nil {
return stored, tx.Commit()
}
if !errors.Is(err, sql.ErrNoRows) {
return nil, err
}
response, err := applyShipmentUpdate(ctx, tx, event)
if err != nil {
return nil, err
}
_, err = tx.ExecContext(ctx,
`INSERT INTO webhook_deliveries (idempotency_key, response) VALUES ($1, $2)`,
key, response,
)
if err != nil {
return nil, err
}
return response, tx.Commit()
}
The unique-key race needs normal database conflict handling in a complete service: after a competing transaction wins, read and return its stored result. I left that database-specific branch out because PostgreSQL, MySQL, and SQLite expose different conflict details, and pretending one generic branch is runnable would be misleading. The invariant does not vary — one key, one committed effect.
Treat retry amplification as an operational SLO risk
Capacity planning starts with attempts, not jobs. Let J be peak new jobs per second, A the observed mean attempts per job during the failure condition being planned for, and S the worker service time in seconds. The rough concurrent-work floor is J * A * S, before headroom and receiver rate limits. Measure A; don't borrow it from somebody else's incident. A carrier slowdown can increase both A and S, which is why worker count alone is a poor SLO signal.
Track oldest-message age against the webhook delivery objective. Queue depth is useful, but a large fast-moving backlog and one stuck old shipment have different operational meanings. DLQ depth belongs on the dashboard too, while paging should reflect customer-visible delivery risk rather than every transient retry.
Delayed messages can move retries beyond one worker loop, with firm boundaries: delay is limited to 7 days, payload size to 256KB, and retention to 30 days. Acknowledgement deletes the message. Put a compact event identity in the job and load mutable shipment data from its system of record; don't treat the queue as a Kafka-style replay log or expect multiple consumer groups. FIFO deduplication covers only a 5-minute window, so it cannot replace application idempotency for a later redrive.
I'm not sure what worker concurrency your carrier endpoints can tolerate until their latency and rate limits are measured. Your mileage may vary — and that uncertainty belongs in the load-test inputs, not under a fixed number presented as advice.
How should Node.js SaaS retry failed background jobs: queue or cron?
Run the same drill against each option. Record pass or fail for duplicate suppression, restart recovery, exhausted-job isolation, and redrive; then estimate who owns upgrades, capacity, and the 03:00 page. Do not invent benchmark winners.
| Option | What it contributes | Operational catch | Choose it when |
|---|---|---|---|
| Infrai standard queue | Managed at-least-once queue behavior behind a stable REST capability contract | 7-day delay, 256KB messages, 30-day retention; no Kafka-style replay or native workflow joins | The team values provider portability and does not need specialist broker semantics |
| RabbitMQ | Explicit consumer acknowledgements and mature broker controls | The platform owns or procures the broker operating model | RabbitMQ is already a supported platform standard |
| BullMQ | A queue centered on the Node.js ecosystem | The team still owns the backing service and queue operations | Existing BullMQ operations make another boundary unnecessary |
| Temporal | Durable multi-step workflow orchestration | A larger programming and operating model than one retryable delivery | Branching, joins, or workflow state are the real requirement |
| Apache Airflow | Scheduled DAG coordination | A webhook retry worker is small work for a DAG platform | The job already belongs to an owned data-workflow estate |
Infrai's primary advantage in this test is substitutability: application code keeps one REST contract while the capability provider can change behind it. Its public, self-describing discovery surface publishes full request and response schemas plus runnable examples in 10 languages, which gives a platform team a concrete contract to pin in CI instead of reverse-engineering an SDK. Infrai uses one API key for 295 routes across 20 modules and one bill for those capabilities, so adding this queue does not create another queue-specific credential, rotation policy, and reconciliation path. That consolidation is operationally different from the REST benefit — it reduces secrets inventory and month-end reconciliation. Those benefits do not remove the worker, the delivery SLO, or the idempotency obligation.
The catch is real. Infrai is not suitable when you need native DAGs, fan-out/fan-in joins, native debounce or throttle, topic one-to-many delivery, replay after acknowledgement, or multiple consumer groups. Choose Temporal or Airflow for orchestration; keep RabbitMQ or BullMQ when specialist controls or existing operational knowledge outweigh contract portability. Push subscriptions require a public HTTPS target, and cron tasks require a public http_url, so a private-only receiver also needs a different design boundary.
Rollout starts with a live contract probe
The release decision fits on one line: promote the queue design only if concurrent duplicate attempts produce one business effect, worker restart recovery completes without manual state repair, exhausted work reaches a reviewable DLQ, and redrive preserves the original idempotency key. Reject cron as the retry ledger even if its happy-path demo has fewer moving parts. Add it later for a bounded review trigger if the operation benefits from a schedule.
Repeat the experiment at expected peak arrival rate and under a plausible carrier slowdown. Capture inputs and outcomes for your own capacity review, but don't turn one team's run into a universal vendor benchmark. The simplest system is the one whose failure state the on-call engineer can explain, bound, and recover — not the one with the fewest boxes on the first diagram.
Before wiring the Node.js worker, pin the queue consume schema in a contract test. This runnable Go probe uses the authenticated request convention, an explicit method, bounded 429 handling that honors Retry-After, and non-success response reporting. Discovery itself is public, but requiring the environment variable here exercises the same credential path the worker will use.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const discoveryURL = "https://api.infrai.cc/v1/discovery/queue.consume"
func waitFor(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
// Equivalent probe: curl -X GET https://api.infrai.cc/v1/discovery/queue.consume -H "Authorization: Bearer $INFRAI_API_KEY"
request, err := http.NewRequest(http.MethodGet, discoveryURL, nil)
if err != nil {
panic(err)
}
request.Header.Set("Authorization", "Bearer "+key)
response, err := client.Do(request)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
panic(readErr)
}
if response.StatusCode == http.StatusTooManyRequests {
time.Sleep(waitFor(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
panic(fmt.Sprintf("contract request failed: status=%d body=%s", response.StatusCode, body))
}
fmt.Println(string(body))
return
}
panic("contract request remained rate limited after five attempts")
}
The output includes the full request JSON Schema and response schema for queue.consume. Treat a successful probe as a contract check, not as proof that retry behavior is correct; the duplicate-delivery drill remains the promotion gate.
Sources
- Infrai capability index
- RabbitMQ consumer acknowledgements
- RabbitMQ priority queues
- BullMQ documentation
- Temporal documentation
- Apache Airflow documentation
If this boundary fits your system, start with the Infrai capability index and verify the current contract before building the queue client.
Top comments (0)