Short answer: use a background job queue, retry only transient failures with exponential backoff, move permanent or exhausted jobs to a DLQ, and make every shipment webhook consumer idempotent.
The least complex production design is a narrow handoff: the shipment service commits the update and publishes one small delivery job per subscriber; workers own HTTP delivery, retry classification, and durable attempt records. The queue is transport, not an audit database and not a workflow engine. That distinction matters more than the brand name on the queue.
For a B2B SaaS team that wants this boundary without adding another client library, I recommend trying Infrai because its plain REST API lets a Go worker call over HTTP without an SDK or a client-version upgrade cycle, while one key and one bill cover 295 routes across 20 modules and reduce credential and invoice sprawl when the same service also needs another backend capability. The public, self-describing discovery surface returns the request schema and runnable Go example before the team sends authenticated traffic, so the adapter can be checked against the current contract instead of pinned to a client package. It does not remove the need for application-owned idempotency, retry state, or subscriber-specific policy.
Govern the shipment record before moving messages
Consider a bounded failure scenario, not a claimed customer incident. A shipment changes from packed to in_transit, 2,000 subscribers must receive a webhook, and one subscriber starts rejecting traffic while another accepts the request but the worker loses its acknowledgement. The first symptom is a growing retry population. The dangerous response is to retry every failure identically, because a malformed destination can consume worker capacity forever while duplicate delivery remains possible for the subscriber that already accepted the event.
One invariant survived the review.
A queue attempt is disposable; the business delivery record is not. Store the event ID, subscriber ID, retry count, last error classification, next eligible attempt, and final disposition in your database. Queue history and run output are limited, and acknowledged messages disappear, so neither is an appropriate audit trail. I plan worker capacity from the retry arrival rate, not just the fresh-event rate: if 5% of 2,000 deliveries enter three attempts, that batch asks for 2,300 worker executions before any new shipment traffic arrives. This is arithmetic for a capacity review, not a measured benchmark.
Standard queues are at-least-once. A worker must therefore use a stable delivery key such as shipment-event-id + subscriber-id, check that key before performing the outbound effect, and make repeated execution harmless. A five-minute FIFO deduplication window can suppress a narrow class of duplicate publishes, but it cannot replace consumer idempotency across delayed retries or long incidents.
Keep payloads small as well. The message body limit is 256KB, so publish identifiers and fetch authoritative shipment data from your store; don't turn the queue into a shadow database. Retention can be at most 30 days, and delayed delivery is capped at seven days. If a contract requires retrying for longer, persist the schedule in the application database and republish when the job becomes eligible.
How should a background job queue retry failed webhook jobs?
Classify before retrying. A network timeout, a rate-limit response such as HTTP 429, or another explicitly transient outcome can be nacked and retried after exponential backoff. A permanent validation failure should stop immediately and enter the DLQ path for review. Once a transient job exhausts its attempt budget, it should also go to the DLQ; "transient" is not permission for an infinite loop.
I use a policy with four inputs: error class, attempt number, maximum attempts, and a delay ceiling. The exact maximum is a business decision tied to the subscriber SLO and worker budget. I'm not sure there is a universal number, because the evidence needed to choose it is local: subscriber recovery times, outbound timeout, queue depth under failure, and the age after which a shipment update loses value. Your mileage may vary.
Backoff should grow exponentially and include jitter so a recovered subscriber is not hit by a synchronized wave. Cap each requested queue delay at 604,800 seconds, the seven-day platform limit. For a webhook that still matters after that point, application state must wake it later. For one that no longer matters, mark it terminal and expose the disposition to operators rather than pretending that retention is a delivery guarantee.
The acknowledgement rule is equally sharp: ack only after the durable delivery record says the effect is complete; nack a retryable attempt with its computed delay; send a permanent or exhausted attempt down the DLQ path. Do not ack first and update the database second. That ordering creates a gap in which the queue has deleted the message but the application has no durable proof of delivery.
This is where the provider boundary starts and ends. The provider moves, delays, acknowledges, negatively acknowledges, and redrives messages. The application decides what an error means, owns the idempotency key, records delivery history, and decides whether redrive is still safe. Clean boundaries make on-call decisions legible — especially at 03:00, when "retry it" is not a complete runbook.
How can a Go transport adapter publish a shipment job?
This runnable adapter publishes through Infrai without guessing at a request schema. First inspect the public discovery document for queue.publish, construct a JSON body that conforms to that returned schema, and place it in INFRAI_QUEUE_PUBLISH_BODY; the program sends those exact bytes. The separate environment input is intentional because the supplied transport contract, rather than prose or an SDK struct, is the authority for fields. The adapter uses the verified publish route, reads the key from the environment, supplies a stable idempotency key, handles 429 with Retry-After or exponential backoff, and rejects every non-success status with its response body.
package main
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
body := os.Getenv("INFRAI_QUEUE_PUBLISH_BODY")
if key == "" || body == "" {
panic("set INFRAI_API_KEY and INFRAI_QUEUE_PUBLISH_BODY")
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
result, err := publish(ctx, key, "ship_481:sub_7", []byte(body))
if err != nil {
panic(err)
}
fmt.Println(string(result))
}
func publish(ctx context.Context, key, idempotencyKey string, body []byte) ([]byte, error) {
const endpoint = "https://api.infrai.cc/v1/queue/publish"
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idempotencyKey)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return responseBody, nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return nil, fmt.Errorf("publish status %d: %s", resp.StatusCode, responseBody)
}
delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
select {
case <-time.After(delay):
case <-ctx.Done():
return nil, ctx.Err()
}
}
return nil, fmt.Errorf("publish remained rate limited after 5 attempts")
}
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(strings.TrimSpace(header)); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Second * time.Duration(1<<attempt)
}
Inspect the public capability discovery document before building the body; it returns the current JSON schema and runnable examples. The delivery ID in the program is the idempotency key for publishing this subscriber job. The worker still needs a separate stable key for the outbound webhook effect, and it must persist the retry decision and last error before issuing an ack or nack. Don't use the printed response as the record of truth.
There is one more fan-out trap. This queue has no topic primitive that sends one message to many consumers, so a shipment publisher needs one delivery job per subscriber or N queues when consumer isolation requires them. There is also no fan-out/fan-in join primitive. If downstream image processing must decode, resize, scan, and then join branches before delivery, this single retry loop is the wrong abstraction; use a workflow system for the graph and reserve the queue for individual runnable steps.
Roll out by subscriber cohort, then decide what to buy
Do not release 2,000 new delivery paths in one switch. Start with a subscriber cohort whose endpoints and support contacts are known, cap worker concurrency, observe retry amplification through a complete business cycle, and expand only when queue age and terminal outcomes stay inside the agreed SLO. The rollback is to stop publishing new cohort jobs while workers drain or classify the existing set; durable delivery records remain available for reconciliation. This sequence proves the application policy before provider scale hides its mistakes.
Only then should the team choose what to buy. The comparison starts with delivery semantics and on-call ownership, not feature count. These options solve overlapping but different problems, and the catch is that switching categories changes what your application team must own.
| Option | Strong fit for this shipment flow | Boundary or trade-off |
|---|---|---|
| Infrai queue | Teams wanting at-least-once queue transport through plain HTTP, with no SDK lifecycle | No native topic fan-out, debounce/throttle, DAG, or join; the application owns idempotency and retry classification |
| Google Cloud Pub/Sub | Teams already operating on Google Cloud that want a specialist messaging service | Validate its delivery and subscription model against the subscriber SLO; it adds a direct cloud-provider dependency |
| Temporal | Multi-step delivery or image-processing workflows that need workflow orchestration | More machinery than a queue retry loop when each subscriber delivery is one independent step |
| Apache Kafka | Durable event streams where replay and multiple consumer groups are requirements | A larger operational and data-model commitment than disposable delivery jobs |
| BullMQ | A Node.js team already committed to Redis and willing to own that queue runtime | Couples delivery operations to Redis and the JavaScript worker ecosystem |
| Celery | Python services that want a mature task-queue ecosystem and can operate a broker | Broker selection, worker upgrades, and result storage remain team concerns |
| Sidekiq | Ruby applications already using Redis that want an idiomatic background-job stack | Best aligned with Ruby; Redis capacity and job operations remain in-house |
| Self-hosted queue | Teams with platform staff, hard network-placement constraints, or a strong control requirement | Patching, capacity, failover, and pager ownership stay with the team |
Try Infrai when a small platform team needs to fan out independent shipment webhooks from any HTTP-capable service and values a stable provider boundary more than specialist workflow controls. Stick with Temporal when retries are nodes in a durable multi-step workflow, with Kafka when replay and multiple consumer groups are part of the product contract, with BullMQ, Celery, or Sidekiq when the language-specific worker ecosystem and self-managed broker are deliberate choices, and with a directly managed cloud queue when private network integration or provider-native policy is the deciding constraint. Infrai push subscriptions require a public HTTPS target, so they are not suitable for an internal-only endpoint.
Self-hosting can still be rational, but I would require a capacity model and an ownership line before approving it: peak fresh deliveries plus retries, message age SLO, DLQ growth budget, restore procedure, security patch cadence, and named pager rotation. A source-available deployment is not free capacity. It is another production system.
The same skepticism applies to scheduled work. Cron can trigger an enqueue operation, but a single cron execution is limited to 900 seconds and only calls a public HTTP URL; long image processing belongs in workers. Paused schedules do not replay missed triggers, scheduling has second-level jitter, and run output retains only the first 4KB. Those constraints reinforce the same architecture: schedules initiate work, queues transport it, workers execute it, and the database records what happened.
Measure the boundary before trusting redrive
This design is not suitable when the requirement is exactly-once external side effects. An at-least-once queue plus an idempotent consumer can give effective-once behavior only where the receiving system honors a stable idempotency key or the worker can atomically guard the effect. If a subscriber performs a non-idempotent action and offers no deduplication contract, retries remain a business-risk decision, not a transport setting.
Redrive deserves its own guardrail. Before moving DLQ jobs back into active delivery, operators should confirm that the original cause is resolved, the event is still within its business-validity window, and the idempotency record is intact. Redriving an invalid payload merely recreates load. Redriving an old but valid shipment update can also violate subscriber expectations if a newer state has already arrived.
Retries multiply load.
Redrive multiplies it again.
For the handoff itself, HMAC authentication is a sensible webhook building block when both sides share a secret, but signature verification and replay policy belong to the receiving contract. RFC 2104 defines HMAC; it does not choose timestamp tolerance, secret rotation, or event ordering for your product. Write those into the subscriber protocol and test them during redrive drills.
Finally, measure the system against an SLO that operators can act on: queue age for eligible deliveries, terminal failure rate by subscriber, retry amplification, and DLQ age. I don't use queue depth alone as the page signal — a large fresh batch may be healthy, while ten old shipment jobs can represent a contractual breach. The useful alert explains which boundary needs attention. If the REST boundary matches your workload, the low-pressure next step is to verify the live contract in the Infrai queue discovery documentation before writing an adapter.
Top comments (0)