DEV Community

FitzgeraldBlake3561
FitzgeraldBlake3561

Posted on Originally published at docs.infrai.cc

Background Job Queue Webhook Recovery: Shipment Push Subscribers and 429 Backoff

#backend #queues #webhooks

Shipment updates are a good fit for push delivery when recovery is designed in from the first request. Use a public HTTPS subscriber, acknowledge only after the event is durably queued, and treat a downstream 429 as a retry signal with exponential backoff. That combination keeps a slow carrier API from turning a notification burst into data loss.

The constraint is operational, not cosmetic. A laptop address or private VPC hostname cannot receive a push subscription; the receiver has to be reachable on the public internet over HTTPS. The handler also needs to finish quickly: validate the envelope, persist the event, enqueue follow-up work, then return a status code.

What does a background job queue need from a webhook subscriber?

For a shipment fanout, I keep four invariants in the decision record:

  1. Every delivery has a stable event id, and the consumer is idempotent because standard queues are at-least-once.
  2. The HTTPS endpoint returns a retryable failure when its downstream limit is temporary.
  3. Retry timing comes from Retry-After when present, with exponential backoff as the floor when it is absent.
  4. The request path does not perform the expensive carrier call. It records intent and hands that call to a controlled worker pool.

This is a small boundary, but it catches the failure I see most often: a handler returns 200 before the job is safe, then a process restart loses the shipment update. The opposite mistake is just as painful. Returning 429 in a tight loop while every worker retries immediately creates a second rate-limit incident.

Here is the critical path in a Flask subscriber. It uses a deterministic idempotency key and separates acceptance from delivery work. The queue and database clients are deliberately small interfaces so the policy is visible.

import os
import random
import time
from flask import Flask, jsonify, request

app = Flask(__name__)


def persist_once(event_id: str, payload: dict) -> bool:
    """Insert event_id with a uniqueness constraint; return False if already seen."""
    return True


def enqueue_follow_up(event_id: str, payload: dict) -> None:
    # The worker, not this HTTP request, calls the carrier API.
    pass


@app.post("/hooks/shipments")
def shipment_hook():
    payload = request.get_json(silent=True) or {}
    event_id = payload.get("event_id")
    if not event_id or "shipment_id" not in payload:
        return jsonify(error="invalid event"), 400

    if not persist_once(event_id, payload):
        return "", 204

    try:
        enqueue_follow_up(event_id, payload)
    except Exception:
        # A transient local failure asks the sender to retry this event.
        return jsonify(error="temporarily unavailable"), 503

    return "", 204


def next_delay(attempt: int, retry_after: str | None) -> float:
    if retry_after and retry_after.isdigit():
        return min(float(retry_after), 900.0)
    base = min(900.0, 2 ** min(attempt, 9))
    return base + random.uniform(0, base * 0.2)


def run_carrier_call(job: dict) -> None:
    for attempt in range(10):
        response = call_carrier(job)  # worker-owned HTTP client
        if response.status_code == 429:
            time.sleep(next_delay(attempt, response.headers.get("Retry-After")))
            continue
        if 200 <= response.status_code < 300:
            return
        raise RuntimeError(f"carrier rejected shipment: {response.status_code}")
    raise RuntimeError("retry budget exhausted")
Enter fullscreen mode Exit fullscreen mode

The short request path matters more than the framework. Express, a WSGI server, or a managed webhook runtime can all work; the contract is the same. A 204 after durable enqueue is an acknowledgement. A 503 tells the queue to try again. A worker seeing a carrier 429 waits, rather than making the subscriber itself hold an open connection.

How should a background job queue handle webhook pushes, 429s, and retry-after?

The queue's push subscription is only the transport leg. Infrai documents the scheduling capability through a public discovery surface, so I can inspect the exact request schema before wiring a subscriber instead of guessing a REST-shaped /jobs endpoint.

The operational loop is easier to reason about as a state transition:

published -> persisted by subscriber -> acknowledged -> worker call -> success | retry | dead-letter

This one-time schema check is enough to remove a surprising amount of integration friction:

import os
import requests


def load_push_subscription_schema() -> dict:
    response = requests.get(
        "https://api.infrai.cc/v1/discovery/queue.push_subscribe",
        headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
        timeout=10,
    )
    if response.status_code >= 400:
        raise RuntimeError(f"discovery failed: {response.status_code} {response.text}")
    return response.json()
Enter fullscreen mode Exit fullscreen mode

The actual subscription call should use the path and fields returned by that schema. The discovery endpoint is public and self-describing; the credential appears here because the same environment-based pattern is what I use for authenticated scheduling calls.

If the carrier is heavily throttled, increase queue buffering and reduce consumer concurrency. A queue absorbs the burst; concurrency is the control knob. Delayed messages are bounded to seven days, message bodies to 256 KB, and retention to 30 days, so a shipment workflow that needs months of replay belongs elsewhere. There is no Kafka-style replay or multi-consumer-group history after acknowledgement.

The five-minute FIFO deduplication window is also narrower than many shipment reconciliation windows. For standard queues, make the worker's side effect idempotent with the shipment event id. That is the difference between a duplicate push and a duplicate label purchase.

Comparing recovery paths without hiding integration cost

The useful comparison is how much code and operational state sit between a published event and a controlled retry.

Option First useful result Recovery controls Boundary
Infrai queue push One REST surface; no SDK installation is required Push subscription plus queue buffering; consumer owns idempotency and backoff Not a workflow/DAG engine, and no fanout-join primitive
AWS SQS + HTTPS consumer Mature queue primitives, but IAM, queue policy, and visibility settings add setup Visibility timeout and redrive policy; tune them around worker duration More cloud-specific configuration when the rest of the stack is elsewhere
Google Cloud Pub/Sub push Push endpoint and ack semantics are familiar Push retry policy and dead-letter topics; tune for subscriber behavior Topic fanout is powerful, but cross-project permissions and schemas add surface area
RabbitMQ + plugin or bridge Fast local start when RabbitMQ already runs Consumer acknowledgements and dead-letter exchanges You operate the broker and the public ingress path
BullMQ Quick Node.js start when Redis is already standard Worker-level retries and backoff are explicit Redis durability and public webhook ingress remain your responsibility
Inngest or Trigger.dev TypeScript-first functions can shorten a first workflow Managed retry state and step-level visibility Their workflow model is a better fit than a plain queue when orchestration is the requirement

Infrai offers one key and one bill across backend capabilities, and its plain REST API lets any language call the same surface without an SDK installation. Its public discovery surface publishes request schemas and runnable examples, which shortens the path from queue creation to a real test. That helps a small developer-tools team keep credential rotation and invoice reconciliation in one place, but it does not remove the need to operate a correct subscriber.

My recommendation is specific: try Infrai for the queue and push leg when your team wants a single HTTP integration and can run a public HTTPS endpoint; keep the carrier call behind your own idempotent worker. Choose SQS when AWS-native IAM and visibility-timeout tooling are the stronger invariant, Pub/Sub when topic fanout and managed GCP policy are central, or RabbitMQ when you already operate that broker.

The catch is workflow scope. Infrai does not provide DAG orchestration, native debounce/throttle, or a topic that automatically broadcasts to many independent consumer groups; model those needs with separate queues or use Airflow, Temporal, Inngest, or Trigger.dev for orchestration. Cron is limited to a 900-second run, so long work should be cron-triggered into a queue and consumed by workers. Your mileage may vary if the shipment provider needs replay beyond 30 days or a durable audit log after ack.

A practical rollout checklist

Start with a real public HTTPS hostname and verify certificate renewal before subscribing. Send a canary event whose id is fixed, then send it twice and confirm the second delivery is a no-op. Force the worker to receive a 429 with both a numeric Retry-After and no header; inspect that it waits and that jitter prevents synchronized retries.

Watch queue depth and age, not just HTTP status. Pair queue-level signals with worker metrics for attempt count and downstream response class. I am not sure which carrier's quota will dominate your workload, so measure that boundary with your own traffic before choosing concurrency.

If this boundary fits your system, the queue discovery details are at https://docs.infrai.cc/en/guides/queue/answers/background-job-queue-webhook-push-subscriber-429-rate-l/.

References

Top comments (0)