DEV Community

ethanbrooks1486
ethanbrooks1486

Posted on

Shipment Dispatch Control: Background Job Queue Backoff for 429 Rate-Limited Webhook Push

Short answer: acknowledge each shipment event only after it is durably queued, then deliver one job per subscriber with an idempotency key; on 429, honor Retry-After and retry that subscriber without replaying successful deliveries. The deciding constraint is fan-out. One slow fan community endpoint must not stall every other recipient.

A public HTTPS endpoint is necessary for a push consumer, but it is not the place to perform the whole broadcast. Keep that request path boring: authenticate, validate, persist, acknowledge. The queue owns delivery time. The delivery record owns duplicate suppression.

This is a retry problem, not an Express routing trick.

Delivery reliability and the failure window

Consider a game studio publishing shipment.dispatched after a collector's edition leaves the warehouse. Fan sites, tournament organizers, community bots, and regional storefronts subscribe to the update. The source event has one ID, while delivery state has one row per (eventId, subscriberId). That distinction matters because subscriber A may return 204, subscriber B may return 429, and subscriber C may time out. Requeueing the source event as one indivisible job makes A receive a duplicate whenever B needs another attempt.

The useful state machine is small: pending, leased, delivered, or retry_at. A worker leases one delivery, sends it, and commits the response outcome. If the worker dies after the remote endpoint accepts the request but before the commit, the queue may redeliver it. Exactly-once network delivery isn't the promise to make. Stable identity is. Send an idempotency key such as shipmentEventId:subscriberId, and require the receiver to store the first successful result for that key.

This also changes what gets measured. Track queue age, attempts per delivery, time until the next eligible retry, and terminal outcomes by subscriber. Don't collapse those into one global success rate — a healthy aggregate can hide a single subscriber that has been throttled for hours. I would benchmark the acknowledgment path and the delivery workers separately, with bursts large enough to reach the configured subscriber limit. I'm not sure which burst shape matches every game launch; production arrival data resolves that, not a prettier backoff formula.

How should a background job queue retry a 429 webhook subscriber?

429 Too Many Requests means the receiver is rate-limiting the request. RFC 6585 defines the status, and HTTP semantics define Retry-After as either seconds or an HTTP date. Parse both forms. If the header is valid, wait at least that long. If it is absent or invalid, exponential backoff with random jitter prevents synchronized workers from striking the endpoint again at the same instant. Cap both attempts and delay so a bad subscription cannot consume unbounded queue capacity.

The catch is that Retry-After is advice about timing, not proof that a replay is harmless. Idempotency still has to work. A subscriber can process a request and lose its response, leaving the sender with a timeout and no knowledge of the remote commit. Retry the same delivery ID and payload; never mint a fresh identity for each attempt.

Avoid sleeping inside a worker. Persist nextAttemptAt, release the lease, and let the scheduler make the job eligible later. Sleeping holds concurrency while doing no work, which becomes config bloat in disguise: more worker processes, more connection pools, more knobs.

No thanks.

Implement the durable acceptance path

The HTTP handler below delegates durability and uniqueness to a repository. enqueueOnce must enforce a unique constraint on (eventId, subscriberId) in the database. The handler returns only after those writes commit. An in-memory array would make the example shorter, but it would erase the exact property the design depends on.

import express, { Request, Response } from "express";
import { createHash } from "node:crypto";

type ShipmentEvent = {
  id: string;
  type: "shipment.dispatched";
  shipmentId: string;
  carrier: string;
};

type Subscriber = { id: string; url: string };

interface DeliveryRepository {
  enqueueOnce(event: ShipmentEvent, subscriber: Subscriber): Promise<void>;
}

declare const deliveries: DeliveryRepository;
declare function subscribersFor(type: ShipmentEvent["type"]): Promise<Subscriber[]>;

const app = express();
app.use(express.json({ limit: "64kb" }));

app.post("/shipment-events", async (req: Request, res: Response) => {
  const event = req.body as Partial<ShipmentEvent>;
  if (event.type !== "shipment.dispatched" || !event.id || !event.shipmentId || !event.carrier) {
    res.status(400).json({ error: "invalid shipment event" });
    return;
  }

  const validEvent = event as ShipmentEvent;
  const subscribers = await subscribersFor(validEvent.type);
  await Promise.all(subscribers.map((subscriber) =>
    deliveries.enqueueOnce(validEvent, subscriber)
  ));

  const receipt = createHash("sha256").update(validEvent.id).digest("hex");
  res.status(202).json({ receipt });
});
Enter fullscreen mode Exit fullscreen mode

The worker needs equally plain rules. This version accepts the queue's attempt count, parses Retry-After, and returns a scheduling decision instead of blocking the process. The repository should atomically mark success or set nextAttemptAt; a lease or visibility timeout makes abandoned work available again.

type Delivery = {
  eventId: string;
  subscriberId: string;
  url: string;
  payload: ShipmentEvent;
  attempt: number;
};

type Decision =
  | { kind: "delivered" }
  | { kind: "retry"; nextAttemptAt: Date }
  | { kind: "failed"; reason: string };

function retryAfterMs(value: string | null, nowMs: number): number | null {
  if (!value) return null;
  const seconds = Number(value);
  if (Number.isFinite(seconds) && seconds >= 0) return Math.ceil(seconds * 1000);
  const dateMs = Date.parse(value);
  return Number.isNaN(dateMs) ? null : Math.max(0, dateMs - nowMs);
}

function jitteredBackoffMs(attempt: number): number {
  const ceiling = Math.min(300_000, 1_000 * 2 ** Math.min(attempt, 8));
  return Math.floor(Math.random() * ceiling);
}

async function deliver(job: Delivery, now = new Date()): Promise<Decision> {
  const response = await fetch(job.url, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "idempotency-key": `${job.eventId}:${job.subscriberId}`
    },
    body: JSON.stringify(job.payload),
    signal: AbortSignal.timeout(10_000)
  });

  if (response.ok) return { kind: "delivered" };
  if (response.status !== 429) {
    return { kind: "failed", reason: `unexpected status ${response.status}` };
  }
  if (job.attempt >= 8) return { kind: "failed", reason: "attempt limit reached" };

  const requestedDelay = retryAfterMs(response.headers.get("retry-after"), now.getTime());
  const delayMs = requestedDelay ?? jitteredBackoffMs(job.attempt);
  return { kind: "retry", nextAttemptAt: new Date(now.getTime() + delayMs) };
}
Enter fullscreen mode Exit fullscreen mode

Real delivery code also classifies timeouts and other status families according to a written policy. Keep that policy beside the tests. A permanent 400 should not churn through eight retries, while a lost connection has an ambiguous outcome and therefore needs the same idempotency protection as a throttled request.

Benchmark the uncertain commit window

First, move fan-out itself off the ingestion transaction when subscriber counts make that transaction too wide. Persist the source event and an outbox record together, then have a dispatcher create delivery rows in bounded batches. That preserves the fast public HTTPS acknowledgment path without quietly dropping the expansion step.

Second, enforce per-subscriber concurrency and rate budgets. A global worker limit is crude: one popular endpoint can occupy every slot, while smaller subscribers wait behind it. Partition or schedule by subscriber ID, and add a dead-letter policy for exhausted attempts. Operations need a deliberate replay command that preserves the original idempotency key.

Third, test ugly timing. Run concurrent duplicates with the same key, kill a worker between the remote response and the local commit, send Retry-After: 120, send an HTTP-date value, omit the header, and advance a fake clock. A useful load test reports acknowledgment latency separately from eventual delivery latency. Benchmarks that mix them produce one smooth number and no diagnosis.

Queue products expose different versions of the same ownership problem. Amazon SQS hides a received message for a visibility timeout; if it is not deleted before that period ends, it can become visible again. Google Cloud Pub/Sub uses an acknowledgment deadline and supports push or pull delivery. RabbitMQ consumers use acknowledgments, and unacknowledged deliveries can be requeued when a channel or connection closes. Those mechanisms affect lease renewal and redelivery, but none removes the need for an application-level delivery ID. Stick with the queue your team already operates when it provides delayed eligibility, bounded redelivery, and observable leases. A queue without delayed scheduling is not suitable when subscriber throttling is routine; adding a separate scheduler may cost more operational attention than changing the queue.

Runtime boundaries for queue fan-out

A durable queue adds storage, workers, leases, dashboards, and replay procedures. For a private integration with one trusted subscriber and tiny traffic, a synchronous call with a strict timeout may be the more honest design. For a fan-out shipment feed, choose the queue once any subscriber's latency or rate limit must be isolated from the source acknowledgment.

The rule is blunt: acknowledge after durable acceptance, track delivery per subscriber, and retry the same identity at a scheduled time. Public HTTPS gets the event into the system. Idempotency and retry state keep it correct after the easy request has ended.

References

Top comments (0)