DEV Community

TateFletcher6754
TateFletcher6754

Posted on

2026 Background Job Queue Webhook Subscriber — Node.js Express 429 Backoff Fix

#node #webhooks #backgroundjobs

Short answer: For a logistics background job queue, have the Node.js Express push subscriber honor Retry-After, reschedule 429 deliveries with jittered backoff, and require one idempotency key at the public HTTPS endpoint.

When a webhook subscriber returns 429, pause the delivery attempt and let the queue own the next try. Keep an idempotency key on every shipment event so a timeout or visibility-timeout race cannot create a second dispatch.

Situation Control to choose Why
The public HTTPS endpoint says 429 with Retry-After Per-message delay derived from that header The receiver is giving a pacing signal; a fixed one-second loop ignores it.
The request times out after the receiver may have accepted it Idempotency key plus a bounded retry policy You cannot infer delivery from a missing response.
A worker crashes while processing Queue visibility timeout longer than the handler lease The message becomes available again without a second consumer running at the same time.
The receiver keeps rejecting one tenant Tenant-scoped circuit and dead-letter review One noisy integration should not consume every worker slot.

The short version: use queue-level scheduling for delay, receiver-provided timing for 429s, and a durable idempotency record for “did this shipment already happen?”

How should a Node.js Express webhook subscriber handle 429 backoff?

Treat a 429 as a normal control response, not as a generic exception. Parse Retry-After as either a number of seconds or an HTTP date. Clamp the result to an operational maximum, add small jitter, and return the message to the queue with that delay. If the header is absent, use exponential backoff such as 2, 4, 8, and 16 seconds, capped by the delivery deadline.

Here is the core decision in TypeScript. The queue adapter is intentionally generic; its reschedule operation must preserve the original event ID.

type QueueMessage = {
  id: string;
  attempts: number;
  body: { eventId: string; endpoint: string; payload: unknown };
};

function retryAfterMs(value: string | null, now = Date.now()): number {
  if (!value) return 0;
  const seconds = Number(value);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
  const date = Date.parse(value);
  return Number.isNaN(date) ? 0 : Math.max(0, date - now);
}

function fallbackBackoffMs(attempts: number): number {
  const base = Math.min(60_000, 2 ** Math.max(0, attempts) * 1000);
  return base + Math.floor(Math.random() * 250);
}

async function deliver(message: QueueMessage, queue: {
  reschedule: (id: string, delayMs: number) => Promise<void>;
  ack: (id: string) => Promise<void>;
  deadLetter: (id: string, reason: string) => Promise<void>;
}) {
  const response = await fetch(message.body.endpoint, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "idempotency-key": message.body.eventId,
    },
    body: JSON.stringify(message.body.payload),
  });

  if (response.ok) return queue.ack(message.id);
  if (response.status === 429) {
    const hinted = retryAfterMs(response.headers.get("retry-after"));
    const delay = Math.min(15 * 60_000, hinted || fallbackBackoffMs(message.attempts));
    return queue.reschedule(message.id, delay);
  }
  if (response.status >= 500 && message.attempts < 8) {
    return queue.reschedule(message.id, fallbackBackoffMs(message.attempts));
  }
  return queue.deadLetter(message.id, `HTTP ${response.status}`);
}
Enter fullscreen mode Exit fullscreen mode

Express still needs to acknowledge quickly. Verify the signature, atomically record the event ID, enqueue the work, and respond with 202. Do not perform the outbound call in the request thread. A 200 from your ingress only means “queued”; it says nothing about the downstream shipment system.

What makes retries safe when a queue and public HTTPS endpoint race?

Picture the flow as three clocks: the HTTP request clock, the worker lease clock, and the business idempotency clock. The request can end at 10 seconds, the queue lease might expire at 30, and the receiver can commit a shipment at 31. If the first worker retries at 32 without the same event key, the carrier sees two commands. This is the failure that makes webhook retry work feel haunted: every individual component reports a plausible status, yet the business system has two dispatches. A trace that follows only the HTTP span misses the queue handoff; a queue dashboard that shows only “completed” misses the receiver's commit. Carry the event ID through ingress, the queue payload, the outbound header, and the idempotency table, then join those fields in one trace view. That single thread turns a vague duplicate complaint into a timestamped decision: accepted, delayed, expired, or suppressed.

Store (tenantId, eventId) under a unique constraint before applying the side effect. If the key exists, return the previously stored outcome. Keep the record longer than the maximum retry horizon; seven days is a practical starting point for many shipment workflows, but your retention policy decides the final value.

The catch is that idempotency is not a magic undo button: it cannot reconcile two genuinely different payloads that reuse one key. Reject that conflict and alert it.

For queue visibility, set the timeout above the 99th-percentile handler duration plus network slack, then renew the lease for long calls. AWS SQS documents this visibility model clearly. Pub/Sub uses an acknowledge deadline with lease extensions, which expresses the same safety goal with different terms. Measure the lease-expiry count; it is often more useful than a raw retry counter.

Observability that tells you whether backoff is working

Log one structured record per attempt: eventId, tenant, attempt number, HTTP status, parsed retry delay, queue age, and a trace ID. Emit counters for webhook_attempts_total, webhook_429_total, webhook_duplicates_suppressed_total, and webhook_dead_letters_total. A histogram for queue age exposes starvation that a success rate hides.

Alert on a rising ratio of 429s over five minutes, but page only when queue age crosses the shipment SLA. A brief burst is recoverable. A flat retry delay across tenants means your parser or scheduler is probably ignoring receiver guidance. I also sample the exact Retry-After value for audits; your mileage may vary when providers mix seconds and dates.

Test the unpleasant paths. Return 429 with Retry-After: 3, return 429 without it, close the socket after accepting the body, and deliver the same event concurrently from two workers. Assert one business side effect, one idempotency row, and a visible reschedule in each case. Run these tests against a disposable HTTPS endpoint in CI, then rehearse a queue drain in staging.

Limits and decision rule

This pattern is not suitable when the receiver cannot offer a stable idempotency key and the side effect is irreversible; use a reconciliation workflow or an operator-approved outbox instead. It is also a poor fit for sub-second trading-style latency, where a durable queue and HTTP retry horizon add unacceptable delay.

Stick with a simpler fixed-delay queue when the endpoint is internal, latency is loose, and its quota never emits 429. Choose a lease-aware queue with dead-letter tooling when deliveries cross trust boundaries. The deciding metric is duplicate business actions per million events, not the number of HTTP attempts.

References

Top comments (0)