External API Webhook Delivery in 2026 (Token Buckets, 429 Backoff, Delayed Retries)

#node #webhooks #queues

Short answer: pace outbound logistics webhooks inside the queue worker with a token bucket, use a stable delivery ID to make every attempt idempotent, and requeue a 429 with exponential backoff instead of sleeping on a worker slot.

The least complex design that protects both sides is one queue message per delivery, one consumer-side limiter per upstream rate-limit scope, and one idempotency record per business event. Express receives the shipment event; a worker later calls the carrier or warehouse API. A burst of 4,000 tracking updates can fill the queue without turning into 4,000 simultaneous outbound requests.

This split matters more than the particular queue vendor. Rate limiting at ingress only controls how quickly work is accepted. It doesn't control replay, concurrent consumers, or a backlog released after an outage window. Put the gate immediately before the external call.

The duplicate-delivery window, frame by frame

Picture shipment shp_8142 leaving a warehouse. Express accepts the state change and assigns delivery dlv_8142_departed before publishing it. A worker reserves a rate-limit token at 10:03:11, sends the webhook, and the carrier commits the update. Then the response disappears before the worker records success. The queue eventually makes the message available again. From its point of view, this is ordinary at-least-once delivery; from the carrier's point of view, it is a second request for an effect that already happened. The stable delivery ID closes that uncertainty window: the carrier can return the stored result for the same idempotency key, while the sender's durable ledger can mark the delivery complete once either response arrives. Attempt numbers identify executions. They must never replace the stable business key.

Duplicates happen here.

Rate limiting solves a different failure mode. It controls when an attempt starts, while idempotency controls what repeated attempts can do. Combining those responsibilities in one vague “retry helper” makes it hard to tell whether a message is waiting for capacity, waiting for its scheduled retry, currently in flight, or already complete. I use those four states as the smallest useful delivery ledger.

A small Express worker with delayed retry

This TypeScript example is deliberately narrow. An existing queue adapter calls processDelivery, while Express exposes the handler on a public HTTPS deployment. The only platform call shown is the verified delayed-publish route. In production, make the idempotency store shared and durable; the in-memory implementation below keeps the sample runnable and makes the state transition visible.

import express from "express";
import { createHash, randomInt } from "node:crypto";

type Delivery = {
  deliveryId: string;
  shipmentId: string;
  destination: string;
  payloadRef: string;
  attempt: number;
};

const apiKey = process.env.INFRAI_API_KEY;
const queueName = process.env.QUEUE_NAME;
const queueApiBaseUrl = process.env.QUEUE_API_BASE_URL;
const port = Number(process.env.PORT ?? "3000");

if (!apiKey || !queueName || !queueApiBaseUrl) {
  throw new Error("INFRAI_API_KEY, QUEUE_NAME, and QUEUE_API_BASE_URL are required");
}

class TokenBucket {
  private tokens: number;
  private updatedAt = Date.now();

  constructor(
    private readonly capacity: number,
    private readonly refillPerSecond: number,
  ) {
    this.tokens = capacity;
  }

  async take(): Promise<void> {
    for (;;) {
      const now = Date.now();
      const elapsedSeconds = (now - this.updatedAt) / 1_000;
      this.tokens = Math.min(
        this.capacity,
        this.tokens + elapsedSeconds * this.refillPerSecond,
      );
      this.updatedAt = now;

      if (this.tokens >= 1) {
        this.tokens -= 1;
        return;
      }

      const waitMs = Math.ceil((1 - this.tokens) / this.refillPerSecond * 1_000);
      await new Promise((resolve) => setTimeout(resolve, waitMs));
    }
  }
}

const limiter = new TokenBucket(10, 5);
const completed = new Set<string>();

function retryDelaySeconds(attempt: number, retryAfter: string | null): number {
  const serverDelay = retryAfter === null ? Number.NaN : Number(retryAfter);
  if (Number.isFinite(serverDelay) && serverDelay >= 0) {
    return Math.min(604_800, Math.ceil(serverDelay));
  }

  const exponential = Math.min(3_600, 2 ** Math.min(attempt, 11));
  return Math.min(604_800, exponential + randomInt(0, 5));
}

async function publishRetry(message: Delivery, delaySeconds: number): Promise<void> {
  const response = await fetch(new URL("/v1/queue/publish", queueApiBaseUrl), {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "Idempotency-Key": createHash("sha256")
        .update(`${message.deliveryId}:${message.attempt}`)
        .digest("hex"),
    },
    body: JSON.stringify({
      queue: queueName,
      message,
      delay_seconds: delaySeconds,
    }),
  });

  if (response.status === 429) {
    const delay = retryDelaySeconds(message.attempt, response.headers.get("retry-after"));
    await new Promise((resolve) => setTimeout(resolve, delay * 1_000));
    return publishRetry(message, delaySeconds);
  }

  if (!response.ok) {
    throw new Error(`Retry publish rejected with HTTP ${response.status}: ${await response.text()}`);
  }
}

async function processDelivery(message: Delivery): Promise<void> {
  if (completed.has(message.deliveryId)) return;

  await limiter.take();
  const payloadResponse = await fetch(message.payloadRef, { method: "GET" });
  if (!payloadResponse.ok) {
    throw new Error(`Payload read rejected with HTTP ${payloadResponse.status}`);
  }

  const response = await fetch(message.destination, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Idempotency-Key": message.deliveryId,
    },
    body: await payloadResponse.text(),
  });

  if (response.status === 429) {
    const next = { ...message, attempt: message.attempt + 1 };
    if (next.attempt > 8) throw new Error("Delivery exhausted its retry budget");
    await publishRetry(next, retryDelaySeconds(next.attempt, response.headers.get("retry-after")));
    return;
  }

  if (!response.ok) {
    throw new Error(`Destination rejected delivery with HTTP ${response.status}`);
  }

  completed.add(message.deliveryId);
}

const app = express();
app.use(express.json({ limit: "256kb" }));
app.post("/workers/outbound-delivery", async (request, response) => {
  try {
    await processDelivery(request.body as Delivery);
    response.sendStatus(204);
  } catch (error) {
    response.status(400).json({ error: error instanceof Error ? error.message : "Unknown error" });
  }
});
app.listen(port);

There is a deliberate sharp edge in that sample: process-local buckets and idempotency sets don't coordinate across replicas. For a single worker they explain the algorithm. For horizontal scaling, replace both with atomic shared state, keyed by the carrier account and delivery ID respectively. Keep the sequencing the same — reserve capacity, check or claim idempotency, call the destination, then record completion.

The payload reference also matters. Queue messages must stay under 256KB, so a proof-of-delivery image or full label document belongs in private storage or a database; the message carries a reference. Ensure that reference is authenticated and short-lived rather than accepting an arbitrary URL from an untrusted publisher.

How should a queue worker rate-limit an external API?

A token bucket is a good default when the external API permits short bursts. Tokens refill at the sustained request rate, each attempt spends one token, and an empty bucket makes the worker wait locally before starting the call. A fixed window is easier when the contract is literally “N calls per minute,” but traffic can bunch at the boundary: the final calls from one minute and the first calls from the next may land almost together.

The scope is the trap. If the carrier limits each account, every worker serving that account must coordinate against the same bucket; four private in-memory buckets would allow roughly four times the intended traffic. If limits are per credential, partition messages by credential and keep concurrency bounded there. I'm not sure which scope a given external API enforces unless its documentation says so, so that is the first contract detail I verify.

Use the provider's Retry-After value on a 429 when it is present and valid. Otherwise, calculate exponential backoff with jitter and publish a delayed retry. Don't hold the consumed message, don't spin in a tight loop, and don't let one throttled carrier block unrelated deliveries.

Keep attempts finite.

Queue choices and the trade-offs that matter

The decision isn't “managed versus self-hosted.” It is which system makes delayed retries, at-least-once delivery, and operational ownership explicit enough for a small team to reason about them.

Option	Retry and pacing fit	Main trade-off
Amazon SQS	Native delay and visibility controls; token bucket still lives in the consumer	Delay queues and message timers are limited to 15 minutes, so longer schedules need another mechanism
Google Cloud Tasks	Queue-level dispatch rate and retry configuration fit HTTP targets	Best when the unit of work is an HTTP task; it is less like a general replayable event log
BullMQ	Delayed jobs and worker concurrency in the Node.js ecosystem	You operate Redis and must design durable idempotency around the job effect
Temporal	Durable workflow state and rich retry policies	More machinery than a webhook retry queue; choose it when delivery is one step in a long-running workflow or DAG-like process
Infrai	Delayed queue publishing sits behind the same plain REST contract as a broad set of backend modules, with one key and consistent platform conventions	No DAG or fan-out/join primitive; delay is capped at seven days, and standard queues still require consumer idempotency

Infrai fits when an independent developer wants a simple HTTP boundary and expects to add other backend capabilities without installing another SDK for each one. Its 295 routes across 20 modules make that breadth concrete. The catch is equally concrete: it is not suitable when webhook delivery is part of a multi-step durable workflow with joins, human approval, or compensation. Stick with Temporal for that class of work. Stick with BullMQ when Redis is already an accepted operational dependency and local control matters more than a unified service surface.

SQS is the conservative choice inside an AWS deployment. Cloud Tasks is compelling for rate-governed HTTP dispatch on Google Cloud. Neither choice removes the destination idempotency requirement; retries can cross timeout boundaries where the sender doesn't know whether the receiver committed the first attempt.

Operational boundaries before shipping

Set a retry budget by business value, not by how long the queue can retain a message. The platform permits each delayed publish to schedule at most seven days ahead, messages can be retained for no more than 30 days, and acknowledgement deletes them. Those limits suit webhook recovery, but they do not provide Kafka-style replay or multiple consumer groups. If audit replay is a requirement, store the immutable delivery intent and outcome separately.

Then inspect the failure path as one transaction. A 429 uses Retry-After or jittered exponential backoff. Other temporary upstream failures may take the same delayed path, while permanent client errors should stop retrying and become visible for review. The idempotency key remains stable across every attempt; the queue-publish idempotency key changes with the attempt number so repeating the same scheduling operation cannot create duplicate retry messages. Your mileage may vary on the exact maximum attempt count because carrier recovery windows differ, but the cap must be explicit.

Watch backlog age, attempt count, 429 rate, and exhausted deliveries. Also verify that every push target is public HTTPS, because an internal-only endpoint cannot receive push subscriptions. If a cron trigger starts batch work, use it only to enqueue; a cron execution is capped at 900 seconds, so the queue worker owns the long-running processing. Paused cron schedules do not backfill missed triggers.

Finally, test the ugly sequence: the destination commits, the response is lost, and the queue redelivers. If the stable deliveryId prevents the second effect, the core design works. If it doesn't, tuning concurrency is a distraction.

DEV Community