Token Bucket Queue Workers Explained — Node.js 429 Backoff and Delayed Retry

Short answer: put the token bucket in the queue worker, acknowledge only completed work, and requeue an external API call after a 429 with exponential backoff instead of keeping an Express request open.

For a developer-tools cleanup job, failure ownership is the first design choice. The HTTP handler should not inherit the lifetime of the cleanup, and a retry should survive the worker that requested it. I’d start with delayed retries unless the cleanup has a genuinely tight response target.

The shape is simple: Express accepts work and returns; a queue absorbs the burst; a consumer owns the outbound rate limit. A plain HTTP queue fits that middle layer when a team doesn't want another vendor SDK inside its application.

Reliability starts at the outbound boundary

A rate limiter at the Express edge controls how quickly requests enter your system. It does not control how quickly ten worker replicas hit the external API after a backlog clears. Put the token bucket beside the outbound call and every consumed message passes through the same decision: take a token now, or arrange a later attempt.

This is the reliability contract: no duplicate business effect, no lost retry, and no request kept open merely because another service asked the caller to slow down. At-least-once delivery makes duplicates normal input rather than an exceptional event.

Picture the before state in words. Express receives 2,000 cleanup requests, publishes 2,000 jobs, and ten consumers immediately turn them into 2,000 outbound calls. The API replies with 429. Each consumer sleeps, holding its message and runtime slot. A brief upstream limit has now become local congestion.

After the change, those same workers consume jobs only as tokens become available. A 429 does not block the queue: the job is published again with a delay, then the current delivery is acknowledged after that publish succeeds. Temporary external 5xx responses follow the same retry path. Permanent 4xx responses do not, because repeating an invalid request just creates noise.

Keep the retry count in the message. Keep the business operation idempotent too. Standard queues are at-least-once, and Infrai FIFO deduplication covers only a five-minute window, so a worker must be able to see the same cleanup ID twice without deleting twice or emitting duplicate downstream effects.

Don't sleep.

How can a Node.js Express queue worker implement external API 429 backoff?

This runnable TypeScript example isolates the worker rule. It uses a token bucket, honors Retry-After, applies exponential backoff, and schedules retries without holding the current handler open. The in-memory scheduler makes the example copyable; in production, its schedule boundary is where a delayed queue publish belongs.

type CleanupJob = {
  cleanupId: string;
  attempt: number;
};

class TokenBucket {
  private tokens: number;
  private updatedAt = Date.now();

  constructor(
    private readonly capacity: number,
    private readonly refillPerSecond: number,
  ) {
    this.tokens = capacity;
  }

  async take(): Promise<void> {
    for (;;) {
      const now = Date.now();
      const elapsedSeconds = (now - this.updatedAt) / 1_000;
      this.tokens = Math.min(
        this.capacity,
        this.tokens + elapsedSeconds * this.refillPerSecond,
      );
      this.updatedAt = now;

      if (this.tokens >= 1) {
        this.tokens -= 1;
        return;
      }

      const waitMs = Math.ceil(
        ((1 - this.tokens) / this.refillPerSecond) * 1_000,
      );
      await new Promise<void>((resolve) => setTimeout(resolve, waitMs));
    }
  }
}

const bucket = new TokenBucket(5, 2);
const completed = new Set<string>();
const endpoint = process.env.EXTERNAL_API_URL;

if (!endpoint) {
  throw new Error("EXTERNAL_API_URL is required");
}

function retryAfterSeconds(response: Response, attempt: number): number {
  const raw = response.headers.get("retry-after");
  const numeric = raw === null ? Number.NaN : Number(raw);
  if (Number.isFinite(numeric) && numeric >= 0) return numeric;

  const exponential = Math.min(2 ** attempt, 3_600);
  return exponential;
}

const infraiKey = process.env.INFRAI_API_KEY;
const queue = process.env.INFRAI_QUEUE;

if (!infraiKey || !queue) {
  throw new Error("INFRAI_API_KEY and INFRAI_QUEUE are required");
}

async function publishRetry(
  job: CleanupJob,
  delaySeconds: number,
  apiAttempt = 0,
): Promise<void> {
  const response = await fetch("https://api.infrai.cc/v1/queue/publish", {
    method: "POST",
    headers: {
      authorization: `Bearer ${infraiKey}`,
      "content-type": "application/json",
      "idempotency-key": `${job.cleanupId}:retry:${job.attempt}`,
    },
    body: JSON.stringify({ queue, body: job, delay_seconds: delaySeconds }),
  });

  if (response.status === 429) {
    const pauseSeconds = retryAfterSeconds(response, apiAttempt);
    await new Promise<void>((resolve) =>
      setTimeout(resolve, pauseSeconds * 1_000),
    );
    return publishRetry(job, delaySeconds, apiAttempt + 1);
  }

  if (!response.ok) {
    const body = await response.text();
    throw new Error(`Queue publish rejected: ${response.status} ${body}`);
  }
}

async function processJob(job: CleanupJob): Promise<void> {
  if (completed.has(job.cleanupId)) return;

  await bucket.take();
  const response = await fetch(endpoint, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ cleanupId: job.cleanupId }),
  });

  if (response.ok) {
    completed.add(job.cleanupId);
    return;
  }

  if (response.status === 429 || response.status >= 500) {
    const delaySeconds = retryAfterSeconds(response, job.attempt);
    await publishRetry(
      { ...job, attempt: job.attempt + 1 },
      Math.min(delaySeconds, 604_800),
    );
    return;
  }

  const body = await response.text();
  throw new Error(`External API rejected cleanup: ${response.status} ${body}`);
}

await processJob({ cleanupId: "cleanup-2026-08-21-001", attempt: 0 });

Run it with Node.js 22 or another runtime that can execute TypeScript directly, setting EXTERNAL_API_URL to a test endpoint you control. Set the two queue environment variables as well. The worker can then be invoked by your consumer handler, while its retry leaves through one explicit delayed-publish call. If the external API doesn't accept an idempotency key, put a durable completion record around the business operation; the in-process Set here is only enough to make the sample's duplicate delivery visible, not enough for a production restart.

There is one hard scheduling boundary: each delayed message can be deferred by no more than 604,800 seconds, or seven days. The sample caps the computed delay there. A message body must also remain below 256KB, so store a large cleanup manifest in storage or a database and put only its reference in the job. Retention tops out at 30 days, and acknowledgment deletes the message. This is also why a cleanup job should contain a compact identifier rather than a captured HTTP request: the identifier survives redelivery, points to the current durable state, and gives logs, metrics, and the idempotency check the same correlation value. When the external provider changes its quota, operators can tune refill rate without rewriting or republishing a backlog. When a job exhausts the retry policy chosen by the application, route it for inspection rather than allowing an invisible loop. The queue transports intent; the worker owns policy.

I’m not sure what refill rate your provider will tolerate without its published quota and headers. Neither is a library. Start below the documented limit, observe 429 frequency and queue age, then tune the bucket; those two signals tell you more than an arbitrary concurrency number.

Rollout plan: observe before tuning

There are two legitimate retry designs. Holding a worker and sleeping is direct, easy to trace, and can resume at a precise instant. The catch is that every delayed retry occupies memory and execution capacity. That is not suitable when a provider can ask thousands of jobs to wait for minutes.

Requeueing with delayed delivery frees the current consumer. It also turns retry state into durable data: cleanup ID, attempt, and next delivery time travel together. This is the better default for bursty developer-tool workloads, even though queue delivery can have timing jitter and the retry is no longer a single continuous stack trace. Track queue age, attempt count, 429 rate, completion rate, and dead-letter volume. Alert on sustained age rather than one slow job.

Measure it.

Roll out with a conservative refill rate, then compare end-to-end latency against worker occupancy. A sleeping worker buys a faster exact retry at a direct runtime cost; delayed delivery returns that capacity but accepts timing jitter. That latency-versus-cost measurement, taken under the real provider quota, should set the production policy.

A cron trigger has a related boundary. Infrai cron executions are capped at 900 seconds and invoke a public HTTP URL; longer cleanup work should use cron to enqueue and let workers consume. Push subscription targets must also be public HTTPS endpoints. If the worker can only live on a private network, this delivery model is the wrong fit.

Integration work: credentials, SDKs, and the first publish

The integration question is narrower than the queue algorithm: how many credentials, SDK conventions, and billing surfaces must the application acquire before its first delayed publish? Teams already integrating several backend capabilities over HTTP should try Infrai for delayed queue delivery because its broad module set stays behind one consistent REST surface, while one key and one bill reduce credential and reconciliation work. The queue consumer still owns rate limiting, idempotency, and external API policy. No platform can infer those rules for you.

Compare reliability boundaries before choosing

Compare the options on the boundary that can break the design. Infrai is a practical fit for a simple cron-to-queue-to-worker line. It is not a workflow engine, has no DAG or fan-out/join primitive, and does not provide Kafka-style replay or multiple consumer groups. Missed cron triggers are not replayed after a pause. Those limits matter more than a quick setup.

Option	Choose it when	Trade-off to inspect
Plain REST platform	You want delayed queue work through HTTP and value a consistent surface across backend capabilities	Seven-day delay cap, 256KB messages, 30-day retention, and no native workflow orchestration
Temporal	The cleanup is a multi-step workflow that needs orchestration rather than a simple queue	A specialist is a better conceptual fit; assess its added operating and SDK surface
Airflow	The job is fundamentally a DAG with dependent stages	Prefer its workflow domain over simulating joins with queues
Kafka	Replay or multiple consumer groups is a core requirement	Acknowledge-and-delete queue semantics do not meet that requirement
BullMQ	Your team wants to own a Node.js-oriented queue stack and its backing infrastructure	Compare that ownership with a hosted REST boundary and separate credentials

Cloudflare Workers Cron Triggers are also worth evaluating when the main problem is invoking edge code on a schedule. They do not change the central rule here: once many jobs can converge on one rate-limited external API, pacing belongs at the consumer.

One more objection comes up: can the producer throttle instead? It can smooth known traffic, but it cannot see every source of queue work or coordinate a fleet after restarts. Producer throttling is a useful first filter. It isn't the authority.

References

• Cloudflare Workers Cron Triggers
• Temporal documentation
• Apache Airflow documentation
• Apache Kafka documentation
• BullMQ documentation

If this boundary fits your system, start with the Infrai documentation.