Failed Jobs Queue Retry — 5 Rate-Limiting and Idempotency Choices Explained

#queues #retries #ratelimiting #observability

Decision table

Short answer: for failed jobs, choose a queue whose retry controls match the rate limit you must enforce, then make each job idempotent; queue-side deduplication is a useful hint, not a correctness guarantee. For an edtech worker pool that drains background jobs against a rate-limited API, this is the practical starting table:

Option	Pick this when	Main trade-off for retries
Standard managed queue	You need durable delivery and can tolerate duplicates	Simple operations; the handler owns deduplication and pacing
FIFO managed queue	Ordering and a short duplicate-suppression window matter	A five-minute dedup window does not cover a long outage or a slow replay
Cloud Tasks	Work is naturally an HTTP request with per-target dispatch controls	Great scheduling knobs, but the worker contract is tied to HTTP semantics
Redis + BullMQ	Your team already runs Redis and needs rich job state in one runtime	More operational surface: Redis durability, failover, and queue metrics are yours

For a course-platform importer, I would begin with a standard queue, a database idempotency record, and a token bucket in the consumer. Move to FIFO only when an explicit ordering invariant is more valuable than throughput.

How should failed jobs use queue retries, rate limits, and idempotency?

Think of the path as a small conveyor belt: the producer writes a job, the queue holds it, a worker claims it, the downstream API answers, and an acknowledgement removes it. A negative acknowledgement or visibility timeout puts the job back in circulation. A dead-letter queue (DLQ) stores messages that exceed a retry budget. That picture makes one rule obvious: a retry is a new attempt at the same business operation, not proof that the previous attempt never happened. In a real course import, a worker can time out after the partner accepted 400 records, reconnect, and receive the same message again. Without a durable operation key, those 400 records can be written twice, notifications can fire twice, and a later reconciliation has no reliable boundary. The queue did exactly what it promised: deliver a message again. The application still has to decide whether the business effect already exists.

Measure it.

A 429 response means the downstream service is asking for slower traffic; MDN documents the status as “Too Many Requests” and allows a Retry-After hint. Treat that hint as data. If it is absent, use bounded exponential backoff with jitter. Do not let every worker sleep for the same exact interval, or a whole cohort wakes up together and recreates the spike.

The idempotency key should identify the business action, such as course-1842:roster-sync:2026-09-03, rather than the queue delivery attempt. Store the key with a result or a durable “in progress” lease. On a duplicate delivery, return the stored result. On a lease that has expired, let a new worker take over. This is the part queue configuration cannot replace.

I've seen teams assume a five-minute deduplication window would cover a retry storm. It did not: an upstream outage lasted 23 minutes, so the same logical sync appeared after the window. The fix was a unique database constraint plus an idempotent upsert. The queue became boring again.

The implementation: pace the worker, then acknowledge

The example below keeps the queue interface generic. It drains a worker pool at 10 requests per second, honors Retry-After, and acknowledges only after the idempotent operation commits. The queue can be backed by a managed service or Redis; that choice stays outside the handler.

type Job = {
  id: string;
  idempotencyKey: string;
  studentId: string;
  attempt: number;
};

type Queue = {
  consume(): Promise<Job>;
  ack(job: Job): Promise<void>;
  nack(job: Job, delayMs: number): Promise<void>;
};

const queue: Queue = createQueue();
const intervalMs = 100; // ten starts per second
let nextPermit = Date.now();

async function takePermit(): Promise<void> {
  const now = Date.now();
  const waitMs = Math.max(0, nextPermit - now);
  if (waitMs > 0) await new Promise(resolve => setTimeout(resolve, waitMs));
  nextPermit = Math.max(Date.now(), nextPermit) + intervalMs;
}

async function handle(job: Job): Promise<void> {
  await takePermit();
  const existing = await idempotencyStore.find(job.idempotencyKey);
  if (existing?.status === "done") {
    await queue.ack(job);
    return;
  }

  await idempotencyStore.start(job.idempotencyKey, { studentId: job.studentId });
  const response = await fetch("https://partner.example/sync", {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ studentId: job.studentId }),
  });

  if (response.status === 429) {
    const retryAfter = Number(response.headers.get("retry-after") ?? 0);
    const delayMs = retryAfter > 0 ? retryAfter * 1000 : Math.min(60_000, 2 ** job.attempt * 1_000);
    await queue.nack(job, delayMs);
    return;
  }
  if (!response.ok) {
    await queue.nack(job, Math.min(300_000, 2 ** job.attempt * 1_000));
    return;
  }

  await idempotencyStore.finish(job.idempotencyKey, await response.json());
  await queue.ack(job);
}

async function workerLoop(): Promise<void> {
  for (;;) {
    const job = await queue.consume();
    try {
      await handle(job);
    } catch (error) {
      // Let the queue retry transient failures; emit the error with job.id first.
      await queue.nack(job, 5_000);
      console.error({ jobId: job.id, error });
    }
  }
}

There are two details worth testing with a fake clock. First, nextPermit is shared by workers, so increasing concurrency does not silently increase outbound rate. Second, the ack happens after finish; if the process dies between those lines, the next delivery sees the completed idempotency record and exits safely. Your mileage may vary with a queue's visibility timeout, so set that timeout longer than the normal handler duration and extend it for genuinely long imports.

Comparing the serious options without cargo culting

SQS Standard gives at-least-once delivery and a visibility timeout; its DLQ and redrive flow fit the “retry later, inspect eventually” model. SQS FIFO adds ordering and message-grouping semantics, with a five-minute deduplication interval documented by AWS. That interval is useful for accidental double-publishes, but it is too short to serve as a business idempotency ledger.

Cloud Tasks makes delayed HTTP dispatch and per-target rate controls explicit. That is a natural fit when every job is already an authenticated request to one service. The boundary is architectural: non-HTTP workers or multi-step workflows need an adapter, and the target endpoint must be designed for repeat delivery.

BullMQ exposes delayed jobs, attempts, and rate limiting beside Redis. It is productive for a Node.js team that already monitors Redis and accepts responsibility for persistence and failover. It is less attractive when the only requirement is durable retry and the team does not want another stateful system to operate.

RabbitMQ offers acknowledgements and dead-letter exchanges, while Pub/Sub is designed for distributed event delivery and retention. Neither removes the need for an idempotent consumer; they simply place different knobs in different layers. The comparison axis is the failure you need to control, not a feature-count race.

Observability and the limits of every choice

Instrument four timestamps: enqueued, claimed, downstream-start, and acknowledged. From them derive queue age, handler latency, retry delay, and time spent waiting for a permit. Tag metrics by outcome (ok, 429, 5xx, permanent_error) and by logical job type, never by an unbounded student or course ID. Alert on oldest-message age and DLQ growth; a healthy throughput number can hide a queue that is quietly aging.

The catch is that no queue can infer whether a remote side effect committed before a network timeout. If the partner cannot accept an idempotency key, use an outbox and a reconciliation job, and document the residual risk. Standard queues are not suitable when strict global ordering is the requirement; stick with FIFO or a workflow engine when that invariant is non-negotiable. Redis-backed queues are not suitable when your team cannot own Redis recovery drills.

Keep payloads small. Put large roster snapshots in your database or object storage and pass a reference in the message. Retention is a retry window, not an event archive: once a message is acknowledged it is deleted, and managed queue retention tops out at 30 days in the documented model.

DEV Community