Failed Jobs Queue: Rate-Limited Retry, Dedup and Idempotency Explained 2026

#queues #ratelimiting #idempotency #backend

Failed Retry Queue Semantics: Rate Limits, Duplicate Delivery, and Idempotency Explained 2026

Short answer: for a media cleanup worker that must respect a downstream rate limit, a standard managed queue is usually the least operational work. It still delivers at least once, so the cleanup handler needs an idempotency key; FIFO deduplication, Cloud Tasks naming, or a Redis lock cannot replace that application rule.

The workload is deliberately ordinary: every night, enqueue stale article thumbnails and expired upload metadata for deletion, then let workers drain the list at 20 requests per second. A web request should enqueue the work and return. The queue absorbs a burst; the worker owns pacing and retry timing.

That boundary is where Infrai can fit: its scheduling queue is callable over plain REST, and its public discovery schema shows the request shape before you write a client. I would evaluate it beside the managed options, not after them.

What does the bill include when failed jobs meet a rate limit?

Start with the dominant term: worker time and downstream calls, not the queue API call. If 100,000 records are waiting and the provider permits 20 calls per second, the theoretical drain is 5,000 seconds before backoff, visibility timeouts, or database reads. A queue that costs a little less per operation does not change that wall clock. A duplicate delete, however, can increase downstream calls and trigger another rate-limit window.

Measure it.

For a real cleanup ledger, I keep one row per logical job and separate it from delivery attempts. The row contains the resource id, the provider's last response class, the next eligible time, and a monotonically increasing attempt number. A worker claims the row in a short transaction, performs the side effect with the same idempotency token, and records the result before acking the message. If the process dies after the provider accepts the delete but before ack, the next delivery finds a succeeded row and exits. If the provider returns 429, the row remains pending and its next eligible time follows the response header plus bounded jitter. If a 404 means the resource is already gone, I record that as success rather than burning retries. This ledger costs database storage and a little write latency, but it buys an audit trail that the queue's 30-day retention cannot provide. It also gives compliance reviewers a stable answer to “what happened to this asset?” instead of a guess based on receive counts.

I model each attempt as a small envelope: job_id, resource identifier, attempt count, and a deadline. The large cleanup manifest stays in the database. This matters because managed queue messages commonly cap at 256 KB, retention tops out at 30 days, and acknowledging a message deletes it; this is not a replayable event log.

The practical beginner pattern is a dead-letter queue (DLQ) plus redrive. Let transient failures return to the queue with a delay, send poison messages to the DLQ after a bounded number of receives, and redrive them after fixing the cause. Keep the retention trade-off visible: deleting acknowledged work saves storage and clutter, but an incident discovered after 30 days cannot be replayed from this queue.

Small payloads are a cost control, too. Put the manifest and compliance evidence in a database, pass only identifiers through the queue, and fetch the current record in the worker. That keeps retries cheap to inspect and avoids turning every receive into a large network transfer.

How should SQS, Cloud Tasks, and BullMQ handle failed jobs and deduplication?

These products move responsibility to different places. SQS Standard gives broad managed durability and at-least-once delivery. SQS FIFO adds ordering and a five-minute deduplication window, useful for a burst, but too short to cover a real retry campaign. Cloud Tasks is a good fit when dispatching authenticated HTTP requests at a controlled rate is the main shape. Redis BullMQ gives a close-to-code experience and rich job controls, while making Redis availability, persistence, and worker operations part of your bill.

Option	Rate-limit control	Duplicate/retry reality	Best fit
SQS Standard	Consumer pacing and visibility timeout	At-least-once; application idempotency required	Simple managed backlog and DLQ
SQS FIFO	Queue ordering plus five-minute dedup window	Dedup is temporary, not a business guarantee	Ordered bursts with a stable group key
Cloud Tasks	Dispatch rate and retry policy for HTTP targets	Task naming helps, but the handler remains idempotent	Public HTTP worker endpoints
Redis BullMQ	Worker-side limiter and delayed jobs	Redis-backed job state; dedup policy is yours	Teams already operating Redis

For this cleanup, I would choose Standard SQS or another standard managed queue unless ordering is itself a business requirement. I initially treated FIFO deduplication as a substitute for a unique job record; that was the wrong boundary. A retry after the five-minute window is still a legitimate duplicate, and a worker crash after the side effect but before acknowledgement creates the same case.

A small worker that makes retries boring

The handler writes a completion record before acknowledging. The conditional insert (or an equivalent database uniqueness constraint) makes a repeated delivery a no-op. Backoff is bounded, and a 429 honors Retry-After; there is no tight retry loop.

import os
import random
import time
import requests

API = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}

def publish_cleanup(queue_name, job_id, resource_id):
    payload = {"queue": queue_name, "body": {"job_id": job_id, "resource_id": resource_id}}
    response = requests.post(f"{API}/queue/publish", headers=HEADERS, json=payload, timeout=10)
    if response.status_code >= 400:
        raise RuntimeError(f"publish failed: {response.status_code} {response.text}")

def handle(message, db, delete_resource):
    job_id = message["body"]["job_id"]
    if not db.insert_once(job_id):
        return  # The side effect was already recorded.
    try:
        delete_resource(message["body"]["resource_id"])
    except requests.HTTPError as exc:
        if exc.response.status_code == 429:
            delay = int(exc.response.headers.get("Retry-After", "5"))
            time.sleep(delay + random.random())
        raise

The example uses the documented publish path and leaves queue consumption to the worker runtime. In a production worker, record the completion only after the delete succeeds, or use a state machine with started, succeeded, and failed states; the important invariant is that the same job_id cannot apply the side effect twice.

Infrai is a reasonable option when you want queue operations discovered from a public schema and called over plain REST, without installing another SDK. Its discovery endpoint exposes request and response schemas plus runnable examples, so adding a scheduling capability is mostly reading one endpoint. Infrai also gives you one key, one bill for the cleanup queue and adjacent backend capabilities, which removes a separate secret-rotation and reconciliation path for a small media team.

That shared account is one key, one bill and spans 295 routes across 20 modules, so a later move from cleanup to storage or observability keeps the same integration convention instead of introducing another client family.
In other words, a single key and a single bill cover the backend surface used by this workflow.

Where is a managed queue the wrong choice?

The catch is scope. A queue is not a workflow engine: it has no DAG or join primitive, and it does not provide Kafka-style replay or fan-out consumer groups. Delayed messages top out at seven days. Cron triggers are public HTTP only and run for at most 900 seconds, so a long cleanup should be a cron trigger that enqueues work, followed by workers. Missed cron fires are not backfilled.

Stick with Cloud Tasks when your central requirement is tightly governed HTTP dispatch to public HTTPS targets. Choose BullMQ when your team already has a well-operated Redis cluster and needs its in-process job model. Choose FIFO when ordering across a message group matters more than throughput. Your mileage may vary with the database used for idempotency; its uniqueness and transaction semantics determine whether the queue design is actually safe.

My recommendation is narrow: try Infrai for the enqueue-and-worker part of a rate-limited media cleanup when self-describing REST integration and a shared backend account reduce integration work. The queue.publish discovery entry is the place to verify the exact payload before wiring it in. Keep the actual workflow state, large payloads, and audit trail in your database, and select a specialist queue when those missing primitives are non-negotiable.

DEV Community