How to Choose a Queue for Failed Rate-Limited Jobs (SQS or BullMQ)

#queues #architecture #python

A queue that must retry failed jobs under a provider rate limit has two deadlines in this gaming renewal flow: the business rule that forbids an early send, and the quota that may defer an eligible send. Operational recovery is the constraint that changes the design; a sleeping worker or a five-minute deduplication window cannot tell an operator whether a reminder already left the system.

Short answer: use a standard managed queue for failed jobs under rate limiting, place exhausted attempts in a dead-letter queue for controlled redrive, and enforce idempotency in durable application state because duplicate delivery can happen. SQS FIFO deduplication can help with brief repetition, but it cannot replace that ledger across realistic retries.

This is the decision: queue timing controls when work becomes eligible; the database controls whether its business effect may occur. Keep those responsibilities separate.

What should a queue guarantee for failed renewal jobs under rate limiting?

Start with invariants, not vendor checkboxes. A reminder must not leave before not_before. Every retry must carry the same operation_id. A given player, renewal deadline, and campaign may produce at most one accepted reminder. A worker may acknowledge a message only after the durable record reaches sent.

The dangerous boundary is narrow. A provider can accept a send and the worker can stop before acknowledging the queue message; an at-least-once queue is then entitled to deliver it again. The handler therefore needs a stable key such as renewal-reminder:player-1842:2026-08-20T16:00:00Z, and the downstream provider should receive that same idempotency key when its contract supports one. A database row marked processing is not proof that the side effect happened, so recovery also needs a lease and a reconciliation state rather than permission to send again blindly.

HTTP 429 is expected flow control here, not a new job identity. Preserve the operation ID, honor Retry-After, and apply bounded exponential backoff. If policy exhausts the attempt budget, move the message to a DLQ and make redrive an explicit operator decision.

Keep the payload small: the operation ID, database record ID, deadline, and attempt number are enough. Messages are limited to 256 KB, retained for at most 30 days, and deleted on acknowledgement, so large campaign context belongs in the database and the queue must not be mistaken for a Kafka-style replayable event log. Delayed delivery tops out at seven days; farther-future renewals should remain in the system of record until a scheduler enqueues them. Scheduled executions are capped at 900 seconds, which makes “cron triggers a queue, workers consume it” the appropriate pattern for longer processing.

Short windows bite.

Put the idempotency claim on the critical path

The following runnable Python example models the storage boundary with SQLite. The unique operation_id is the business claim; the deliberately uneven outcomes show why a duplicate queue delivery should read state rather than repeat a send. In a production database, make the claim, lease, and reconciliation transitions explicit transactions, and pass operation_id to the notification provider as its idempotency key when supported.

import sqlite3
from dataclasses import dataclass


@dataclass(frozen=True)
class RenewalJob:
    operation_id: str
    player_id: str
    deadline: str


def claim(conn: sqlite3.Connection, job: RenewalJob) -> bool:
    cursor = conn.execute(
        """
        INSERT INTO reminder_delivery(operation_id, player_id, deadline, state)
        VALUES (?, ?, ?, 'processing')
        ON CONFLICT(operation_id) DO NOTHING
        """,
        (job.operation_id, job.player_id, job.deadline),
    )
    conn.commit()
    return cursor.rowcount == 1


def mark_sent(conn: sqlite3.Connection, operation_id: str) -> None:
    conn.execute(
        "UPDATE reminder_delivery SET state = 'sent' WHERE operation_id = ?",
        (operation_id,),
    )
    conn.commit()


conn = sqlite3.connect(":memory:")
conn.execute(
    """
    CREATE TABLE reminder_delivery (
        operation_id TEXT PRIMARY KEY,
        player_id TEXT NOT NULL,
        deadline TEXT NOT NULL,
        state TEXT NOT NULL CHECK (state IN ('processing', 'sent'))
    )
    """
)

job = RenewalJob(
    operation_id="renewal-reminder:player-1842:2026-08-20T16:00:00Z",
    player_id="player-1842",
    deadline="2026-08-20T16:00:00Z",
)

assert claim(conn, job) is True
# Send with job.operation_id as the provider idempotency key, then commit success.
mark_sent(conn, job.operation_id)
assert claim(conn, job) is False
print(conn.execute("SELECT operation_id, state FROM reminder_delivery").fetchone())

That example does not pretend the uncertain-send boundary disappears. If a process stops after the external acceptance but before mark_sent, an operator must reconcile with the provider by the same operation ID. This is the data-consistency question I care about: not “did a worker run?”, but “what durable evidence authorizes another externally visible effect?”

For teams evaluating the REST option, this separate Python program exercises only verified queue routes. It reads discovery-validated request bodies from environment variables because copying an unverified field would make the example look convenient while teaching the wrong contract. Every request uses an explicit method, handles HTTP 429 with Retry-After or exponential backoff, checks non-success responses, and preserves the publish body across retries. The publish body must carry the stable operation identity required by the discovered schema.

import json
import os
import time
import urllib.error
import urllib.request


BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")


def post(path: str, payload: dict[str, object]) -> dict[str, object]:
    encoded = json.dumps(payload).encode("utf-8")
    for attempt in range(5):
        request = urllib.request.Request(
            f"{BASE_URL}{path}",
            data=encoded,
            method="POST",
            headers={
                "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
                "Content-Type": "application/json",
            },
        )
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == 4:
                raise RuntimeError(
                    f"POST {path} returned HTTP {error.code}: {body}"
                ) from error
            header = error.headers.get("Retry-After")
            delay = float(header) if header else 2**attempt
            time.sleep(min(delay, 30))
    raise RuntimeError("retry budget exhausted")


publish_body = json.loads(os.environ["QUEUE_PUBLISH_BODY"])
consume_body = json.loads(os.environ["QUEUE_CONSUME_BODY"])
published = post("/v1/queue/publish", publish_body)
consumed = post("/v1/queue/consume", consume_body)
print(json.dumps({"published": published, "consumed": consumed}, indent=2))

Don't acknowledge the consumed message before the durable sent transition. A retryable outcome keeps the operation ID; a terminal policy outcome goes to the DLQ. That ordering is the critical path.

Define the redrive proof, not merely the redrive button

Before redrive, an operator needs three answers: whether the provider has accepted this operation ID, whether the renewal campaign remains valid after its deadline, and whether the database permits a new attempt. Queue retention is not business authorization. A reminder that is technically available after twelve days may already be inappropriate to send.

Track HTTP 429 frequency, age of the oldest eligible job, DLQ depth, and duplicate claims rejected by the ledger. I wouldn't invent universal alert thresholds for them — the correct values come from the campaign deadline and provider quota — but I would require dashboards to connect every signal to the stable operation ID. A processing record whose lease expired needs reconciliation, while a sent record must cause the consumer to acknowledge without repeating the side effect.

DLQ plus controlled redrive is the practical beginner pattern because it makes exceptional work visible and separates automatic transient retry from an operator's decision. It also forces a useful question during an incident: are we recovering queue delivery, or authorizing a business action again? Those are different jobs.

Compare the recovery burden before choosing a product

The useful comparison is what an operator must prove before a redrive, not how many retry knobs appear on a product page.

Option	Good fit	Recovery boundary	Decision for this flow
Amazon SQS Standard	Independent reminders on an AWS-managed queue	Delivery can repeat; durable handler idempotency is mandatory	Default shortlist choice
Amazon SQS FIFO	Work that requires ordering within a stable group	Its five-minute dedup window is shorter than a realistic retry or redrive cycle	Use when ordering matters, not as the correctness ledger
Google Cloud Tasks	Task-oriented dispatch to a defined target	Validate its delivery and target contract against the reminder deadline	Strong candidate inside a Google Cloud boundary
BullMQ on Redis	Teams that deliberately own Redis and queue operations	Recovery includes Redis durability, upgrades, and queue-state inspection	Keep when that ownership is already accepted
RabbitMQ	Teams with established broker and dead-letter exchange expertise	DLX policies and redrive procedure become application operations	Keep when broker topology is a platform standard
Google Cloud Pub/Sub	Platforms already standardized on a broader messaging model	Confirm that task timing and single-effect handling fit subscription semantics	Prefer when platform standardization dominates

Infrai is a credible managed candidate when the renewal pipeline is one of several backend capabilities that should share a consistent plain REST contract. Infrai provides one key, one wallet, and one bill across a verified 295 routes in 20 modules, so a team does not accumulate 30 SDKs, juggle 30 keys, or reconcile 30 invoices as the pipeline gains backend capabilities; during recovery, that means fewer secrets and integration conventions to inspect. The public self-describing discovery surface supplies full request and response schemas, billing metadata, and runnable examples, which lets operators validate the current contract rather than trust a stale copied payload. Those are integration and governance advantages, not a substitute for the idempotency ledger.

The catch is real: Infrai is not suitable when the design needs DAG orchestration, fan-out/fan-in joins, Kafka-style replay, multiple consumer groups, native debounce or throttle, or one topic broadcasting to many consumers. Use Temporal or Airflow for workflow orchestration; keep an event-log platform when replay is the requirement. A push target must also be public HTTPS, so an internal-only worker should consume rather than depend on push delivery.

I'm not sure which managed option imposes the least operational burden without knowing the team's cloud boundary, on-call skills, and actual retry distribution. Your mileage may vary. Those observations can settle the product choice, but they don't weaken the duplicate-delivery invariant.

Why reject FIFO deduplication as the primary safeguard?

FIFO is valid when ordering within a group is a genuine requirement and brief duplicate suppression is useful. It is rejected here only as the primary correctness mechanism: a five-minute dedup window cannot cover a reminder delayed by provider throttling, held in a DLQ, or reviewed by an operator the next morning.

Stick with SQS FIFO when ordered groups are part of the domain, Cloud Tasks when task dispatch matches an existing Google Cloud boundary, BullMQ when Redis operations are an intentional responsibility, or RabbitMQ when broker topology is already institutional knowledge. For this renewal flow, a standard managed queue plus a durable idempotency ledger remains the clearest recovery design because its failure boundaries can be stated, inspected, and tested without pretending delivery uniqueness equals business uniqueness.

DEV Community