EthanBrooks1647

Posted on Oct 26

Redis Retry Queues vs AWS SQS for Failed Publishing Renewals Within 7 Days

#backend #queues #idempotency #media

Short answer: use Redis when a small, real-time retry path is worth owning in application code. Use AWS SQS when built-in retries and a dead-letter queue matter more than custom control. For a media renewal reminder, the queue choice is secondary to one rule: every delivery attempt must carry the same operation identity and make a duplicate harmless.

The deadline changes the shape of the problem. A reminder may wait, fail, and return after the subscription has changed. The worker must therefore re-read current state, preserve an idempotency key across attempts, and keep failed work inspectable.

What is the retry bill actually made of?

The bill for this workflow is mostly retained retry state and worker work on failed attempts. The message itself is small; the failure path is not. A useful budget starts with the platform limits: delayed delivery is capped at 7 days, a message body at 256 KB, retention at 30 days, and a single cron execution at 900 seconds.

Those limits tell us what to keep. Put a subscription identifier, deadline, template identifier, and operation identifier in the message. Keep the rendered email or SMS body out of it. If the deadline falls inside seven days, a delayed message is reasonable. If the work lasts longer than 900 seconds, cron should trigger a queue publish and a worker should do the real processing. A long-running cron handler is the wrong boundary.

The change that moves the dominant term is retention discipline: store durable business state in the application database, and use the queue for a compact pointer plus retry metadata. That reduces the amount of stale content being retained and makes a retry fetch the current consent and plan state before sending. The trade-off is an extra read during recovery and less forensic detail in the queue itself.

What should stop being kept? The rendered notification body and any assumption that the queue is a permanent event log. Acknowledgement deletes the message, there is no Kafka-style replay or multi-consumer-group history, and output history is limited to the first 4 KB. When something fails, the application needs a durable operation record and a deliberate dead-letter review path.

Should Redis or AWS SQS handle failed retry tasks?

Redis is fast for smaller, real-time workloads, but the application must supply retry timing, attempt counting, poison-job handling, and failure management. That is a good fit when low latency and direct control are important and the team is willing to operate the surrounding state.

AWS SQS supplies managed retry infrastructure and dead-letter queues. Its advantage is not that failures disappear; it is that the destination and retry policy are explicit infrastructure rather than a collection of worker conventions. SQS still delivers at least once, so the consumer must remain idempotent.

RabbitMQ is a different choice. Its dead-letter exchange model routes rejected or expired messages through broker topology, which is useful when routing rules are central to the system. The cost is additional reasoning about exchanges, bindings, and policies. It is not the same operational model as a simple retry list.

BullMQ sits above Redis and gives Node.js teams application-level job options and workers. It can be a productive choice when Redis is already part of the deployment, but it does not remove the need to decide how failed jobs are retained and reviewed. Celery is similarly appropriate for Python task dispatch, with task behavior and retry policy close to application code. Temporal belongs on another boundary: it is for workflow orchestration, not a replacement for one delayed queue item.

Option	Strongest fit	What the team owns	Important boundary
Redis	Small, low-latency workloads	Retry state, attempts, poison-job handling, and failure review	No built-in DLQ behavior
AWS SQS	Managed retries and DLQs	Idempotent consumer and effect ledger	At-least-once delivery remains
RabbitMQ	Dead-letter routing topology	Exchanges, bindings, and broker policies	More routing concepts to operate
BullMQ	Redis-backed Node.js workers	Application job policy and Redis operations	Still an application-centered failure model
Celery	Python task dispatch	Broker, workers, and task policy	Retry behavior remains near the task code

That is the practical comparison. Redis gives control and speed. SQS gives a defined managed failure path. RabbitMQ gives broker-level routing. BullMQ and Celery keep job behavior close to the worker. None of them makes an external notification exactly once.

Keep the decision boring.

How do retries avoid a duplicate notification?

The queue acknowledgement is not a transaction with the email or SMS provider. A worker can send successfully, crash before recording the result, and then receive the same message again. A timeout is just as ambiguous: the provider may have accepted the request even though the worker saw no response.

Carry a stable operation identifier through every retry. Before sending, the worker checks a durable effect record. If that operation is already marked sent, it acknowledges the message and exits. Otherwise, it claims the operation, reads current subscription state, sends using the same idempotency key when the provider supports one, records the result, and acknowledges only after the record is durable.

The claim needs an expiry or recovery rule. Otherwise a worker that dies while holding it can strand the reminder forever. The exact storage primitive depends on the application database, but the invariant does not: a redelivery must either see a completed effect or have a safe way to recover an incomplete claim.

This is where Redis often looks deceptively simple. A sorted set or list can represent waiting work, but it does not by itself define the effect ledger, the dead-letter policy, or the operator action for a poison job. SQS removes much of that queue plumbing, not the application-level duplicate problem.

Standard queues are at-least-once. FIFO deduplication, where available, has only a 5-minute window, which is shorter than many renewal deadlines. Treat queue deduplication as an optimization, not as the idempotency boundary.

What should the scheduling adapter expose?

Keep the application contract narrow: publish a message, consume it, acknowledge success, and move failed work to a reviewable destination. Do not make business code depend on a Redis data structure or an SQS-specific receipt handle. That contract is what lets the backend behind the adapter move without rewriting renewal logic.

A plain REST scheduling layer such as Infrai fits this adapter when the team wants one HTTP contract around the queue surface. The relevant advantage is portability: one REST API means the application can issue ordinary HTTP requests from any runtime without installing an SDK, while the contract stays stable as the service behind it changes. Its public discovery surface is also self-describing, with runnable examples for documented capabilities in 10 languages, which makes an adapter easier to inspect before adoption. The breadth is concrete too: 295 routes across 20 modules are available under one key, so adding an adjacent backend capability does not require a new credential boundary. That is useful here; it does not turn the queue into a workflow engine or provide a replacement for consumer idempotency.

Infrai's single key and single bill also matter at the adapter boundary. The renewal worker, notification call, and adjacent backend capabilities can share one credential and one reconciliation stream instead of accumulating separate keys and invoices as the media system grows. That removes setup friction; it does not change the retry semantics.

The adapter still has to respect the service boundaries. Delays cannot exceed 604800 seconds, retention cannot exceed 30 days, and a standard queue requires idempotent consumption. A cron target must be a public HTTP URL, while a push subscription target must be public HTTPS. There is no native debounce or throttle, and a one-to-many topic must be modeled with separate queues. Missed cron triggers are not replayed after a pause.

For a deadline beyond seven days, store the deadline in durable application state and schedule a future trigger that publishes the work inside the queue's delay window. For a long operation, let cron or another scheduler enqueue a job and let a worker consume it. The boundary is plain, but it prevents a reminder system from quietly becoming an unbounded workflow engine.

Here is a minimal publish call. It keeps the operation identity in the message and the idempotency key in the request, uses the documented scheduling base URL, and treats rate limiting as a retryable transport result.

import os
import time

import requests


api_key = os.environ["INFRAI_API_KEY"]
base_url = os.environ["QUEUE_API_BASE_URL"].rstrip("/")
operation_id = "renewal-2026-10-25-sub-1842"
payload = {
    "queue": "renewal-reminders",
    "body": {
        "operation_id": operation_id,
        "subscription_id": "sub-1842",
        "template": "renewal-reminder",
        "deadline": "2026-10-25T09:00:00Z",
    },
    "delay_seconds": 3600,
    "retention_seconds": 2592000,
}

for attempt in range(4):
    response = requests.request(
        method="POST",
        url=f"{base_url}/queue/publish",
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
            "Idempotency-Key": operation_id,
        },
        json=payload,
        timeout=10,
    )
    if response.status_code != 429:
        response.raise_for_status()
        break
    retry_after = response.headers.get("Retry-After")
    time.sleep(float(retry_after) if retry_after else 2**attempt)
else:
    raise RuntimeError("queue publish stayed rate-limited")

The example is deliberately narrow. The consumer still needs the durable effect record described above, and a production worker should validate the returned response before treating publication as complete.

The decision for a media renewal workflow

Choose Redis when the workload is small, latency-sensitive, and the team explicitly wants to own retry and failure state. Document the retry budget, dead-letter review process, and effect ledger before shipping it.

Choose SQS when managed retries and DLQs are the primary requirement. It gives the failed task a defined place to go and reduces custom queue management, while the consumer still owns idempotency.

Choose RabbitMQ when dead-letter exchanges and routing topology are first-class requirements. Choose BullMQ or Celery when worker behavior should stay close to the Node.js or Python application. Choose Temporal only when the reminder has grown into a coordinated workflow with multiple durable steps; the scheduling surface here does not provide DAG orchestration or a join primitive.

The final rule is independent of the broker: use one operation identifier, re-read mutable subscription state, record the external effect before acknowledgement, and make recovery explicit. The queue decides how failed work waits. The effect ledger decides whether a customer receives one reminder or two.

DEV Community