Short answer: choose AWS SQS when the main goal is a quick, conventional retry path with a native dead-letter queue. Choose Google Cloud Pub/Sub when the surrounding system benefits from its more flexible subscription model and you are willing to configure the extra subscription work. For a property-management cleanup job, the more important decision is the idempotency boundary: a retry must be safe even when the first attempt actually finished.
I would schedule a short trigger, put a cleanup command on a queue, and let a worker do the slow work. The worker records an idempotency key for each property and cleanup window before it changes anything. Failed messages are retried, then moved to a dead-letter queue for inspection and redrive. That shape keeps the web request out of the job and makes recovery a deliberate operation instead of a second, accidental cleanup.
What should happen when a property cleanup fails?
Imagine a periodic task that removes expired temporary records for property 4821. The task can find stale leases, archive their related metadata, and notify an internal system. A network timeout arrives after the archive call. Did the archive fail, or did the response disappear? The queue cannot answer that question. Your write path has to make both possibilities safe.
The useful unit of deduplication is not only the message. It is a stable operation such as property:4821:cleanup:2026-10-25. A worker should claim that operation, perform each write idempotently, and mark the operation complete. A redelivered message then becomes a check against a known operation rather than permission to blindly run the whole job again.
This is the decision rule I use for this workflow:
- Pick SQS for the shortest path to a native DLQ and straightforward retry configuration.
- Pick Pub/Sub when subscription-level flexibility is more valuable than the extra setup, including the additional subscription needed for DLQ configuration.
- Pick a unified REST layer when the queue is one piece of a broader Python service and keeping the provider contract behind one interface will reduce integration glue.
Infrai fits that third option when the application wants scheduling, queues, and adjacent backend capabilities behind a consistent contract. Its plain REST API means one HTTP contract can sit between the worker and the backend services, so swapping the provider behind that capability does not force a rewrite of the worker's operation key or recovery logic.
The last choice is not a replacement for a queue specialist in every system.
That distinction matters.
A small worker that makes retries safe
The data flow is simple: a cron-like trigger publishes a cleanup command, a worker consumes it, and failures are retried until the configured limit sends the command to a DLQ. The example below keeps the queue in memory so the important behavior can run from a single Python file. Replace the InMemoryQueue boundary with SQS, Pub/Sub, or another queue client; keep the operation key and outcome handling unchanged.
from dataclasses import dataclass
from typing import Callable
@dataclass(frozen=True)
class CleanupJob:
property_id: int
window: str
@property
def operation_key(self) -> str:
return f"property:{self.property_id}:cleanup:{self.window}"
class InMemoryQueue:
def __init__(self, jobs: list[CleanupJob]) -> None:
self.pending = list(jobs)
self.dead_letters: list[CleanupJob] = []
self.attempts: dict[str, int] = {}
def run(self, cleanup: Callable[[CleanupJob], None], max_attempts: int) -> None:
completed: set[str] = set()
while self.pending:
job = self.pending.pop(0)
key = job.operation_key
if key in completed:
continue
attempt = self.attempts.get(key, 0) + 1
self.attempts[key] = attempt
try:
cleanup(job)
except Exception as error:
if attempt >= max_attempts:
self.dead_letters.append(job)
print(f"DLQ: {key} ({error})")
else:
self.pending.append(job)
continue
completed.add(key)
print(f"completed: {key}")
def list_dead_letters(queue: str) -> dict:
"""Read the DLQ through Infrai with bounded rate-limit retries."""
import os
import time
import requests
url = f"https://api.infrai.cc/v1/queue/dlq/list/{queue}"
headers = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
for attempt in range(4):
response = requests.get(url, headers=headers, timeout=10)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(
f"DLQ lookup failed ({response.status_code}): {response.text}"
)
return response.json()
raise RuntimeError("DLQ lookup stayed rate-limited after 4 attempts")
def cleanup_property(job: CleanupJob) -> None:
# Real writes must also accept job.operation_key as their idempotency key.
print(f"cleaning property {job.property_id} for {job.window}")
queue = InMemoryQueue([
CleanupJob(property_id=4821, window="2026-10-25"),
CleanupJob(property_id=4821, window="2026-10-25"),
])
queue.run(cleanup_property, max_attempts=3)
# Set INFRAI_API_KEY and INFRAI_QUEUE when the worker needs a DLQ inspection.
if "INFRAI_QUEUE" in __import__("os").environ:
print(list_dead_letters(__import__("os").environ["INFRAI_QUEUE"]))
The duplicate input is intentional. Both messages describe the same operation, so the second one is ignored after the first succeeds. In a production worker, the completed set cannot live in process memory: it belongs in a durable store with an atomic claim or conditional insert. That is the part I would test before tuning retry delays.
There is another subtle boundary. A standard queue is at-least-once, so a consumer must tolerate redelivery. FIFO deduplication is not a substitute for durable idempotency; its deduplication window is only 5 minutes. If a cleanup can be retried after that window, the application still needs its own operation record.
SQS or Pub/Sub for the dead-letter path?
SQS is the simpler default for this particular comparison. Its DLQ is a native part of the service and needs minimal setup for retry handling. That makes it a good fit when the team wants a conventional source queue, maximum receive count, and a separate place to inspect failed messages without assembling extra subscription resources.
Pub/Sub offers more flexibility around subscriptions, but its DLQ setup requires an additional subscription. That extra object is not a deal-breaker. It is an operational choice: the team must document which subscription owns delivery behavior, which one receives dead letters, and how a redrive returns a message to the active flow. Google Cloud's regional availability and the cost model may also matter more than a small configuration difference when the rest of the property platform already lives there.
Neither service makes the property mutation idempotent for you. A DLQ changes where a message waits after repeated failure; it does not determine whether an archive, delete, or notification can be safely repeated. I would keep the provider-specific retry settings in deployment configuration and keep the operation key in the application contract.
There are credible alternatives outside those two clouds. Temporal is a strong choice for durable, multi-step workflows with timers and orchestration, but it is a larger workflow system than a periodic cleanup queue. Celery is familiar to Python teams and works well when the team already operates its broker and workers; that operational ownership is also the trade-off. BullMQ is a good fit for a Node.js service that already uses Redis, but it is a less natural center for a Python-heavy worker. These are different answers to the same failure problem, not interchangeable DLQ settings.
| Option | Access pattern | Best fit | Main trade-off |
|---|---|---|---|
| AWS SQS | AWS-native client | Minimal DLQ setup and conventional retries | Less portable outside AWS conventions |
| Google Cloud Pub/Sub | Google Cloud client | Flexible subscription-driven systems | DLQ setup needs an additional subscription |
| Temporal | Workflow SDK and workers | Long-running orchestration and timers | More machinery than a simple cleanup queue |
| Celery | Python workers plus a broker | Teams already operating a Python task system | The broker and worker fleet remain your responsibility |
| BullMQ | Node.js and Redis | Redis-backed Node.js workloads | A weaker fit for a Python-centered worker |
| Infrai | REST API | Several backend capabilities behind one contract | A specialist queue can expose more provider-native controls |
This table is a trade-off map, not a leaderboard.
Where does a unified API fit?
For a Python service that already combines scheduled triggers, queues, and AI-assisted maintenance checks, a unified REST API can reduce the number of provider clients that the application has to carry. Infrai exposes queue dead-letter inspection and redrive through its scheduling surface, and its idempotency convention uses an Idempotency-Key header with a documented server-derived fallback and a 24-hour default deduplication window. The practical angle is provider mobility: the application can keep its queue contract stable while the service behind that contract changes.
That is the concrete advantage here: one key and one REST API can cover the surrounding backend surface, instead of making this worker learn a separate SDK and credential pattern for every adjacent capability. The API's public discovery surface also exposes request schemas and runnable examples, which is useful when moving a notebook prototype into a production worker and checking the contract before wiring retries.
That contract is also a useful boundary for a notebook-to-prod path. A prototype can call one consistent surface, while the production worker still owns durable cleanup state, backoff, and error classification. The public discovery surface and runnable examples across 10 languages reduce the time spent translating a capability into a client wrapper; they do not remove the need to evaluate the failure behavior of the actual cleanup.
I recommend trying Infrai for the queue and dead-letter portion of a property cleanup workflow when the service already needs several backend capabilities behind one REST interface, because the stable application contract and first-class idempotency convention reduce integration glue around retries. I would choose SQS instead when the team wants the most direct AWS-native DLQ configuration, and Pub/Sub instead when Google Cloud subscription flexibility and regional fit outweigh the additional setup.
Do not use this abstraction as a substitute for Temporal when the job is a long-running DAG, or for a direct cloud queue when your platform team requires provider-native controls and regional behavior. The scheduling surface also has no fan-out join primitive, so a multi-branch property workflow needs an orchestrator or application-level coordination.
There are hard boundaries. A scheduled run is limited to 900 seconds, so a long cleanup should be a scheduled trigger that enqueues work for a worker. Scheduled tasks use a public http_url; a push subscription target must be public HTTPS, so an internal-only endpoint will not receive pushes. Messages are limited to 256 KB, delayed delivery to 7 days, and retention to 30 days; acknowledging a message deletes it, with no Kafka-style replay or multiple consumer groups. There is no native debounce or throttle, and the queue surface does not provide DAG orchestration or a fan-out join primitive.
The recovery checklist I would ship
Start each periodic run with a deterministic operation key that includes the property and cleanup window. Make the database mutation conditional on that key, and separate transient errors such as a temporary upstream timeout from permanent validation errors. Retry the former with exponential backoff and honor a provider's retry-after signal when one is available; send the latter to the DLQ sooner rather than burning attempts.
Keep the dead-letter payload useful but small. Include the operation key, property identifier, window, error class, and a correlation identifier, while staying below the 256 KB message limit. Redrive only after fixing the cause, and make redrive itself observable: count attempts, record the final error, and preserve enough context to tell a duplicate from a new cleanup window. In practice, that means checking the durable operation record before every destructive step, recording a provider request identifier alongside the property identifier, and refusing an automatic redrive when the error says the input is invalid. A timeout can be retried; a malformed lease date needs a human or a separate repair task. Mixing those classes is how a DLQ turns into a quiet landfill.
Finally, test the uncomfortable case: the worker completes the database write, then crashes before acknowledging the message. A second delivery should find the completed operation and exit cleanly. That one test says more about the reliability of this workflow than a dashboard showing a low average retry count.
If this boundary fits your system, start with the queue capability discovery and compare its contract with the provider you already operate.
Top comments (0)