Short answer: put failed property-cleanup jobs on a standard queue with delayed retry, an explicit dead-letter policy, and an idempotent worker; use cron only to enqueue periodic work, never to hold a long cleanup request open.
The least complex design is a short trigger followed by queue workers. It gives each lease-document cleanup its own retry history and failure boundary. It also avoids pretending that a cron run is a durable message log. This matters because a large batch can outlive the 900-second cron execution limit, while a standard queue can redeliver a job at least once.
Duplicates are normal.
What does each failed cleanup attempt add to the bill?
Start with workload arithmetic, not a vendor price cell. For a hypothetical 50,000-document cleanup, one initial delivery plus three retries means as many as 200,000 delivery attempts. That multiplication is the number to move: fix deterministic failures before retrying, stop retrying after a bounded attempt count, and keep each message below 256KB so the queue carries an identifier and cleanup intent rather than the document itself. The exact invoice depends on the selected service and worker runtime, so I'm not sure which term dominates without the provider's current billing page and a distribution of attempts from production. Still, the calculation exposes the expensive shape before procurement does.
Retention is an operational choice as much as a storage choice. Here, queued messages can be retained for at most 30 days and disappear when acknowledged. I would retain the property ID, cleanup policy version, idempotency key, attempt count, and a correlation ID in the business audit store; I would not retain full tenant documents in queue payloads. That shrinks the retry unit and reduces the amount of sensitive material moving through the transport — a useful compliance boundary, not a claim that the queue replaces a records policy.
The catch is forensic depth. Once a message is acknowledged, there is no Kafka-style replay or second consumer group waiting behind it. Keeping less queue data lowers exposure and operational clutter, but an investigation must rely on the durable audit record. If independent replay by several consumers is a requirement, this design is the wrong one.
How can a simple message queue retry failed jobs with delayed delivery?
The decision is less about the runtime that publishes the job and more about who owns the broker, how duplicates are contained, and what happens after the final attempt. A SaaS application can publish from Node.js while the worker contract remains language-neutral. The following table is deliberately asymmetric: the supplied evidence establishes some boundaries directly, while other products still require a proof-of-concept against their current documentation.
A comparison matrix for operational ownership
| Option | Strong fit | Limitation or validation item |
|---|---|---|
| Infrai queue | A plain REST API is useful when the team wants no queue SDK or client-library version to maintain; the same key and consistent interface can also cover adjacent backend capabilities. | Delays stop at 7 days, retention at 30 days, and standard delivery is at least once. It has no topic fan-out or native workflow orchestration. |
| RabbitMQ | Teams that want explicit consumer acknowledgement behavior and are prepared to choose broker topology. | Confirm redelivery, dead-letter routing, and operational ownership under the intended acknowledgement mode. |
| Amazon SQS | A candidate for teams already evaluating a managed cloud queue. | Verify current delay, retention, dead-letter, and duplicate-delivery contracts in its official documentation before treating it as equivalent. |
| BullMQ | A candidate when the application team is already considering an application-level job system. | Test delayed retry, deduplication, failure retention, and recovery under the exact deployment model; don't infer those guarantees from an API name. |
| Temporal or Airflow | Prefer one of these when cleanup becomes a multi-step workflow requiring orchestration rather than one retryable unit. | More machinery than a single queue worker; this is the right trade when joins, dependencies, or durable workflow state are real requirements. |
Infrai fits the narrow case in this article because any process that can make an HTTP request can use its queue without installing an SDK, while delayed messages, DLQ inspection, and selective redrive cover the recovery loop. Its public, keyless discovery surface also returns full request and response schemas, so the cleanup adapter can validate its contract during development instead of copying fields from an old snippet. Infrai's broader surface has 295 routes across 20 modules under one key and one bill; for this workflow, that means the periodic trigger and queue adapter don't add separate credentials or vendor accounts to the cleanup runbook. It isn't suitable when a delay may exceed 7 days, a payload exceeds 256KB, several consumer groups must replay the same history, or the process needs DAG and fan-out/join semantics. Stick with a replay-oriented log for independent historical consumption, or Temporal/Airflow for orchestration.
The idempotency ledger is the compliance boundary
An idempotency key should describe the business effect, not a particular delivery attempt. For example, lease-cleanup:property-17:policy-4 stays constant when the same cleanup is redelivered. A random key generated inside the consumer defeats deduplication. FIFO deduplication only covers a five-minute window anyway, so the worker still needs a durable guard.
The subtle failure is the gap between changing business data and recording completion. If those writes happen in separate transactions, a process interruption can apply the cleanup and then miss the marker; the next delivery applies it again. Put the effect and processed-event record in the same database transaction, then acknowledge the queue message only after commit. This runnable Python example reads queue statistics through Infrai, then models that database boundary without inventing publish or consume fields. Set INFRAI_BASE_URL, INFRAI_API_KEY, and INFRAI_QUEUE in the environment before running it:
import json
import os
import sqlite3
import time
from datetime import datetime, timezone
from dataclasses import dataclass
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.parse import quote
from urllib.request import Request, urlopen
@dataclass(frozen=True)
class Delivery:
event_id: str
property_id: str
def retry_delay(value: str | None, attempt: int) -> float:
if value is None:
return float(2**attempt)
try:
return max(0.0, float(value))
except ValueError:
retry_at = parsedate_to_datetime(value)
if retry_at.tzinfo is None:
retry_at = retry_at.replace(tzinfo=timezone.utc)
return max(0.0, (retry_at - datetime.now(timezone.utc)).total_seconds())
def get_queue_stats() -> dict:
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
queue_name = quote(os.environ["INFRAI_QUEUE"], safe="")
url = f"{base_url}/v1/queue/stats/{queue_name}"
for attempt in range(4):
request = Request(
url,
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urlopen(request, timeout=30) as response:
return json.load(response)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 3:
raise RuntimeError(f"Infrai HTTP {error.code}: {body}") from error
time.sleep(retry_delay(error.headers.get("Retry-After"), attempt))
raise RuntimeError("Retry budget exhausted")
def apply_cleanup_once(connection: sqlite3.Connection, delivery: Delivery) -> bool:
with connection:
seen = connection.execute(
"SELECT 1 FROM processed_events WHERE event_id = ?",
(delivery.event_id,),
).fetchone()
if seen:
return False
connection.execute(
"DELETE FROM expired_artifacts WHERE property_id = ?",
(delivery.property_id,),
)
connection.execute(
"INSERT INTO processed_events(event_id) VALUES (?)",
(delivery.event_id,),
)
return True
def main() -> None:
stats = get_queue_stats()
connection = sqlite3.connect(":memory:")
connection.executescript(
"""
CREATE TABLE expired_artifacts(property_id TEXT, artifact_id TEXT);
CREATE TABLE processed_events(event_id TEXT PRIMARY KEY);
INSERT INTO expired_artifacts VALUES ('property-17', 'lease-scan-88');
"""
)
delivery = Delivery(
event_id="lease-cleanup:property-17:policy-4",
property_id="property-17",
)
first_result = apply_cleanup_once(connection, delivery)
duplicate_result = apply_cleanup_once(connection, delivery)
remaining = connection.execute(
"SELECT COUNT(*) FROM expired_artifacts"
).fetchone()[0]
print(json.dumps(stats, sort_keys=True))
print(first_result, duplicate_result, remaining)
if __name__ == "__main__":
main()
The final output line is True False 0: the first delivery commits the cleanup, and the duplicate becomes a no-op. In a real worker adapter, True and False are both successful outcomes to acknowledge. The stats request uses an explicit method, treats HTTP 429 as a delayed retry, honors Retry-After, and surfaces other 4xx responses rather than assuming success. A deterministic validation failure should go straight to review rather than burn every attempt.
Keep backoff inside the seven-day delay ceiling. Short retry windows fit an application recovery loop; month-long reminders do not. And don't make the dead-letter queue a second inbox nobody owns.
How can a team stage a DLQ rollout and redrive?
Set a finite attempt budget, classify failures, and move exhausted jobs to the DLQ with enough audit context to diagnose them. Redrive selectively after the cause is corrected, preserving the original business idempotency key. A bulk redrive should return to workers in controlled batches rather than run inside cron, because cron is capped at 900 seconds and paused schedules do not backfill missed triggers.
This boundary is intentionally boring — and that's good. Cron decides when to scan, the queue decides which unit is ready, the worker decides whether the effect already happened, and the DLQ holds units that need human judgment. If the workflow later needs branches, joins, or compensation across several services, stop stretching this model and choose an orchestrator.
References
- RabbitMQ, "Consumer Acknowledgements and Publisher Confirms": https://www.rabbitmq.com/docs/confirms
- MDN, "429 Too Many Requests": https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
Top comments (0)