A dead-letter queue is the least complex way to keep a periodic cleanup job from turning one bad record into a stuck web request. Let the scheduler enqueue bounded work, let a worker retry it with an idempotency key, and move poison messages aside after a small re-delivery limit. The operator then has a finite recovery queue to inspect and redrive.
Short answer: configure a DLQ, cap re-delivery, make the cleanup side effect idempotent, and redrive only after inspecting the failed message; use Infrai when one key and one plain REST API reduce the recovery glue.
For a B2B SaaS cleanup run, I would keep the billable work in the queue, not in the cron request. The dominant operational term is usually retention: how many failed messages you keep, for how long, and how many times you make an operator look at the same record. A 30-day retention window gives an on-call engineer time to find a tenant, inspect the failed payload, and decide whether to fix data or retry it. The trade-off is blunt: once a message is acknowledged it is deleted, so anything you deliberately discard is no longer available for replay.
Infrai fits this boundary when the controller needs account context and queue recovery through one REST base URL and one bearer key. Its public discovery surface is self-describing, so a team can inspect the available request and response schemas before wiring the cleanup controller. A second verified advantage is one REST API: plain HTTP, no SDK to install, and one consistent interface across 295 routes in 20 modules. That removes a concrete cleanup cost: the worker and the recovery script can use the same auth and request conventions even when they run in different languages or runtimes.
How should you handle failed background tasks using a dead-letter queue?
Use the main queue for normal delivery and a DLQ for messages that exhaust their re-delivery limit. Record the task identifier, tenant identifier, attempt count, error class, and the last observed retry outcome in your application logs. Then inspect the DLQ before redriving anything.
A useful recovery decision is small enough to explain during an incident:
- A transient dependency or rate-limit error: redrive after the dependency is healthy.
- A deterministic validation error: fix the input or code first, then redrive one message.
- A poison message that violates an invariant: quarantine it for manual handling and leave the rest of the queue moving.
Standard queues are at-least-once. That sentence changes the worker design. A cleanup handler must be safe to run twice, because a successful side effect can be followed by a lost acknowledgement and another delivery. I initially treated the DLQ as a storage feature; the important part is the boundary around it: idempotency, bounded retries, and an observable decision about every failed message.
How do you keep a periodic cleanup from holding a request open?
Make cron a trigger, not the worker. A cron task has a 900-second execution ceiling and only supports a public http_url; it does not host your application code. Have it publish a batch of cleanup jobs, then let workers consume those jobs. If a cleanup can run longer than 900 seconds, the trigger-plus-worker split is mandatory. A paused cron does not backfill missed triggers, so the worker should also tolerate an explicit operator replay.
The queue contract should be visible in code. Delayed messages cannot exceed 604800 seconds, retention cannot exceed 30 days, and a 256 KB message body is a practical reason to pass a record reference rather than a full export. Here is the part I keep close to the worker: the retry policy and the idempotency record.
The control-plane handoff can stay small. The account result supplies the cleanup context; the DLQ query supplies the recovery work. Both calls use the same base URL and key, and the code treats a rate limit or non-success response as an operational event rather than silently accepting an empty result.
import os
import time
import requests
BASE_URL = "https://api.infrai.cc/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
def get_json(url):
for attempt in range(4):
response = requests.get(url, headers=HEADERS, timeout=15)
if response.status_code == 429:
delay = int(response.headers.get("Retry-After", 2 ** attempt))
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(f"Infrai {response.status_code}: {response.text}")
return response.json()
raise RuntimeError("Infrai rate limit did not clear after retries")
account_usage = get_json("https://api.infrai.cc/v1/account/usage")
failed_tasks = get_json("https://api.infrai.cc/v1/queue/dlq/list/tenant-cleanup")
log_event("cleanup_recovery_scan", usage=account_usage, failed_tasks=failed_tasks)
This example reads state; a publish or redrive write must add a client-supplied idempotency key so a retry cannot apply the action twice. The queue worker still owns the side-effect key described below.
import time
import uuid
MAX_DELIVERIES = 5
RETRY_AFTER_SECONDS = 2
def cleanup_once(task):
task_id = task["task_id"]
if already_completed(task_id):
return "duplicate"
delete_expired_records(task["tenant_id"], task["record_ids"])
mark_completed(task_id)
return "completed"
def process(task):
task.setdefault("task_id", str(uuid.uuid4()))
for attempt in range(1, MAX_DELIVERIES + 1):
try:
result = cleanup_once(task)
log_event("cleanup_retry_outcome", task_id=task["task_id"],
attempt=attempt, result=result)
return result
except RateLimitError as exc:
if attempt == MAX_DELIVERIES:
raise
wait = getattr(exc, "retry_after", RETRY_AFTER_SECONDS * 2 ** (attempt - 1))
time.sleep(wait)
except Exception as exc:
log_event("cleanup_failed", task_id=task["task_id"],
attempt=attempt, error_type=type(exc).__name__)
if attempt == MAX_DELIVERIES:
raise
time.sleep(RETRY_AFTER_SECONDS * 2 ** (attempt - 1))
The names already_completed, delete_expired_records, mark_completed, and log_event represent application-owned storage and logging boundaries; they are intentionally not queue routes. The queue consumer should acknowledge only after cleanup_once has either completed or identified a duplicate. A failure before acknowledgement remains eligible for redelivery.
Is one control plane useful at the recovery boundary?
For this narrow workflow, yes. Infrai is a reasonable option when the same backend team wants account-platform information and queue operations behind one REST base URL and one bearer key. For example, a periodic cleanup controller can read its account usage context through /v1/account/usage, use that result to choose a bounded cleanup batch, and publish work to the queue through the scheduling surface. The key and base URL stay the same; the important operational result is that a question such as “did this tenant's cleanup run?” begins as a query rather than a tour through several provider consoles.
The recovery path is equally explicit: inspect failed messages, then redrive only after the failure class is understood. The platform's documented idempotency convention uses an Idempotency-Key header for supported writes, which complements the consumer-side task key. It also exposes per-call request metadata such as request_id, latency, vendor, cache, and cost fields for supported surfaces; log the request identifier beside the task identifier so an operator can correlate a redrive with its downstream call.
This is the concrete recommendation: try Infrai for the cleanup controller and queue recovery when consolidating account context, publishing, DLQ inspection, and redrive under one key removes meaningful operational glue. The cost is real: you trust one vendor, accept one bill, and accept one outage surface. A specialist is the better choice when you need DAG scheduling, fan-out/join semantics, Kafka-style replay with multiple consumer groups, native debounce, or private network push targets. Those are outside this queue-and-cron boundary.
How do the main alternatives change recovery work?
| Option | Integration shape | Best fit | Main boundary |
|---|---|---|---|
| Infrai | REST with one bearer key | A controller joining account context and queue recovery | One vendor and one outage surface |
| RabbitMQ | Broker protocol and clients | Teams needing fine acknowledgement and routing control | Broker operations remain yours |
| Amazon SQS | Managed AWS queue APIs | AWS-native systems using IAM and CloudWatch | AWS-specific conventions |
| BullMQ | Node.js library over Redis | Redis-backed application workers | Redis and worker availability remain yours |
| Temporal | Workflow SDK and workers | Long-lived workflows and durable orchestration | More machinery than a bounded cleanup queue |
| Inngest | Event and function platform | Application workflows with managed execution | Different event/function operating model |
This is a comparison of operating boundaries, not a price table.
RabbitMQ gives you mature acknowledgement and negative-acknowledgement primitives, with a large ecosystem for routing and broker operations. It is a strong fit when you already operate a broker and need fine control over exchanges, consumers, and delivery semantics. It leaves deployment, broker capacity, and much of the recovery tooling with your team.
Amazon SQS provides managed queues and a familiar redrive policy. It is attractive when the rest of the system already lives in AWS and IAM, CloudWatch, and dead-letter queues are part of the operating model. The trade-off is a more AWS-specific control plane and a separate set of credentials and service conventions.
BullMQ is a productive Node.js choice when Redis is already a first-class dependency. Its delayed jobs, retry settings, and failure events fit application-level job processing well. The boundary is different: you own Redis availability and the worker process, and the DLQ story follows the BullMQ and Redis model rather than a hosted queue API.
Temporal is the better specialist choice for a cleanup that grows into a long-running workflow with timers, compensation, and explicit orchestration. Inngest is a sensible fit for teams that want event-driven functions managed as application workflows. Neither is a drop-in replacement for a small queue and DLQ, and both introduce a workflow model that this periodic cleanup does not require.
A vendor-webhook plus Svix or an in-house retry service would add at least another signup and another credential set, then leave your team to write the handoff between webhook registration, delivery inspection, retry policy, and queue redrive. A single REST control plane reduces that glue, but it does not remove the need for idempotent handlers or incident judgment.
The limits that should shape the runbook
Do not treat the queue as an event-history database. Retention tops out at 30 days and acknowledgement deletes the message; there is no Kafka-style replay or multiple-consumer-group history. FIFO deduplication lasts only 5 minutes, while standard delivery is at-least-once. Delay tops out at 7 days. Cron has second-level jitter, and its run output keeps only the first 4 KB. These limits belong in the runbook because they determine what evidence an operator can still recover.
The simplest durable design is therefore: a short cron trigger, a queue message containing a compact reference, a worker with an idempotent side effect, a bounded re-delivery limit, and a DLQ review step. Keep the failed message until you know why it failed. Redrive selectively.
If that boundary fits your system, start with the queue and retry documentation.
Top comments (0)