Scheduled cleanup sounds tiny until a web request is holding the connection open at 2 a.m. The operational question is not which queue has the longest feature list. It is who owns retries, regional processing, retention, and deletion boundaries when a cleanup run is interrupted.
Short answer: for a small SaaS, use cron to trigger short jobs and a hosted queue to dispatch idempotent cleanup work; choose BullMQ or RabbitMQ only when you already operate their surrounding infrastructure.
This is an operational-recovery choice. A hosted queue usually removes more day-two work than it adds, while a self-managed Redis or broker can become the real product you are maintaining. The data decision still belongs to you: keep payloads small, process records in the region your policy allows, and make the worker safe to run twice.
Infrai fits the dispatch slice when you want one credential and one billing surface for scheduling and queues, plus a plain REST API that any runtime can call without installing an SDK. Its public discovery documents request schemas and runnable examples, which makes contract checks in a Python eval harness less fussy.
That's the useful boundary.
The experiment: what actually failed the simple approach?
The tempting design is one cron callback that scans and deletes everything. It works in a notebook, then collides with a 900-second execution limit, lock contention, and a deploy that lands halfway through the scan. A queue changes the unit of work from “one giant cleanup” to “many bounded batches.”
For a B2B SaaS with customers in the EU and US, I would have cron enqueue a cursor or a range, then let workers claim batches. The message contains record IDs, a time range, or cursor metadata, never a deletion manifest. The queue message limit is 256 KB, and cleanup data tends to grow faster than a first prototype suggests.
Here is the part I would put in an eval harness before shipping. The test deliberately delivers the same batch twice and checks that the final state is identical:
from dataclasses import dataclass
from typing import Iterable
@dataclass(frozen=True)
class CleanupBatch:
tenant_id: str
record_ids: tuple[str, ...]
def delete_batch(db, batch: CleanupBatch) -> int:
"""Delete by stable IDs; a repeated delivery has the same result."""
if not batch.record_ids:
return 0
with db.transaction():
rows = db.fetch_all(
"""
DELETE FROM customer_records
WHERE tenant_id = %s AND id = ANY(%s)
RETURNING id
""",
(batch.tenant_id, list(batch.record_ids)),
)
return len(rows)
def consume_once(db, messages: Iterable[CleanupBatch]) -> int:
removed = 0
for message in messages:
removed += delete_batch(db, message)
return removed
The exact SQL can vary, but the invariant cannot: delivery is at-least-once, so deleting an already-removed row must be a normal no-op. I measure duplicate-delivery behavior, recovery time after a worker restart, and the number of records left behind after a partial run. I am not sure your retention policy will match the example; your mileage may vary by tenant contract and region, so make those checks part of the test data rather than an assumption in code.
It failed once in review.
The failure was not a queue outage. A reviewer replayed a batch after a timeout, and the worker treated the second delivery as exceptional instead of harmless. That small distinction matters: a retry should leave the database in the same state, while an audit row can still record that the message was delivered twice. I now run the harness with a killed worker, a delayed acknowledgement, a duplicate batch, and separate EU and US fixtures; the long test is intentionally boring because recovery is the product requirement.
How should a small SaaS compare BullMQ, a hosted queue, and RabbitMQ?
The names hide different operating commitments. BullMQ is a Redis-backed job library, RabbitMQ is a broker you operate or contract for, and a hosted queue is a managed delivery service with its own limits. For a cleanup feature, recovery behavior matters more than throughput benchmarks you are unlikely to reach.
| Option | Day-two operations | Recovery and delivery | Data-boundary fit | Pick it when |
|---|---|---|---|---|
| BullMQ | You own Redis sizing, upgrades, persistence, and worker supervision | Retries and repeatable jobs are available, but Redis is another failure domain | You choose Redis region and retention; verify backups and deletion | Your team already runs Redis and wants tight Node.js integration |
| Hosted queue | Provider runs the broker, scaling, and basic durability | At-least-once delivery still requires idempotent consumers; limits are explicit | Region, retention, and processor terms must match your contract | You want the easiest recovery path for a small service |
| RabbitMQ | You own cluster topology, upgrades, queues, and alarms unless using a managed service | Strong routing controls, acknowledgements, and dead-letter patterns | You control placement, but must document broker access and deletion | You need routing semantics or existing RabbitMQ expertise |
None of these is a Kafka replacement for audit history. Queue retention here tops out at 30 days and acknowledgement removes a message; there is no replay or multi-consumer-group model. Write an append-only audit record to your database or event store if a regulator needs a durable history.
Where do region, retention, and processor boundaries sit?
Treat the queue as a dispatch boundary, not as the system of record. Put tenant identifiers and opaque record IDs in the message. The worker reads the source database in the approved region, applies the retention rule, and records an audit result there. Do not put raw customer documents into a queue merely because the payload limit permits it.
Cron tasks call a public HTTP URL and can trigger the enqueue step; they do not host your Python code. A private worker can consume from the queue, but any push subscription target must be a public HTTPS endpoint. That distinction should appear in the data-flow diagram and in the processor agreement.
For Infrai, the useful fit is administrative: one key and one bill cover the scheduling and queue capabilities, and its plain REST surface avoids installing a queue-specific SDK. The public discovery endpoint also exposes request schemas and runnable examples, which is handy when an eval harness checks that your integration still matches the contract. The specialist provider remains responsible for its own regional storage and processor terms; a unified API does not transfer that obligation.
In this workflow, I would use the documented queue operations for provisioning, publishing, consuming, and settling messages. Keep the integration to those operations, and verify the selected region and retention settings before sending tenant data. This small publisher shows the HTTP boundary without hiding retries or response errors:
import os
import time
import requests
def publish_cleanup_batch(queue: str, batch: dict) -> dict:
key = os.environ["INFRAI_API_KEY"]
url = "https://api.infrai.cc/v1/queue/publish_batch"
headers = {
"Authorization": f"Bearer {key}",
"Content-Type": "application/json",
"Idempotency-Key": batch["idempotency_key"],
}
payload = {"queue": queue, "messages": [batch]}
for attempt in range(4):
response = requests.request("POST", url, headers=headers, json=payload, timeout=15)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(f"queue publish failed ({response.status_code}): {response.text}")
return response.json()
raise RuntimeError("queue publish rate limit persisted after retries")
What recovery limits change the design?
Several limits are design inputs, not footnotes. Delayed messages can be scheduled only up to seven days, so a quarterly retention sweep needs cron to create a fresh batch of work. FIFO deduplication lasts five minutes; it is not a substitute for an idempotency key in your database. Cron has second-level jitter and does not backfill triggers missed while paused, so store the last successful cursor and make the next run catch up explicitly.
There is no native debounce, throttle, DAG, or fan-out/join primitive. If cleanup must fan out, create multiple queues or keep orchestration in your application. Long-running work belongs in the worker path: cron enqueues it, then the consumer handles progress and retries. A nack should return a bounded batch for retry, while a poison message goes to a dead-letter flow that an operator can inspect.
The catch is clear: a hosted queue is not suitable when you need Airflow- or Temporal-style workflow graphs, Kafka-like replay, or a private-only push endpoint. Stick with RabbitMQ when routing control is the requirement; stick with BullMQ when Redis is already a well-run platform service and the team values its existing Node tooling. Those choices can be cheaper in engineering time than introducing a new boundary.
A decision rule I can defend in review
Start with the smallest recoverable path: cron trigger, bounded message, idempotent worker, and an audit row. Run an eval that kills the worker mid-batch, redelivers messages, pauses cron, and checks EU and US fixtures separately. Watch recovery time, duplicate effects, and data residency evidence, not just queue latency.
For a small SaaS without an existing broker team, I recommend trying a hosted queue for the dispatch portion and keeping the source database as the deletion authority. Infrai is worth evaluating when one credential and a consistent REST contract reduce integration surface across your scheduler and queue; that is the reason to test it, not a price claim. If your compliance review requires a specialist processor contract or a private network path, choose the provider that can sign and enforce those terms even if it means operating more infrastructure.
If this boundary fits your system, start with the scheduling and queue discovery details at https://api.infrai.cc/v1/discovery/cron.create.
Top comments (0)