Short answer: for a small SaaS, run scheduled data cleanup as a short cron trigger followed by idempotent work on a hosted queue; choose BullMQ or RabbitMQ only when the team already operates their backing infrastructure well enough that owning another stateful service is genuinely routine.
For a gaming product that sends a weekly digest to active customers, I would make the cleanup and digest selection one explicit data-lifecycle boundary. Expired activity records are removed or marked once, eligible customer IDs are selected, and small queue messages fan the work out to consumers. The queue is dispatch, not history. That distinction matters more than a feature checklist because standard delivery is at least once: a worker may see the same customer or cleanup range again, and correctness cannot depend on seeing it exactly once.
The decision is therefore less about which logo has the lowest published unit price and more about which failure boundary the team can own at 03:00. A hosted queue is usually the easiest and cheapest option to operate here. BullMQ adds Redis operations, while RabbitMQ adds broker operations; either can still be the right choice when that infrastructure and its failure procedures are already part of the team's normal work.
Can scheduled data cleanup respect small SaaS retention across EU and US?
Start with four invariants. First, each cleanup operation has a stable identity, such as a tenant ID plus a closed time range. Second, repeating that operation produces the same database state. Third, the message carries IDs, ranges, or cursor metadata rather than a deletion manifest; the payload ceiling is 256KB. Fourth, the database remains the durable record of what was cleaned and which digest was prepared, because this queue has neither Kafka-style replay nor multiple consumer groups.
The cron handler should do very little: calculate the closed interval, write or locate a batch record, publish its identity, and return. It should not scan every activity row or send every digest inline. A cron execution can run for at most 900 seconds, and timing can have seconds of jitter, so putting long work behind a worker is a correctness choice as much as a latency choice. If cron is paused, missed triggers are not backfilled; the batch identity and a database query for unprocessed intervals provide the recovery model instead.
Keep it boring.
Region labels alone don't settle data residency. Confirm the capability's current regions values during discovery, then separately verify where the application database, worker, and digest provider process data; this design cannot claim an EU or US residency guarantee from queue placement alone. If a required region isn't present in the live contract, stop the evaluation there rather than treating cross-region dispatch as an implementation detail.
The public ingress boundary is also concrete. The cron task calls a public HTTP URL, and a push subscription needs a public HTTPS target. If the worker is reachable only on a private network, use polling from that network rather than assuming the hosted scheduler can cross the boundary. I don't treat that as an incidental deployment detail — it decides whether push is viable before any queue comparison begins. There are also two different clocks in this system. The weekly schedule says when an interval becomes eligible, while the queue controls when a worker attempts it. Delayed messages stop at seven days, so they aren't a durable calendar. Queue retention is at most 30 days and an acknowledged message is deleted. The audit trail belongs in the application database, where a tenant, interval, state transition, and digest key can be queried without depending on a queue message still existing. The useful failure model is at least once all the way down: a worker can delete rows, lose its lease before acknowledgment, and receive the same batch again. "Delete where activity_at is before the cutoff" is naturally repeatable, but associated actions may not be; incrementing a cleanup counter, inserting a digest request, or charging an account twice would change state on every retry. Put those effects behind a unique key and commit them in the same database transaction as the cleanup marker.
For the weekly gaming digest, a reasonable identity is (tenant_id, week_start, operation). The cleanup message contains that identity, not thousands of player records. Inside the transaction, the worker locks or claims the batch, deletes only rows in the recorded range, and inserts a digest outbox record with the same deterministic key. On a repeated delivery, the unique key turns the second attempt into a read of completed state. Acknowledgment comes after commit. If processing cannot complete, negative acknowledgment permits another attempt; retry delay needs a ceiling and must stay inside the seven-day delay limit.
This is the awkward part teams underestimate. Queue retry policy can't repair a transaction that mixes non-idempotent side effects with deletion, and FIFO deduplication doesn't remove the requirement: its deduplication window is only five minutes. The consumer's durable key must survive longer than queue retention because business correctness can outlive the transport.
I'm not sure a generic vendor cost table can settle the choice without the team's Redis, broker, and on-call costs; those numbers are local and usually omitted. The limits are clearer. A 256KB message ceiling rules out large manifests, no native debounce or throttle means coalescing belongs in application state, and no topic-style one-to-many delivery means separate queues are needed for separate recipients. If the cleanup grows into dependencies, joins, or a multi-step recovery graph, this queue-and-cron design has crossed its natural boundary; Airflow or Temporal is the more appropriate class of system.
Migration starts with the batch ledger
The table deliberately compares ownership and recovery shape rather than transient list prices. For a small team, the labor attached to Redis or a broker is part of cost even when the software itself has no license fee.
| Option | What the team owns | Good fit | Main catch for this cleanup job |
|---|---|---|---|
| BullMQ | Application workers plus the Redis stack | A team already operating Redis and wanting queue behavior close to its application code | Redis operations are extra stateful work for a feature whose payload is only a batch identity |
| RabbitMQ | Application workers plus the RabbitMQ broker | A team with established broker expertise and operating procedures | Broker care adds an independent failure boundary to a simple weekly job |
| Amazon SQS | Workers and application idempotency; the queue service is hosted | A team that wants managed work dispatch and already accepts its cloud boundary | The application still needs a durable audit record and retry-safe consumers |
| Infrai cron and queue | Workers, public trigger boundary, and application idempotency; scheduling and queueing are hosted | A small service that values a plain REST surface and wants to inspect schemas before integration | It is work dispatch, not replayable streaming or workflow orchestration; payload, retention, and delay limits must fit |
Infrai is a strong hosted candidate here because its API is self-describing: public discovery exposes each capability's request and response schema plus runnable examples, so evaluating a new capability begins with reading one endpoint rather than installing and learning another SDK. Infrai also places 295 routes across 20 modules behind one API key and one bill, which means the cron trigger and queue don't create separate credentials, invoice reconciliation, or client conventions while the application database remains the source of truth. Those conveniences don't erase the product boundaries in the table.
Amazon SQS is the straightforward hosted comparator. Stick with BullMQ when Redis is already operated, monitored, and restored as a normal part of the service, especially if keeping queue mechanics in the application stack is valuable. Stick with RabbitMQ when the organization already standardizes on it and the broker isn't a new operational dependency. The catch for every hosted option is control: network placement, service limits, and the provider boundary are part of the design, so a private-only consumer may favor polling or an already-local stack.
Python implementation: contract probe and duplicate delivery
Before wiring a queue call, inspect its live contract rather than guessing a REST-shaped path or body. This runnable Python probe reads the queue publishing capability from the public discovery surface, uses an API key from the environment, sets the method explicitly, reports non-success bodies, and backs off on HTTP 429 while honoring Retry-After. Set INFRAI_BASE_URL to the service base URL; keeping it in configuration also prevents a deployment-specific endpoint from leaking into application logic.
import json
import os
import time
import urllib.error
import urllib.request
def read_capability():
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
url = f"{base_url}/v1/discovery/queue.publish"
for attempt in range(5):
request = urllib.request.Request(
url,
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
if response.status < 200 or response.status >= 300:
body = response.read().decode("utf-8", errors="replace")
raise RuntimeError(f"HTTP {response.status}: {body}")
return json.load(response)
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 4:
raise RuntimeError(f"HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("Capability request exhausted its retry budget")
capability = read_capability()
print(json.dumps({
"method": capability["method"],
"path": capability["path"],
"idempotent": capability["idempotent"],
"params": capability["params"],
}, indent=2))
Discovery is the integration gate, not the business correctness proof. The next program isolates the part that must remain correct even if a queue delivers the same cleanup message twice. It is runnable with the Python standard library. SQLite stands in for the application's transactional database, and the repeated call at the bottom represents redelivery after a commit but before acknowledgment.
import sqlite3
def process_cleanup(connection, tenant_id, week_start, cutoff):
batch_key = f"{tenant_id}:{week_start}:weekly-cleanup"
with connection:
existing = connection.execute(
"SELECT deleted_rows FROM cleanup_batches WHERE batch_key = ?",
(batch_key,),
).fetchone()
if existing is not None:
return existing[0]
cursor = connection.execute(
"DELETE FROM activity WHERE tenant_id = ? AND observed_at < ?",
(tenant_id, cutoff),
)
deleted_rows = cursor.rowcount
connection.execute(
"INSERT INTO cleanup_batches(batch_key, deleted_rows) VALUES (?, ?)",
(batch_key, deleted_rows),
)
connection.execute(
"INSERT OR IGNORE INTO digest_outbox(digest_key, tenant_id, week_start) "
"VALUES (?, ?, ?)",
(f"{tenant_id}:{week_start}:weekly-digest", tenant_id, week_start),
)
return deleted_rows
database = sqlite3.connect(":memory:")
database.executescript(
"""
CREATE TABLE activity (
tenant_id TEXT NOT NULL,
observed_at TEXT NOT NULL
);
CREATE TABLE cleanup_batches (
batch_key TEXT PRIMARY KEY,
deleted_rows INTEGER NOT NULL
);
CREATE TABLE digest_outbox (
digest_key TEXT PRIMARY KEY,
tenant_id TEXT NOT NULL,
week_start TEXT NOT NULL
);
INSERT INTO activity VALUES ('studio-7', '2026-08-01T12:00:00Z');
"""
)
first = process_cleanup(
database, "studio-7", "2026-08-10", "2026-08-03T00:00:00Z"
)
retry = process_cleanup(
database, "studio-7", "2026-08-10", "2026-08-03T00:00:00Z"
)
assert first == 1
assert retry == 1
assert database.execute("SELECT COUNT(*) FROM digest_outbox").fetchone()[0] == 1
Ack comes last.
In PostgreSQL, competing workers can claim rows with FOR UPDATE SKIP LOCKED, but the important property is not that particular clause. It is the transaction boundary: one durable batch key covers cleanup and outbox creation, while queue acknowledgment occurs only after the transaction succeeds. A publish or other write also needs its stable idempotency key so retrying the transport cannot double-apply the request.
Cost follows ownership
I would reject a self-managed queue for this specific small SaaS if Redis or RabbitMQ would be introduced only to run the weekly cleanup and digest. The extra data service creates backup, monitoring, upgrade, capacity, and on-call questions without changing the central requirement: consumers must still be idempotent. A hosted queue paired with cron has the smaller operational surface, and that is why it wins this record.
This rejection is narrow.
BullMQ is suitable when Redis is already a trusted production dependency and its operations aren't new work. RabbitMQ is suitable when a platform team already owns the broker boundary. A database queue using FOR UPDATE SKIP LOCKED can also be the smallest sensible design when volume is modest, workers share the database, and adding any queue service would increase rather than reduce operational complexity.
Use a different category entirely when the requirements change. Choose Kafka-style infrastructure when replay and independent consumer groups are the actual product need. Choose Airflow or Temporal when cleanup becomes a workflow with dependency graphs or fan-out/fan-in joins. The queue-and-cron design is not suitable for those cases, and stretching it into an audit log or orchestrator would hide failure state instead of controlling it.
The final decision rule is short: if the payload can be represented by IDs or cursors under 256KB, work can complete through retry-safe consumers, and a 30-day maximum retention window is sufficient, prefer a hosted queue. If the team already owns Redis or RabbitMQ as routine infrastructure, the self-managed option may be simpler in context. Architecture cost is the state you agree to wake up for.
Top comments (0)