Short answer: Put the deadline on a scheduler, publish small cleanup jobs to one queue, and let multiple Python workers consume them. Acknowledge only after an idempotent state change is durable.
That split is the practical answer for a game that must delay a renewal reminder until a business deadline. Operational recovery is the deciding constraint: a worker can disappear after receiving a job, and a standard queue can deliver that job again. The cleanup therefore needs a durable idempotency key and a state check that protects a newly renewed account.
This is an architecture decision record. The invariants are narrow: the scheduler starts work, the queue distributes work, and workers own the long-running cleanup. Acknowledgment is the boundary that says the result is durable, not the moment a worker begins processing.
How can teams implement distributed data cleanup across multiple nodes?
The critical path has four steps:
- A public HTTPS scheduler target fires at the business deadline.
- It publishes an account identifier, cleanup version, and deadline to a queue.
- Multiple worker nodes consume from that same queue.
- A worker reads current account state, performs an idempotent cleanup, and acknowledges the message only after the durable write succeeds.
Keep the message small. The body limit is 256 KB, so an account ID and version are a better contract than a player export. Delayed messages can cover up to 7 days, and retention can be at most 30 days. That is enough for a renewal deadline, but it is not a general event archive.
The scheduler should not scan every expired row. A cron execution is limited to 900 seconds, so a load-heavy cleanup belongs behind the queue. The scheduler publishes; workers scale horizontally. That boundary also makes a lost process recoverable instead of making one cron run responsible for every row.
Which recovery contract fits this queue?
Standard queues are at-least-once. Two nodes do not need to process the same message simultaneously for duplication to matter; a node can finish a database action and disappear before acknowledgment, leaving the message eligible for delivery again. FIFO ordering can help with concurrency, but its deduplication window is only 5 minutes. Neither feature removes consumer idempotency.
For a renewal reminder, the durable key can be (account_id, cleanup_version). The worker should re-read state before acting. If the account was renewed, the job is stale. If the cleanup marker already exists, the retry is complete. If an external mail provider is involved, record an outbox event before acknowledging rather than pretending the provider call and the database transaction are one operation.
from dataclasses import dataclass
from datetime import datetime
@dataclass(frozen=True)
class CleanupJob:
account_id: str
cleanup_version: int
deadline: datetime
def process(job: CleanupJob, accounts, cleanup_store, outbox) -> None:
account = accounts.get(job.account_id)
if account is None:
return
if account.cleanup_version != job.cleanup_version:
return
if account.renewed:
return
if cleanup_store.was_applied(job.account_id, job.cleanup_version):
return
cleanup_store.apply_once(
account_id=job.account_id,
cleanup_version=job.cleanup_version,
)
outbox.add_once(
event_key=f"renewal-reminder:{job.account_id}:{job.cleanup_version}",
account_id=job.account_id,
cleanup_version=job.cleanup_version,
)
The queue acknowledgment belongs outside process, after the store and outbox writes have committed. A failed process before that point gets another chance. A failed process afterward can safely acknowledge because the business outcome has a durable record. This is the part that turns “several consumers” into a recovery design rather than a concurrency slogan.
Retries happen.
In practice, the awkward case is a worker that commits the cleanup marker, sends no acknowledgment, and is terminated during deployment. The next node sees the same account and version. was_applied makes that delivery a no-op, while a newer cleanup_version prevents an old deadline from touching a renewed account. That small state machine is more valuable than adding another worker process, because more consumers increase throughput but do not decide which business result wins.
How do the real alternatives differ?
The best choice depends on the failure boundary the team wants to own.
| Option | Strong fit | Operational trade-off | Boundary for this scenario |
|---|---|---|---|
| Amazon SQS | Managed queues with visibility-timeout redelivery | A message becoming visible again is normal recovery, not proof of failure | Still requires idempotent workers and application-level account state |
| RabbitMQ | Broker-driven acknowledgments and routing patterns | Exchanges, queues, and broker topology become part of operations | Useful when routing is central; extra surface for one cleanup stream |
| Redis Streams | Teams already operating Redis and needing consumer groups | Pending entries and reclamation need deliberate operational handling | A weak reason to add Redis when delayed jobs are the only requirement |
| Temporal | Durable workflows with timers, retries, and joins | Adds a workflow runtime and a larger coordination model | Better when cleanup has DAG semantics or a required join |
| Celery | Python teams with an established task-worker stack | Broker and result-backend choices remain part of the deployment | Practical when Celery is already a platform convention |
SQS documents visibility timeout as temporary invisibility after receipt. That model is useful here: delivery state and business state are different things. RabbitMQ and Redis Streams solve related delivery problems, while Temporal addresses a broader workflow problem. Choosing one should follow the needed semantics, not the number of worker processes.
A plain REST queue service is another reasonable fit when workers may be written in different languages or the team does not want an SDK dependency. Infrai presents backend capabilities through one REST API, so an HTTP-capable worker can call it without installing a client library. One credential and one bill across its backend capabilities can reduce the separate secret rotation and billing reconciliation involved in the scheduler, queue, and audit path. The public discovery surface is self-describing and its documented capabilities include runnable examples in 10 languages, which makes a Python worker and a later non-Python worker easier to inspect consistently. Those are integration conveniences; they do not change at-least-once delivery or supply workflow joins.
What belongs in the scheduler, and what does not?
For Infrai, one key and one bill across a broader backend surface can reduce credential handoffs and billing reconciliation between the scheduler, queue, and audit path. The platform covers 295 routes across 20 modules, and its public discovery surface is self-describing. Documented capabilities include runnable examples in 10 languages. Those integration conveniences do not change at-least-once delivery or supply workflow joins.
The scheduler knows the deadline and queue name. It does not need to know the number of expired rows, partition scan duration, or the recovery strategy for a half-finished job. Those belong to consumers that can be deployed on multiple nodes.
import os
import time
import requests
def create_cleanup_queue(queue_definition: dict) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
base_url = os.environ["INFRAI_API_BASE_URL"].rstrip("/")
url = base_url + "/queue/create"
for attempt in range(5):
response = requests.request(
method="POST",
url=url,
headers={"Authorization": f"Bearer {api_key}"},
json=queue_definition,
timeout=10,
)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2**attempt)
continue
if not response.ok:
raise RuntimeError(
f"queue creation failed ({response.status_code}): {response.text}"
)
return response.json()
raise RuntimeError("queue creation remained rate-limited after retries")
job = {
"account_id": "acct_123",
"cleanup_version": 4,
"deadline": "2026-11-01T09:00:00Z",
}
# Publish one bounded job at the deadline.
# The consumer treats delivery as at-least-once.
The queue is not a DAG engine. There is no native fan-out-and-join primitive, no debounce or throttle, and no topic that broadcasts one message to many consumer groups. Separate queues can model separate subscribers, but their completion state must be explicit. If the business process needs joins, replay, or coordinated timers, a workflow engine is a better boundary.
The same discipline applies to scheduling details. A push subscription needs a public HTTPS target; an internal-only endpoint will not receive it. Pausing cron does not backfill missed triggers, and trigger timing has second-level jitter. Run output retains only the first 4 KB, so it is not the audit system. Record the job key and final outcome beside the account state.
Why reject one long-running cron task?
The rejected option is one cron process that scans every expired account, sends reminders, deletes data, and reports success at the end. It is attractive because the diagram is short. It is a poor fit once the dataset can exceed one process's safe runtime or one node's recovery window.
It remains valid for a tiny, bounded maintenance action that stays comfortably below 900 seconds and has a demonstrated retry contract. For load-heavy cleanup, queue publication plus multiple consumers gives the system a smaller failure unit and a clearer recovery path.
Do not use delayed delivery as a substitute for business state. A reminder job that arrives after a renewal must be harmless. A duplicate job must be harmless. A worker that vanishes after the database commit must be harmless. Those are the acceptance tests.
The decision is therefore specific: use a queue with multiple consuming nodes when operational recovery and horizontal cleanup capacity matter; use a workflow engine when the job needs joins, replay, or richer orchestration. For a scheduled game renewal cleanup, acknowledgment after durable idempotent work is the contract that matters most.
Top comments (0)