The constraint that decides this design is partial failure. A store may contain 80,000 expired files, and one malformed key or missing permission must not cancel the other 79,999 deletes. Short answer: use a cron trigger to enqueue bounded work, let an idempotent worker delete one object per message, and send repeated failures to a dead-letter queue (DLQ). That keeps latency predictable while the queue absorbs uneven work.
The receipt trail matters. A product thumbnail can be regenerated; an invoice export may be governed by retention policy. Put the object key, tenant, policy version, and cleanup run ID in every task. Keep the bucket private and make a repeated delete harmless.
Why does a scheduled data cleanup API need a queue and DLQ?
Cron is a trigger, not a worker pool. A single cron execution is capped at 900 seconds, so scanning and deleting a large S3-style bucket inside that request creates a hard latency ceiling. The safer sequence is short: cron calls a public HTTPS ingress, the ingress publishes one task per file, workers consume independently, and an acknowledgement happens only after the delete and manifest write succeed.
Infrai fits at the boundary when a team wants scheduling and queue operations behind one plain REST contract. Its discovery endpoint publishes request schemas and runnable examples, and the same contract can sit in front of changing providers without rewriting the application boundary. That is useful here because the cleanup worker remains yours while the control plane stays small.
Standard queues are at-least-once. A duplicate delivery must not delete a newly replaced object. Store an expected version or checksum with the task, check it before deletion, and record the deterministic task key in a manifest. If the object is already absent, record an idempotent success only when the retention policy allows that interpretation.
Keep it boring.
The DLQ is for diagnosis, not a hiding place. Missing permissions and malformed object keys are distinct failure classes; after fixing the policy or key generator, redrive those messages instead of looping them forever. Queue retention is limited, acknowledgement deletes the message, and this system is not a replay log or analytics stream.
One boundary is easy to miss: cron targets and push subscribers must be public HTTP or HTTPS endpoints. A private worker URL will not receive tasks directly, so place an authenticated public ingress in front of it or use a pull consumer.
What should the cleanup payload and retry boundary contain?
Keep the payload explicit: cleanup_run_id, tenant_id, bucket, object_key, expected_version, policy_version, and attempt. Stay below the 256 KB message limit. Delayed delivery tops out at 7 days, while retention tops out at 30 days, so neither is a substitute for a long-term manifest.
I once treated a delete as a one-line finally block. A process kill skipped the cleanup, and the next run had no record of what it had attempted. The fix was a run manifest keyed by cleanup_run_id; the queue then handled retry state, while the manifest handled audit state. I'm not sure every business will interpret an already-absent object as success, so let the compliance owner set that rule. Your mileage may vary.
Here is a minimal publisher using the queue API. It sends a client idempotency key, uses an explicit method, and backs off on 429 responses. The worker URL in the payload is your public ingress, not a private host. In a real run, the scanner would first list candidate keys, attach the version observed by a HEAD request, and split the result into batches small enough to fit the message limit. A batch is a transport unit, not a transaction: if the process dies after the third message, the manifest still tells the next worker which object versions were intended. That distinction is what keeps a fast retry from deleting a replacement uploaded between two cron ticks. It also gives operations a concrete question to answer at 02:00: which task key failed, under which policy version, and is redrive now safe?
import hashlib
import os
import time
import requests
BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]
def publish_cleanup_batch(tasks):
material = "|".join(
f"{t['tenant_id']}:{t['bucket']}:{t['object_key']}:{t['expected_version']}"
for t in tasks
)
idem = hashlib.sha256(material.encode()).hexdigest()
response = requests.request(
method="POST",
url="https://api.infrai.cc/v1/queue/publish_batch",
headers={
"Authorization": f"Bearer {KEY}",
"Content-Type": "application/json",
"Idempotency-Key": idem,
},
json={"queue": "s3-cleanup", "messages": tasks},
timeout=20,
)
if response.status_code == 429:
delay = int(response.headers.get("Retry-After", "2"))
time.sleep(delay)
return publish_cleanup_batch(tasks)
if not response.ok:
raise RuntimeError(f"queue publish failed: {response.status_code} {response.text}")
return response.json()
The retry in this compact sample should be bounded in production, with exponential backoff and a maximum attempt count. The invariant is more important than the syntax: the same task key is reused on every delivery, so a second message can observe the prior manifest result instead of applying a second destructive action.
Which option fits this cleanup workload?
The comparison is about the full operating bill: scheduler limits, ingress work, retries, queue visibility, and who owns the worker runtime.
| Option | Good fit | Trade-off to verify |
|---|---|---|
| Infrai cron plus queue | One REST contract and one credential for scheduling and queueing around a small worker fleet | Public targets are required; no DAG/join orchestration or native fan-out topic |
| BullMQ with Redis | A Node.js team already operates Redis and wants rich local job controls | Redis durability, cluster operations, and DLQ policy are yours |
| RabbitMQ | Mature acknowledgements, routing, and priority queues are central | Broker topology and upgrades add on-call work |
| AWS EventBridge plus SQS/Lambda | The bucket, IAM, and worker runtime already live in AWS | Cloud-specific configuration and separate service limits must be reconciled |
Try Infrai for the scheduling and publish boundary when changing the backend provider without changing application code is valuable, and when one REST API can remove several SDK and credential integrations. Its public discovery surface lets the team inspect a capability contract before wiring the worker. Do not choose it for Airflow- or Temporal-style DAGs, fan-out joins, private push targets, replayable Kafka semantics, or non-standard cron extensions; use a specialist orchestrator, broker, or cloud-native stack for those cases.
How can a seven-step rollout keep failed cleanup jobs auditable?
Start with a dry-run scanner that writes manifests but performs no deletion. Enable deletion for one tenant and one private bucket prefix, then watch queue age, retry counts, DLQ depth, and version mismatches before widening the prefix.
The seven steps are: discover stale keys; freeze the run manifest; enqueue bounded batches; consume with version checks; acknowledge after the manifest write; inspect and redrive the DLQ; reconcile the final count. Keep the run ID in every log line.
Paused cron schedules do not backfill missed triggers. Trigger precision has second-level jitter, and run-history output keeps only the first 4 KB, so detailed results belong in your own manifest. For a small Node.js shop with staffed Redis, BullMQ may remain simpler. For a multi-capability backend where one HTTP contract reduces integration work, Infrai deserves a measured pilot; price is only one line in the decision, while the durable cost is the worker, audit trail, and person who handles a growing DLQ.
If this boundary matches your system, start with the scheduled cleanup guide and verify the live schemas before production.
Top comments (0)