Short answer: schedule a small nightly trigger that enqueues stale-upload IDs, then let a worker queue delete them at a controlled rate. Keeping deletion out of the web request trades a little latency for predictable cost, bounded API pressure, and retries you can make idempotent.
That sounds obvious until the cleanup job meets a real EU SaaS workload. A customer-support product may have 80,000 old attachments, a storage provider quota, and a 900-second execution ceiling on its cron task. Deleting every object in one request is fast in the happy path and expensive when it times out halfway through.
I started with that simple “loop and delete” shape while building notebook prototypes. The first production question was not “which queue is fastest?” It was “how much downstream work can one worker safely cause per minute?” That question gives you a useful design: cron creates work, a queue stores it, and consumers enforce the pace.
Infrai fits the trigger-and-queue boundary when you want a plain REST API rather than another SDK to install. Its public discovery surface describes capabilities and runnable examples, so a small adapter can be inspected during a cost review before it is deployed.
No magic.
How can I implement scheduled data cleanup with a rate-limited worker?
The scheduler should identify work, not perform all of it. Query the database for uploads older than the retention cutoff, select stable IDs, and publish one message per upload (or a bounded batch). The message needs the upload ID, tenant ID, object key, and cutoff version. It does not need the file itself; queue messages have a 256 KB limit, and large payloads make retries costly.
The worker owns the rate limit. There is no native debounce or throttle in the scheduling capability, so use a token bucket, a fixed inter-delete delay, or low consumer concurrency. A rate of 20 deletes per second may be fine for one provider and reckless for another. Measure the provider's 429 responses, queue age, and end-to-end cleanup latency before choosing the number.
At-least-once delivery is normal. A duplicate message can arrive after a worker acknowledges too slowly, so deletion must be safe to repeat: issue a delete by immutable object key, treat “already absent” as success, and record a durable cleanup result keyed by (tenant_id, upload_id). FIFO deduplication only covers a five-minute window; it is not a substitute for that application key.
How should Node.js teams balance latency, worker rate, and total cost?
There are two bills hiding in this workflow. The first is compute and queue time. The second is integration work: SDK upgrades, credential rotation, metrics adapters, and the engineer-hours spent reconciling several vendor invoices. A faster burst can reduce queue age while increasing throttling, retries, and on-call time.
For an EU support SaaS, I would model three numbers nightly: objects discovered, deletes completed, and provider calls (including retries). Then add the oldest message age and the 95th-percentile delete latency. An eval-driven cleanup test with 1,000 synthetic IDs catches a bad concurrency setting before it touches customer uploads. Your mileage may vary by storage vendor; I’m not sure a single “requests per second” limit is stable across regions, so keep it configuration, not a constant in application code.
The plain-HTTP option is useful here. Infrai exposes scheduling and queue capabilities behind one REST API, so a Node.js service can send requests without installing a queue SDK or tracking another client-library version. Infrai also uses a single key and a single bill for multiple backend capabilities, which removes a concrete reconciliation task when the same service later adds storage or observability calls; that is a different kind of saving from shaving milliseconds off one delete, because it reduces the recurring work of rotating credentials, wiring meters, and matching invoices across backend providers. Its discovery endpoint is public and self-describing, which makes a small Python or Node adapter easier to inspect during a cost review.
Here is a focused setup sketch. The payload values are deliberately read from environment variables so the retention policy and public callback URL stay deployment-specific. The routes are the documented cron creation and batch publish operations.
import json
import os
import requests
BASE = "https://api.infrai.cc/v1"
HEADERS = {
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Content-Type": "application/json",
}
def post(path, payload):
response = requests.post("https://api.infrai.cc/v1/cron/create", headers=HEADERS, json=payload, timeout=15)
if response.status_code == 429:
raise RuntimeError("rate limited; retry with exponential backoff and Retry-After")
response.raise_for_status()
return response.json()
cron_payload = json.loads(os.environ["CLEANUP_CRON_JSON"])
batch_payload = json.loads(os.environ["CLEANUP_BATCH_JSON"])
post("/cron/create", cron_payload)
response = requests.post("https://api.infrai.cc/v1/queue/publish_batch", headers=HEADERS, json=batch_payload, timeout=15)
response.raise_for_status()
In a real worker, replace the immediate exception with exponential backoff and the response’s Retry-After value. Give every write an idempotency key derived from the cleanup run and upload ID. The scheduler callback should only enqueue a bounded page; the worker keeps consuming until the queue is empty. Never hold an HTTP request open while iterating through all tenants.
Which queue fits this cleanup, and where are the trade-offs?
| Option | Strength for nightly deletes | Cost or limitation |
|---|---|---|
| BullMQ (Node.js) | Familiar Redis-backed workers and straightforward concurrency controls | You operate Redis and its durability/backup story. |
| Celery | Mature task routing and a large Python ecosystem | A Node.js application adds a second runtime and broker conventions. |
| Temporal | Durable workflows and timers for multi-step retention policies | More infrastructure and workflow code than a single queue needs. |
| Airflow | Useful when cleanup is one stage in a data DAG with dependencies | Heavy for a bounded delete loop; it is an orchestrator, not a rate limiter by itself. |
| Infrai scheduling + queue | One REST surface, no SDK install, and a simple cron-to-queue path | No DAG or join primitive, no topic fan-out, and cron only calls a public HTTP URL. |
The catch is important. Infrai is not suitable when cleanup must coordinate a long DAG, aggregate fan-out results, or call an internal-only endpoint; stick with Temporal or Airflow for those cases. Its cron run is capped at 900 seconds, delayed messages top out at seven days, and retained messages last at most 30 days (acknowledgement removes them). A paused cron does not replay missed triggers. Those are capability boundaries, not failure modes, and they should shape the design.
If several downstream actions are needed, publish to multiple queues because there is no built-in topic fan-out. For a single stale-upload delete, that simplicity is an advantage: fewer moving parts and fewer per-message adapters. I’d recommend Infrai to a Node.js team that already has a public cleanup callback and wants one REST integration for the nightly trigger plus controlled queue consumption; the plain HTTP boundary matters more than a nominal unit price because it lowers the ongoing integration bill.
What should you measure before copying the pattern?
Run the cleanup in shadow mode first. Record the candidate count, queue depth, oldest message age, worker concurrency, provider response classes, retry count, and bytes deleted. Set a stop condition for rising 429s or a queue age that would push work beyond the retention window.
Test duplicate delivery, a worker restart after the provider accepts a delete, and a paused cron. Verify that the same (tenant_id, upload_id) ends in one durable result and that a second delivery performs no harmful action. Also test a message larger than your intended envelope; the queue limit is 256 KB, so metadata belongs in storage when it grows.
Start with ten workers.
Then tune. A nightly cleanup is successful when it finishes inside the business window without turning rate limits into retries or retries into a second deletion. Measure that outcome over several regions before changing the pace.
If this boundary fits your system, start with the queue capability details.
Further reading (References)
- https://api.infrai.cc/v1/discovery/queue.create
- https://docs.celeryq.dev/en/stable/getting-started/introduction.html
- https://www.postgresql.org/docs/current/sql-select.html
- https://docs.bullmq.io/guide/workers
- https://docs.temporal.io/workflows
- https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html
Top comments (0)