Short answer: run a short nightly scheduler that enqueues stale-upload IDs, then let a bounded worker queue delete them at a rate your storage and partner APIs can tolerate. Do not make the scheduler perform the deletes. A cron invocation has a 900-second ceiling, while a queue gives you retries, back-pressure, and a place to enforce latency versus cost.
This is the shape I use for a European SaaS cleanup job: select uploads older than the retention policy, enqueue one small message per object, and consume at a fixed concurrency. The delete operation must be safe to repeat because standard delivery is at least once. A second attempt should see an already-absent object as success, not as a new incident.
Why the nightly trigger should stay small
The scheduler has one job: wake up and publish work. It should not scan a million rows and synchronously call a storage provider. That design turns a slow partner response into a missed run, and the run history only keeps the first 4 KB of output anyway.
Keep the message boring: upload ID, tenant ID, region, and an idempotency key. Keep it below the 256 KB message limit. The worker can look up the current object location, check the retention decision again, and issue the delete. Rechecking matters when a customer restored an export after the original scan.
There is no native debounce or throttle in this scheduling surface. Put the limit in the consumer: a small worker pool, a token bucket, or both. In my tests I start with 4 concurrent deletes and a 200 ms interval between starts, then adjust from observed p95 latency and provider quota responses. Your mileage may vary; the right number depends on the storage account and the size distribution of uploads. That tuning is deliberately boring: I would rather miss a nightly deadline than trigger a provider ban while trying to save a few worker minutes.
Measure first.
A retry is not a duplicate business action if the action is idempotent. Use a deterministic key such as cleanup:${tenantId}:${uploadId}:${retentionDate} and make the database state transition conditional. Never acknowledge a message before the delete and state update have both completed.
How do you implement scheduled data cleanup with a rate limited worker queue in Node.js?
The following worker-facing example uses two scheduling calls: batch publishing from the nightly job, and consuming from the queue. The scheduler itself can be any public HTTPS task that runs the query and invokes the publisher. The API uses plain HTTP, so there is no SDK glue to install.
type CleanupItem = {
tenantId: string;
uploadId: string;
retentionDate: string;
};
const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!baseUrl || !apiKey) throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");
async function publishRequest(body: unknown) {
const response = await fetch(`${baseUrl}/v1/queue/publish_batch`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (!response.ok) {
const detail = await response.text();
throw new Error(`${response.status}: ${detail}`);
}
return response.json();
}
async function consumeRequest(body: unknown) {
const response = await fetch(`${baseUrl}/v1/queue/consume`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (!response.ok) {
const detail = await response.text();
throw new Error(`${response.status}: ${detail}`);
}
return response.json();
}
async function publishNightly(items: CleanupItem[]) {
await publishRequest({
queue: "stale-upload-cleanup-eu",
messages: items.map((item) => ({
body: item,
idempotency_key: `cleanup:${item.tenantId}:${item.uploadId}:${item.retentionDate}`,
})),
});
}
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
async function consumeBatch() {
const result = await consumeRequest({
queue: "stale-upload-cleanup-eu",
max_messages: 4,
});
for (const message of result.messages ?? []) {
try {
const item = message.body as CleanupItem;
await deleteIfStillStale(item); // Treat an absent object as success.
await markDeleted(item); // Conditional on the same idempotency key.
await acknowledge(message.id);
await sleep(200); // Simple start-rate limit.
} catch (error) {
console.error("cleanup attempt will be retried", message.id, error);
await negativeAcknowledge(message.id);
}
}
}
deleteIfStillStale, markDeleted, acknowledge, and negativeAcknowledge are application functions, not hidden platform endpoints. They should carry the same idempotency key into your storage adapter and database transaction. In production I also honor a provider's Retry-After value and use exponential backoff for HTTP 429 responses; a tight retry loop only converts a quota warning into an outage.
The nightly query should claim rows in small pages. PostgreSQL's FOR UPDATE SKIP LOCKED is useful when several scheduler instances might overlap, but it is still your responsibility to record a claim and release it if publishing fails. A queue does not provide a topic fan-out primitive, so if cleanup also needs billing or audit work, publish separate messages to separate queues.
What changes when latency, cost, and operational scope change?
Here is the comparison I would put in an architecture review. These are different tools, not interchangeable price stickers.
| Option | Good fit | Trade-off for nightly cleanup |
|---|---|---|
| BullMQ + Redis | Node.js teams already running Redis and wanting rich local job controls | Extra stateful infrastructure and Redis operations; quotas still need worker code |
| Celery | Python-heavy services with mature task routing | Excellent worker ecosystem, but it adds a Python runtime and broker choices to a Node.js service |
| Temporal | Long, branching workflows with durable history and joins | More operational and conceptual weight than a queue for one idempotent delete |
| A managed cron plus queue API | Small teams that want one HTTP contract and little glue | Less workflow orchestration; you own idempotency, metrics, and retention policy |
Infrai's concrete advantage is one REST API over plain HTTP, with no SDK to install, and one key, one bill covering the scheduler and queue; that makes it a reasonable fit when swapping vendors without changing code matters. Its discovery surface exposes request schemas and runnable examples. That contract is useful for a CLI or SDK because the code stays plain HTTP while the service behind it can change. The other practical advantage here is a broad capability surface with a consistent interface, so storage and scheduling calls can share the same integration conventions. In plain terms, you can swap vendors without changing every call site.
The catch is scope. This is not a DAG engine, and there is no fan-out/join primitive. Delayed messages top out at seven days, retention tops out at 30 days, and acknowledged messages are removed rather than replayed through Kafka-style consumer groups. Cron expressions do not have nonstandard L extensions, missed triggers are not backfilled after a pause, and trigger timing has second-level jitter.
Stick with Temporal or Airflow when cleanup is one stage in a workflow that needs joins, human approval, or a durable event history. Stick with BullMQ when Redis is already your platform and you need its ecosystem. Choose the managed HTTP option when time-to-first-call and low configuration overhead beat those extra controls.
What I would change at scale
At 10,000 uploads, four consumers and a 200 ms start interval may be enough. At 10 million, that same setting becomes a calendar problem. I would shard queues by region or provider quota, expose queue depth and delete latency, and autoscale consumers within a hard concurrency ceiling. The scheduler would only enqueue a bounded page and leave the next page to another run or a continuation message.
I would also separate policy from execution. A policy table answers “is this upload stale?” while the worker answers “can I delete it now?” That split makes legal holds and customer restores explicit, and it keeps a retry from accidentally turning a changed retention rule into data loss.
Start with a dry-run queue in staging. Measure p50 and p95 delete latency, 429 frequency, duplicate delivery rate, and the age of the oldest pending message. Then choose the concurrency that meets the EU tenant's nightly deadline without treating provider quotas as an afterthought.
Top comments (0)