Short answer: use cloud cron to discover expired support data, then put each cleanup unit on a queue when the batch can outlive one run or must meet a latency target without exceeding an API rate limit. For a small batch that always finishes comfortably inside one invocation, cron alone is cheaper to operate and easier to understand.
That is the decision I would ship first. The scheduler answers when should discovery begin? The queue answers how quickly may work leave? Treating those as one concern looks tidy until a customer support workspace has a few unusually large tenants and one sweep has to share a limited downstream API budget with live requests.
The interesting trade-off isn't “serverless versus servers.” It is elapsed cleanup time versus the extra storage, worker, and observability cost needed to make that elapsed time predictable. I care about the oldest eligible record, not raw worker speed.
Before choosing machinery, I write an acceptance test around the oldest eligible support artifact. The test starts its clock when the retention rule makes that artifact eligible, not when a worker happens to receive it. That distinction exposes a common measurement gap: worker duration can look excellent while discovery delay and backlog wait consume most of the promised cleanup window.
| Observed condition | Smallest design I would test | Reason to reject it |
|---|---|---|
| One bounded page per interval | Scheduled loop with a durable checkpoint | The worst-case page misses the deadline |
| Bursty discovery, one API allowance | Scheduled discovery plus paced queued workers | Backlog age keeps rising at the safe request rate |
| Several producers sharing an allowance | Queue plus a shared rate limiter | The limiter's scope cannot match the provider's limit scope |
Then I run the clock forward with a fake API that accepts requests only at the configured pace. I double the largest plausible tenant, overlap two discovery ticks, and replay one delivered operation. The choice passes only if the oldest-item age remains within the target and the same operation still completes once. This is a design experiment, not a production benchmark, so the actual deadline and dataset distribution belong to the application team.
Failure injection decides more than happy-path throughput
My initial default is always the smallest deployable system: one scheduled Node.js function, one query, one loop. That is still the right baseline. The mistake is assuming that await in a loop provides operational control. It limits one invocation, but overlapping schedules, manual replays, and multiple regions can each create another loop. The aggregate request rate is then outside that function's view.
The second trap is using “run completed” as the success signal. A discovery run can complete while thousands of cleanup operations remain queued. Conversely, a run can stop after handing off every operation correctly. The useful state machine is discovered, claimed, applied, and completed; the useful latency is from eligibility at the retention cutoff to completed deletion. Measure those transitions separately.
Retries need classification as well. A rate-limit response should return the job to delayed work according to the provider's guidance. An authentication or malformed-request response needs review rather than rapid repetition. A network interruption leaves the outcome uncertain, so the stable operation ID and completion record prevent blind duplication. None of this requires a particular queue, but every implementation needs an explicit policy for delivery attempts and work that can no longer make progress.
This is where a cheap-looking cron loop can become expensive in engineering time. I don't mean vendor pricing; I mean waking up to answer a basic question with no data: “Did we delete the eligible attachment, or did the request end after the remote side accepted it?” Durable handoff plus an operation ledger gives a concrete answer. The catch is that it adds state, migrations, replay tooling, and a worker deployment. Those costs are real.
How do cloud cron and a Node.js queue pace rate-limited API batch processing?
A periodic trigger should perform a bounded discovery pass. It selects support artifacts that have crossed the retention boundary, records a stable cleanup operation for each one, and exits. Workers consume those operations under a concurrency and dispatch-rate limit. A record's cleanup key must stay stable across retries so redelivery cannot create a second logical operation.
This separation matters because discovery rate and service rate are different variables. Suppose a run discovers 12,000 stale attachment records. That number is an example capacity test, not a benchmark. A single cron handler that walks the full set has only two bad knobs: send faster and risk the API limit, or send slower and keep the invocation open. A queue lets the producer finish after durable handoff while consumers pace the calls. Backlog age then tells the operator whether the chosen pace can meet the cleanup deadline.
Keep the unit of work narrow: one tenant plus one bounded page or object range, not “clean this entire tenant.” Large tenants otherwise become head-of-line blockers. Include the retention cutoff in the payload, because a retry must apply the same policy decision made during discovery rather than recomputing against a later clock.
One boundary is easy to miss. Don't enqueue faster merely because the queue accepts writes quickly. Admission needs a ceiling too — otherwise a scan can create a huge backlog whose storage and retry traffic cost more than the useful work. Pause discovery when the oldest-message age crosses the service objective, then let workers drain it before the next full scan.
Clock products and queue products therefore occupy different slots. Vercel Cron, GitHub Actions cron, and Google Cloud Scheduler are candidates for initiating the bounded scan; they are not the pacing algorithm in this design. Amazon SQS documents a visibility timeout that temporarily hides a received message while it is being processed. RabbitMQ documents priority queues, which can order urgent cleanup differently from routine retention work. Those are objective mechanism differences, not a ranking: visibility, priority, and time-based initiation solve separate parts of the system.
A TypeScript contract I can fake
I keep the application code behind three tiny interfaces. The infrastructure adapter can change later, while the rate policy and idempotency contract stay visible in tests.
type CleanupJob = {
operationId: string;
tenantId: string;
cutoffIso: string;
cursor?: string;
};
interface CleanupQueue {
send(job: CleanupJob): Promise<void>;
}
interface CleanupStore {
claim(operationId: string): Promise<boolean>;
complete(operationId: string): Promise<void>;
release(operationId: string): Promise<void>;
}
interface SupportApi {
deleteExpiredPage(job: CleanupJob): Promise<{ nextCursor?: string }>;
}
export async function runCleanup(
job: CleanupJob,
store: CleanupStore,
api: SupportApi,
queue: CleanupQueue,
): Promise<void> {
if (!(await store.claim(job.operationId))) return;
try {
const result = await api.deleteExpiredPage(job);
if (result.nextCursor) {
await queue.send({
...job,
operationId: `${job.tenantId}:${job.cutoffIso}:${result.nextCursor}`,
cursor: result.nextCursor,
});
}
await store.complete(job.operationId);
} catch (error) {
await store.release(job.operationId);
throw error;
}
}
The queue adapter should acknowledge only after complete succeeds. Its delivery timeout must exceed the normal processing window, and long operations need a deliberate extension policy; the SQS documentation explains that an expired visibility timeout makes an unacknowledged message visible again. Duplicate delivery is therefore an input the handler must tolerate, not an exotic corner case.
Notice what this sample leaves out. It doesn't hide rate limiting inside setTimeout, because sleeping workers consume capacity and coordinate poorly. It also doesn't prescribe a token-bucket implementation: a single worker with fixed concurrency may be enough, while several worker instances require a shared limiter if they consume one account-wide API allowance. Your mileage may vary because the decisive fact is the downstream provider's actual limit scope. I'm not sure which limiter belongs in a system until that scope — per credential, tenant, route, or region — is confirmed.
Short code. Long contract.
The queue complexity removal test
Cron alone is suitable when discovery and processing are both bounded, overlap is prevented, the work is safe to retry, and the worst credible batch finishes well within the invocation budget. For example, a cleanup that selects at most one small page per run and can wait for the next interval after rate limiting may not justify another moving part. Keep the checkpoint in durable storage and advance it only after the page succeeds.
A queue is not suitable when the team cannot operate its retry and dead-letter policy. Moving work into durable messages without ownership merely turns a visible timed-out run into an invisible growing backlog. Stick with the bounded cron loop until the batch size or latency objective proves that it cannot keep up.
The reverse boundary is clearer. Add queued workers when any one of these becomes true: the worst-case batch can exceed the run window, several producers share one API allowance, individual items need isolated retries, or the oldest eligible item has a deadline that a once-per-period loop cannot reliably meet. For urgent and routine support tasks sharing a broker, a priority mechanism such as the one RabbitMQ documents can help, but priority is no substitute for capacity; low-priority retention work still needs an explicit maximum wait.
No universal winner exists.
Small stays small.
Four assertions before deployment
Start with four measurements: items discovered per interval, sustained successful service rate, oldest eligible-item age, and retry count by reason. Compare arrival rate with service rate over the busiest credible support-retention window. If work arrives faster than it can be completed, queueing only records the debt; it does not remove it.
Then test the ugly sequence. Trigger two discovery runs at once, deliver the same job twice, interrupt a worker after the downstream call but before local completion, and fill a batch with one oversized tenant. The expected result is stable operation identity, bounded outbound concurrency, no premature checkpoint, and an observable backlog that later drains. These are deterministic integration tests with fake clocks and a fake rate-limited API, not claims about production measurements.
My decision rule is conditional: start with one bounded scheduled loop, instrument its completion latency, and introduce durable queued work when the measured workload threatens the deadline or makes retries too coarse. The scheduler remains the clock. The queue becomes the pace setter. That split costs more to operate, so it should earn its place with a concrete latency or isolation requirement.
Top comments (0)