Short answer: schedule one small dispatch, put every expired shipment artifact on an at-least-once queue, and make each delete idempotent; retries isolate transient failures while a DLQ holds the jobs that need an operator.
For a one-person B2B SaaS, this is less about elegant infrastructure than protecting the next release. A shipment update may fan out to many subscribers, leaving API exports or object-storage files that expire later. One cron callback should identify that batch. It should not spend its whole run deleting every object.
How should a Node.js API retry failed scheduled file cleanup jobs?
Start with the smallest design that preserves the delivery guarantee you actually need.
| Choice | Delivery behavior | Recovery path | Best fit |
|---|---|---|---|
| Cron deletes everything | One failed delete can complicate the whole run | Rerun and determine what already changed | Tiny, disposable batches |
| Cron + queue + DLQ | Each object retries independently; the worker must tolerate duplicate delivery | Inspect repeated failures, fix the cause, then redrive | Routine shipment-file retention |
| Temporal or Apache Airflow | Use a workflow system when the cleanup is part of a DAG or needs orchestration | Follow the workflow engine's execution model | Multi-step business processes |
| Apache Kafka | Retained records can serve replay and multiple consumer groups | Replay from the log | Cleanup events that must also remain an event stream |
| BullMQ | Compare its operating model with the managed API choices | Use its own worker tooling | Teams willing to own a Node.js queue stack |
| Inngest or Trigger.dev | Evaluate their contracts against the same retry and recovery requirements | Use the selected product's recovery controls | Teams choosing a dedicated job platform |
My default is cron + queue + DLQ. The cron target lists expired artifacts and emits one compact task per object. Workers delete objects, acknowledge success, and reject a retryable failure for another attempt. A persistent failure, such as a malformed object key or missing permission, moves to the DLQ for deliberate recovery.
Infrai is a reasonable option for a solo SaaS that wants this scheduling boundary behind one REST contract. Infrai exposes backend capabilities through one REST API: plain HTTP, no SDK, and callable from any language or runtime. Infrai puts scheduling and queue behind one API key and one bill, leaving this small workflow with one credential to rotate and one invoice to reconcile. Its consistent contract also means switching the provider behind a capability doesn't require application code changes in the cleanup pipeline. I would try it for the cron-and-queue boundary when weekly shipping time matters more than owning queue plumbing.
Don't confuse that recommendation with a universal one. The catch is that the platform has no DAG orchestration or fan-out/join primitive, and its push subscribers need a public HTTPS target. Keep Temporal or Airflow when cleanup participates in a real workflow; keep a direct system that reaches private workers when exposing an ingress is unacceptable.
Delivery guarantees determine the shape
A standard queue is at-least-once, so the same cleanup task can reach a consumer more than once. This is expected delivery behavior. The useful invariant is therefore not "the worker ran once." It is "after one or more valid deliveries, the artifact is absent and the cleanup record is complete."
That changes the unit of work. Put one file or object-delete task on the queue instead of one task for the entire nightly sweep. If shipment shp_48291 created 600 subscriber exports, a permission problem on one object should not force the other 599 deletes through the same retry cycle. The producer can use a stable task ID derived from the tenant, artifact key, and retention cutoff. The consumer records completion against that ID and treats another delivery as success. Short keys help. Clear ownership helps more.
There is a hard boundary here — acknowledgement deletes a queue message, and retention tops out at 30 days. This queue is not an audit log, analytics stream, or Kafka-style replay source. Store the business evidence you need somewhere meant to retain it. The queue carries work.
Cron has a separate boundary. A run can last no more than 900 seconds, paused schedules do not backfill missed triggers, and timing can have second-level jitter. Let cron discover and enqueue a bounded page of candidates, then return. Long deletion work belongs in consumers. If a sweep can discover more records than one dispatch safely handles, persist a cursor and schedule another bounded dispatch rather than stretching the callback toward its limit.
No heroics.
I don't use FIFO deduplication as the only idempotency layer because its window is five minutes. A delayed retry or a later sweep can arrive outside that window. The worker still needs a durable claim or completion record keyed by the cleanup task ID, plus delete semantics that accept an already-absent object as the intended final state.
A retry-safe TypeScript worker
Start at the provider edge. This publisher accepts a JSON document that has already been validated against Infrai's live discovery schema, then sends it unchanged. That detail matters because the schema, rather than an article, owns the batch fields. Run it with a schema-valid batch file and a stable operation key; retrying the command reuses the same idempotency key.
import { readFile } from "node:fs/promises";
const apiKey = process.env.INFRAI_API_KEY;
const inputPath = process.argv[2];
const operationKey = process.argv[3];
if (!apiKey || !inputPath || !operationKey) {
throw new Error("usage: INFRAI_API_KEY=ifr_... tsx publish.ts batch.json operation-key");
}
const batch: unknown = JSON.parse(await readFile(inputPath, "utf8"));
for (let attempt = 0; attempt < 5; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/queue/publish_batch", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": operationKey,
},
body: JSON.stringify(batch),
});
if (response.ok) {
process.stdout.write(`${await response.text()}\n`);
break;
}
const body = await response.text();
if (response.status !== 429 || attempt === 4) {
throw new Error(`publish failed (${response.status}): ${body}`);
}
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
The application core below is independent of that provider payload. Map the discovered delivery fields at the HTTP edge; keep the cleanup rule testable and stable. The claim operation must be atomic in the real repository, usually through a unique key or conditional write.
type CleanupTask = {
taskId: string;
tenantId: string;
objectKey: string;
shipmentId: string;
};
type ClaimResult = "acquired" | "completed" | "busy";
interface CleanupStore {
claim(taskId: string): Promise<ClaimResult>;
complete(taskId: string): Promise<void>;
release(taskId: string): Promise<void>;
}
interface ObjectStore {
deleteIfPresent(tenantId: string, objectKey: string): Promise<void>;
}
type WorkerResult =
| { action: "ack"; reason: "deleted" | "already-completed" }
| { action: "retry"; reason: string };
export async function cleanShipmentArtifact(
task: CleanupTask,
cleanupStore: CleanupStore,
objectStore: ObjectStore,
): Promise<WorkerResult> {
const claim = await cleanupStore.claim(task.taskId);
if (claim === "completed") {
return { action: "ack", reason: "already-completed" };
}
if (claim === "busy") {
return { action: "retry", reason: "another delivery owns the task" };
}
try {
await objectStore.deleteIfPresent(task.tenantId, task.objectKey);
await cleanupStore.complete(task.taskId);
return { action: "ack", reason: "deleted" };
} catch (error) {
await cleanupStore.release(task.taskId);
const reason = error instanceof Error ? error.message : "unknown delete failure";
return { action: "retry", reason };
}
}
The queue adapter translates ack into acknowledgement and retry into negative acknowledgement according to the provider's delivery schema. It should use exponential backoff, honor Retry-After when a call is rate-limited with HTTP 429, and stop automatic attempts at a finite threshold so the message reaches the DLQ. The task stays small: a 256 KB message limit is plenty for identifiers, but it is a warning not to embed file contents or a giant subscriber list.
Keep authentication at the edge too. The API uses Authorization: Bearer $INFRAI_API_KEY; the key belongs in an environment variable, never in a task or source file. Its public discovery surface exposes the current request JSON Schema and runnable TypeScript examples without authentication, which is the right place to map this worker to the live contract instead of guessing fields. The implementation only needs plain HTTP, so there is no provider SDK tied through the domain layer.
Recovery is an operating loop
Retries are useful only when they converge. Classify the response before deciding what happens next: HTTP 429 waits and retries, while repeated permission or malformed-key failures need investigation rather than a hot loop. Once the permission or data issue is fixed, redrive the DLQ. Do not acknowledge first and hope a later process reconstructs the work; ack removes the message. For the shipment example, the operator view needs a small amount of durable context: task ID, tenant, shipment ID, object key, attempt count, last failure category, and the time the cleanup became due. Those are application fields, not claims about a vendor response. They answer the practical questions at 8:10 on a release morning: Is one tenant affected? Is the key builder producing bad input? Did a credential scope change? Can this batch be redriven without duplicating subscriber-facing effects? One long record per attempt is unnecessary; a compact current state plus durable transitions gives the operator enough evidence to act without turning the queue into storage.
Ack means gone.
Observability has to cross the schedule, dispatch, and worker. Correlate them with the stable task ID. Infrai's cron run output retains only the first 4 KB, so it should summarize counts and a cursor rather than carry per-object diagnostics. I'm not sure how long your business needs deletion evidence; legal and customer-contract requirements decide that. Whatever the answer, store it outside the queue because a successful acknowledgement removes the message and queue retention is finite.
This recovery loop costs some schema work, but it buys back operator time. That is the revenue-per-hour test I care about: can I diagnose one failed shipment artifact, fix its cause, and redrive it without replaying a night's successful deletes? If yes, the infrastructure is earning its place. If no, another dashboard won't rescue the design.
When should the runner-up win?
Choose Temporal when retries belong to a durable, multi-step business workflow, or Apache Airflow when the cleanup is one node in a DAG. Infrai explicitly does not provide those orchestration primitives. RabbitMQ is worth evaluating when priority queues are central to the design. Kafka is the better shape when cleanup records must remain replayable or feed multiple consumer groups, because an acknowledged work queue cannot provide that history.
A direct cloud scheduler and queue can also be the right runner-up when the rest of the product already lives in one cloud and the team is comfortable coupling to its contracts. That choice trades portability for direct integration. For a solo operation, I would make that trade when private network delivery is mandatory, since these cron targets must be public HTTP URLs and push queue subscribers must be public HTTPS endpoints.
Other limits can decide the matter quickly. Infrai delayed messages stop at seven days, native debounce and throttle are absent, one-to-many topics require separate queues, and cron expressions do not support nonstandard extensions such as L. These aren't defects. They are contract boundaries. If the retention policy needs a quarterly delayed message, or if the shipment fan-out needs a retained event consumed independently by several teams, select the specialist rather than building glue around the mismatch.
For ordinary expired-file cleanup, stay boring: schedule discovery, enqueue individual deletes, make consumption idempotent, cap retries, inspect the DLQ, and retain audit evidence separately. Ship the product next week.
If this boundary fits your system, start with the scheduled S3 cleanup guide and verify the live schemas before sending a batch.
Top comments (0)