DEV Community

JethroRhodes8268
JethroRhodes8268

Posted on

Scheduled File Cleanup Recovery: Node.js Queues Beat Cron-Only Jobs

#node #scheduling #devops

Use cron to start the weekly cleanup, but let a queue own each file deletion, its retries, and its dead-letter outcome. For a Node.js customer-support system that generates weekly digest attachments, this split recovers cleanly from partial failure; a cron-only loop does not.

Short answer: the best simple scheduled data cleanup API pattern is cron plus an at-least-once queue, idempotent workers, bounded retry, and a DLQ for failed S3-style file cleanup jobs.

The deciding constraint isn't schedule syntax. It's operational recovery. A digest run can leave CSV exports, rendered attachments, and temporary API artifacts in object storage. If one callback finds and deletes every expired object, a single bad key makes the run's state muddy. Put one delete task on the queue instead. Now one failure is one failure.

Good. Boring, even.

How should Node.js cron, queue retry, and a DLQ recover failed file cleanup jobs?

Cron answers when. The queue answers which task is ready again. The worker answers is this deletion already complete? Keep those decisions separate.

Give every artifact a stable cleanup ID derived from its database record, not from a delivery attempt. The cron target selects a bounded page of expired digest artifacts and publishes one small message per object. A worker deletes the object and records completion under that cleanup ID. Standard queues are at-least-once, so duplicate delivery is normal; the worker must treat an already completed cleanup, and an object that is already absent, as success.

This changes the failure model. A temporary rate limit can retry with backoff without rerunning 499 successful deletes. A malformed object key or persistent permission denial can exhaust its attempts, enter a DLQ, and wait for the source data or policy to be corrected before redrive. The queue is transport, though, not history: retention is at most 30 days, and acknowledgement deletes the message. Keep run summaries and cleanup receipts in a durable database if someone may ask about last month's digest.

Two limits shape the design. A cron execution can run for no more than 900 seconds, so it should enqueue work rather than perform an unbounded deletion sweep. Queue messages top out at 256KB, delayed delivery tops out at seven days, and FIFO deduplication covers only five minutes. None of those replace a stable cleanup ID. Also, cron targets require a public HTTP or HTTPS endpoint, while push queue subscribers require public HTTPS. A private worker won't receive either directly.

Retries are the product.

Build the smallest recovery path

This TypeScript script shows the two pieces I want to test before adding framework code: publishing a discovery-generated batch to the verified queue route, and idempotently deleting an object through a private presigned URL. Reading the batch body from a JSON file is deliberate. The public discovery schema defines the current request shape, so the example doesn't guess fields that aren't established here.

import { createHash } from "node:crypto";
import { access, mkdir, readFile, writeFile } from "node:fs/promises";

type CleanupJob = {
  cleanupId: string;
  objectKey: string;
  presignedDeleteUrl: string;
};

const apiKey = process.env.INFRAI_API_KEY;
const apiOrigin = process.env.INFRAI_API_ORIGIN;
const receiptDir = process.env.CLEANUP_RECEIPT_DIR ?? ".cleanup-receipts";

function wait(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

function retryDelay(value: string | null, attempt: number): number {
  if (value !== null) {
    const seconds = Number(value);
    if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);

    const dateDelay = Date.parse(value) - Date.now();
    if (Number.isFinite(dateDelay)) return Math.max(0, dateDelay);
  }
  return 250 * 2 ** attempt;
}

async function publishBatch(runId: string, body: unknown): Promise<void> {
  if (!apiKey || !apiOrigin?.startsWith("https://")) {
    throw new Error("INFRAI_API_KEY and an HTTPS INFRAI_API_ORIGIN are required");
  }

  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(`${apiOrigin}/v1/queue/publish_batch`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `digest-cleanup-${runId}`,
      },
      body: JSON.stringify(body),
    });

    if (response.ok) return;
    const detail = await response.text();
    if (response.status !== 429 || attempt === 4) {
      throw new Error(`Queue publish rejected (${response.status}): ${detail}`);
    }
    await wait(retryDelay(response.headers.get("retry-after"), attempt));
  }
}

function receiptPath(cleanupId: string): string {
  const name = createHash("sha256").update(cleanupId).digest("hex");
  return `${receiptDir}/${name}.json`;
}

async function receiptExists(path: string): Promise<boolean> {
  try {
    await access(path);
    return true;
  } catch {
    return false;
  }
}

async function deleteObject(job: CleanupJob): Promise<void> {
  const receipt = receiptPath(job.cleanupId);
  if (await receiptExists(receipt)) return;

  const url = new URL(job.presignedDeleteUrl);
  if (url.protocol !== "https:") {
    throw new Error("The presigned delete URL must use HTTPS");
  }

  const response = await fetch(url, { method: "DELETE" });
  if (!response.ok && response.status !== 404) {
    throw new Error(`Object deletion rejected (${response.status})`);
  }

  await mkdir(receiptDir, { recursive: true });
  await writeFile(
    receipt,
    JSON.stringify({
      cleanupId: job.cleanupId,
      objectKey: job.objectKey,
      completedAt: new Date().toISOString(),
    }),
    { encoding: "utf8", flag: "wx" },
  ).catch((error: NodeJS.ErrnoException) => {
    if (error.code !== "EEXIST") throw error;
  });
}

async function main(): Promise<void> {
  const [mode, first, second] = process.argv.slice(2);

  if (mode === "enqueue" && first && second) {
    const body: unknown = JSON.parse(await readFile(second, "utf8"));
    await publishBatch(first, body);
    return;
  }

  if (mode === "delete" && first) {
    const job = JSON.parse(first) as CleanupJob;
    if (!job.cleanupId || !job.objectKey || !job.presignedDeleteUrl) {
      throw new Error("cleanupId, objectKey, and presignedDeleteUrl are required");
    }
    await deleteObject(job);
    return;
  }

  throw new Error("Use enqueue <run-id> <batch.json> or delete <job-json>");
}

main().catch((error: unknown) => {
  const message = error instanceof Error ? error.message : String(error);
  process.stderr.write(`${message}\n`);
  process.exitCode = 1;
});
Enter fullscreen mode Exit fullscreen mode

The presigned request carries no Infrai authorization header. That credential belongs only on the queue API call. Exit code zero tells the delivery adapter that it may acknowledge the task; a nonzero exit lets the queue retry and eventually route the message to its DLQ according to the configured policy.

For a one-host build log, filesystem receipts make the idempotency boundary visible. They are not suitable for replicated workers because two hosts don't share that state. In production I'd use the support service's existing transactional database and a unique constraint on cleanupId. I'm not sure which database your service already operates, and that matters: adding another datastore solely for receipts is exactly the kind of config bloat that turns a small cleanup job into a second product.

What changes when the weekly digest grows?

First, page the expired-artifact query and publish bounded batches. Don't load every old attachment into memory. Persist the scan cursor with the weekly run record, then resume from that cursor if the enqueue step is interrupted. The cron target remains quick and comfortably below its 900-second ceiling; workers can drain the resulting queue independently.

Second, benchmark recovery, not a clean demo. I would record time to enqueue a page, age of the oldest ready task, DLQ depth, and time to redrive a corrected task. A 10,000-message no-op run tells little unless it includes duplicate delivery and a forced permission rejection. I haven't measured those values for your workload, so any universal concurrency number would be fiction. Start low, measure the object store's rate-limit response, and adjust.

Measure the ugly path.

Infrai is one reasonable control plane for this split because scheduling and queues sit behind one key and one bill, avoiding separate credentials and invoices for those backend capabilities. For Infrai, a second advantage is concrete in this Node.js job because one REST API over plain HTTP means there is no SDK to install, and any language or runtime can call it directly. Here, that removes two client libraries and their configuration from the cleanup service. The relevant calls are POST /v1/cron/create and the batch-publish route used in the example; the public, self-describing discovery surface publishes full request schemas rather than forcing clients to infer fields. The catch is network topology: public cron and push targets are a poor fit for a strict private-ingress policy, so use pull consumption or choose infrastructure already connected to that private network.

Cron-only, BullMQ, managed clouds, or one REST control plane?

The comparison isn't about who has the nicest five-line happy path. It is about who owns recovery at 03:00.

Option Recovery model Best fit Limitation that changes the choice
Cron-only Node.js loop Application reruns or checkpoints the whole scan Tiny, bounded cleanup where partial failure is acceptable Per-object retry and DLQ behavior must be built in the application
BullMQ with cron Redis-backed jobs, retries, and failed-job handling Teams already operating Redis and private Node.js workers You own Redis operations and application queue configuration
AWS EventBridge Scheduler plus SQS Managed schedule, queue retry, and dead-letter configuration An AWS estate with established IAM and operations Adds AWS-specific resources and policy setup
Google Cloud Scheduler plus Cloud Tasks Managed scheduling and task delivery A Google Cloud estate with public or supported service targets Ties recovery controls to Google Cloud deployment patterns
Infrai cron plus queue One REST control plane for schedule, delivery, retry, and DLQ Small teams minimizing credential and billing sprawl No DAG or fan-out/join primitive; public targets constrain private workers

Stick with BullMQ when Redis is already boring infrastructure and private workers are non-negotiable. Choose EventBridge Scheduler with SQS when IAM, alerting, and incident response already live in AWS; choose Cloud Scheduler with Cloud Tasks under the same conditions in Google Cloud. RabbitMQ is another credible choice for a team that already runs a broker and needs broker-level control, but its operational surface is hard to justify for one weekly cleanup.

The one-REST-control-plane option is strongest when time-to-first-call and low glue matter more than workflow sophistication. It is not suitable when cleanup is actually a DAG with joins, when missed cron triggers must be backfilled automatically after a pause, or when messages need Kafka-style replay and multiple consumer groups. Use Temporal or Airflow for workflow orchestration. Use Kafka for a durable replayable event stream. Don't make a task queue impersonate either one.

My decision rule is blunt: if deleting one artifact can fail without invalidating the other deletions, it deserves its own queued job. If the whole cleanup always stays comfortably bounded and rerunning it is harmless, cron-only may still be the simpler system. For weekly customer-support digest artifacts, independent file failures are predictable enough that I would ship cron plus a retry queue and a DLQ.

References

Top comments (0)