Node.js Daily Cleanup Job Deletes Old Uploads with Scheduled Retention

#node #scheduling #database

A media platform running a nightly reconciliation against a payment provider should separate the calendar trigger from the retention decision. A daily cleanup job is a reasonable latency-versus-cost compromise when the policy permits a full-day delay: it removes old reconciliation uploads and processing logs without putting deletion work on the request path. The scheduler is only the bell. The durable run record is the control.

TL;DR: use one UTC trigger, a shared lease, a fixed cutoff, and small idempotent batches. Select the simplest trigger that can call that worker, but judge it by missed-run visibility and overlap behavior rather than by a brand name or a posted price.

For this workload, the expensive failure is usually quiet retention drift. A duplicate invocation wastes work; an unobserved gap can leave provider exports, import logs, or settled-record artifacts past the intended retention boundary. The correct policy duration comes from the organization responsible for those data classes, not from the scheduler.

Start with the deletion boundary

The first design decision is to name what is eligible. A nightly payment reconciliation may produce a provider report, a normalized import row, a discrepancy record, and an operational log. They are related, but they do not automatically have the same retention period or recovery value. A single DELETE WHERE created_at < cutoff turns that distinction into an assumption.

Store a retention class and a policy-derived expiry time with each deletable record. At run start, persist a run identifier, the UTC cutoff, and a lease expiry. Persist candidate, deleted, failed, and deferred counts as the run advances. A retry then uses the original cutoff instead of silently widening the selection after midnight. This also gives an operator a concrete answer to a basic question: did the reconciliation cleanup finish, or did a process merely begin?

Keep the cutoff in UTC. A five-field cron expression such as 0 2 * * * describes a calendar time, while RFC 3339 provides a common timestamp representation for stored run state. The expression does not prevent two application replicas from starting work.

A completion record by itself is incomplete evidence. Pair it with the age of the oldest eligible record or with an independently calculated eligible count. A zero-row run may mean the policy is satisfied, but it can also expose a broken selector, an incorrect tenant scope, or a delayed data feed. Those possibilities call for different investigation paths.

How should a daily cleanup job delete old uploads?

Claim a bounded set of database rows before touching their associated objects. The claim must be durable and must expire, because a worker can stop after it has claimed records but before it has completed the associated delete. PostgreSQL documents SKIP LOCKED as suitable for avoiding contention with multiple consumers of a queue-like table; it is useful here when the table is deliberately being used as a work queue. It is not a substitute for defining a retention policy.

The following Python sketch describes the unit of work even when the surrounding web service is Node.js. It uses a stable cutoff supplied by the run record and relies on parameter binding instead of constructing SQL from timestamps. 100 is an example batch size, not a universal setting; choose it from lock duration, object-delete latency, and the database's normal workload.

from datetime import datetime


def claim_upload_batch(cursor, cutoff: datetime, limit: int = 100):
    cursor.execute(
        """
        WITH candidates AS (
            SELECT id
            FROM reconciliation_uploads
            WHERE expires_at <= %s
              AND deleted_at IS NULL
              AND claim_expires_at < NOW()
            ORDER BY expires_at, id
            FOR UPDATE SKIP LOCKED
            LIMIT %s
        )
        UPDATE reconciliation_uploads AS upload
        SET claim_expires_at = NOW() + INTERVAL '15 minutes'
        FROM candidates
        WHERE upload.id = candidates.id
        RETURNING upload.id, upload.object_key
        """,
        (cutoff, limit),
    )
    return cursor.fetchall()

There is no atomic commit that covers a normal database row and an independent object deletion. That limitation should shape the procedure. Claim the row, issue an idempotent delete to the object store, and then mark the row deleted. If the process stops between steps, the expired claim becomes visible to a later run for reconciliation. This accepts temporary retention after a crash in exchange for avoiding a record that claims an object still exists when it does not.

Small batches help.

They make cancellation, timing, and partial progress legible. They also limit the time a cleanup operation competes with reconciliation queries, which matters when a faster schedule reduces policy lag but creates more overlapping database work. A cleanup action should be reversible only where the data policy allows it; a deletion log is not a restoration mechanism.

Choose the trigger after the worker contract

The worker needs authentication, an exclusive lease, a disable switch, and a durable ledger regardless of who emits the schedule. Once those are defined, service selection becomes a failure-domain decision. A process-local timer can be sufficient for one intentionally managed process, but application restarts and horizontal replicas make its ownership ambiguous. A container-oriented scheduler moves invocation outside that process, while a managed scheduler moves it outside the deployment environment. Neither proves that cleanup completed.

Trigger family	Useful boundary	Limitation to account for
Kubernetes CronJob	Schedules a Job in a Kubernetes cluster	Concurrency policy and missed-start behavior still need explicit configuration and monitoring
Amazon EventBridge Scheduler	Emits a scheduled target invocation outside the application deployment	The target must authenticate requests and keep its own idempotency and run ledger
Google Cloud Scheduler	Sends scheduled HTTP, Pub/Sub, or App Engine targets	Delivery creates a trigger, not proof that downstream deletion reached completion

This is not a ranking. The important comparison is where each mechanism stops being responsible. Kubernetes CronJobs expose job-oriented scheduling in the cluster. EventBridge Scheduler and Cloud Scheduler act as external invocation systems. The cleanup worker owns the cutoff, lease, retries, and evidence in all three cases.

For a nightly media reconciliation, run more often only when the policy's allowed lag requires it. A shorter interval lowers the maximum wait before a stale upload is selected. It also increases wakeups, lease contention, and the chance that the next run sees work from the previous one. Choose the widest interval that satisfies the required retention window, then monitor the actual completion age.

Make the rollout observable before it becomes destructive

Start in report-only mode. Calculate candidates using the same cutoff and tenant filters, write the intended run ledger, and compare the count with a manually reviewed sample. This gives the team a chance to catch a policy mapping mistake before an object delete is enabled.

Then enable deletion for one retention class and retain enough run detail to answer what was selected, claimed, deleted, deferred, and failed. Exercise two simultaneous triggers, a worker stopped after claiming rows, an already-absent object, a database timeout, and a run that crosses the UTC day boundary. RabbitMQ's acknowledgement documentation illustrates the broader delivery principle: acknowledgement signals that a consumer handled a delivery, not that every business-side effect is automatically safe. The same distinction matters for scheduler invocations.

Alert on evidence of policy drift: a last successful completion older than the allowed execution window together with an aging eligible backlog. Do not page merely because a scheduler emitted an invocation. An operator needs the cutoff, run identifier, lease owner, and oldest eligible timestamp to decide whether to wait, investigate, or disable the next run.

The rollout can stay compact: establish the ledger, shadow the selection, enable one class, test interruption paths, then expand. The outcome is a cleanup job whose cost and latency are deliberate, and whose deletes can be explained later.

Sources

References:

DEV Community