DEV Community

magnusberg2958
magnusberg2958

Posted on

A Node.js API Pattern for Scheduled Data Cleanup: Cron, Queue Retry, and a DLQ

#scheduled #cleanup #api

Short answer: run scheduled data cleanup as a small control-plane operation that freezes an eligibility cutoff, publishes idempotent object-file jobs to a durable queue, and sends exhausted or invalid work to a DLQ for review. A cron callback that lists and deletes files directly has no useful recovery boundary once a deployment, duplicate trigger, or slow storage operation intervenes.

The distinction matters because retention work is destructive and usually quiet until it is late. A successful scheduler tick only says that something woke up; it says nothing about which files were eligible, how much work was accepted, whether a retry can still meet the cleanup SLO, or whether a late worker might delete a newly written object under an old rule. Treat the schedule as a signal, not proof of completion.

Nothing else follows from a green tick.

Keep the Node.js API in the control plane if that is where policy changes and operator actions already live. The API can validate retention rules, create an immutable run record, and expose run status. Discovery and deletion belong in workers with a durable handoff. The implementation language is secondary to those boundaries.

What should a scheduled data cleanup API do with S3 files, cron queue retry, failed jobs, and a DLQ?

Start a run with a policy identifier, a policy version, a cutoff timestamp, and a unique run ID. Discovery must use that fixed cutoff even if it takes longer than the next scheduling interval. Each published job should carry the object key, its version identifier when storage is versioned, the run ID, the policy version, and the cutoff. On receipt, the worker checks eligibility again before it deletes. That last check is cheap insurance against key reuse, a corrected policy, and a job delivered twice.

There are four terminal decisions worth making explicit: delete completed, safe skip because the file is no longer eligible, retry after a transient dependency result, and dead-letter after an invalid job or an exhausted retry budget. Put the reason and the immutable job identity in the terminal record. Operators cannot reason about a DLQ full of anonymous payloads.

For externally triggered runs, authenticate the raw request bytes with an HMAC, include a timestamp and unique request ID, and reject stale or replayed requests. RFC 2104 defines the HMAC construction; the replay window and its storage are application choices. Signing parsed JSON on one side and raw bytes on the other is a classic mismatch, because the two sides are not authenticating the same message.

The catch is that a durable queue does not make an unsafe retention policy safe. It gives the policy a recoverable execution path. A small, disposable task on one replica may be suitable for in-process cron, provided missed runs and duplicate execution are acceptable. It is not suitable when cleanup has a strict deadline, multiple replicas, or a deletion policy that requires an audit trail. In that case, use a separately owned scheduler and queue, or retain work in a database outbox when the database already owns the policy state.

Design Good fit Operational cost Question to settle
In-process cron One process and low-consequence cache cleanup Deployments and replica count complicate ownership Can a run disappear without violating the SLO?
Scheduler plus durable queue Bounded work with independent producers and consumers Queue retention and DLQ ownership need an on-call path Who owns delayed and dead-lettered jobs?
Database outbox plus workers Policy and run state already live in one database Polling and row retention must be capacity planned Can the database absorb peak discovery?
Self-hosted scheduler and broker Portability is a hard requirement Upgrades, backups, and recovery become platform work Is that control worth another system to operate?

Priority queues rarely rescue a flawed retention design. RabbitMQ documents that priority queues consume resources and have a performance cost, with a small number of levels recommended. Separate capacity budgets for urgent and ordinary cleanup often make backlog age and SLO ownership easier to see.

Implement a bounded cleanup state machine

The worker below is deliberately transport-neutral. It models the data-plane behavior behind a Node.js cleanup API without claiming that an in-memory structure is durable. Its queue adapter is responsible for making Complete, Retry, and DeadLetter durable state transitions; its storage adapter is responsible for comparing current metadata against the fixed cutoff and policy before deletion.

package cleanup

import (
    "context"
    "errors"
    "time"
)

var ErrPermanent = errors.New("invalid cleanup job")

type Job struct {
    RunID, PolicyVersion, Key, VersionID string
    Cutoff                             time.Time
    Attempt                            int
}

type Store interface {
    Eligible(context.Context, Job) (bool, error)
    Delete(context.Context, Job) error
}

type Queue interface {
    Complete(context.Context, Job) error
    Retry(context.Context, Job, time.Duration) error
    DeadLetter(context.Context, Job, string) error
}

func Handle(ctx context.Context, store Store, queue Queue, job Job) error {
    eligible, err := store.Eligible(ctx, job)
    if err == nil && !eligible {
        return queue.Complete(ctx, job)
    }
    if err == nil {
        err = store.Delete(ctx, job)
    }
    if err == nil {
        return queue.Complete(ctx, job)
    }

    job.Attempt++
    if errors.Is(err, ErrPermanent) || job.Attempt >= 5 {
        return queue.DeadLetter(ctx, job, err.Error())
    }

    delay := time.Duration(1<<min(job.Attempt, 6)) * time.Second
    return queue.Retry(ctx, job, delay)
}
Enter fullscreen mode Exit fullscreen mode

This is the part that deserves design review. A consumer acknowledgement may happen only after a terminal record or the next durable queue state exists. If acknowledgement precedes that transition, a process exit turns a known cleanup candidate into invisible work. If the record precedes a failed delete but has no way to distinguish intent from completion, the audit trail lies. Consider a run that has discovered 10,000 candidates: the producer has already committed the cutoff, some workers have completed deletions, and a consumer receives the same message again after its delivery lease expires. The job must retain enough identity to decide that the object version has already reached a terminal outcome, rather than treating a duplicate delivery as a new authorization to delete whichever object now owns the same key. Conversely, a retry record cannot declare success merely because it was enqueued; it needs a result tied to that run, policy version, cutoff, and object version. The exact transaction mechanism depends on the broker and storage boundary, but the invariant does not.

Durable first.

Use capped exponential backoff with jitter in production. Five immediate attempts are just load amplification wearing a retry label. The retry window must leave time for DLQ review and replay inside the cleanup objective; otherwise the configured attempt count is an arbitrary number. Invalid authorization, malformed payloads, and a policy-version mismatch should move directly to a reviewable terminal path rather than consuming the transient retry budget.

Set an SLO before capacity planning the queue

Define completion in operational terms: for example, eligible files reach a terminal outcome within an agreed window after the run cutoff. The key leading signal is oldest_pending_age, not a green scheduler metric. Track candidate discovery rate, queue depth, enqueue and completion rates, retry attempts, DLQ age, and run duration alongside it. A producer can fire on time while the cleanup service falls farther behind.

The steady-state floor is straightforward. If A files become eligible per second and workers complete R files per second, worker capacity must exceed A / R; then reserve headroom for retry traffic, tail latency, deployments, and normal maintenance. Bound discovery by object count and elapsed time as well. Without those bounds, a delayed run can fill the queue just as the next one starts, which hides the source of the backlog and makes each later recovery harder to estimate.

Make producer overlap harmless. A lease or unique run key can reduce duplicate discovery, but it cannot prove uniqueness across clocks, restarts, and redelivery. Idempotency belongs at the worker: the same object version and policy cutoff must have one effective terminal outcome. For a versioned object store, a bare key is insufficient because the name can refer to different content over time.

How should teams verify and roll back scheduled cleanup safely?

Begin in report-only mode. Discover candidates, publish intended state transitions, and compare a sample against the retention policy without mutating storage. Measure listing and worker throughput separately, estimate drain time from the observed distribution rather than the mean, and enable real deletion for a narrow data boundary. A canary is a prefix, tenant, or policy cohort here; one application replica is not a meaningful canary for a distributed queue.

Then rehearse four cases: deliver the same job twice and expect one effective deletion; pause consumers until backlog age crosses a warning threshold and verify recovery within the SLO; force a retryable adapter result through the attempt budget and confirm that the DLQ preserves run ID, object identity, cutoff, and attempt count; submit a newer object under an old job and expect a safe skip.

Rollback must be boring.

Disable the schedule, then pause consumers. Do not purge queued work, because the queued jobs are evidence and may still describe a valid policy decision. If the policy was wrong, invalidate the affected run IDs and issue fresh jobs only after correction. If capacity caused the delay but the policy and cutoff remain valid, resume the existing jobs. Object deletion is irreversible unless a separately tested recovery process exists, so recovery testing belongs before activation, not after the first incident.

The design is complete when the current on-call rotation can identify the oldest unprocessed work, explain why it is delayed, contain a policy mistake, and resume valid work without guessing. That is a better standard than asking which cron library looks simplest in a Node.js example.

References

Top comments (0)