Short answer: use a standard five-field cron trigger for a small daily cleanup job, keep the deletion policy inside the application, and introduce a queue only when uploads, logs, or records need independent retries, concurrency control, or durable acknowledgement.
The scheduler is a clock, not a retention engine. An Express process may receive the trigger, but the correctness boundary belongs to a separately invocable cleanup operation with a fixed cutoff, bounded batches, an idempotency key, and an audit record. The simplest service selection is therefore the one that satisfies those boundaries with the least new operational state; picking by the prettiest scheduling screen answers the wrong question.
Decision and scope
Use 0 2 * * * for a once-daily run at 02:00 in the scheduler's explicitly configured time zone, then call one application command that computes its cutoff once and processes a bounded amount of work. The command must be safe to repeat with the same run identity. It must also be safe for tomorrow's run to encounter an item left halfway through today's run.
That recommendation covers ordinary retention work: expired uploads, application logs already eligible for deletion, and database records whose policy state makes them disposable. It doesn't imply that all three datasets belong in one transaction or even one job. Their evidence requirements differ. A database can record a state transition atomically with an audit row, while deleting an object from blob storage crosses a system boundary; a log archive may be governed by a retention control that application code shouldn't bypass. One schedule may initiate three independently reconciled sweeps, each with its own cutoff and authorization.
The governing invariant is more precise than "run every day": for any eligible item, repeated execution produces the same durable end state, and an operator can later explain which policy, cutoff, run, and outcome applied. Exactly-once execution is an attractive phrase, but the useful engineering target is an exactly-once effect constructed from idempotent operations and durable evidence. Delivery systems can redeliver. Processes can stop between an external delete and a local commit. The design must remain correct anyway.
Keep it dull.
How should a daily Node.js Express cleanup job delete old uploads, logs, and records?
First, define "old" as data, not as an expression scattered across handlers. A retention rule needs a timestamp field, a comparison operator, a duration or fixed policy boundary, and any state exclusions such as legal hold, active upload, unsettled ledger entry, or pending export. Compute the cutoff once at the beginning of the run. If each page evaluates now - retention, the eligibility boundary moves while the scan is in progress, which complicates both replay and audit review. Next, assign the run a stable idempotency key. The trigger may derive it from the job name and the intended schedule window, for example upload-retention:2026-08-06, while the cleanup ledger enforces uniqueness. A retry using that key resumes or confirms the same logical run rather than creating a second one. This isn't proof that every side effect happened once; it is the anchor that lets each side effect be correlated and reconciled.
Selection must be deterministic and bounded. Query by immutable eligibility fields, order by a stable key, and claim a finite batch. Offset pagination is a poor fit when rows disappear during scanning because later offsets can shift; a stable cursor or claimed-work state gives the next attempt an unambiguous place to continue. The long paragraph is intentional here because these details form one failure boundary: a movable cutoff, unstable traversal, and unrecorded progress can combine to skip data without producing an obvious error, even though each local query appeared successful.
Then separate logical disposition from physical removal. For a record with an associated upload, the database may first record that the item is eligible and claimed by a specific run; the worker attempts the external deletion; and the database records completion. If execution stops after physical removal, retrying the same item must regard "already absent" as the desired state, not as evidence that a second destructive action is required. If policy requires an immutable audit trail, record decisions and outcomes in an append-only structure rather than relying on mutable application logs.
Finally, reconcile. Count claimed, completed, excluded, and failed items by run; retain the cutoff and policy version; and compare incomplete claims against the underlying stores. An HTTP 200 from a trigger endpoint proves very little about a multi-stage sweep. The evidence should answer a harder question: can an operator distinguish "not eligible," "eligible but not claimed," "claimed but incomplete," and "completed" without reconstructing history from prose logs?
Invariants, failure boundaries, and service selection
The options differ less in cron syntax than in the state they introduce. A conventional five-field expression is widely portable enough for the timing requirement, but scheduler time zones and overlap behavior still need explicit configuration and tests. The operational choice begins with the maximum credible batch, not the average one.
| Option | Appropriate boundary | State you must own | Not suitable when |
|---|---|---|---|
| In-process timer | One process, disposable development data, no availability promise | Process lifetime and overlap guard | Multiple replicas can fire independently or restarts can miss a run |
| External cron trigger plus bounded worker | Daily work finishes within a predictable window and one retry unit is acceptable | Run ledger, authentication, timeout, overlap policy | Individual items need independent retry or controlled parallelism |
| Cron trigger plus queue | Items have separate outcomes, workers scale independently, or work exceeds one invocation | Message identity, acknowledgement, redelivery, dead-letter handling, reconciliation | The batch is tiny and the extra delivery state would dominate operations |
| Durable workflow | The process spans long waits, ordered steps, approvals, or compensations | Workflow history, versioning, activity idempotency | A single bounded sweep has one observable outcome |
No row is a universal winner. The catch with an in-process timer is replica coordination: two Express instances may both believe they own 02:00, while a deployment around that time may leave neither one responsible. It remains valid for a local tool or a single, deliberately nonredundant process where missed execution has no material consequence. An external trigger removes dependence on the web process's lifetime, but it does not remove the need for overlap prevention, authentication, or a durable run ledger.
A queue changes the unit of recovery. Consumer acknowledgements communicate when a broker may treat a delivery as handled, and publisher confirms concern the publisher-to-broker side; they solve different portions of the delivery path. That distinction matters because acknowledgement after deletion creates a redelivery window, while acknowledgement before deletion creates a loss window. The usual answer is acknowledgement after a durable, idempotent effect, coupled with a processed marker or state transition. Asynchronous messaging also decouples producers from consumers, which is useful when the daily scanner and deletion workers need different scaling or deployment schedules, but decoupling adds observability and reconciliation obligations.
Cost belongs in the decision record, though not as a slogan. Account for the trigger, queue operations, worker runtime, retained workflow history, audit storage, on-call burden, and the engineering cost of testing redelivery. A service with more machinery can be economical when it replaces bespoke recovery work; for a hundred bounded rows per night, the same machinery may be the largest source of failure states. The threshold isn't universal. Measure the worst policy-eligible backlog, deletion latency distribution, lock impact, and recovery objective before selecting it.
Critical path in Go
The scheduling surface should invoke a command with a run key and a policy cutoff. Although the surrounding application is Node.js Express, keeping the contract language-neutral prevents the HTTP framework from becoming the retention architecture. The Go example below shows the critical path, not a vendor API: claim an item, perform an idempotent deletion, and persist an auditable result before acknowledging completion to the caller.
package cleanup
import (
"context"
"errors"
"fmt"
"time"
)
var ErrAlreadyAbsent = errors.New("object already absent")
type Candidate struct {
ID string
ObjectKey string
}
type Ledger interface {
BeginRun(ctx context.Context, runKey string, cutoff time.Time) error
Candidates(ctx context.Context, runKey string, limit int) ([]Candidate, error)
Claim(ctx context.Context, runKey, itemID string) (bool, error)
Complete(ctx context.Context, runKey, itemID, outcome string) error
}
type ObjectStore interface {
Delete(ctx context.Context, key string) error
}
func Sweep(ctx context.Context, ledger Ledger, objects ObjectStore, runKey string, cutoff time.Time) error {
if err := ledger.BeginRun(ctx, runKey, cutoff); err != nil {
return fmt.Errorf("begin run: %w", err)
}
items, err := ledger.Candidates(ctx, runKey, 250)
if err != nil {
return fmt.Errorf("select candidates: %w", err)
}
for _, item := range items {
claimed, err := ledger.Claim(ctx, runKey, item.ID)
if err != nil {
return fmt.Errorf("claim %s: %w", item.ID, err)
}
if !claimed {
continue
}
err = objects.Delete(ctx, item.ObjectKey)
if err != nil && !errors.Is(err, ErrAlreadyAbsent) {
return fmt.Errorf("delete %s: %w", item.ID, err)
}
if err := ledger.Complete(ctx, runKey, item.ID, "deleted"); err != nil {
return fmt.Errorf("record completion %s: %w", item.ID, err)
}
}
return nil
}
BeginRun should enforce uniqueness for the run key and preserve the original cutoff on retry. Claim should prevent concurrent workers from owning the same item, subject to a documented lease or recovery rule. Complete should append or durably record the outcome. Those implementations are storage-specific, so pretending one transaction recipe fits every database would be misleading.
Test the contract at the boundaries. Run the same key twice and assert one logical audit run. Stop after object deletion but before completion, then verify that retry converges on deleted. Start two workers against the same candidates and verify exclusive claims. Advance the wall clock during pagination and verify that the fixed cutoff does not change. Inject a held record and verify that no physical delete is attempted. These tests provide more assurance than asserting that the cron parser accepted five fields.
Deployment needs a similarly narrow sequence: begin with a dry run that records candidates without deleting them, compare the results with the approved retention rule, cap the first live batches, and monitor incomplete claims rather than only trigger success. Don't let a generic admin endpoint accept arbitrary cutoffs from the public request. The schedule should select a named policy, and authorization should constrain which policy can run.
Rejected option and its valid use case
For the stated daily cleanup, reject a durable workflow engine as the default. A single scan with bounded, independently idempotent items does not inherently need workflow history, activity versioning, durable timers, or compensation graphs; adding those concepts before the failure model demands them makes routine retention harder to operate and audit.
The rejection is conditional.
Choose a durable workflow when deletion requires a legal approval, a waiting period, an export that must finish first, ordered removal across several systems, or compensating action after a later step fails. Stick with cron plus a bounded worker when the policy is already decided and the only job is to make eligible items converge on a deleted state. Use cron plus a queue when the sweep is simple but the recovery unit must be each upload, log segment, or record. These boundaries preserve the simplest design without confusing "simple" with "fewest visible components."
References
- RabbitMQ, "Consumer Acknowledgements and Publisher Confirms": https://www.rabbitmq.com/docs/confirms
- Google Cloud, "Pub/Sub overview": https://cloud.google.com/pubsub/docs/overview
Top comments (0)