For a scheduled data cleanup API that removes expired game files, delivery guarantees matter more than the timer syntax. If losing one run is acceptable, a cron trigger is the smaller design. If every eligible object must eventually be handled, put a durable work record behind the trigger and make the delete operation idempotent.
Short answer: choose cron-only for best-effort housekeeping; choose a durable queue with bounded retries and a dead-letter path when delivery guarantees matter. The scheduler should create work, not perform all of it while a web request waits. That distinction is the useful boundary.
I judge this at 3am, when an alert has already made me ask what page fired and whether the alarm describes lost work or merely delayed work. A gaming service deleting expired replay files, screenshots, or match artifacts needs that answer before the first failure, because a successful HTTP response from a scheduler says very little about the objects left in storage.
What failed: the timer ran, but did the cleanup happen?
The bounded incident pattern is familiar: a periodic task wakes up, scans a set of expired asset records, starts deleting files, and then loses its process or connection partway through the batch. A cron-only implementation can report that the invocation started while leaving the remaining records for the next interval. That may be fine for disposable thumbnails. It is a poor contract for an expiry policy that must be auditable or that controls access to private match data.
The invariant I want is narrower than “the job succeeded”: every selected cleanup item has a durable state transition, and a later attempt can safely repeat it. The scheduler owns the time boundary. A worker owns one item. The database or queue owns the fact that the item still needs attention.
There is a cost to that invariant. A queue adds visibility, retry policy, retention, and another operational surface. I accept that cost only when a missed deletion has a meaningful consequence. A queue does not create exactly-once deletion, either; crashes can occur after the storage service accepts a delete and before the worker records completion. The practical target is at-least-once delivery plus an idempotent operation.
How should a Node.js API schedule data cleanup for game files?
Ask this before choosing a scheduler: what happens if one 15-minute run never reaches a particular object?
For cache-like game previews, the answer might be “the next scan handles it.” A timer and a query for expired rows are enough, provided the query is bounded and repeated scans are safe. I initially treated that as the default for every asset class; then I noticed that a private match recording and a disposable thumbnail have very different consequences when the 15-minute scan stops halfway through. For regulated exports, player-generated reports, or access-controlled recordings, the answer may require an explicit pending state, an attempt count, and a human-readable dead-letter record. The word “cleanup” hides these different contracts.
A useful decision table is deliberately boring:
| Requirement | Cron-only sweep | Durable work queue |
|---|---|---|
| Missing one interval is acceptable | Good fit | Usually unnecessary |
| Every selected item needs eventual handling | Weak unless the sweep is durable | Good fit |
| Duplicate delivery is possible | Must make the sweep repeatable | Must make each item idempotent |
| Operators need failed-item replay | Requires a separate record | Natural dead-letter workflow |
| Work can outlive the request | Only if the process persists it elsewhere | Built into the worker boundary |
This is also where “simple” gets defined honestly. A cron expression is simple to type. A delivery contract is simple to reason about only after it has been written down.
The timer did run.
How should a worker make deletion safe?
The preventative path is to enqueue a stable cleanup identity rather than a mutable blob of scan results. The identity can be the object key plus an expiry version or record identifier. The worker verifies that the item is still eligible, deletes the storage object, and records the outcome using a conditional state transition. If the acknowledgment is lost, the same identity comes back and the delete is harmless or is treated as already complete.
Here is the core state machine in Go. The interfaces are intentionally generic; the important part is the order and the idempotency boundary, not a provider-specific client.
type CleanupItem struct {
ID string
ObjectKey string
ExpiryEpoch int64
}
type Store interface {
IsStillEligible(context.Context, string, int64) (bool, error)
DeleteObject(context.Context, string) error
MarkComplete(context.Context, string) error
}
func Handle(ctx context.Context, store Store, item CleanupItem, now int64) error {
eligible, err := store.IsStillEligible(ctx, item.ID, item.ExpiryEpoch)
if err != nil {
return fmt.Errorf("check eligibility: %w", err)
}
if !eligible || item.ExpiryEpoch > now {
return nil
}
if err := store.DeleteObject(ctx, item.ObjectKey); err != nil {
return fmt.Errorf("delete %q: %w", err)
}
if err := store.MarkComplete(ctx, item.ID); err != nil {
return fmt.Errorf("mark %q complete: %w", err)
}
return nil
}
The awkward ordering is intentional. Marking the database row complete first can strand the object if the subsequent storage delete fails. Marking it complete after deletion means a crash can cause a duplicate attempt, so the storage delete must tolerate that outcome. If the storage API distinguishes “missing” from other errors, classify missing as success and send permission, throttling, and malformed-key failures through different retry policies.
A retry count is not a guarantee. It is a budget. Transient failures deserve backoff and a cap; permanent failures should stop consuming worker capacity and land in a dead-letter record with the original item, last error, attempt count, and timestamps. The operator needs a replay action that reuses the same identity after the underlying cause is fixed.
What does the scheduler need to observe?
The dashboard is not the contract. I want separate signals for scheduled runs created, work items accepted, items completed, retry age, and dead-letter depth. A green scheduler metric beside a growing queue is a delayed incident, not a healthy system. Alert on oldest pending age and dead-letter growth, not only on process liveness.
Log the cleanup identity, object key in a privacy-conscious form, attempt number, result class, and correlation identifier. Avoid logging player content or signed URLs. If a caller can create a cleanup request, authenticate it and protect any signed payload against tampering; HMAC is specified for keyed message authentication in RFC 2104.
Tests should exercise the crash windows, not just the happy path: the worker exits after deletion, the record update times out, the object is already absent, the expiry is extended before processing, and a poison item reaches its dead-letter limit. A deployment should also prove that old workers and new message schemas can coexist long enough to drain outstanding work.
Priority is a policy decision, too. If urgent moderation cleanup shares capacity with bulk expiry, a priority-capable queue can express that preference, but priority does not replace fairness or a retry budget. RabbitMQ documents priority queues as an ordering feature with operational trade-offs; similar semantics must be verified for whichever queue is selected rather than assumed from the word “priority.”
When is the smaller design the right one?
Cron-only wins when the cleanup is repeatable, bounded, and explicitly best effort. Keep a durable cursor or query on expiry timestamps if the dataset is large, use a lease so two invocations do not duplicate expensive scans, and make each delete safe to repeat. That is a reasonable design for disposable game previews whose delay has no user-visible consequence.
A durable queue wins when the system needs per-item retries, replay, isolation from request lifetime, or evidence that a particular object was handled. It still needs a scheduler, a worker pool, storage semantics, and observability. The queue is not a magic reliability coating. It is a place to preserve the work that a timer alone cannot remember.
The advice does not apply unchanged to irreversible player-data deletion, legal holds, or a cleanup that spans several databases. Those cases need an explicit retention policy, authorization checks, and a reconciliation process; treating a storage delete as the whole transaction would be an unsafe simplification.
My choice for the game-asset scenario is therefore conditional: cron-only for disposable, replayable housekeeping; a durable queue and dead-letter workflow for cleanup whose delivery must be demonstrated item by item. The incident lesson is not “always add a queue.” It is to decide what a missed deletion means, then choose the smallest architecture that can prove the answer.
Sources
- RFC 2104, HMAC: https://www.rfc-editor.org/rfc/rfc2104
- RabbitMQ priority queues: https://www.rabbitmq.com/docs/priority
- Amazon S3 DeleteObject API reference: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html
- PostgreSQL transaction isolation documentation: https://www.postgresql.org/docs/current/transaction-iso.html
Top comments (0)