Short answer: let cron enqueue one dated cleanup run, let idempotent queue workers delete bounded batches, and retain renewal delivery records until their replay windows have closed.
A marketplace renewal reminder creates an awkward retention problem. Delivery attempts, deduplication keys, and expired leases accumulate, but deleting them too early weakens the evidence that prevents a second reminder. Running the whole delete from a Node.js cron callback, or holding an HTTP request open, makes the scheduler part of the data plane. The safer invariant is small: the trigger requests work; it does not perform the sweep.
I've been paged by both failure classes: missed jobs and duplicate deliveries. The useful postmortem question wasn't which cron library started the run. It was whether one retry could create two logical runs, and whether cleanup could erase the state needed to recognize a repeated delivery. Those are two different symptoms with the same control failure — execution was treated as exactly-once even though ownership and durable state could not prove it. Protect the delivery invariant before reclaiming storage.
How should scheduled Node.js data cleanup move from cron to an HTTP queue?
Treat a nightly cleanup as an at-least-once command even if the scheduler normally fires once. A retry, a process restart, or two schedulers during a deployment must converge on the same logical run. Use a stable key such as renewal-retention/2026-08-12, backed by a unique constraint, and acknowledge the HTTP request after that run exists durably. The date illustrates key shape; it does not imply that every marketplace should use the same timezone or cutoff.
The worker then claims a bounded batch. Each transaction selects rows older than the configured retention boundary, deletes only those rows, records progress, and commits before the queue message is acknowledged. If the worker stops after commit but before acknowledgment, redelivery finds that the selected rows are already gone and continues safely. If it stops before commit, the rows remain eligible.
No guesswork.
Retention governance comes before deletion
Do not put pending reminders in that predicate. Keep three lifecycles explicit: the business deadline for the reminder, the replay or deduplication horizon for delivery, and the retention period for operational history. Cleanup is allowed only after the relevant horizons expire. I'm not sure there is a universal retention duration; legal requirements, dispute handling, queue redrive policy, and the longest credible interruption determine it for a given system.
| State | Cleanup rule | Delivery reason |
|---|---|---|
| Pending renewal reminder | Never select it | It is still business work |
| Delivery deduplication key | Retain it through the replay horizon | A redelivery must resolve to the same outcome |
| Delivery attempt history | Delete bounded batches after policy expiry | It supports diagnosis until its retention duty ends |
| Expired worker lease | Reclaim it only when no active owner can renew | Another worker may safely resume the run |
Spend the queue complexity budget deliberately
The catch is that a queue adds a dispatcher, worker ownership, redrive policy, and more telemetry. It is not suitable when cleanup is provably tiny, completes well inside the scheduler's execution window, and a single transactional delete has acceptable lock impact. In that narrow case, keep a direct cron task, retain the same unique run key, and still make the delete replay-safe. At the other extreme, use a database-native partition drop or lifecycle policy when retention aligns exactly with partitions and per-row business checks are unnecessary; a row-by-row worker would add cost without adding safety.
Use an HTTP endpoint only as an enqueue boundary. Choose queue workers when cleanup may outlive a request, needs bounded concurrency, or must resume after interruption. Keep direct cron execution for small, measured jobs. Prefer partition lifecycle operations for large, cleanly partitioned retention sets.
Implement acceptance, publication, and replay as one contract
The cron callback should make one authenticated request to an internal endpoint. That handler validates the requested business date, derives the idempotency key, inserts the cleanup run if it does not exist, and publishes that run to workers. Return quickly. Don't stream deletion progress through the HTTP response.
A Node.js service can implement this contract with its normal router and database client. The example is in Go to make the interface easy to isolate: one request becomes one durable run, and duplicate requests return the same run ID. InsertRun must enforce uniqueness on RunKey; Publish must tolerate the same run being announced more than once.
package cleanup
import (
"encoding/json"
"net/http"
"time"
)
type Store interface {
InsertRun(runKey string, cutoff time.Time) (runID string, created bool, err error)
}
type Publisher interface {
Publish(runID string) error
}
type Handler struct {
Store Store
Queue Publisher
Now func() time.Time
}
type request struct {
BusinessDate string `json:"business_date"`
}
func (h Handler) Enqueue(w http.ResponseWriter, r *http.Request) {
var in request
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
http.Error(w, "invalid request", http.StatusBadRequest)
return
}
day, err := time.Parse("2006-01-02", in.BusinessDate)
if err != nil || day.After(h.Now().UTC()) {
http.Error(w, "invalid business date", http.StatusBadRequest)
return
}
runKey := "renewal-retention/" + day.Format("2006-01-02")
runID, created, err := h.Store.InsertRun(runKey, day)
if err != nil {
http.Error(w, "request could not be accepted", http.StatusServiceUnavailable)
return
}
if created {
if err := h.Queue.Publish(runID); err != nil {
http.Error(w, "request could not be accepted", http.StatusServiceUnavailable)
return
}
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusAccepted)
_ = json.NewEncoder(w).Encode(map[string]string{"run_id": runID})
}
There is an important detail behind that compact handler: a database row that exists but was not published must be recoverable by a dispatcher scanning undispatched runs, or the insert and publish must share a transactional outbox. Otherwise a process exit between those operations creates a durable run that no worker sees. Walk that interval during design review: request accepted, run key inserted, process interrupted, dispatcher resumes, worker claims the dated run, batch commits, acknowledgment is delayed, and the same message arrives again. At every arrow, identify the durable record that assigns ownership and the operation that is safe to repeat. The public response doesn't need to expose that machinery, but the runbook does.
A long-running job should be a series of short transactions, not one enormous transaction. Select a stable batch using an immutable primary key and a strict cutoff. Delete by those selected IDs in the same transaction. Record a cursor only if the ordering and snapshot semantics are understood; repeatedly selecting eligible IDs can be simpler because successful deletes naturally remove completed work.
The preventative path below puts the safety condition in the repository boundary. DeleteExpiredAttempts must exclude any attempt whose deduplication key is still inside the replay horizon, and the database should enforce that predicate in the transaction rather than trusting an in-memory filter.
package cleanup
import (
"context"
"time"
)
type Repository interface {
DeleteExpiredAttempts(ctx context.Context, before time.Time, limit int) (int, error)
}
type Worker struct {
Repo Repository
BatchSize int
}
func (w Worker) Run(ctx context.Context, cutoff time.Time) error {
for {
deleted, err := w.Repo.DeleteExpiredAttempts(ctx, cutoff, w.BatchSize)
if err != nil {
return err
}
if deleted == 0 {
return nil
}
select {
case <-ctx.Done():
return ctx.Err()
default:
}
}
}
Keep the batch size configurable and start conservatively. Larger batches finish with fewer round trips but hold locks longer and create burstier replication and I/O pressure; smaller batches cost more dispatch overhead but give cancellation and rate control more opportunities. Your mileage may vary because index shape, row width, concurrent writes, and database isolation all change the useful size. Measure transaction duration and lock wait rather than copying a number from another system.
When a downstream HTTP dependency answers 429 Too Many Requests, respect Retry-After when it is present and reschedule the unit of work rather than spinning in the worker. A dead-letter queue can isolate messages that repeatedly fail processing, but it is not an archive and should have an explicit inspection and redrive runbook. Those mechanisms protect the cleanup control loop; they do not make a non-idempotent delete safe.
Retries happen.
Observe age, then rehearse recovery
Alert on the oldest eligible row age, the age of the oldest unfinished cleanup run, repeated redelivery count, dead-letter depth, and rows deleted per batch. A "cron fired" metric proves only that a trigger executed. It cannot prove that every batch committed or that the retention boundary was correct.
Deploy predicate changes with a dry-run query that reports counts by record state and age bucket, then review a sample of primary keys. The actual deletion path should use the same predicate. Before widening a cutoff, verify backups and restore procedures, cap the first run, and watch database saturation. Afterward, reconcile eligible rows against completed work. This is runbook work, not ceremony.
Test the ugly boundaries as one recovery drill rather than isolated unit cases: enqueue the same business date twice and confirm one run ID; stop a worker before commit and verify the batch remains eligible; stop it after commit but before acknowledgment and verify redelivery makes no additional business change; cancel between batches and verify another worker can resume; advance the clock across the marketplace timezone boundary and verify the cutoff is unchanged; then redeliver a completed message. The expected result is one logical cleanup run, no pending renewal removed, and eventual removal of every eligible history row. Record the run key, cutoff, claimed batch range, delete count, attempt number, and terminal state in logs so an operator can reconstruct that chain without querying application tables during an incident.
The timer is innocent.
The deciding question is not "Can cron run this nightly?" It is "After any retry or interruption, can we prove that renewal reminders are neither lost nor duplicated and that only expired operational data was removed?" If the answer depends on the scheduler firing exactly once, the design isn't ready.
References
- AWS, "Amazon SQS dead-letter queues": https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- MDN, "429 Too Many Requests": https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
Top comments (0)