Short answer: to implement scheduled data cleanup safely, treat expiry as a delivery problem, not a timer problem. Store the reservation's deadline, enqueue an idempotent expiry command, claim work with a lease, and make the final state transition conditional on the deadline still having passed. A rate-limited nightly sweep is the backstop, not the only clock.
The page I want at 3am is not cleanup ran. It is something with a consequence: reservations past their hold window are accumulating, the expiry queue is older than its service-level objective, or the worker is rejecting work faster than it completes it. A dashboard full of green scheduler ticks tells me very little. What page fired?
That is the whole point.
For a concrete example, suppose a developer-tools marketplace holds a name reservation for 30 minutes while an organization completes setup. The reservation should expire after that window, even if the process that created it disappeared. The important guarantee is usually at-least-once delivery with an idempotent state transition, not exactly-once execution. Exactly-once is an attractive label for a system that still has crashes between the database commit and the acknowledgement.
What should the first alert tell you?
Start at the alert and work backward. If the on-call sees 18,000 expired reservations still marked held, the useful questions are: when did the oldest deadline pass, how many expiry commands were attempted, how many were claimed, and which state transition was refused? Those dimensions distinguish a scheduler outage from a queue backlog, a rate limit, and a data-integrity guard doing its job.
I distrust a single count because it hides time. A count of 18,000 may be normal after a batch import and dangerous when the oldest item is 42 minutes late. Track age of the oldest eligible item and a histogram of lateness. Alert on both queue age and the ratio of successful terminal transitions to claimed jobs. A scheduler heartbeat can remain healthy while every worker is stuck on a downstream quota.
The signal should fire before users feel the problem. If the hold window is 30 minutes and the operational target is to expire work within 2 minutes of eligibility, an alert at 5 minutes of queue age gives someone time to inspect the consumer. That number is a policy example, not a universal threshold; choose it from the business consequence of retaining a name too long and the amount of noise the team can actually handle.
Measure the delay, not the intention.
The instrumentation change is small but consequential: emit eligible_at, claimed_at, completed_at, outcome, and attempt for every command, and attach a stable reservation identifier rather than a payload that can drift. Keep the scheduler's run identifier separate from the command identifier. One run may enqueue many commands, and retrying a run must not make a command look new.
A false positive has a cost too. Page on every brief spike and the responder learns to ignore the pager; page too late and the retention rule becomes an informal suggestion. The threshold is part of the delivery guarantee because it determines when a human is asked to restore it.
Why is the deadline more reliable than the nightly sweep?
The database row should contain the authoritative deadline, such as hold_until, and the transition should verify it at the moment of mutation. A cron-like trigger can run every night, every minute, or after a deployment; none of those schedules proves that a particular reservation is eligible. The deadline does.
A sweep can select rows where the state is held and hold_until <= now(), then publish expiry commands. It must be repeatable. If the process dies after publishing 500 commands but before recording its progress, the next sweep should find the same eligible rows without corrupting them. A cursor based on mutable status is easier to reason about than an offset that shifts as rows change, but either approach needs a bounded batch and an index that matches the eligibility predicate.
The same pattern can delete stale uploads in a SaaS service, but the reservation example makes the safety rule visible: a late cleanup action must not remove data whose owner renewed it. In a Node.js implementation the scheduler and consumer can be separate processes, yet that split changes no database invariant. The queue is an execution detail; hold_until remains the authority.
The final update must be conditional. A reservation can be renewed, cancelled, or completed while an expiry command is waiting in the queue. The worker must not turn a newer state back into expired merely because an old command arrived. This is why the command carries the reservation id and an observed deadline, while the database decides whether the transition is still valid.
Here is the core shape in Go. The repository and queue interfaces are deliberately generic; the important part is the predicate and the distinction between retryable and terminal outcomes.
type ExpireCommand struct {
ReservationID string
HoldUntil time.Time
Attempt int
}
func (w *Worker) Handle(ctx context.Context, cmd ExpireCommand) error {
result, err := w.repo.ExpireIfDue(ctx, cmd.ReservationID, cmd.HoldUntil)
if err != nil {
return Retryable(err)
}
switch result {
case Expired:
return nil
case AlreadyFinal, DeadlineMoved:
return nil
default:
return Retryable(fmt.Errorf("unexpected transition result: %v", result))
}
}
In SQL terms, the update needs a predicate equivalent to state = 'held' AND hold_until <= current_time, with the reservation id in the key. If the command is duplicated, the second execution observes a non-eligible row and succeeds as a no-op. That is idempotency in the place that matters: the state transition, not a hopeful de-duplication cache.
How should I implement scheduled data cleanup with rate limits?
A worker queue smooths work; it does not remove an external quota. Set concurrency and pacing from the strictest dependency, then leave room for ordinary application traffic. A token bucket can enforce a rate, while a bounded worker pool limits simultaneous requests. They solve different problems. Ten workers each making one request per second still create ten requests per second, and bursts can violate a quota even when the average looks safe.
Claiming is another boundary. With a relational store, workers can claim a bounded set of eligible rows using a transaction and row locks, often with FOR UPDATE SKIP LOCKED, then record a lease expiration. The lease must be finite so a crashed worker does not own work forever. The command can be delivered again after a lease expires; the conditional transition still protects the row.
The transaction should be short. Do not hold database locks while waiting on a remote API. Claim rows, commit, perform the external operation if one exists, then record the result with a compare-and-set condition on the lease token. If the remote side has its own idempotency key, use the reservation id plus operation name. If it does not, acknowledge the uncertainty explicitly and make the local operation safe to repeat where possible.
A simplified consumer loop makes the pacing visible:
func (w *Worker) Run(ctx context.Context) error {
ticker := time.NewTicker(w.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
cmd, err := w.queue.Receive(ctx)
if err != nil {
return err
}
err = w.Handle(ctx, cmd)
if err == nil {
if ackErr := w.queue.Ack(ctx, cmd); ackErr != nil {
return ackErr
}
continue
}
if w.retryable(err) {
if nackErr := w.queue.Retry(ctx, cmd, backoff(cmd.Attempt)); nackErr != nil {
return nackErr
}
} else {
if deadErr := w.queue.DeadLetter(ctx, cmd, err); deadErr != nil {
return deadErr
}
}
}
}
}
The example intentionally acknowledges only after the handler returns success or after the command has been placed in a deliberate terminal path. A process crash between the state commit and the acknowledgement creates a duplicate. That is acceptable under at-least-once delivery because the handler is designed for it. A process crash after an acknowledgement but before the state commit is not acceptable, which is why the acknowledgement belongs after the transition.
The restart case deserves its own rehearsal. Imagine a deployment drains half the workers while a backlog contains reservations with deadlines spread across the previous hour. Some commands have been received but not acknowledged, some rows have leases that will expire in 90 seconds, and the sweep is starting its next batch. A healthy design does not require an operator to remember which category each row belongs to: the lease can be reclaimed, the duplicate command can be safely handled, and the deadline predicate can reject a renewal race. The operational record should show those outcomes separately so that a successful no-op is not confused with a failed expiry. This is a longer path than “run the job again,” but it is the path that makes a restart ordinary rather than an incident.
Backoff should include a cap and jitter. Retries for a 429 or a transient network failure should not synchronize every worker into another burst. A malformed command, an unknown reservation, or a deadline that moved forward is a different class: retrying it forever converts bad data into an outage. Send it to a dead-letter path with enough context to repair or discard it deliberately.
There is a real tradeoff here. A fixed hold window with a durable sweep is a poor fit for work that needs sub-second expiry, and a queue is needless machinery for a tiny dataset that can be checked inside the owning transaction. Conversely, a single request from a Node.js cron process is a poor fit when a restart can interrupt thousands of stale records. Choose the smallest mechanism that still gives the required delivery guarantee.
Which tests prove the behavior instead of the timer?
Test the state machine with a controllable clock, but do not stop there. The important tests are crash points and reorderings: publish twice, receive twice, renew after enqueue, acknowledge after a committed transition, and kill the worker after claiming. The expected result is one expired reservation or a safe no-op, never a resurrection of an already completed reservation.
Run a small integration test against the actual transaction isolation and lock behavior used in production. A mock will happily let two workers claim the same row unless the test asks the database to prove otherwise. Include a batch larger than one page and verify that the sweep resumes after a partial failure. Test an empty sweep as a normal success, not as an exceptional condition.
Deployment deserves a test too. During a rolling restart, old and new workers may process the same command schema. Add a versioned command envelope only when the payload really needs evolution, and make unknown fields ignorable where the serializer permits it. Schema changes to hold_until or state values should be expanded, deployed, backfilled if necessary, and contracted later. Expiry is a background workflow; it should not block a safe application rollout.
The runbook should answer four questions without a database archaeology project: how to pause production expiry, how to inspect the oldest eligible reservation, how to replay a dead-letter command, and how to distinguish a downstream quota from a local transaction failure. A manual replay must use the same idempotent handler. The emergency procedure is part of the system's delivery guarantee.
What survives a worker restart?
Use a deadline in durable storage, a repeatable sweep, at-least-once commands, conditional state transitions, bounded claims, and explicit pacing. Choose the sweep interval from lateness tolerance, choose the lease from the longest expected handler time, and choose the retry policy from the dependency's failure semantics. Those values should be reviewed together; changing one in isolation often moves the failure into another queue.
The cleanest success metric is not scheduler uptime. It is the age distribution of eligible reservations that have not reached a terminal state, split by outcome and attempt count. When that metric is healthy, duplicate delivery is boring, retries are finite, and the alert names a user-visible consequence. That is the standard I want from a job that runs while everyone else is asleep.
Do not promise more than the system can prove.
Further reading
- Celery introduction and task delivery concepts: https://docs.celeryq.dev/en/stable/getting-started/introduction.html
- PostgreSQL
SELECT, including row locking andSKIP LOCKED: https://www.postgresql.org/docs/current/sql-select.html - PostgreSQL transaction isolation: https://www.postgresql.org/docs/current/transaction-iso.html
- OpenTelemetry semantic conventions: https://opentelemetry.io/docs/specs/semconv/
Top comments (0)