Short answer: put the reservation-expiration decision in an idempotent worker, put the retry policy on the message, and make the cleanup record its recovery state. A scheduler should create work. It should not be responsible for proving that a failed execution eventually recovered.
For a property-management system, the job is concrete: release a stale reservation after its fixed hold window so an available unit can be booked again. The dangerous failure is not a late log line. It is a reservation that remains active after its hold has expired, or a retry that releases a newer reservation by mistake.
The decision table for queued cleanup
| Approach | Pick this when | Recovery trade-off |
|---|---|---|
| Cron starts a worker directly | The cleanup is small, local, and a missed run can be safely replayed | Simple scheduling, weak visibility into individual failures |
| Cron publishes one message per cleanup window | The application already has a durable queue and workers need independent retries | Better isolation; the publisher must avoid duplicate windows |
| A scheduler publishes reservation IDs | Each reservation needs its own deadline, attempt count, and audit trail | Best diagnosis; more messages and a stricter idempotency contract |
For stale reservations, the third option is usually the cleanest boundary. A message should name the reservation and the deadline that made it eligible. A worker then rechecks current state before changing anything. That second check is the difference between expiring an old hold and damaging a new booking.
The cron format itself is only a trigger description. The Linux crontab(5) manual documents fields, matching rules, and environment behavior; it does not provide a durable retry ledger for the work launched by the command. Treat that distinction as an architectural decision, not a configuration detail.
What should happen when a cleanup message fails?
A useful message has four pieces of business context: reservationId, eligibleAt, attempt, and a stable operation key. The queue can add delivery metadata, but the worker needs enough information to decide whether a retry is still safe.
The state transition is easier to reason about as a short diagram in words:
scheduled hold -> eligible message -> state recheck -> expire, skip, or retry -> terminal record
A transient database timeout should return the message to the retry path. A malformed identifier should go to a dead-letter path after the message is recorded. A reservation that has already been canceled should be acknowledged as a no-op. Retrying all three cases forever turns a data-quality problem into an operational queue problem.
Use bounded exponential delay with jitter. For example, a policy can allow 5 attempts, with delays of 10 seconds, 30 seconds, 90 seconds, 270 seconds, and 810 seconds, then move the message to a review queue. Those values are policy examples, not universal defaults. The important properties are bounded work, visible terminal state, and enough delay for a short dependency outage to clear.
How do you make the Node.js worker safe to retry?
The worker should claim no more than it can finish within its visibility or lease interval. It should also use a conditional update: expire the reservation only when the stored hold token and eligibility timestamp still match the message. A duplicate delivery then becomes a harmless no-op.
That is the boundary.
type CleanupMessage = {
reservationId: string;
holdToken: string;
eligibleAt: string;
operationKey: string;
attempt: number;
};
type CleanupResult = "expired" | "skipped";
async function expireReservation(message: CleanupMessage): Promise<CleanupResult> {
const reservation = await reservations.findById(message.reservationId);
if (!reservation || reservation.holdToken !== message.holdToken) {
await audit.record({ key: message.operationKey, result: "skipped" });
return "skipped";
}
if (reservation.status !== "held" || reservation.holdUntil > message.eligibleAt) {
await audit.record({ key: message.operationKey, result: "skipped" });
return "skipped";
}
const changed = await reservations.expireIfHeld({
id: message.reservationId,
holdToken: message.holdToken,
holdUntil: message.eligibleAt,
});
await audit.record({
key: message.operationKey,
result: changed ? "expired" : "skipped",
});
return changed ? "expired" : "skipped";
}
The conditional operation belongs in the data store transaction where possible. A read followed by an unconditional write leaves a race: a guest can renew or replace the hold between those two calls. The operationKey also needs a uniqueness rule in the audit store, so a redelivered message cannot create a second “successful” history entry.
Keep retry classification close to the worker boundary. Errors caused by a temporary dependency can be retried. Validation failures and stale business state can be acknowledged or quarantined. Logging the classification matters because “message failed” is not a useful diagnosis by itself.
Metrics should answer three different questions: how much work is waiting, how old the oldest eligible message is, and how often attempts end in a terminal failure. One counter cannot answer all three. Track queue depth, age of oldest message, retry count, dead-letter count, and successful versus skipped expirations.
Logs should carry the reservation ID, operation key, attempt, scheduled eligibility time, and final classification. Avoid putting guest names, payment data, or full request payloads into those fields. A trace can connect the scheduler publish step to the worker and the database transaction, but the audit record remains the durable business evidence.
Alerts need recovery thresholds, not just error thresholds. A brief spike of retries may recover by itself. An oldest-message age that exceeds the hold window is more serious because it describes customer-visible delay. Page on sustained backlog age or a growing dead-letter queue; route isolated malformed messages for review. This is where the design earns its extra message and audit fields: an operator can distinguish a delayed queue from a business-state no-op, and can distinguish both from a dependency failure that is consuming the retry budget.
This is the operational recovery axis: the team should be able to tell whether work is delayed, being retried, permanently rejected, or safely skipped without opening application code.
Where does this design stop helping?
A queue does not solve an incorrect eligibility rule. Store timestamps with an explicit timezone policy, use a consistent clock strategy, and test the boundary where the hold expires exactly as a worker starts. Also test duplicate delivery, a changed hold token, a database timeout after commit, and a worker restart during processing.
Cron remains reasonable for a small maintenance command when the command is idempotent and the schedule can be replayed. A per-reservation message is heavier, so it should earn its place through better recovery and diagnosis. The decision is not “which scheduler is best?” It is “which state and evidence must survive a worker failure?”
A sound implementation has one scheduler trigger, one durable message per unit of work, a bounded retry policy, a conditional state change, and an operator-visible terminal outcome. That pattern keeps stale reservations from becoming invisible inventory and keeps retries from becoming a second source of data corruption.
Top comments (0)