Short answer: put the retry timestamp in the queue, cap exponential backoff with jitter, and make reservation expiry idempotent. A worker sleep is the wrong place to hold a property-management reservation: a deploy or a stalled process can erase the only recovery signal.
In this example, a reservation gets a fixed hold window of 15 minutes. The expiry job may fail because the database is briefly unavailable, but a late retry must never cancel a booking that has since been paid. That operational recovery rule matters more than picking a fashionable queue library.
What should a failed-job queue remember?
Treat each message as a small recovery record. It needs a reservation id, the hold deadline, an attempt count, and an idempotency key. The deadline is business data; the next delivery time is transport data. Keeping both prevents a retry from silently extending a guest's hold.
The worker below uses a generic queue interface. publish accepts a delayed delivery timestamp, and ack removes a successfully handled message. Those are ordinary queue capabilities, so the same shape can sit behind a hosted queue, a database-backed poller, or a self-hosted broker.
type ExpiryJob = {
reservationId: string;
holdUntil: string;
attempt: number;
idempotencyKey: string;
};
type Queue = {
publish: (job: ExpiryJob, deliverAt: Date) => Promise<void>;
ack: (messageId: string) => Promise<void>;
deadLetter: (job: ExpiryJob, reason: string) => Promise<void>;
};
const BASE_DELAY_MS = 5_000;
const MAX_DELAY_MS = 15 * 60_000;
const MAX_ATTEMPTS = 7;
function retryAt(attempt: number, now = Date.now()): Date {
const exponential = Math.min(MAX_DELAY_MS, BASE_DELAY_MS * 2 ** attempt);
const jitter = Math.floor(Math.random() * Math.max(1, exponential * 0.2));
return new Date(now + exponential + jitter);
}
async function handleExpiry(
queue: Queue,
messageId: string,
job: ExpiryJob,
expireIfStillHeld: (id: string, holdUntil: string, key: string) => Promise<"expired" | "already-done" | "changed">
) {
try {
const result = await expireIfStillHeld(job.reservationId, job.holdUntil, job.idempotencyKey);
if (result === "changed") {
// A payment or extension won the race; acknowledge without expiring it.
await queue.ack(messageId);
return;
}
await queue.ack(messageId);
} catch (error) {
if (job.attempt >= MAX_ATTEMPTS) {
await queue.deadLetter(job, "attempt budget exhausted");
await queue.ack(messageId);
return;
}
await queue.publish(
{ ...job, attempt: job.attempt + 1 },
retryAt(job.attempt)
);
await queue.ack(messageId);
}
}
The conditional operation behind expireIfStillHeld should update a row only when its status is still held and its stored deadline is less than or equal to the current time. A unique idempotency key makes a redelivery harmless. In practice, this is one SQL transaction: read the current state, perform the conditional transition, and record the key.
A five-minute in-worker sleep is easy to write, so it is worth testing the failure path before shipping it. Kill the process during the sleep, then compare the reservation table with the worker logs and deploy timestamps. The timer has no queue receipt, no retry count, and no durable clue for an operator. Treating “wait five minutes” as a scheduling guarantee confuses an instruction to one process with a persisted deadline. The replacement writes an outbox row in the same transaction as the hold, and a relay publishes the row with the absolute deadline. If the relay restarts halfway through a batch, it may send a duplicate, but the idempotency key reduces that duplicate to a no-op. During a deployment drill, terminate a worker with messages in flight, wait for the visibility timeout, and verify that another worker reclaims them. This turns recovery into a normal deployment check instead of an incident-only ritual. The fix is boring: persist the schedule and let another worker claim it.
No receipt. No retry.
Measure twice.
How do delayed messages, exponential backoff, and failed jobs fit together?
Backoff should describe transient work, not business timing. The initial expiry message is scheduled for holdUntil; only a failure gets the short 5-second base delay. Each retry doubles up to 15 minutes, then adds up to 20% random jitter. Jitter spreads a database recovery across workers instead of creating a synchronized thundering herd.
Do not retry every exception. A validation error, a missing reservation, or a deliberate cancellation is terminal and should be acknowledged or sent to a dead-letter stream immediately. Network timeouts and temporary database contention are candidates for retry. Your mileage may vary when the provider's error taxonomy is coarse; log the original error class so that policy can be tightened later.
The attempt budget is part of the contract. Seven attempts with this cap keep a poison message from cycling forever, while the dead-letter record preserves enough context for an operator to inspect it. A replay tool should create a new idempotency key only after a human confirms the reservation is still eligible.
Where does Node.js scheduling actually fail?
Node.js timers are process-local. setTimeout is useful for a quick demo, but it cannot provide durable delayed delivery across crashes or deploys. A queue consumer also needs a visibility timeout longer than the expected database transaction; otherwise a slow worker can be redelivered while the first attempt is still running.
Use a monotonic lease for the worker lock and wall-clock timestamps for the business deadline. Clock skew between hosts can make a reservation appear early, so compare deadlines in the database that owns the reservation. Keep the payload small and put customer-facing details in the database; stale copies in a message are hard to reconcile.
For a multi-write flow, create the reservation and its initial expiry message through a transactional outbox. The outbox row commits with the reservation, then a relay publishes it and marks the row sent. This avoids the split-brain case where the booking exists but the process crashes before enqueueing its expiry. The pattern is described at microservices.io and remains useful even when the queue itself is reliable.
Which trade-offs should an operator choose?
There is no universally best queue. A managed delayed queue reduces maintenance but may limit ordering or redrive controls. A database poller is easy to inspect and often sufficient at modest volume, but polling adds load and needs careful indexes. A broker with native dead letters gives stronger throughput controls, at the cost of another operational surface. Choose using recovery evidence: replayability, visibility into attempt history, deadline precision, and the blast radius of a duplicate.
The catch is that this pattern is not suitable when expiry must be sub-second, when a strict global ordering is required, or when the queue cannot persist delayed delivery. In those cases, use a scheduler with the needed clock guarantees or keep the state transition inside the system that already owns ordering. Stick with a simple database sweep when reservations are low volume and a few seconds of lateness is acceptable.
I measure four signals: age of the oldest due message, retry count by error class, dead-letter rate, and the gap between holdUntil and the actual transition. Alert on the gap, not only on queue depth. A healthy-looking queue can still be late if consumers are acknowledging messages before the database commit.
Top comments (0)