Short answer: for a small Node.js marketplace, choose a scheduled sweep before a delayed message queue when periodic cleanup is the only job; switch when retry traffic needs isolation from the primary database or delay accuracy becomes an explicit requirement. In both designs, keep attempts, the next eligible time, and the idempotency result in durable storage. The scheduler or queue should wake a worker, not become the authority on whether a cleanup already happened.
Picture the data flow without the infrastructure labels. A timer finds expired reservations and creates stable job IDs. A worker claims one job, removes the temporary reservation exactly once at the business layer, then records success. A retryable response schedules another attempt; exhausted or invalid work enters a dead-letter state for a person to inspect. No seller or buyer request waits for that sequence.
That split is the decision. The database owns truth; the wake-up mechanism owns timing.
How can Node.js SaaS code handle failed jobs and delayed delivery?
Start with the state machine, because changing a timer into a queue later is easy compared with recovering retry history that existed only in logs. The following TypeScript keeps transport concerns behind Wakeup, while JobStore owns claim, completion, retry, and dead-letter transitions. The sample delays of 30, 120, and 600 seconds are policy choices for this example, not universal defaults.
type CleanupJob = {
id: string;
reservationId: string;
attempt: number;
dueAt: Date;
};
type Failure =
| { kind: "rate_limited"; retryAfterMs?: number }
| { kind: "temporary" }
| { kind: "invalid"; reason: string };
interface JobStore {
claim(jobId: string): Promise<CleanupJob | null>;
wasApplied(reservationId: string): Promise<boolean>;
markApplied(reservationId: string, jobId: string): Promise<void>;
complete(jobId: string): Promise<void>;
retry(jobId: string, attempt: number, dueAt: Date): Promise<void>;
deadLetter(jobId: string, reason: string): Promise<void>;
}
interface Wakeup {
schedule(jobId: string, dueAt: Date): Promise<void>;
}
interface Marketplace {
releaseExpiredReservation(reservationId: string): Promise<void>;
}
const retryDelayMs = [30_000, 120_000, 600_000] as const;
async function runCleanup(
jobId: string,
store: JobStore,
wakeup: Wakeup,
marketplace: Marketplace,
): Promise<void> {
const job = await store.claim(jobId);
if (!job) return;
if (await store.wasApplied(job.reservationId)) {
await store.complete(job.id);
return;
}
try {
await marketplace.releaseExpiredReservation(job.reservationId);
await store.markApplied(job.reservationId, job.id);
await store.complete(job.id);
} catch (error: unknown) {
const failure = classifyFailure(error);
if (failure.kind === "invalid" || job.attempt >= retryDelayMs.length) {
await store.deadLetter(
job.id,
failure.kind === "invalid" ? failure.reason : "retry budget exhausted",
);
return;
}
const delay =
failure.kind === "rate_limited" && failure.retryAfterMs !== undefined
? failure.retryAfterMs
: retryDelayMs[job.attempt];
const dueAt = new Date(Date.now() + delay);
await store.retry(job.id, job.attempt + 1, dueAt);
await wakeup.schedule(job.id, dueAt);
}
}
function classifyFailure(error: unknown): Failure {
if (error instanceof MarketplaceError && error.status === 429) {
return { kind: "rate_limited", retryAfterMs: error.retryAfterMs };
}
if (error instanceof MarketplaceError && error.status >= 400 && error.status < 500) {
return { kind: "invalid", reason: `request rejected with ${error.status}` };
}
return { kind: "temporary" };
}
class MarketplaceError extends Error {
constructor(
readonly status: number,
readonly retryAfterMs?: number,
) {
super(`marketplace request failed with ${status}`);
}
}
The important line isn't the delay array. It's wasApplied. At-least-once delivery allows the same job to arrive again, so the handler needs a durable record keyed by the business operation, such as release-reservation:<reservationId>. A unique constraint around that key can make two workers converge on one outcome. Keep the check and the business mutation in one transaction when they share a database; if the side effect crosses a service boundary, make the receiving service enforce the same key. A process crash between the remote effect and the local completion record is otherwise enough to repeat the effect.
No magic here.
Backoff can wait.
The claim operation also needs exclusive ownership with an expiry. A worker that disappears cannot hold a job forever, while a slow worker must not be silently joined by another one before its claim expires. Pick the claim duration from observed high-percentile cleanup time and leave headroom. I'm not sure what that duration should be for a marketplace I haven't measured; a production-shaped load test and job-duration histogram resolve it.
Govern the durable job ledger
A retry ledger should answer five questions from one row: what business action is pending, who currently owns it, how many attempts have run, when it may run again, and why it stopped. Suggested fields are job_id, reservation_id, idempotency_key, state, attempt, due_at, claim_until, last_failure_class, and finished_at. This is deliberately plain. Operators should be able to distinguish pending, claimed, succeeded, and dead-letter work without reconstructing a timeline across application logs.
Failure classification belongs beside that state. An invalid reservation reference is terminal. A temporary dependency failure may be retried with a bounded backoff. HTTP 429 means the caller has sent too many requests in a period, and the response may include Retry-After; when it is present, preserve that instruction rather than replacing it with a locally convenient delay. After the attempt budget is exhausted, stop automatic delivery and retain the envelope plus a sanitized failure class for review.
Don't auto-replay the dead-letter set on a timer. Replay is a new operational decision: validate that the underlying cause is gone, retain the original idempotency key, assign a replay ID for audit, and release a bounded batch. Ten bad records are an investigation. Ten thousand released at once are fresh load.
This is also where cost enters the design, but not as a vendor price comparison. A sweep spends database reads while idle and can create lock pressure under load; a queue adds another operated component plus message traffic. Measure queries per sweep, rows examined per claimed job, oldest-ready age, attempts per completion, and dead-letter growth. Those numbers show which resource is becoming expensive. Guessing does not.
Migrate wake-ups without moving business truth
Treat the two options as replaceable wake-up adapters around the same ledger. A scheduled sweep queries a bounded page where state = 'pending' and due_at <= now, claims those rows, and exits. A delayed queue publishes the stable job ID for later consumption, but the consumer still claims the ledger row before changing marketplace data. That extra lookup can look redundant. It is what prevents an old delivery from reviving completed work.
| Decision signal | Scheduled sweep | Delayed message queue |
|---|---|---|
| Idle workload | Repeated indexed queries | No application poll loop |
| Timing | Bound by sweep interval | Bound by the queue's delivery behavior |
| Burst isolation | Shares database capacity | Buffers work outside the primary database |
| Retry inspection | Ledger query is direct | Ledger remains the inspection source |
| Duplicate defense | Claim plus idempotency record | Claim plus idempotency record |
| Operational surface | Database and scheduler | Database, publisher, consumer, and queue |
Choose the sweep while cleanup volume is modest, the primary database has measured headroom, and minute-level timing is acceptable. It's a short path to production and keeps failure state close to the reservation records. The catch is that frequent polling trades timing precision for database work, and concurrent sweepers require careful claims. This option is not suitable when cleanup bursts can compete with buyer-facing queries or when the retry stream needs an independently controlled concurrency budget.
Choose a delayed queue at that boundary, not because a broker grants exactly-once business effects. RabbitMQ's acknowledgement documentation describes the key failure window: deliveries that were not acknowledged can be requeued and redelivered, so consumers must be ready for redelivery. The same business-layer idempotency rule remains. A queue is also a poor fit when nobody on the team can operate and observe the extra component; stick with the sweep until its measured limits, rather than adopting infrastructure for an imagined scale problem.
I care more about that migration boundary than feature counts. Keep Wakeup.schedule(jobId, dueAt) narrow, deploy the queue publisher without changing job semantics, compare ledger counts against consumed wake-ups, then disable the poller. The job ID, attempt budget, and replay procedure stay put — only the alarm clock changes.
Only the alarm clock.
Test concurrency budgets before launch
Before deployment, exercise the four transition edges that tend to expose bad assumptions: two workers claim the same job, the process stops after the business effect but before completion is recorded, a 429 carries a retry delay, and the final attempt fails. The second case deserves extra attention. Let worker A release reservation r-1842, then stop it before complete persists; after the claim expires, let worker B receive the same stable job ID. Worker B should find the durable application record, skip the release, and complete the ledger row. If it calls the business mutation again, the test has found an idempotency gap rather than a queue problem. Each test should end with one business effect, a bounded next step, and a ledger row an operator can explain. Use a fake clock so delayed cases finish immediately in tests, then run a smaller integration suite against the real claim implementation.
In production, graph oldest eligible job age rather than queue depth alone. Depth can rise during a healthy burst; age tells you work is not progressing. Track claim expirations, completions by attempt number, dead-letter additions, and replay outcomes. Page on sustained age beyond the cleanup promise, not on every individual retry. A correlation ID should connect the scheduler, ledger transition, and marketplace mutation, while logs should avoid storing buyer details that are irrelevant to cleanup.
Then rehearse recovery. Pause new claims, inspect the failure class, fix the dependency or input policy, and replay a small cohort with the original idempotency keys before widening the batch. Keep the procedure in the repository next to the state transitions. If the team cannot explain who may replay dead letters and how a replay is stopped, the system isn't ready, regardless of transport.
For periodic marketplace cleanup, the practical default is a durable retry ledger plus a scheduled sweep. Move wake-ups to a delayed message queue when measurements show database contention, isolation needs, or tighter timing requirements. Neither choice earns trust by itself. Idempotent effects, bounded retries, visible dead letters, and a rehearsed replay path do.
Top comments (0)