Short answer: to implement distributed data cleanup across multiple worker nodes, make the database decide which reservation is due, then let any worker process that claim. The scheduler should create opportunities to work, not own the work. For a customer-support system, the important rule is simple: a hold that has passed its deadline must become expired once, even if several worker nodes see it and one node disappears halfway through.
I would use a short polling loop, an atomic claim, an idempotent expiration command, and an explicit retry policy. I would measure claim age, completion age, duplicate attempts, and permanent failures before choosing a queue. Delivery semantics are the decision axis. A timer alone is not a reliability design.
How can you implement distributed data cleanup across multiple worker nodes?
Suppose a support agent places a callback slot, escalation slot, or inventory hold while a case is being handled. The hold lasts 15 minutes. After that window, the system should release it so another case can use it. That sounds like a timer. It is actually a distributed state transition.
The reservation has at least three relevant states: held, expired, and consumed. The cleanup worker may try to move held to expired; it must not expire a reservation that was consumed in the meantime. It also must not assume that one successful network response means the whole operation happened.
The practical invariant is:
A worker may attempt expiration more than once, but only an eligible
heldreservation can make the transition, and the transition is safe to repeat.
Race conditions are the point.
That is at-least-once work delivery paired with an idempotent state change. Exactly-once execution is a poor premise for a process that can be killed between a database commit and an acknowledgement. The state machine is where correctness belongs.
I start by writing the state transition before choosing a scheduler. That choice adds a little design work up front, but it prevents a queue's delivery behavior from quietly becoming the business rule.
Here is the smallest useful contract. claimExpired must be implemented as one database transaction or one atomic operation in the chosen store. It must check the deadline and status while changing the claim fields.
type ReservationStatus = "held" | "expired" | "consumed";
type Reservation = {
id: string;
status: ReservationStatus;
holdUntil: Date;
claimToken: string | null;
claimUntil: Date | null;
};
type ReservationStore = {
claimExpired(limit: number, now: Date, token: string, claimForMs: number): Promise<Reservation[]>;
expireIfHeld(id: string, now: Date): Promise<"expired" | "already-done" | "not-eligible">;
releaseClaim(id: string, token: string): Promise<void>;
};
A claim is a lease, not ownership forever. If a worker stops, another worker can reclaim the item after claimUntil. The lease should be longer than the normal operation, but short enough to recover from a dead node. There is no universal value. Record the duration you chose and revisit it from observed completion times.
The lease is the recovery boundary.
The smallest worker loop that works
The scheduler only wakes workers. Each worker asks for a bounded batch, processes it, and keeps going. A random delay prevents a fleet restart from sending every node into the store on the same millisecond.
type ExpirationResult = {
id: string;
outcome: "expired" | "already-done" | "not-eligible";
elapsedMs: number;
};
const sleep = (ms: number): Promise<void> =>
new Promise((resolve) => setTimeout(resolve, ms));
const jitter = (maxMs: number): number =>
Math.floor(Math.random() * maxMs);
export async function runExpirationWorker(
store: ReservationStore,
workerId: string,
signal: AbortSignal,
): Promise<void> {
const batchSize = 50;
const claimForMs = 60_000;
while (!signal.aborted) {
const now = new Date();
const token = `${workerId}:${now.getTime()}:${Math.random()}`;
const reservations = await store.claimExpired(batchSize, now, token, claimForMs);
if (reservations.length === 0) {
await sleep(1_000 + jitter(2_000));
continue;
}
for (const reservation of reservations) {
const started = Date.now();
try {
const outcome = await store.expireIfHeld(reservation.id, new Date());
const result: ExpirationResult = {
id: reservation.id,
outcome,
elapsedMs: Date.now() - started,
};
recordExpiration(result);
} catch (error) {
recordFailure(reservation.id, error);
} finally {
await store.releaseClaim(reservation.id, token);
}
}
}
}
function recordExpiration(result: ExpirationResult): void {
// Emit a metric and a structured event. Do not log customer message bodies.
void result;
}
function recordFailure(id: string, error: unknown): void {
// Include the reservation id and error class, with a bounded retry counter.
void id;
void error;
}
The final transition needs the same guard as the claim. In SQL terms, it looks like UPDATE reservations SET status = 'expired' WHERE id = ? AND status = 'held' AND hold_until <= ?. Check the affected-row count. One row means this worker performed the transition. Zero rows means another worker won, the reservation was consumed, or the deadline condition changed. All three are normal outcomes, not reasons to retry forever.
The code deliberately does not delete the row. A support operation needs an audit trail: which hold existed, when its deadline passed, and which state won. Cleanup of old history is a separate retention job with a different risk profile. Combining them makes incident review harder.
How should retries behave after a worker dies?
There are two failure points to design for. A worker can fail before the state transition, leaving a claim that will expire. It can also fail after the transition but before the delivery acknowledgement or lease release. The first case needs a retry. The second must not reverse or repeat the business effect.
This is why the expiration command checks current state instead of trusting a cached reservation. If the worker receives the same id twice, the second call returns already-done. If a customer-support agent consumed the reservation after a stale worker read it, the guarded update returns not-eligible. The worker records that result and moves on.
A queue with a visibility timeout follows the same general shape: delivery hides a message for a period, and a message can become visible again when the consumer does not finish in time. The timeout is a redelivery control, not proof that the business operation ran exactly once. Set it from measured processing time, and extend or divide work when a single operation can exceed it.
Do not retry every error with the same delay. A temporary database connection failure deserves bounded exponential backoff with jitter. A malformed payload should go to a dead-letter path after a small retry budget. A state conflict is usually a successful no-op from the cleanup job's perspective. Treating all three as identical creates either noisy retries or silent data loss.
type RetryDecision =
| { action: "retry"; delayMs: number }
| { action: "dead-letter"; reason: string }
| { action: "ack" };
export function decideRetry(
error: unknown,
attempt: number,
maxAttempts: number,
): RetryDecision {
if (isStateConflict(error)) return { action: "ack" };
if (!isTransient(error) || attempt >= maxAttempts) {
return { action: "dead-letter", reason: classify(error) };
}
const capped = Math.min(30_000, 500 * 2 ** attempt);
return { action: "retry", delayMs: capped + jitter(Math.floor(capped / 2)) };
}
function isStateConflict(error: unknown): boolean {
return error instanceof Error && error.name === "StateConflict";
}
function isTransient(error: unknown): boolean {
return error instanceof Error && ["TimeoutError", "ConnectionError"].includes(error.name);
}
function classify(error: unknown): string {
return error instanceof Error ? error.name : "UnknownError";
}
A dead-letter record needs an owner and an alert. It is not a trash can. For support data, include the reservation identifier, attempt count, first-seen time, last error class, and a replay command. Avoid copying private case text into logs or messages.
Which delivery model fits this cleanup job?
A database polling loop is often enough when the reservation table already owns the state and the cleanup volume is moderate. It has less glue: one query, one guarded update, and metrics. Its cost is contention and polling load, so index the eligibility query on status and deadline, and bound each claim batch.
A managed queue such as Amazon SQS exposes a visibility timeout. That is useful when producers and workers need to be decoupled, but the consumer still needs an idempotency key and a guarded state transition. The timeout controls when an unfinished message may be delivered again; it does not make the reservation update transactional with message acknowledgement.
RabbitMQ uses acknowledgements and can redeliver messages that are not acknowledged. That gives explicit consumer control, while the application still owns duplicate handling and durable state coordination. Kafka commits consumer offsets and is strong for an ordered, replayable event stream, but a cleanup task that only needs one due item at a time may gain operational machinery without gaining a stronger business guarantee.
Those are different tools, not a leaderboard. Pick based on the failure you need to absorb. If a queue already exists for support events, publishing a reservation-expired command may fit the existing operational model. If the source of truth is a relational table and the job is small, polling can be easier to test and operate. In either design, at-least-once delivery plus idempotent mutation is the honest baseline.
What I would change at scale
I would first separate discovery from execution when the due set becomes large. A partitioned due-time index lets workers claim disjoint ranges. A bounded batch avoids one slow customer record holding a lease for the whole backlog. A concurrency limit protects the database and the downstream notification path. I would measure before adding another moving part.
The dashboard needs more than a success count:
- oldest unclaimed eligible reservation
- claim-to-start and claim-to-finish latency
- duplicate attempt rate
- retry and dead-letter counts
- state-conflict count
- backlog size by deadline age
A useful alert is based on age, not just worker liveness. Ten healthy workers can still be too slow if the oldest eligible reservation is already outside the support policy. Also alert when the database clock and worker clock drift beyond the tolerance used by the deadline query. Use UTC timestamps and compare server-side time where possible.
Test the ugly sequences directly: two workers claim the same row, a process exits after the update, a lease expires during a long downstream call, the reservation is consumed at the deadline, and the queue acknowledgement fails after success. These tests are more valuable than a happy-path timer test. The expected result is a legal final state and a bounded retry, not a promise that a callback ran once.
The trade-off is intentional. A polling worker is not glamorous, but it makes the state transition visible. A queue may improve isolation and burst handling, but it adds acknowledgement, visibility, replay, and dead-letter decisions. Choose the smallest mechanism that can state and enforce your delivery guarantee.
Top comments (0)