DEV Community

JaggerBlack5781
JaggerBlack5781

Posted on

Pick Queue Versus Cron for Scheduled Marketplace Data Cleanup Recovery

A scheduled cleanup should start with cron when the work is a bounded scan, and move to a queue when each record needs independent retry, backoff, or ownership. For marketplace webhooks, I use that rule with one extra guard: the cleanup path must make a retry attempt safe to repeat before it can create another delivery.

The practical split is straightforward. Cron decides when to look for stale webhook attempts. A queue decides how individual retry jobs are claimed, delayed, retried, and observed. A small marketplace can use both. Making cron carry per-record recovery logic is where the design usually gets brittle.

That distinction saves a lot of guessing.

How do I pick queue versus cron for scheduled data cleanup?

The word cleanup hides two different jobs. One job finds records that should no longer be active: expired carts, abandoned listings, old webhook attempts, or temporary locks. The other job performs work that may fail and needs another chance. Those jobs have different failure surfaces.

A cron invocation is a time trigger. The cron model described in the Unix tradition is a recurring command schedule, expressed as time fields rather than as a durable record for every future unit of work. That makes it a good fit for a sweep such as “find attempts whose next retry time has passed.” It does not, by itself, tell you whether attempt 481 was claimed by one worker, whether it failed halfway through, or whether it needs a delay before the next try.

A queue is a work handoff. It gives each cleanup or retry item a place to wait and a state transition that can be monitored. That extra machinery has a cost in operations and in code. It earns its keep when records have different retry histories or when one slow destination must not hold up unrelated marketplace events. The limitation is just as important: a queue adds another stateful system to deploy, observe, and recover when its own handoff fails.

I write the decision down in a small matrix before adding infrastructure:

Question Cron-led sweep Queue-led work
Is the job one bounded batch? Strong fit Usually unnecessary
Does each record need its own retry delay? Awkward Strong fit
Can the work be safely repeated? Required Still required
Must one slow item be isolated? Weak unless partitioned Strong fit
Is delayed recovery the main requirement? Needs a second mechanism Native design concern
What happens when a run overlaps? You must prevent it The claim protocol helps, but does not remove idempotency work

The recommendation is to keep the scheduler responsible for discovery and let a queue own item-level recovery. If a scheduled cleanup only deletes or marks a bounded set of records and can finish within its interval, cron alone is often the cleaner choice. If it sends outbound work, treats records differently, or needs independent retries, enqueue one durable job per unit of work.

Cron is not a fit when the real requirement is per-record recovery.

That is a revenue-per-hour decision. A one-person SaaS cannot spend a week operating a distributed workflow for a ten-minute nightly sweep. It also cannot quietly lose marketplace deliveries because a single destination returned an error. I ship weekly, so I outsource the undifferentiated part to well-understood primitives and keep the business rule in my code.

Why do duplicate webhook deliveries change the choice?

Outbound webhooks are naturally at-least-once work in many practical systems. A sender may finish the network request while losing the response, or a worker may crash after the receiver has accepted the message. Retrying can then produce a second delivery. A queue does not make that duplicate impossible. Cron does not either.

The receiver needs an idempotency boundary. Give every logical event a stable event identifier, persist the receiver's processed identifiers, and make the side effect conditional on that record. The exact storage mechanism depends on the application, but the invariant is stable: the same event can be presented more than once without applying its business effect more than once.

For a marketplace, the cleanup record should distinguish at least the event identity, delivery state, next eligible time, and attempt count. A retry worker claims one record, sends the event, and records the outcome. If the worker loses its lease after the receiver accepted the event, the next worker may send it again. That is acceptable only if the receiver treats the event identifier as a deduplication key.

The cleanup sweep still matters. It finds attempts whose lease expired, whose next retry time has arrived, or whose retention window has ended. It should not blindly replay every record it sees. A query that means “eligible and not currently claimed” is part of the recovery protocol, not a convenience filter.

The claim is a contract.

One trap is deleting old rows before proving that no retry can refer to them. Another is marking a row complete before the outbound request has a defined result. I prefer explicit terminal states and a retention policy that is longer than the period in which a delayed worker can return. The exact duration is an application policy, not a universal cron or queue setting.

How should a small implementation divide the work?

Start with a scheduled discovery pass. It selects a bounded page of due or expired records, writes a job for each selected record, and advances a claim field in the same transaction as the selection where the database supports that pattern. The queue consumer then handles one event. The next scheduled pass can continue discovery without replaying records already claimed.

Here is deliberately generic TypeScript. The interfaces are the important part: discovery, claim, delivery, and outcome recording remain separate so the recovery policy can be tested without a particular queue product.

type DeliveryState = "pending" | "claimed" | "delivered" | "dead";

interface WebhookAttempt {
  id: string;
  eventId: string;
  nextEligibleAt: Date;
  state: DeliveryState;
  attempts: number;
}

interface RetryStore {
  claimDue(limit: number, now: Date): Promise<WebhookAttempt[]>;
  recordOutcome(
    attemptId: string,
    outcome: "delivered" | "retry" | "dead",
    nextEligibleAt?: Date,
  ): Promise<void>;
}

interface WorkQueue {
  enqueue(attemptId: string): Promise<void>;
}

async function scheduleWebhookCleanup(
  store: RetryStore,
  queue: WorkQueue,
  now = new Date(),
): Promise<number> {
  const due = await store.claimDue(100, now);

  for (const attempt of due) {
    await queue.enqueue(attempt.id);
  }

  return due.length;
}
Enter fullscreen mode Exit fullscreen mode

The number 100 is a page-size example, not a recommended production limit. Pick a bound that protects the database and keeps one run predictable. If enqueueing fails after claiming, the claim must expire or be repaired; otherwise the record can become invisible. That is why a claim lease and a recovery sweep belong in the design even when the queue is reliable.

The worker should be boring. Load the attempt by identifier, verify that it is still eligible, send the event with its stable event identifier, and record one of three outcomes: delivered, retry later, or dead. Do not bury retry timing in the cron expression. Time-based discovery and per-item retry policy are separate concerns.

Keep it boring.

The awkward case is worth spelling out because it is where a tidy diagram tends to lie: a cleanup run claims an attempt, publishes its job, the worker sends the webhook, the receiver accepts it, and the worker process dies before recording success. The next sweep sees an expired lease and schedules the same event again. If the receiver has stored the event identifier before applying its side effect, the second delivery becomes a harmless observation; if it has not, the marketplace may create the same business effect twice. No queue setting can repair that missing boundary. The scheduler, queue, and receiver each have a different job, and the recovery design only works when the handoff between them is explicit.

The trade-off is operational complexity for operational recovery.

Which failure modes deserve a queue?

A queue becomes the better center of gravity when the cleanup has uneven work. One destination may respond quickly while another times out. One event may be on attempt 1 while another is on attempt 6. One merchant may have a temporary outage while the rest of the marketplace is healthy. Independent jobs let the system isolate those cases and expose them as data.

It is also a better fit when recovery must continue after a process restart. A queue can hold uncompleted jobs while consumers come and go. That does not eliminate the need for leases, visibility timeouts, poison-message handling, or a dead-letter policy. It does make those decisions explicit instead of hiding them in a long-running cron process.

Cron remains the runner-up when the workload is a compact maintenance pass: remove records past a retention boundary, compact a small table, or mark expired sessions. In those cases, a queue can add more moving pieces than the job needs. Use a lock or a database coordination primitive to stop overlapping runs, keep the query bounded, and emit the count of examined, changed, and failed records.

The difficult middle case is a large sweep that does not need delivery semantics. Split it into deterministic pages, run pages on a schedule, and make each page repeatable. Adding a queue just because the table is large is not enough; add one when the work units need independent lifecycle and recovery. A queue is the wrong fit if it only hides an unbounded database query; fix the page boundary first.

What should I measure before changing the architecture?

Measure recovery, not only throughput. I want the age of the oldest eligible attempt, the number of claimed items with expired leases, delivery outcomes by destination, retry counts, and the size and age of dead work. For a cleanup-only job, I also want the duration of each run and the number of rows examined versus changed.

Alert on a recovery invariant. For example, an eligible record that remains unclaimed beyond its service objective is more useful than a generic “cron succeeded” signal. A cron process can exit successfully after selecting zero rows while the query is wrong. A queue can report healthy consumers while jobs are repeatedly failing.

Testing should exercise the awkward boundary cases: two cleanup runs overlap, enqueueing stops after a claim, the worker crashes after delivery, the receiver returns a timeout after accepting the event, and a retry becomes due while a lease is still valid. These are not exotic cases for outbound webhooks. They are the normal reasons to make the state machine visible.

I would not buy or build a second system until these tests and measurements show a real gap. The weekly shipping constraint matters. A small, explicit state table plus a scheduled sweep can carry a surprising amount of workload. When item-level recovery becomes the product requirement, introduce the queue around that boundary and keep the scheduler narrow.

The final rule is simple: use cron to discover time-based work; use a queue to recover independent work. For marketplace webhook cleanup, choose the queue when duplicate-safe retries and isolation are central. Choose cron alone when the job is a bounded, repeatable maintenance pass. In both cases, idempotency is the part that protects the business.

References

Top comments (0)