AlgernonCross4103

Posted on Oct 19

Scheduled Cleanup Jobs: 4 Boundaries for Retries, Dead Letters, Latency, and Cost

#queues #scheduling #backend

Short answer: trigger one idempotent sweep on a schedule, enqueue one small cleanup job per tenant or partition, and let a rate-limited worker own retries and dead-letter routing; keep cron out of the retry loop.

For a B2B SaaS cleanup pipeline, that split is the simplest setup once missed work has customer or compliance consequences. The scheduler decides when a sweep becomes eligible. The queue absorbs bursts and records what remains unfinished. Workers decide how fast the downstream dependency may be called. A dead-letter queue is an operator inbox, not another automatic retry tier.

The decision axis is latency versus cost. A permanently warm fleet can drain promptly but spends money while idle; a scaled-to-zero fleet trades some pickup delay for lower idle cost. That choice can change without changing the delivery contract below.

Should scheduled cleanup retries use a background job queue with dead letters?

The system has four ownership boundaries: schedule, admission, execution, and recovery. Mixing any two makes failure ambiguous. If a timer both finds expired records and deletes them, a process exit leaves no durable indication of which records were skipped. If a worker reschedules itself, duplicate timers can quietly multiply the offered load. If a dead-letter consumer automatically republishes every message, poison work becomes an expensive loop.

The schedule boundary emits a sweep identifier such as cleanup:2026-08-21T02:00Z. Admission expands that sweep into bounded partitions and records a stable job key. Execution leases one job, applies the current rate limit, and makes an idempotent state transition. Recovery stores exhausted work with enough context to classify it, while keeping credentials and sensitive record contents out of the payload.

Four invariants make that division testable:

Enqueuing the same (sweep_id, partition_id) twice creates one logical job.
A lease expiring may cause another delivery, so cleanup is safe to repeat.
A retry never runs before its recorded available_at time.
Exhausted jobs stop consuming worker capacity until an operator makes a new, auditable decision.

Exactly-once delivery isn't the target. A queue can redeliver after a worker completes the database change but loses its acknowledgement. The useful contract is at-least-once delivery around an idempotent effect. For deletion, that often means an atomic predicate such as “delete rows whose retention deadline passed and whose legal-hold flag is false,” plus a completion record keyed by the logical job ID. The legal-hold predicate belongs in the write transaction, not only in the earlier scan, because policy state can change while a job waits.

Payloads should carry identifiers, policy version, attempt count, and timestamps. Don't embed a user-controlled callback URL for the worker to fetch. OWASP's SSRF guidance recommends allowlisting trusted destinations when the application can identify them; a cleanup worker usually needs no arbitrary destination at all. This is also a deliverability lesson in another coat: when capacity is scarce, accepting an unconstrained destination turns one tenant's bad input into everyone else's delay.

Retention policy is executable governance

Start by separating retryable outcomes from terminal ones. A downstream 429 Too Many Requests is an explicit rate-limit signal defined by RFC 6585. When a valid Retry-After value is present, RFC 9110 defines it as either a delay in seconds or an HTTP date; honor it, then add local admission control so every worker doesn't wake at once. A timeout before the effect is known is also retryable, but it demands an idempotency check on the next attempt. Invalid policy, a missing tenant, or a record under legal hold should finish as a classified no-op or terminal result rather than burn the retry budget.

Keep two budgets. The attempt budget caps how many executions one logical job may consume. The age budget caps how long old cleanup may compete with new work. A concrete starting policy might allow five attempts over six hours, with exponential delay and jitter, but those are configuration examples rather than universal values. Set them from the retention objective, the dependency's recovery pattern, and the number of partitions the pool can drain before the next sweep.

Backpressure is a queueing problem before it is a retry problem. Let A be admitted jobs per second, W the number of workers, and R the allowed calls per worker per second. The pool's nominal service ceiling is W × R, before accounting for execution time and retries. If A stays above that ceiling, shorter backoff only increases contention. Reduce admission, add permitted capacity, or accept a longer drain time. Don't hide overload behind more retries.

Capacity first.

Consider a hypothetical 240-partition nightly sweep sharing a dependency that permits ten cleanup calls per second. Admission can create all 240 logical jobs immediately without pretending all 240 are executable. The limiter releases work at the permitted rate, while per-tenant concurrency prevents the first large account in key order from occupying every lease. If 24 partitions receive a retryable response, their delayed jobs remain visible and the other 216 can continue; the sweep doesn't sleep as one process behind the earliest failed partition. Now add the compliance edge case: a tenant receives a legal hold after discovery but before its partition runs. The worker reads the current policy inside the mutation boundary, classifies the partition as a no-op, and acknowledges it. It doesn't retry stale intent, and it doesn't delete based on a payload snapshot. Finally, imagine the dependency publishes a Retry-After value. Those 24 jobs become eligible at the recorded time, with jitter to avoid a synchronized wave, but they still pass through the same global and per-tenant gates. This example does not predict a drain time; execution duration, retry timing, database work, and worker startup delay are deliberately unspecified. It shows why schedule, admission, execution, and recovery need separate state even in a setup that starts with one worker process.

Fairness matters in multi-tenant SaaS. Partition work by tenant and cap concurrent leases per tenant, or a customer with millions of expired rows can occupy the entire pool. Use small deletion batches so a lease has a predictable duration, but don't claim a magic batch size: row width, indexes, lock behavior, and downstream quotas decide it. Your mileage may vary until a production-shaped load test captures those variables.

Dead-letter handling begins after the final allowed attempt, not after every terminal business decision. Store the logical job ID, tenant and partition identifiers, policy version, first-seen and last-seen timestamps, attempt count, and a bounded error category. Avoid raw response bodies; they can contain personal data or secrets. Alert on dead-letter age and growth rate, then require a reason and a new job ID for redrive. No blind loops.

Cost envelope across four queue mechanisms

The following options can all implement the contract. The useful comparison is where durability lives, how delayed work is represented, and which operational burden the team accepts. Product names are examples of mechanisms, not a ranking.

Mechanism	Concrete example	Useful boundary	Cost and latency trade-off
Database-backed leasing	PostgreSQL with `FOR UPDATE SKIP LOCKED`	Queue state and cleanup state can share transactions; the team owns polling, retention, and contention control	Reuses an existing database, but aggressive polling spends database capacity to reduce pickup delay
Redis-backed worker queue	BullMQ	Delayed jobs, worker concurrency, and retry policy sit in an application-oriented queue layer	Adds a service and memory-capacity planning; warm workers can provide low pickup latency
Message broker routing	RabbitMQ dead-letter exchanges	Dead-lettering is configured through exchange and queue policies	Strong routing flexibility adds broker operations and configuration review
Managed message queue	Amazon SQS with a dead-letter queue	Visibility timeout, receive count, and redrive policy are managed primitives	Usage-based operations reduce broker ownership; polling and worker startup choices still shape latency and cost

This table doesn't select a winner. A team already operating PostgreSQL may rationally accept polling code for a modest nightly workload. A team with a broker platform may prefer its established alarms and access controls. A bursty system with little broker expertise may value managed operations. Measure queue age, lease contention, duplicate-effect prevention, and total worker time under the same production-shaped dataset before choosing.

There is a catch with the “fewest services” answer: placing the queue in the application database is not suitable when cleanup polling or lock pressure threatens the transactional workload. Move the queue boundary when isolation is worth the extra service. Conversely, a dedicated broker is difficult to justify for a small workload if nobody can own upgrades, capacity, recovery drills, and authorization policy. Stick with the database mechanism while its measured contention stays inside the database budget.

Implementation model: lease, limit, commit, acknowledge

The following Python is a reference contract, not a client library. Its interfaces deliberately hide the queue implementation so a Node.js service can express the same state machine with its own driver. The important part is the ordering: lease durable work, acquire rate capacity, perform an idempotent cleanup, commit the outcome, then acknowledge the delivery.

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Protocol


class Result(Enum):
    DONE = "done"
    RETRY = "retry"
    TERMINAL = "terminal"


@dataclass(frozen=True)
class Job:
    job_id: str
    sweep_id: str
    tenant_id: str
    partition_id: str
    policy_version: str
    attempt: int
    first_seen_at: datetime


class Queue(Protocol):
    def lease(self, seconds: int) -> Job | None: ...
    def acknowledge(self, job_id: str) -> None: ...
    def retry_at(self, job_id: str, available_at: datetime) -> None: ...
    def dead_letter(self, job_id: str, category: str) -> None: ...


class CleanupStore(Protocol):
    def apply_if_eligible(self, job: Job) -> Result: ...


class RateLimiter(Protocol):
    def acquire(self, tenant_id: str) -> None: ...


def retry_delay(attempt: int) -> timedelta:
    # Example policy: bounded exponential delay; add queue-supported jitter.
    seconds = min(30 * (2 ** max(attempt - 1, 0)), 30 * 60)
    return timedelta(seconds=seconds)


def run_once(queue: Queue, store: CleanupStore, limiter: RateLimiter) -> bool:
    job = queue.lease(seconds=120)
    if job is None:
        return False

    limiter.acquire(job.tenant_id)
    result = store.apply_if_eligible(job)

    if result is Result.DONE:
        queue.acknowledge(job.job_id)
    elif result is Result.TERMINAL:
        queue.dead_letter(job.job_id, category="policy_or_input")
    elif job.attempt >= 5:
        queue.dead_letter(job.job_id, category="retry_budget_exhausted")
    else:
        now = datetime.now(timezone.utc)
        queue.retry_at(job.job_id, available_at=now + retry_delay(job.attempt))
    return True

Two details deserve more attention than the queue adapter. First, apply_if_eligible must combine the current retention rule, legal-hold check, mutation, and completion marker in the store's correct atomic boundary. Second, the limiter needs a global ceiling and a per-tenant ceiling. Worker concurrency alone isn't a rate limiter: ten workers completing variable-length jobs can produce a lumpy request rate, and synchronized retries make the spike worse.

Test the state machine with clock control. Cover duplicate enqueue, lease expiry immediately before acknowledgement, a 429 carrying Retry-After, timeout after the cleanup commit, policy changes between admission and execution, exhausted attempt and age budgets, and redrive under a new job ID. During deployment, stop taking new leases, allow bounded in-flight work to finish, and let remaining leases expire naturally. Observability should expose oldest-ready-job age, leased count, completion rate, retry rate by category, dead-letter age, and tenant fairness. Queue depth alone can't distinguish a healthy scheduled burst from a stalled pool.

Rollout and the rejected single-task design

The rejected design runs discovery and deletion inside one scheduled process. It has appealing properties: one deployment unit, no queue service, and an obvious manual command. It remains valid when cleanup is tiny, bounded, idempotent, comfortably finishes before the next run, and missing one invocation can wait until the next run without breaching a retention or customer objective.

It was rejected for this rate-limited worker pool because execution time is coupled to the sweep window. Retries extend that window, one large tenant delays the rest, and a process interruption leaves recovery dependent on another complete scan. GitHub Actions illustrates a broader scheduler limitation: its documentation says scheduled events can be delayed during periods of high load, some queued jobs may be dropped, scheduled workflows run from the latest commit on the default branch, and the shortest interval is five minutes. Those properties may be acceptable for a sweep trigger. They are a poor substitute for durable per-partition work state.

The final decision is intentionally narrow: use scheduling for eligibility, a durable queue for ownership, a limiter for admission to the dependency, and a dead-letter path for human recovery. Choose the underlying mechanism from measured pickup latency, idle and operational cost, isolation needs, and the team's ability to run it. The architecture should survive that mechanism changing.

DEV Community