Scheduled Data Cleanup API: Idempotent Media Files Beyond Cron Retries

Short answer: a scheduled data cleanup API for media files should let cron discover work, let a durable queue retry failed jobs, and let a state ledger control both deletion and the outbound webhook.

Cron must never delete accountable media files directly. The deciding artifact is evidence: object deletion and webhook delivery cross separate systems, so no queue can turn them into one atomic transaction, and every transition must remain explainable after either system has forgotten its request. For a scheduled data cleanup API, the defensible delivery promise is at-least-once execution with idempotent effects, not an unsupported claim of exactly-once delivery.

The design below treats every expired media object as a small state machine. It also retains evidence after the bytes are gone, because deletion without an audit record is operational amnesia.

How can cron, a queue, retries, and a DLQ clean media files?

It should separate eligibility, execution, and notification. The scheduled scan decides which files are eligible according to an already-recorded retention policy; it does not perform destructive I/O. A queue worker claims a stable command, checks the ledger, deletes the exact stored object version, records the outcome, and creates a webhook outbox entry. A different worker delivers that outbox entry.

Keep one durable identity from beginning to end. A practical command key can be derived from the tenant, media asset, object version, and retention-policy revision. The queue message carries that key rather than treating a delivery attempt ID as business identity. If a visibility timeout expires after deletion but before acknowledgement, the next worker sees the same command key and resumes from recorded state.

The invariants are small enough to review:

• One cleanup command names one immutable object version, never a mutable prefix.
• A command can move forward through states, but a retry cannot move it backward.
• The delete result is recorded before notification becomes eligible.
• One logical event has one stable event ID across every webhook attempt.
• A dead-letter entry preserves the command key, failure class, attempt count, and next operator action.

The hard boundary is between the storage service and the ledger. There is no shared transaction. If the process loses its lease after issuing the delete and before recording completion, the next attempt has to reconcile: check the ledger, inspect the exact version through the storage adapter, and converge on deleted. Don't infer success merely because a key lookup no longer returns live bytes; versioning, retention controls, and provider semantics belong behind that adapter.

This is also where a media pipeline differs from a generic cache sweeper. The expired object may have derivatives, legal-hold metadata, a playback record, and downstream subscribers waiting for a deletion notice. Eligibility must be frozen before execution, and the command must name every resource whose lifecycle is coupled. Otherwise a retry can remove a newly written thumbnail that happened to reuse an old path.

Short jobs are deceptive.

Begin with the deletion receipt

Treat the terminal ledger row as a deletion receipt. It should bind the immutable object version, retention-policy revision, eligibility time, command key, terminal state, and logical event ID. There are two independently retryable effects behind that receipt: deleting bytes and sending the outbound webhook. Combining them in one handler makes the common happy path shorter, but it leaves an auditor nowhere precise to stand. A timeout after the webhook request leaves the worker unable to distinguish a lost request from a processed response. Re-sending is valid under at-least-once delivery, yet the receiver needs the stable event ID to deduplicate it.

Retries happen.

Consider one concrete timeline. At 01:00, the scheduler selects version v17 of a media file and inserts command tenant-8:asset-42:v17:policy-3; two workers receive duplicate queue deliveries, but only one obtains the ledger lease. That worker deletes v17, then loses its lease before it can commit the deletion state. The second delivery cannot blindly issue every downstream effect again. It claims the same command key, asks the storage adapter about that exact version, records the converged deleted state, and inserts event media.deleted:tenant-8:asset-42:v17:policy-3 into the outbox in the same local transaction. Later, the webhook receiver accepts the event but its acknowledgement is lost, so the dispatcher sends it again. The receiver's uniqueness constraint recognizes the stable event ID and returns its normal acknowledgement without applying the business event twice. Three deliveries occurred across two boundaries, yet the durable record still describes one file version, one logical deletion, and one logical notification. This is the failure sequence the architecture must explain; “the queue retries it” is not an explanation.

Sign each webhook over the exact transmitted bytes, a timestamp, and the stable event ID with HMAC. RFC 2104 defines HMAC as keyed message authentication; it does not provide replay protection by itself. The receiver still needs a freshness window and a uniqueness check for the event ID. The sender should keep the signature inputs stable for a given payload, while generating a separate attempt ID for tracing.

The DLQ is not a slower retry queue. It is a quarantine boundary for commands that automatic policy can no longer advance: for example, an authorization denial needs a credential or policy change, while a malformed object locator needs data repair. Those classes should not consume an unlimited retry budget. Transient transport failures can use capped exponential backoff with jitter; permanent policy failures should move directly to review. The exact caps depend on the queue lease, retention objective, and downstream recovery target. I'm not sure there is a defensible universal number, and a load test plus the receiver's published limits are what would resolve it.

Ordering deserves skepticism too. Global ordering turns unrelated tenants into one failure domain. Require ordering only within the narrow key that can conflict, such as one media asset and its versions. Priority queues can help urgent legal deletion outrank routine expiration, but priority also changes scheduling behavior and operating cost; the referenced queue documentation describes that mechanism, not proof that every cleanup system needs it.

Choose the control plane by the evidence it preserves

Option	Delivery boundary	Recovery evidence	Operational trade-off	Suitable case
Cron deletes objects inline	Scheduler process	Logs and storage response	Few components, but a crash mixes discovery with destructive work and makes partial batches ambiguous	Small, reconstructable datasets where missed work can be found by the next full scan
Database job ledger plus workers	Durable row and claim lease	State transitions, command key, attempt records	Clear audit path; requires careful leasing, indexes, and contention control	Retention systems already anchored in a transactional database
Broker queue plus database outbox	Queue acknowledgement and ledger transaction	Queue metadata plus durable business state	Separates throughput from truth; adds reconciliation between broker and database	Media workloads with bursts, multiple workers, and webhook delivery
Workflow engine	Persisted workflow history	Step history and timers	Expressive recovery; more runtime concepts and governance	Long waits, approvals, or multi-day legal-hold workflows

The table is an evidence inventory, not a ranking. Start by writing the questions an operator must answer after the raw queue message, storage request log, and scheduler process are gone. Then choose the smallest control plane that preserves those answers. Here “smallest” means fewest ambiguous states the team must operate, not fewest boxes on a diagram. A database-backed queue can be simpler than a broker when the database already owns eligibility and throughput is moderate. A broker becomes useful when dispatch pressure, isolation, or worker fleets justify another consistency boundary.

For a Node.js API, public endpoints should expose commands and status rather than run cleanup inside the request. An application can accept a cleanup-run command, expose its status by run ID, and provide a separately authenticated replay command for reviewed DLQ items. Return an existing run for the same idempotency key, and never make an HTTP client wait for an S3 file sweep to finish. These are interface responsibilities, not an invitation to invent provider routes.

Cost follows retained evidence and scan shape more than syntax. Listing an entire bucket on every tick, retaining every attempt forever, and retrying permanent failures all create avoidable work. Partition eligibility by expiration time, keep compact terminal records for the audit period, and measure queue age, attempts by failure class, deletion lag, webhook acknowledgement lag, and DLQ depth. A low average hides the oldest object; alert on age distributions and explicit deadlines.

Encode the receipt in the worker boundary

The following code is deliberately an interface-level example. In a Node.js implementation, the same state transitions belong behind storage, ledger, queue, and webhook interfaces; changing the runtime must not change the delivery contract. The transaction named below covers only ledger state and outbox insertion. It cannot include the object store.

from dataclasses import dataclass
from enum import Enum
from typing import Protocol


class State(str, Enum):
    READY = "ready"
    DELETED = "deleted"
    NOTIFIED = "notified"
    DEAD = "dead"


@dataclass(frozen=True)
class CleanupCommand:
    command_key: str
    tenant_id: str
    asset_id: str
    bucket: str
    object_key: str
    object_version: str
    policy_revision: str


class ObjectStore(Protocol):
    def delete_version(self, bucket: str, key: str, version: str) -> None: ...


class Ledger(Protocol):
    def claim(self, command_key: str) -> CleanupCommand | None: ...
    def state(self, command_key: str) -> State: ...
    def record_deleted_and_enqueue_event(
        self, command_key: str, event_id: str, payload: bytes
    ) -> None: ...
    def release_for_retry(self, command_key: str, failure_class: str) -> None: ...
    def move_to_dlq(self, command_key: str, failure_class: str) -> None: ...


class RetryableFailure(Exception):
    pass


class PermanentFailure(Exception):
    pass


def process_cleanup(command_key: str, store: ObjectStore, ledger: Ledger) -> None:
    command = ledger.claim(command_key)
    if command is None:
        return

    if ledger.state(command_key) in {State.DELETED, State.NOTIFIED, State.DEAD}:
        return

    try:
        store.delete_version(
            command.bucket, command.object_key, command.object_version
        )
        event_id = f"media.deleted:{command.command_key}"
        payload = build_deleted_payload(command, event_id)
        ledger.record_deleted_and_enqueue_event(
            command.command_key, event_id, payload
        )
    except RetryableFailure as error:
        ledger.release_for_retry(command.command_key, type(error).__name__)
        raise
    except PermanentFailure as error:
        ledger.move_to_dlq(command.command_key, type(error).__name__)

record_deleted_and_enqueue_event must be one local transaction. That prevents a committed deletion record from losing its notification intent. The outbox dispatcher can then claim unsent events, calculate the HMAC signature, send the request, and record the acknowledgement. On an ambiguous timeout it sends the same event ID again. This is intentional. The receiver's deduplication record turns repeated transport delivery into one accepted business event.

The sample leaves payload construction to the media domain because its schema depends on the catalog. Define and version that schema before deployment, test canonical byte serialization for signature verification, and make consumers ignore additive fields they do not understand. Contract tests should submit the same event twice, reorder two events for the same asset, reject an invalid signature, and verify that an expired timestamp is not accepted.

Deployment needs one more guardrail — overlap. Run two scheduler instances in a test environment, pause one after it writes commands, and verify that the uniqueness constraint suppresses duplicate command keys. Then pause a worker immediately after the storage adapter returns and before the ledger transaction; recovery should converge without creating another logical event. Fault injection at each boundary is more informative than a green test that only exercises the straight line.

Reserve direct cron for disposable data

The rejected design is a cron callback that lists expired S3 files, deletes them, and posts webhooks in one loop. It is attractive because the control flow fits on one screen. The catch is that its checkpoint usually describes a batch position rather than the state of one immutable object version, so a process restart can repeat a wide slice of work while still leaving operators unable to prove which webhook corresponds to which deletion.

It is not suitable when deletion has compliance deadlines, multiple derivatives must move together, subscribers require signed events, or operators need selective DLQ replay. Use the ledger-and-outbox design in those cases.

Stick with direct cron when the data is disposable, a complete rescan is cheap, no external notification is required, and deletion is independently idempotent. Temporary render caches are a reasonable shape for it. Even then, cap each run, prevent overlapping scans, expose the oldest eligible-object age, and keep the retention predicate separate from the delete call so it can be tested without touching storage.

The queue design has limits as well. It adds leases, reconciliation, schema evolution, and a new class of stale-state alarms. A workflow engine is the better choice when cleanup pauses for human approval or legal review, while a database ledger is often easier than a broker for modest workloads that already depend on one transactional store. Your mileage may vary because operational familiarity is real: a theoretically elegant queue that nobody can inspect at 02:00 is the wrong queue.

The final acceptance test is blunt. Given any command key, an operator must be able to answer which immutable file version was selected, why it was eligible, whether deletion was confirmed, which event ID represents it, how many delivery attempts occurred, and whether the next action is automatic retry or human review. If the system cannot answer all six from durable state, its delivery guarantee is a slogan.

References

• https://www.rfc-editor.org/rfc/rfc2104
• https://www.rabbitmq.com/docs/priority