Short answer: put the cleanup request on a durable background queue, acknowledge it only after an idempotent business effect is committed, and make retry and dead-letter handling part of the data model. A cron expression can start the work; it cannot prove that a health record was safely handled.
In a healthtech system, the periodic job might remove expired export bundles, compact audit indexes, or redact temporary files after a retention window. The dangerous assumption is that “the worker ran” means “the cleanup happened exactly once.” Queue delivery is normally at-least-once, and a process can disappear after deleting an object but before acknowledging its message. Design for that interruption instead of hoping it is rare.
Define the cleanup message contract
Begin with a failure drill, not a queue vendor. Kill a worker after it deletes a temporary export but before it acknowledges the message. Then deliver the same message to a second worker. The correct result is one business outcome, one durable decision record, and a visible redelivery—not a second deletion attempt that can surprise an auditor.
Make the drill concrete. Seed one export with a retention date that has passed, one with an active legal hold, and one whose policy version is unknown. Publish four deliveries: the eligible export once, that same export again, the held item, and the malformed policy item. Stop the process immediately after the storage adapter returns success on the first delivery. On restart, the duplicate should be classified from the claim and outcome rows, the held item should be acknowledged as retained, and the malformed item should appear in the dead-letter queue with no clinical payload copied into it. This test exercises the boundary that matters: a broker redelivery must not become a second business decision, while a bad message must remain visible to the team that owns policy changes.
At-least-once delivery is the practical baseline. Exactly-once effects require a transaction boundary spanning the database, object storage, and broker, which a normal queue does not provide. Write down the actual promise: “at-least-once messages, idempotent cleanup effects, and alerting for incomplete runs.”
The scheduler only requests evaluation. A policy service or database query decides whether an item is eligible, and that decision must be repeatable. A later retry should reach the same answer for the same policy version, even if the wall clock has advanced.
One line is enough to start a run. The rest is evidence.
How should a Node.js queue worker consume, ack, nack, retry, and dead-letter health jobs?
Treat the queue protocol as a state machine. A consumer receives a message, validates its schema, claims an idempotency key, applies the cleanup effect, records the outcome, and then acknowledges. If validation fails permanently, reject the message without requeue and route it to a dead-letter queue. If a dependency is temporarily unavailable or a rate limit is active, reject with requeue only when the broker's redelivery behavior is bounded; otherwise publish a delayed retry with an attempt number and a next-visible timestamp.
RabbitMQ documents this distinction for consumer acknowledgements: an acknowledgement confirms successful processing, while a negative acknowledgement can requeue or discard a delivery. The same principle applies to other brokers, although names differ. Never acknowledge before the database commit that makes the deletion idempotent.
The following Python sketch shows the contract behind a Node.js worker. The transport adapter can be replaced by the broker client used by your service; the business functions are deliberately generic.
from dataclasses import dataclass
from datetime import datetime, timezone
@dataclass(frozen=True)
class CleanupMessage:
run_id: str
item_id: str
policy_version: str
attempt: int
def consume(message, db, storage, clock=datetime.now):
item = CleanupMessage(**message)
if item.attempt > 5:
return "dead-letter: retry budget exhausted"
if not db.claim_once(item.run_id, item.item_id, item.policy_version):
return "ack: effect already claimed"
decision = db.retention_decision(item.item_id, item.policy_version)
if not decision.eligible:
db.record_outcome_once(item.run_id, item.item_id, "held")
return "ack: retained"
storage.delete_if_present(item.item_id)
db.record_outcome_once(
item.run_id,
item.item_id,
"deleted",
completed_at=clock(timezone.utc),
)
return "ack: committed"
The broker callback should map ack: ... to an acknowledgement, a transient dependency exception to a bounded retry, and a schema or policy violation to dead-letter handling. Keep that mapping in one adapter so application code does not accidentally acknowledge a message while an exception is still being classified.
I once treated a 429 response as a generic failure and immediately retried every item. That turned a provider limit into a traffic spike. Read the Retry-After guidance for HTTP 429, preserve the same idempotency key, and schedule the next attempt after the indicated delay when present. Your mileage may vary when a provider omits that header, so use a capped exponential delay and a queue-level rate limit rather than a tight loop.
What makes retries and idempotency safe for periodic cleanup?
Use a uniqueness constraint on (run_id, item_id, policy_version) or an equivalent durable key. An in-memory “seen” set disappears on deploy and cannot protect against a redelivery handled by another worker. Claiming and outcome recording should be atomic in the database; storage deletion is then an external side effect that must tolerate repetition through delete_if_present or a provider operation with the same semantics.
Do not hold one database transaction open while waiting on object storage. Commit the claim, perform the remote deletion, and record a terminal outcome with a compare-and-set transition. If the process stops between those steps, a reconciler can inspect claimed-but-incomplete rows and publish the same run and item again. That is a deliberate recovery path, not a hidden workaround.
Retry only failures that can plausibly change. A missing object is a successful terminal state if the decision record says it was eligible. A malformed message, unknown policy version, or identifier that fails validation should go straight to a dead-letter queue with a reason code. Dead letters are an investigation queue, not a trash can: retain the opaque payload, policy version, first-seen time, and failure classification while excluding sensitive health data.
Short paragraphs help here.
Which delivery guarantees and operational checks matter?
At-most-once delivery lowers duplicate work but can lose a cleanup request during a crash. At-least-once delivery makes recovery observable, at the price of duplicate deliveries that idempotency must absorb. Exactly-once effects require a transaction boundary spanning every external system, which a normal queue and object store do not provide. State the guarantee precisely: “at-least-once messages, idempotent cleanup effects, and alerting for incomplete runs.”
Test the failure edges, not just the happy path. Kill a worker after the storage call and before acknowledgement. Deliver the same message to two workers. Return HTTP 429 for several attempts, then allow success. Advance the retention clock across a daylight-saving transition. Confirm that one item has one terminal outcome and that the run can be replayed without deleting a newly protected item.
Monitor the latest completed business date, oldest claimed-but-incomplete item, retry age, dead-letter count, and the ratio of held to deleted records. A green process dashboard does not detect a scheduler that never published yesterday's run. Alert on business-date lag and on a dead-letter reason that crosses its normal baseline.
| Failure or constraint | Queue behavior | What to verify |
|---|---|---|
| Worker dies after the side effect | Redelivery | One terminal outcome for the item |
| Provider returns HTTP 429 | Delayed, bounded retry |
Retry-After is respected and attempts are capped |
| Invalid schema or policy version | No requeue; dead letter | Reason code is retained without clinical payload |
| Human approval is required | Pause before effect | Approval identity is part of the decision record |
There is a limit to this pattern. It is not suitable when a regulator requires a human approval for every deletion, or when the data is a disposable cache with no recovery value; a direct batch with a small audit table may be clearer. Stick with a workflow engine when each item needs long waits, approvals, or fan-out across many independent systems. The queue is a delivery mechanism, not a compliance policy.
The dominant cost is usually retained data and the operational evidence around it, not the scheduler call. Keeping every temporary export, raw delivery log, and retry payload forever increases storage, backup, and audit-search work. Quantify those categories for one representative month before changing queue infrastructure. If temporary exports are 80% of retained bytes, moving the trigger from a timer to a queue will not change the bill; shortening their policy-approved lifetime will.
That change has a cost. Deleting evidence too early can make an incident or a patient-access request impossible to explain. Keep a compact decision record: subject identifier, policy version, eligibility date, hold status, and the resulting action. Do not copy clinical payloads into the queue message or log stream. The message should carry an opaque cleanup-run ID and a bounded set of object IDs.
The scheduler only requests evaluation. A policy service or database query decides whether an item is eligible, and that decision must be repeatable. A later retry should reach the same answer for the same policy version, even if the wall clock has advanced.
Which rollout choice keeps deletion reversible?
Ship the schema, unique keys, and decision ledger first. Run several scheduled dates in report-only mode, publish messages, and record what would be deleted without touching storage. Have compliance review examples containing legal holds, unmatched exports, and duplicate deliveries.
Enable one low-risk artifact class next. Keep the raw audit trail and clinical payloads under their existing retention controls until lag alerts and replay tests are boring. A rollback is then simple: stop creating new deletion decisions and replay incomplete runs after correcting policy state. Existing terminal outcomes remain explainable.
The clock should be replaceable. The evidence should not be.
Top comments (0)