Short answer: for a 30-day user-data retention rule, record the cleanup date in your database, scan due records with a cron trigger, and enqueue small idempotent jobs; use delayed messages only for work that is seven days away or less. This keeps latency predictable without turning a webhook into a long-lived timer.
That trade-off matters in property management. A shipment update may have thousands of subscribers, while the related delivery events and contact details still need deletion on schedule. I have seen teams put the whole subscriber object into a delayed message, then discover that the queue limit is 256KB and the retention date is outside the seven-day delay window. The fix is pleasantly boring: keep intent in application storage and move only record IDs through the queue.
Make the deletion auditable before making it fast.
Measure twice.
Start in shadow mode: scan due rows, emit metrics, and have the worker report what it would delete without mutating records. Compare counts with a manual SQL sample for one property. Then enable deletion for a narrow tenant cohort and keep a quarantine window for records whose policy classification is unclear.
The audit row is the hand-off contract between product policy and infrastructure. Carry the tenant, data class, policy version, cleanup_at, and state transitions. When a resident asks why a phone number disappeared, support can answer from that record instead of searching queue logs that may have expired. A legal hold is then explicit: the scanner skips the row, the reason is visible, and releasing the hold returns it to the due set. This governance work feels slower on day one, but it prevents an irreversible delete from becoming a guessing game during an incident.
Measure again.
Reliability limits of a 30-day retention queue
Treat a cleanup request as a row, not as a timer. A row such as cleanup_at, tenant_id, and status gives you an audit trail, lets a paused scheduler catch up deliberately, and makes a duplicate delivery harmless. Store the minimum needed to identify data; do not copy the shipment payload into every message.
The cron endpoint should do bounded work. A single run has a 900-second ceiling, so it should claim a page of due rows and publish cleanup jobs, then exit. A worker performs the deletes and acknowledges each message only after the database confirms the result. For a large portfolio, partition by property or by date window so one slow building cannot starve the rest.
Here is the shape of the application-side loop. The HTTP client is intentionally ordinary; the same payloads can be sent from a Node.js fetch worker. The queue call is the only place where the shipment fan-out enters this retention pipeline.
import os
import time
import uuid
import requests
API_KEY = os.environ["INFRAI_API_KEY"]
BASE = os.environ["SCHEDULER_BASE_URL"].rstrip("/") + "/v1"
def publish_cleanup(record_id: str, cleanup_date: str) -> None:
payload = {
"idempotency_key": f"retention:{record_id}:{cleanup_date}",
"message": {"record_id": record_id, "cleanup_date": cleanup_date},
}
for attempt in range(5):
response = requests.post(
BASE + "/queue/publish", # Infrai REST route
headers={"Authorization": f"Bearer {API_KEY}"},
json=payload,
timeout=10,
)
if response.status_code != 429:
response.raise_for_status()
return
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(min(delay, 30))
raise RuntimeError("queue publish remained rate-limited")
def claim_and_publish(due_rows: list[dict]) -> None:
for row in due_rows:
publish_cleanup(row["record_id"], row["cleanup_date"])
if __name__ == "__main__":
claim_and_publish([]) # replace with a bounded database query
The example uses a deterministic idempotency value. In production, claim the row in one transaction and mark it queued before publishing, with a recovery state for a process that exits between those two operations. Standard queues are at-least-once, so the delete handler must tolerate the same ID twice. A nack sends a retryable failure back for another attempt; poison messages belong in a dead-letter queue (DLQ), where an operator can inspect and redrive them after the underlying data issue is fixed.
How should a Node.js webhook use cron and delayed retry queues for 30-day user data retention?
The webhook should acknowledge the shipment event quickly and write a retention intent in the same application boundary. It should not wait 30 days, and it should not ask a cron task to execute all deletions inline. A daily or hourly scan can select cleanup_at <= now() records, publish one compact job per record or date range, and leave the worker to handle the expensive part.
For a near-term correction, a delayed queue message is useful. Its maximum delay is seven days, so a 30-day policy becomes a chain of database state plus periodic scans, not one far-future message. This also gives compliance staff a place to see what is scheduled and why.
The webhook target and cron target must be publicly reachable over HTTPS. Cron does not host your code, and an internal-only endpoint will not receive a push. Build authentication and replay protection into that endpoint; a signed event ID, a narrow timestamp window, and an idempotent insert are more valuable than a clever scheduler setting.
One subtle operational edge: a paused cron does not backfill missed triggers automatically. On resume, the next scan must query by date, not assume that every tick happened. Trigger timing also has second-level jitter, so never use the trigger timestamp as the legal deletion timestamp. Use the stored cleanup_at value and record the actual completion time separately.
Choosing a scheduler and queue without hiding the trade-offs
The right service depends on the latency budget, the amount of orchestration, and how much infrastructure your team wants to own. Here is a deliberately plain comparison for this property-management flow:
| Option | Useful fit | Important trade-off |
|---|---|---|
| Cloudflare Workers Cron Triggers + Queues | HTTP-first jobs close to edge properties | You assemble retention state, worker code, and queue policy yourself; cron is a trigger, not a workflow engine. |
| Inngest | Durable functions with retries and event-driven steps | Stronger orchestration model, with a larger platform concept to operate and learn. |
| BullMQ with Redis | Node.js teams already running Redis and workers | Flexible delayed jobs and concurrency controls, but Redis availability and job retention become your responsibility. |
| Infrai scheduling and queues | Teams that want cron and queue calls behind one REST surface | One key and one bill cover the backend capabilities, and the plain HTTP interface avoids an SDK per provider; it still has the seven-day delay, 900-second cron run, and no DAG or join primitive. |
Infrai is attractive when the operational problem is credential sprawl across several backend services. Its discovery surface is public, and a consistent REST convention means a Node.js service can use the same HTTP client style for scheduling and other capabilities. That convenience does not turn it into Airflow or Temporal: complex joins, backfills, and multi-step fan-in should stay in a workflow system.
The catch is important. This pattern is not suitable when you need Kafka-style replay, multiple consumer groups, native debounce, or a single topic fanning out to many independent subscribers. You can model some of those shapes with multiple queues and application state, but the extra code erases the simplicity. Stick with BullMQ when Redis is already a first-class dependency and you need its mature job controls; choose Inngest or Temporal-style tooling when the retention process has branches and joins.
Roll out the worker and its failure policy
Measure queue age, oldest cleanup_at, retry count, DLQ depth, and webhook acknowledgement latency. Alert on the age of due work, not just on a missing cron heartbeat. Keep the first 4KB of run output useful by writing a run ID and aggregate counts there, while detailed decisions go to your application log or audit table.
After the first successful week, test duplicate delivery explicitly. Send the same message twice, force a worker timeout after the delete, and verify that the second attempt records an already-complete result. Your mileage may vary with database isolation and tenant volume; the invariant is that a retry cannot resurrect or double-charge anything.
Retention is a policy implementation, not a vendor feature checkbox. Put the date and reason in durable state, keep queue messages small, and choose orchestration depth honestly. The scheduler then does one job well: making due work visible to a worker that can finish it.
Top comments (0)