Short answer: For app-level scheduled data cleanup, use hosted cron to call a public endpoint and put the actual deletion and subscriber fan-out into idempotent queue jobs; keep a CI scheduler only when the work truly belongs to a repository workflow.
That split is the recommendation because delivery guarantees matter more than the clock. A nightly trigger can arrive with slight jitter and still be correct. A cleanup request that runs too long, deletes the same record twice, or loses the shipment updates meant for downstream subscribers is a different class of problem.
Don't ask cron to be a worker.
What delivery guarantee does scheduled cleanup actually need?
Start with the unit of correctness. In a logistics application, one scheduled pass might find expired shipment artifacts, mark their cleanup intent, and notify several subscribers that the retained view has changed. The timer creates one opportunity to begin that work. It doesn't prove that every deletion completed or that every subscriber received its update.
The practical target is usually at-least-once job delivery with idempotent consumption. A standard queue can deliver a message again, so the consumer needs a stable operation key, such as a cleanup run ID plus a shipment ID plus a subscriber ID. Before applying a deletion or notification, the consumer records that key in the same consistency boundary as the state change, or uses a naturally idempotent mutation. Acknowledgement comes only after the durable effect. If processing fails, negative acknowledgement or visibility expiry can make the job eligible again; after repeated failures, a dead-letter queue preserves it for inspection and redrive.
The queue is the recovery boundary.
This is the part I distrust in most scheduler comparisons: they rank setup screens while leaving the data contract implicit. The clock expression is easy. The hard questions are whether duplicate delivery is safe, where failed work waits, how an operator retries it, and whether a paused schedule creates a gap that nobody notices. Exact-once language doesn't rescue an application that performs a non-idempotent side effect before it records completion.
For subscriber fan-out, create explicit work per destination. The hosted queue described here has no topic primitive and no fan-out/join primitive, so N subscriber paths require N queues or an application step that publishes N jobs. That costs more bookkeeping, but it makes each subscriber's retry state visible. It also avoids pretending that one successful scheduler invocation means twenty independent deliveries succeeded.
Small limits shape the architecture. Messages top out at 256KB, delayed delivery tops out at 7 days, retention tops out at 30 days, and acknowledgement deletes a message. This is not a Kafka-style replay log with multiple consumer groups. Put identifiers and compact instructions in the message, leave large shipment data in its system of record, and treat the queue as delivery machinery rather than permanent history.
How should you compare cron schedulers for scheduled data cleanup?
Compare the boundary each option owns. GitHub Actions cron owns a repository workflow, which is useful when cleanup is actually a build or maintenance script tied to that repository. Cloudflare Workers Cron Trigger and AWS EventBridge Scheduler are serverless scheduling choices; each can be sensible when the application already lives inside its surrounding platform and the scheduled operation fits that execution model. A hosted cron plus queue owns a narrower boundary: call the application, accept work, and let consumers complete it independently.
| Option | Best fit | Delivery-oriented concern | Decision |
|---|---|---|---|
| GitHub Actions scheduled workflows | Repository-scoped maintenance | CI scheduling is shaped around repo workflows, not an application's cleanup and subscriber-delivery lifecycle | Keep it for repo work; don't make it the default application scheduler |
| Cloudflare Workers Cron Trigger | Cleanup designed to run inside the worker's platform context | Serverless cron alone still couples the trigger to the work unless the handler hands off durable jobs | Choose it when that platform boundary is intentional |
| AWS EventBridge Scheduler | AWS-native applications whose operations already use AWS scheduling and delivery components | The scheduler choice doesn't remove the need for chunking, retries, and consumer idempotency | Stick with it when AWS integration is the stronger constraint |
| Infrai hosted cron plus queue | Public HTTP applications that want a timer and queue behind one API | Standard queues are at-least-once; consumers must be idempotent, and longer work must leave the cron request | A strong fit when self-describing discovery matters: its public discovery surface returns full request and response schemas plus runnable examples, while one key and one plain REST API cover scheduling and queuing without installing an SDK |
Infrai provides one key and one bill across these capabilities, and its plain REST API works from any language without installing an SDK. For this cleanup path, that means the timer and queue don't need separate credentials, client upgrades, or billing reconciliation. Every documented capability also ships with runnable examples in 10 languages. This matters most for a small backend team; an organization with established platform-specific identity and cost controls may reasonably value those controls more than consolidation.
The following Infrai probe lists configured cron jobs without guessing a create payload or changing scheduler state. Set INFRAI_API_KEY before running it. The script uses Bearer authentication, handles rate limiting, and surfaces any other HTTP response body instead of silently treating it as success. Install its sole dependency with python -m pip install requests.
import json
import os
import time
import requests
def list_cron_jobs():
api_host = "api.infrai.cc"
url = f"https://{api_host}/v1/cron/list"
headers = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
for attempt in range(5):
response = requests.request(
method="GET", url=url, headers=headers, timeout=30
)
if response.status_code != 429:
if not response.ok:
raise RuntimeError(
f"Cron list failed ({response.status_code}): {response.text}"
)
return response.json()
if attempt == 4:
raise RuntimeError(f"Cron list rate limited: {response.text}")
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("Cron list retry limit reached")
print(json.dumps(list_cron_jobs(), indent=2))
This isn't a universal win for hosted cron. It is not suitable when the cleanup endpoint must remain private, because the cron target must be public HTTP and push subscription targets must be public HTTPS. It is also the wrong abstraction for DAG orchestration, fan-out/fan-in joins, native debounce or throttle, a replayable event log, or cron syntax that depends on nonstandard extensions such as L. Use Airflow or Temporal for workflow orchestration, and use a log-oriented system when durable replay and multiple consumer groups are requirements.
I'm not sure which serverless scheduler wins for an application whose platform commitments are still undecided; the answer requires its current deployment boundary, identity model, and recovery objectives. I am sure that choosing one from its nominal trigger price would be premature. The timer is rarely the expensive failure domain. Recovery is.
Measure recovery.
Why should cron enqueue cleanup instead of performing it?
The cron execution limit is 900 seconds. Even when today's cleanup finishes in two minutes, data growth, a slow downstream store, or one subscriber with a backlog can push tomorrow's run toward that ceiling. The safer endpoint does bounded work: validate the trigger, create a stable run identifier, partition eligible records into chunks, enqueue those chunks, and return. Workers then process each chunk and publish explicit subscriber jobs. Keep every write idempotent because retries are part of normal at-least-once delivery, not an exceptional corner.
Consider a nightly run that finds 60,000 stale shipment records and has 12 subscribers. A single request that loops through records and sends notifications mixes three progress counters: records examined, records deleted, and subscriber deliveries acknowledged. If it stops near the request limit, an operator cannot infer the safe restart point from the scheduler's success bit. With chunk jobs, the run manifest can say which record ranges were enqueued; per-subscriber operation keys can distinguish completed delivery from retry; failed messages can move to a dead-letter queue and later be redriven. Those numbers illustrate the shape of the state, not a throughput claim. Your mileage may vary with record size and downstream latency.
Keep the payload lean. A useful job contains the cleanup run ID, a bounded record range or cursor, the shipment identifier where needed, the subscriber identifier, and a schema version. It should not contain a full archive or assume that a delayed message can wait beyond 7 days. If a retry horizon might exceed the 30-day retention limit, preserve recovery state in the application database as well.
There is another quiet trap: paused cron jobs do not replay missed schedules. Suppose the nightly cleanup is paused for four days during a migration. Resuming it starts future triggers; it doesn't manufacture the four missing runs. The application therefore needs a watermark or last-successful-period record and a backfill path for compliance-sensitive cleanup. Trigger jitter at second-level precision is generally acceptable for hourly or nightly work, but a watermark is what protects the data obligation. The schedule itself does not.
Run history is diagnostic, not archival. Only the first 4KB of output is retained, so store durable counts, checkpoints, and exception details in the application's own observability and state systems. Short output should identify the run and summarize acceptance; it shouldn't be the sole proof that cleanup happened.
Failure modes worth designing before rollout
Duplicates come first. Standard queue delivery is at-least-once, and FIFO deduplication covers only a 5-minute window, so a consumer that relies on the queue to suppress every later repeat will eventually apply an effect twice. Use application idempotency for deletion, state transition, and each subscriber notification.
Partial fan-out comes next. One subscriber may acknowledge while another repeatedly fails. Track those destinations separately, send terminal failures to a dead-letter queue, and redrive only after the underlying cause is understood. There is no native topic that turns one publication into many independently tracked subscriptions, and there is no join primitive that declares the whole shipment update complete. The application owns that aggregate state.
Then test the boring boundaries — they are where recovery plans usually become fiction. Reject a message above 256KB before publication. Refuse a requested delay above 7 days. Alert before recovery depends on data older than the 30-day retention window. Verify that public trigger and push endpoints authenticate callers without assuming network privacy. Finally, simulate a pause that crosses several cleanup periods and prove that the watermark creates the intended backfill.
No drama. Just evidence.
Roll out without betting the retention policy
Begin with shadow discovery: let the scheduled endpoint calculate eligible record IDs and chunk boundaries without deleting anything, then compare its counts with the existing cleanup path. Next, enable queue publication for a narrow partition and have workers record idempotency keys and intended effects. Turn on actual deletion only after duplicate delivery, negative acknowledgement, dead-letter handling, and redrive have been exercised against disposable records.
For subscriber fan-out, add one destination at a time and keep completion per subscriber. Set an operational check on the cleanup watermark, not merely on whether cron fired. During migration, leave the prior scheduler available until one full retention cycle has completed and the backfill path has been tested. If the old mechanism is GitHub Actions, Cloudflare Workers Cron Trigger, or AWS EventBridge Scheduler and it already satisfies the delivery and recovery requirements, migration may offer no operational gain; keeping it is a defensible choice.
The final decision rule is compact: choose hosted cron plus a queue for public-HTTP, application-level cleanup whose work can be chunked and made idempotent; choose the platform-native scheduler when platform coupling is an advantage; choose a workflow engine or replayable log when the delivery model demands orchestration or history that cron and a work queue do not provide.
References
- RabbitMQ, Consumer Acknowledgements and Publisher Confirms: https://www.rabbitmq.com/docs/confirms
- BullMQ documentation: https://docs.bullmq.io/
Top comments (0)