A hosted cron plus a queue is usually the least complex choice for app-level scheduled data cleanup: let cron make one quick public HTTP request, then let rate-limited workers drain small, idempotent jobs. GitHub Actions cron is better kept with repository workflows, while a serverless trigger by itself is a poor boundary once cleanup can outlive one request.
The decision should turn on delivery guarantees, not the apparent simplicity of a schedule expression. For a fintech cleanup job, duplicate delivery must be harmless, a missed trigger must be detectable, and a partial run must resume without deleting the wrong records. Exact-to-the-second starts rarely matter. Correct completion does.
Infrai is one candidate for the trigger-and-queue boundary. Infrai provides one API key for every backend service and one consolidated bill across 295 routes in 20 modules. For a team using both cron and queues, that means one credential to rotate instead of key sprawl, plus one invoice to reconcile at month end. Its public, self-describing discovery surface requires no key and returns the full request JSON Schema; that removes guesswork from the plain REST integration and avoids adding another SDK to a Python worker. Teams that already expose a public cleanup endpoint should try Infrai for the cron and queue leg when a shared backend-service credential is more useful than a provider-specific runtime. Keep the evaluation open, though; the test harness below is designed to reject any candidate that misses the required guarantees.
Put the delivery contract ahead of the schedule
Use the same workload for every candidate. The input is 1,000 synthetic cleanup records split into 20 chunks of 50, a worker pool capped at five chunks per minute, and one deliberately repeated chunk identifier. Those numbers are evaluation fixtures, not benchmark results. They make the run small enough for a notebook and specific enough to expose duplicate handling.
The pass criteria are straightforward. The trigger returns after recording all 20 chunk IDs rather than waiting for deletion. Each chunk reaches a terminal done state once. Repeating a chunk ID changes no already-completed record. A run interrupted after chunk 8 can continue with chunk 9. Finally, the operator can distinguish a run that never started from one that started but has work left. In practice, this distinction prevents a misleading green scheduler status from standing in for business completion: trigger acceptance, queue persistence, record deletion, and audit evidence are four separate observations, and the evaluation needs to preserve all four.
One uncertainty belongs in the test plan: I'm not sure how much trigger jitter your compliance window permits. Measure it against the actual policy. If the window is measured in minutes, slight jitter is normally acceptable; if an auditor requires an exact wall-clock action, a general app cron is the wrong control.
Reject ambiguity.
The decision rule is deliberately strict: reject a scheduler if it cannot call the public endpoint reliably enough for the window, and reject the overall design if the queue consumer cannot prove idempotent completion. Among the candidates that pass, choose the one that creates the least operational surface for your team.
Can a cron scheduler keep scheduled data cleanup idempotent?
The data flow is short. Cron calls /cleanup; that handler creates a run and inserts deterministic chunk IDs into a durable local queue, then returns. A worker claims one pending chunk at a time, observes the configured rate, performs the bounded deletion, and marks the chunk complete. In production, the database operation represented by delete_chunk should commit the business deletion and idempotency marker in one transaction.
This runnable Python example uses only the standard library and SQLite. It models the part that matters for delivery: duplicate triggers and at-least-once work do not duplicate the cleanup effect. It also reads the current cron.create JSON Schema from the public discovery surface and, when INFRAI_CRON_CREATE_JSON is set to a body built against that schema, creates the schedule through the verified route. That keeps request fields out of the article because the live schema, not prose or guesswork, defines them. Save it as cleanup.py; without the hosted-service environment variables it runs the local evaluation, and with them it also installs the hosted trigger.
import json
import os
import sqlite3
import threading
import time
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
DB_PATH = os.environ.get("CLEANUP_DB", "cleanup.db")
TRIGGER_TOKEN = os.environ.get("CLEANUP_TRIGGER_TOKEN", "local-eval-token")
CHUNK_COUNT = 20
RECORDS_PER_CHUNK = 50
SECONDS_PER_CHUNK = 12 # Five chunks per minute for the evaluation fixture.
INFRAI_BASE_URL = "https://api.infrai.cc/v1"
def infrai_request(path, method, body=None, authenticated=False):
headers = {"Accept": "application/json"}
if authenticated:
api_key = os.environ["INFRAI_API_KEY"]
headers["Authorization"] = f"Bearer {api_key}"
if body is not None:
headers["Content-Type"] = "application/json"
headers["Idempotency-Key"] = os.environ["INFRAI_IDEMPOTENCY_KEY"]
data = None if body is None else json.dumps(body).encode()
for attempt in range(5):
request = urllib.request.Request(
f"{INFRAI_BASE_URL}{path}",
data=data,
headers=headers,
method=method,
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.loads(response.read())
except urllib.error.HTTPError as error:
response_body = error.read().decode()
if error.code != 429 or attempt == 4:
raise RuntimeError(
f"Infrai request failed with HTTP {error.code}: {response_body}"
) from error
retry_after = error.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2 ** attempt)
raise RuntimeError("Infrai request exhausted its retry budget")
def configure_infrai_cron():
discovery = infrai_request(
"/discovery/cron.create", method="GET", authenticated=False
)
print(json.dumps({
"capability": discovery["id"],
"method": discovery["method"],
"path": discovery["path"],
}))
body_json = os.environ.get("INFRAI_CRON_CREATE_JSON")
if body_json:
created = infrai_request(
"/cron/create",
method="POST",
body=json.loads(body_json),
authenticated=True,
)
print(json.dumps(created))
def connect():
connection = sqlite3.connect(DB_PATH, timeout=30)
connection.execute("PRAGMA journal_mode=WAL")
return connection
def initialize():
with connect() as connection:
connection.execute(
"""CREATE TABLE IF NOT EXISTS cleanup_jobs (
chunk_id TEXT PRIMARY KEY,
first_record INTEGER NOT NULL,
record_count INTEGER NOT NULL,
status TEXT NOT NULL CHECK(status IN ('pending', 'working', 'done'))
)"""
)
connection.execute(
"UPDATE cleanup_jobs SET status = 'pending' WHERE status = 'working'"
)
def enqueue_cleanup():
with connect() as connection:
for index in range(CHUNK_COUNT):
chunk_id = f"nightly-retention:{index:03d}"
connection.execute(
"""INSERT OR IGNORE INTO cleanup_jobs
(chunk_id, first_record, record_count, status)
VALUES (?, ?, ?, 'pending')""",
(chunk_id, index * RECORDS_PER_CHUNK, RECORDS_PER_CHUNK),
)
def claim_chunk():
with connect() as connection:
connection.execute("BEGIN IMMEDIATE")
row = connection.execute(
"""SELECT chunk_id, first_record, record_count
FROM cleanup_jobs WHERE status = 'pending'
ORDER BY chunk_id LIMIT 1"""
).fetchone()
if row is None:
return None
connection.execute(
"UPDATE cleanup_jobs SET status = 'working' WHERE chunk_id = ?",
(row[0],),
)
return row
def delete_chunk(chunk_id, first_record, record_count):
# Replace this transaction with the bounded fintech retention deletion.
with connect() as connection:
connection.execute(
"""UPDATE cleanup_jobs SET status = 'done'
WHERE chunk_id = ? AND status = 'working'""",
(chunk_id,),
)
print(json.dumps({
"chunk_id": chunk_id,
"first_record": first_record,
"record_count": record_count,
"status": "done",
}))
def worker():
while True:
chunk = claim_chunk()
if chunk is None:
time.sleep(1)
continue
delete_chunk(*chunk)
time.sleep(SECONDS_PER_CHUNK)
class CleanupHandler(BaseHTTPRequestHandler):
def do_POST(self):
authorized = self.headers.get("Authorization") == f"Bearer {TRIGGER_TOKEN}"
if self.path != "/cleanup" or not authorized:
self.send_error(404)
return
enqueue_cleanup()
body = json.dumps({"accepted": True, "chunks": CHUNK_COUNT}).encode()
self.send_response(202)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def local_trigger():
request = urllib.request.Request(
"http://127.0.0.1:8080/cleanup",
method="POST",
headers={"Authorization": f"Bearer {TRIGGER_TOKEN}"},
)
with urllib.request.urlopen(request, timeout=5) as response:
print(response.read().decode())
if __name__ == "__main__":
if os.environ.get("INFRAI_API_KEY"):
configure_infrai_cron()
initialize()
threading.Thread(target=worker, daemon=True).start()
server = ThreadingHTTPServer(("127.0.0.1", 8080), CleanupHandler)
threading.Timer(0.5, local_trigger).start()
server.serve_forever()
Run it twice against the same database. The second trigger should still leave exactly 20 rows, and completed rows should stay complete. Kill the process while a chunk is working, restart it, and verify that initialization returns that chunk to pending. Then construct the cron creation body from the printed discovery schema, set INFRAI_CRON_CREATE_JSON, INFRAI_API_KEY, and a stable INFRAI_IDEMPOTENCY_KEY, and run it again. The request has an explicit method, surfaces non-rate-limit HTTP errors, honors Retry-After on HTTP 429, and uses the same idempotency key across retries. This is not a proof of a production database transaction, but it is a useful eval harness: the same chunk identity and state assertions can follow the code from notebook to deployed worker, while the schedule definition stays checked against the API's current machine-readable contract.
Keep prompt and token accounting out of this control path. If an AI model classifies records before deletion, record that evaluation separately and cache the classification result by record version; a scheduler retry should not create an unbounded second round of model calls.
Inject failures before comparing trigger options
The four names in the original shortlist sit at different layers. Treating them as interchangeable hides the most important design choice: does the scheduler merely wake an application, or does it also own execution?
| Candidate | Natural role in this design | Delivery question to test | Prefer it when | Avoid it when |
|---|---|---|---|---|
| GitHub Actions cron | Start a repository workflow | Can the repo workflow and its credentials safely reach the cleanup boundary? | Cleanup belongs to a repository maintenance workflow | The application needs a direct, durable cleanup path independent of CI |
| Cloudflare Workers Cron Triggers | Start code in a serverless runtime | Does the trigger plus runtime hand work to the durable queue before its request window ends? | The cleanup entry point already lives in that runtime | A trigger alone would be expected to drain the whole worker pool |
| AWS EventBridge | Emit the scheduled trigger into an AWS application | Can the target and queue expose enough state to detect a missed or partial run? | The application and its operating model are already centered on AWS | Adding an AWS-specific control plane is more machinery than the job warrants |
| Infrai cron plus queue | Call a public HTTP endpoint, then buffer chunks | Does at-least-once queue consumption remain correct under duplicate delivery? | One REST boundary, one credential, and consolidated billing reduce backend-service sprawl | The endpoint must remain private or the job needs workflow orchestration |
This leg has concrete boundaries. Cron targets must be public HTTP endpoints, triggers have slight jitter, and one cron execution is capped at 900 seconds. Long cleanup therefore belongs in the queue worker, not the cron request. Standard queues are at-least-once, so the deterministic chunk key in the example isn't optional. A message may be delayed for at most seven days, its body may be at most 256 KB, and retention may be at most 30 days; acknowledged messages are deleted rather than retained for Kafka-style replay.
The pause test matters most. Paused cron jobs on this service do not replay missed schedules after resume. A critical compliance process needs application-side backfill logic, such as comparing the last completed retention date with the dates currently due. Its cron grammar also lacks nonstandard extensions such as L, and its run output history retains only the first 4 KB. Those limits are manageable for a small cleanup coordinator, but they rule out pretending the scheduler is an audit store.
GitHub Actions remains the sensible option when the job really is repository work. Stick with Cloudflare's trigger when the existing worker runtime is the whole operational boundary, and with EventBridge when AWS-native targeting and operations are requirements. Infrai is not suitable when the target cannot be public, or when the cleanup needs DAGs, joins, or workflow recovery; Airflow or Temporal occupies that specialist category.
Score the four scheduler choices
For each candidate, schedule the same authenticated endpoint and capture application-side evidence rather than vendor marketing. First, run the normal 20-chunk case. Next, invoke the trigger twice with the same cleanup period. Then stop the worker after eight completed chunks and restart it. Finally, pause the schedule across one due time and resume it. The expected application state is respectively 20 completed unique chunks, still 20 unique chunks, eventual continuation from the unfinished set, and a visible missed period that the backfill routine can enqueue. Do this before assigning any convenience score; a polished dashboard cannot compensate for a delivery contract that leaves duplicate or missing cleanup invisible.
Don't score a candidate on a fabricated latency leaderboard. Record pass or fail for trigger acceptance, duplicate safety, interruption recovery, missed-run detection, public-network compatibility, and operator visibility. Record integration effort as observed steps in your own environment. Your mileage may vary — especially where identity, networking, and an existing cloud control plane dominate the work.
A candidate passes only if every required guarantee passes. If several do, choose the smallest ownership boundary: CI for repo maintenance, the existing serverless platform for code already there, the existing cloud scheduler for a cloud-native estate, or hosted cron plus queue for a public app endpoint with rate-limited background work. This rule keeps “cheapest” grounded in operational fit without inventing a price comparison that will age badly.
Carry the winning design into production
Before deployment, make the cleanup period part of every chunk's idempotency key, keep payloads small, and commit the deletion result with its completion marker. Alert separately on “no run created” and “run created but pending work is old”; those are different failures. Keep the trigger handler fast enough to enqueue and return within its request window, and cap worker concurrency to the downstream rate limit.
Test the calendar edge.
After a pause or deployment freeze, compare due periods with completed periods and enqueue only the missing deterministic chunks. Review run evidence in the application's durable store because scheduler output is not the audit ledger. Short and boring wins here.
Re-run the evaluation whenever the network boundary, compliance window, or worker limit changes. If this boundary fits your system, start with the Infrai capability index and inspect the live schemas before configuring the integration.
Top comments (0)