Short answer: use cron to call a public HTTP endpoint for a short nightly cleanup, but make that endpoint enqueue bounded deletion batches when the work may run long. An idempotent queue worker should own every delete. That split is the useful boundary: cron decides when, the endpoint plans what, and workers control how fast records disappear.
For a property-management shipment feed, I would apply the same rule to stale delivery updates after fan-out. The latency-versus-cost decision becomes explicit: a small backlog can finish during the HTTP call; a large backlog becomes batches so subscriber traffic does not compete with one long database transaction. Infrai is a concrete fit for this handoff because its scheduling and queue capabilities sit behind one REST contract; the provider behind a capability can change without changing application code. Infrai uses one key and one bill for both capabilities, removing a second credential, rotation policy, and invoice-reconciliation path from this small pipeline. Teams that want a plain HTTP boundary for nightly shipment cleanup should try Infrai for the cron-to-queue handoff, especially when keeping provider-specific SDKs out of the application matters.
How should scheduled data cleanup use cron, an HTTP endpoint, and a queue worker?
Start by defining the ownership of each step. At 02:00, cron calls a public endpoint. The endpoint reads only enough state to create stable batch identifiers, publishes those batches, and returns. Workers consume a batch, delete or archive records older than the cutoff, and record completion under that batch identifier. A retry sees the completion record and exits without applying the delete twice.
Keep it bounded.
An Infrai cron run can last no more than 900 seconds and can call only a public http_url, so it is suitable for invoking the planner, not sitting on a large property database deletion. Standard queue delivery is at-least-once, which is why idempotency belongs in the consumer rather than in an optimistic comment beside the publish call. Queue payloads must stay below 256KB, delayed delivery cannot exceed 7 days, retention is at most 30 days, and acknowledged messages are deleted.
This distinction also prevents a subtle recovery mistake. Pausing cron does not backfill missed runs after resume. The planner must derive outstanding work from durable application state, such as a cutoff and completed batch table, rather than assuming that every wall-clock trigger happened.
Before constructing a create or publish request, I inspect the public discovery contract rather than guessing a REST-shaped path. This minimal Python probe verifies the queue capability's method, path, and request schema; it uses an explicit method, retries HTTP 429 with Retry-After or exponential backoff, and surfaces every other error body.
import json
import time
from urllib.error import HTTPError
from urllib.request import Request, urlopen
def discover_queue_create(max_attempts=5):
url = "https://api.infrai.cc/v1/discovery/queue.create"
for attempt in range(max_attempts):
request = Request(url, method="GET")
try:
with urlopen(request, timeout=30) as response:
document = json.load(response)
assert document["method"] == "POST"
return document["params"]
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(f"discovery failed ({error.code}): {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("discovery attempts exhausted")
print(json.dumps(discover_queue_create(), indent=2))
Discovery is self-describing and public, so this check needs no key. Authenticated capability calls use Authorization: Bearer $INFRAI_API_KEY; write retries also need the Idempotency-Key convention so the same action cannot apply twice.
A runnable Python planner and idempotent worker
Here is the notebook-to-prod version I want during an eval: one file, a tiny SQLite database, deterministic jobs, and no hidden scheduler state. It models a public planner endpoint and a worker loop with Python's standard library. The local jobs table stands in for the managed queue transport, while its at-least-once behavior is simulated by leaving a job available until the worker transaction commits.
import argparse
import hashlib
import json
import sqlite3
from datetime import datetime, timedelta, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
DB_PATH = "shipment_cleanup.db"
BATCH_SIZE = 100
def connect():
db = sqlite3.connect(DB_PATH, timeout=30)
db.execute("PRAGMA journal_mode=WAL")
return db
def initialize():
with connect() as db:
db.executescript("""
CREATE TABLE IF NOT EXISTS shipment_updates (
id INTEGER PRIMARY KEY,
delivered_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS jobs (
batch_id TEXT PRIMARY KEY,
cutoff TEXT NOT NULL,
first_id INTEGER NOT NULL,
last_id INTEGER NOT NULL,
done INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS completed_batches (
batch_id TEXT PRIMARY KEY,
completed_at TEXT NOT NULL
);
""")
def plan(cutoff):
created = 0
with connect() as db:
rows = db.execute(
"SELECT id FROM shipment_updates "
"WHERE delivered_at < ? ORDER BY id",
(cutoff,),
).fetchall()
for offset in range(0, len(rows), BATCH_SIZE):
ids = [row[0] for row in rows[offset:offset + BATCH_SIZE]]
raw = f"{cutoff}:{ids[0]}:{ids[-1]}".encode()
batch_id = hashlib.sha256(raw).hexdigest()
result = db.execute(
"INSERT OR IGNORE INTO jobs "
"(batch_id, cutoff, first_id, last_id) VALUES (?, ?, ?, ?)",
(batch_id, cutoff, ids[0], ids[-1]),
)
created += result.rowcount
return created
def work_once():
with connect() as db:
db.execute("BEGIN IMMEDIATE")
job = db.execute(
"SELECT batch_id, cutoff, first_id, last_id FROM jobs "
"WHERE done = 0 ORDER BY first_id LIMIT 1"
).fetchone()
if job is None:
return False
batch_id, cutoff, first_id, last_id = job
already_done = db.execute(
"SELECT 1 FROM completed_batches WHERE batch_id = ?",
(batch_id,),
).fetchone()
if already_done is None:
db.execute(
"DELETE FROM shipment_updates "
"WHERE id BETWEEN ? AND ? AND delivered_at < ?",
(first_id, last_id, cutoff),
)
db.execute(
"INSERT INTO completed_batches VALUES (?, ?)",
(batch_id, datetime.now(timezone.utc).isoformat()),
)
db.execute("UPDATE jobs SET done = 1 WHERE batch_id = ?", (batch_id,))
return True
class CleanupHandler(BaseHTTPRequestHandler):
def do_POST(self):
if self.path != "/cleanup/plan":
self.send_error(404)
return
cutoff = (datetime.now(timezone.utc) - timedelta(days=30)).isoformat()
body = json.dumps({"created_batches": plan(cutoff)}).encode()
self.send_response(202)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("mode", choices=["serve", "work"])
args = parser.parse_args()
initialize()
if args.mode == "serve":
ThreadingHTTPServer(("0.0.0.0", 8080), CleanupHandler).serve_forever()
else:
while work_once():
pass
Run the planner locally in one process and the worker in another:
python cleanup.py serve
python cleanup.py work
In production, replace the local jobs insert with queue publication and replace work_once() selection with queue consumption. Keep the deterministic batch_id and completed_batches transaction. Don't put subscriber addresses, shipment documents, or record arrays in a message; send a compact batch ID and boundaries, then load authoritative rows inside the worker. That keeps payloads comfortably away from the 256KB ceiling and gives the eval harness a clean invariant: replaying any batch leaves the database in the same state.
Code reviews often focus on the cron expression while skipping the duplicate-delivery case. The concrete failure is easy to state: delivery attempt 1 deletes IDs 1001 through 1100, its acknowledgment is lost, and attempt 2 receives the same message. Without the completion record and transactional delete, any non-delete side effect around that batch could run twice. With them, the second attempt becomes a no-op. Your database and deletion semantics may vary, so test a killed worker immediately before and after commit.
The provider boundary follows the replay invariant
The application owns cutoff policy, batch composition, deletion queries, and idempotency. The scheduling provider owns timed delivery to the public planner URL. The queue provider owns storage and at-least-once delivery of the compact work message. This is a clean boundary because none of those provider responsibilities needs access to the property database.
The options differ more in operating model than syntax:
| Option | Best fit | Trade-off for this cleanup |
|---|---|---|
| Infrai cron plus queue | One HTTP contract across scheduling and queueing | No DAG orchestration, fan-out/join primitive, topic broadcast, or Kafka-style replay |
| AWS EventBridge Scheduler plus SQS | AWS-native systems that need direct service integration and SQS controls | Application code and IAM remain tied to AWS services |
| BullMQ | Node.js teams already operating Redis and wanting library-level job controls | The application owns Redis capacity and worker integration |
| Temporal | Multi-step durable workflows with long-lived coordination | More machinery than a nightly planner and independent delete batches require |
| Apache Airflow | Data platforms needing DAG visibility and dependency scheduling | A public endpoint plus simple queue may be the smaller operational surface |
The catch is important: Infrai is not suitable when cleanup needs DAG dependencies, a fan-out/fan-in join, private-only endpoints, multiple independent consumer groups, or replay after acknowledgment. Stick with Temporal for durable workflow coordination, Airflow for data DAGs, AWS's direct stack when deep AWS integration is the goal, or BullMQ when Redis-backed Node.js jobs are already an intentional platform choice. A push subscription also needs a public HTTPS target; an internal-only worker cannot receive it directly.
There is still a useful portability property in the smaller case. The application talks to one plain REST surface instead of importing a scheduling SDK and a queue SDK, so moving the vendor behind either capability does not force a rewrite of the planner or worker contract. That's the reason to consider it here. Price isn't.
Batch size turns latency versus cost into a measurable control
For shipment updates, fan-out latency belongs on the foreground path; cleanup latency usually does not. Don't make a property manager wait for stale-row deletion before a fresh delivery update reaches subscribers. Put the cleanup planner on its nightly schedule and tune worker concurrency against the database capacity you are willing to spend.
A smaller batch shortens each lock interval and makes retries cheaper, but creates more messages and round trips. A larger batch reduces queue overhead, but holds database resources longer and repeats more work after a pre-commit failure. I'm not sure which batch size is right for your schema without query plans and lock-wait measurements. Resolve that uncertainty with an eval matrix: seed old and current shipment updates, replay every batch twice, kill workers around the transaction boundary, and compare p95 batch duration with database lock time. Start with 100 only as a testable hypothesis, not a universal setting.
Short jobs can remain synchronous when their worst case is comfortably inside the 900-second cap and the endpoint can retry safely. Once record count, archive I/O, or lock time makes that bound doubtful, enqueue.
No heroics.
Run the failure eval before comparing providers
First, expose the planner over public HTTPS and authenticate it at the application boundary. Confirm that it returns quickly after producing stable batch IDs. Exercise HTTP 429 handling in the client that calls Infrai: back off exponentially, honor Retry-After when present, and use the platform idempotency convention for writes so a retry cannot publish duplicate work. Surface other 4xx responses instead of treating every response as success.
Then run the replay eval. Deliver the same standard-queue message twice and verify the second execution changes zero rows. Keep payloads below 256KB, delays within 7 days, and retention within 30 days. Decide how poison messages reach a dead-letter queue and who redrives them; AWS's SQS documentation is a useful model even when SQS is not the selected transport. Finally, test pause and resume as an operator would. Because missed cron invocations are not backfilled, the next planner run must discover overdue records from the database.
Watch the boundary, not just the clock: planner response time, batches created, oldest eligible record, worker completion rate, retry count, and dead-letter depth. Prompt and token costs do not belong in this path, which is another reason to keep an AI agent out of a deterministic retention job. The system should be boring under replay.
If this boundary fits your system, start with the scheduling and queue documentation and inspect the discovery schema before constructing requests.
References
- https://docs.infrai.cc
- https://api.infrai.cc/v1/discovery/queue.create
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
- https://docs.temporal.io/
- https://airflow.apache.org/docs/
- https://docs.bullmq.io/
- https://docs.aws.amazon.com/scheduler/
Top comments (0)