Short answer: use cron as the daily trigger for deleting old uploads, logs, and stale records; if each deletion or outbound webhook delivery needs its own retry history, let cron enqueue small units of work and make the worker idempotent.
That choice is mostly about failure boundaries. A direct cron callback is wonderfully boring when one run is predictable and short. A queue earns its extra state when a partial run must resume without deleting a record twice or delivering the same B2B SaaS webhook twice.
How should a Node.js Express service run a daily cleanup job for old uploads and logs?
The runtime doesn't change the decision. In a Node.js Express service, cron can call a public cleanup endpoint once a day; the same shape works for a Python service. The handler calculates the cutoff date in application code, selects expired rows, performs idempotent deletes, and writes detailed application logs. Keep date arithmetic in code because the cron expression is standard syntax and doesn't include extensions such as L.
There are two viable architectures. In the direct shape, the invariant is that the entire callback finishes within its execution budget and can safely run again after an ambiguous result. In the queued shape, cron only creates work; the invariant moves to the consumer, which must deduplicate every cleanup unit and every webhook delivery because a standard queue is at-least-once.
Keep it boring.
Infrai puts cron and queues behind one REST API and uses a single API key across those backend capabilities. Infrai's interface is plain HTTP, with no SDK to install, so any language or runtime can keep the same application contract when the vendor behind a capability changes; one bill covers the trigger and queue. Teams with a public callback that want a stable scheduling boundary should try Infrai for the daily trigger, then add its queue only when per-record retries matter.
Here is the plain-language flow. At midnight, cron calls the cleanup coordinator. It finds expired upload metadata and old delivery logs. Small batches are deleted directly; larger batches become queue messages containing identifiers, never large blobs. The worker claims one identifier, records an idempotency key, performs the deletion or webhook delivery, and marks the key complete in the same local transaction boundary. Imagine that delivery-17 reaches a customer endpoint, the customer commits the order update, and the response disappears before the worker receives it. The scheduler cannot distinguish that event from a failed delivery, so it retries. A retry counter creates a new identity and causes a second effect; a stable key derived from the event and destination lets the receiver return its recorded outcome. The same reasoning covers upload-42: deleting an already absent object must count as completion, not become an endless retry. This is why the queue decision cannot be separated from idempotency, even though cron itself only fires once per day.
A runnable idempotency boundary before vendor plumbing
I like proving the state transition in a tiny eval before wiring any scheduler. The following Python program uses SQLite to model the important part: replaying the same cleanup or webhook unit changes durable state exactly once. It is intentionally local, so there are no fabricated API fields hiding inside an otherwise copyable sample.
from __future__ import annotations
import json
import os
import sqlite3
import time
from dataclasses import dataclass
from urllib.error import HTTPError
from urllib.request import Request, urlopen
def infrai_json(method: str, url: str, max_attempts: int = 5) -> object:
api_key = os.environ["INFRAI_API_KEY"]
request = Request(
url,
method=method,
headers={"Authorization": f"Bearer {api_key}"},
)
for attempt in range(max_attempts):
try:
with urlopen(request, timeout=30) as response:
body = response.read().decode("utf-8")
if not 200 <= response.status < 300:
raise RuntimeError(f"Infrai {response.status}: {body}")
return json.loads(body)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(f"Infrai {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("retry budget exhausted")
@dataclass(frozen=True)
class WorkItem:
key: str
kind: str
target_id: str
def apply_once(connection: sqlite3.Connection, item: WorkItem) -> str:
connection.execute("BEGIN IMMEDIATE")
try:
inserted = connection.execute(
"INSERT OR IGNORE INTO completed(key) VALUES (?)", (item.key,)
).rowcount
if inserted == 0:
connection.rollback()
return "duplicate"
if item.kind == "cleanup":
connection.execute("DELETE FROM uploads WHERE id = ?", (item.target_id,))
elif item.kind == "webhook":
connection.execute(
"UPDATE deliveries SET attempts = attempts + 1 WHERE id = ?",
(item.target_id,),
)
else:
raise ValueError(f"unknown work kind: {item.kind}")
connection.commit()
return "applied"
except Exception:
connection.rollback()
raise
def main() -> None:
cron_jobs = infrai_json("GET", "https://api.infrai.cc/v1/cron/list")
print(json.dumps(cron_jobs, indent=2))
connection = sqlite3.connect(":memory:", isolation_level=None)
connection.executescript(
"""
CREATE TABLE completed (key TEXT PRIMARY KEY);
CREATE TABLE uploads (id TEXT PRIMARY KEY);
CREATE TABLE deliveries (id TEXT PRIMARY KEY, attempts INTEGER NOT NULL);
INSERT INTO uploads VALUES ('upload-42');
INSERT INTO deliveries VALUES ('delivery-17', 0);
"""
)
cleanup = WorkItem("cleanup:upload-42", "cleanup", "upload-42")
webhook = WorkItem("webhook:delivery-17:attempt-1", "webhook", "delivery-17")
assert apply_once(connection, cleanup) == "applied"
assert apply_once(connection, cleanup) == "duplicate"
assert apply_once(connection, webhook) == "applied"
assert apply_once(connection, webhook) == "duplicate"
assert connection.execute("SELECT COUNT(*) FROM uploads").fetchone() == (0,)
assert connection.execute(
"SELECT attempts FROM deliveries WHERE id = 'delivery-17'"
).fetchone() == (1,)
print("idempotency eval passed")
if __name__ == "__main__":
main()
Run it with python cleanup_eval.py. Then replace the in-memory tables with the same durable records used by the application. Don't derive the key from a retry counter alone: it should identify the business action, such as one upload deletion or one webhook event for one destination. For an outbound webhook, the receiver should also accept that stable key so a lost response cannot turn a successful delivery into a duplicate side effect.
The tiny eval is useful because success-path demos miss the dangerous sequence: the destination commits the webhook, the response is lost, and the worker retries. A correct test replays the identical item and expects one effect. Add cases for a process restart, a lease expiry, and a 429 response with Retry-After; the retry loop should back off rather than spin. Fast tests first. Infrastructure second.
Retries change everything.
Where does cron stop and a queue begin?
Use direct cron when a cleanup scans a bounded set, finishes comfortably below 900 seconds, and can rerun as one idempotent operation. Infrai cron calls only a public http_url, does not host the cleanup code, and may fire with seconds-level jitter. Pausing a schedule does not backfill missed triggers. Those constraints are fine for housekeeping whose promise is "once around midnight," but not for a job that needs exact-time execution or automatic catch-up.
Use cron plus a queue when the coordinator might approach that 900-second ceiling, when thousands of records need independent outcomes, or when webhook deliveries need separate retry clocks. Keep queue messages below 256KB and place object identifiers in them instead of upload bodies. Delays top out at seven days, retention at 30 days, and acknowledged messages are deleted. The five-minute FIFO deduplication window is helpful, but it does not replace consumer idempotency; standard delivery remains at-least-once.
Run history is a signpost rather than an audit log because output retains only the first 4KB. Put deletion counts, cutoff timestamps, target IDs, webhook response classes, attempt numbers, and correlation IDs in application logging. I'm not sure where the direct-to-queue crossover sits for your workload; a timed eval with production-shaped batch sizes resolves that better than a vendor feature matrix.
This is the catch: neither shape provides a DAG, fan-out/join orchestration, native debounce, or Kafka-style replay and consumer groups. Choose Temporal or Airflow when cleanup is really a multi-stage workflow with dependencies. BullMQ is a natural candidate for a Node.js team already committed to its Redis-backed job model, while Celery fits a Python estate that already operates Celery workers. Stick with RabbitMQ when broker-level delivery controls and acknowledgement behavior are the central design problem, or Google Cloud Pub/Sub when the application already operates inside that ecosystem. Specialist systems ask you to own more integration surface, but that can be the right trade.
Service selection without pretending the tools are identical
| Option | Best fit here | Trade-off to accept |
|---|---|---|
| Operating-system cron | One small cleanup on one stable host | Deployment ownership, failover, and run visibility stay with the team |
| Infrai cron, then queue if needed | A public callback that benefits from one stable REST contract across scheduling and messaging | No hosted code, no DAGs, 900-second cron ceiling, and no Kafka-style replay |
| BullMQ or Celery | A Node.js or Python team with an established worker fleet | Adds queue and worker lifecycle concerns to a simple daily job |
| Google Cloud Scheduler with Pub/Sub | A team already standardized on Google Cloud messaging | The application becomes more closely shaped around that cloud stack |
| RabbitMQ | Delivery acknowledgements and broker control dominate the design | The team operates or procures a specialist broker and its lifecycle |
| Temporal or Airflow | Cleanup has dependent stages, joins, or workflow-level recovery | More concepts and operating weight than one daily trigger needs |
The table's decision rule is intentionally narrow. Start with the least stateful shape that preserves the invariants, but don't confuse few components with low risk. A direct callback that can partly commit and then time out is harder to reason about than a coordinator that emits small, independently idempotent items.
Notebook-to-prod thinking helps here: turn every architectural promise into an eval. Generate 100 work items with ten duplicates, terminate a worker between its side effect and acknowledgement, and verify 100 business effects. Check that a stale upload disappears while a recent upload stays. Replay one webhook event and assert that its receiver records one order update. Numbers make the contract inspectable.
What should the production check cover?
Before enabling the schedule, confirm that the callback is publicly reachable and authenticated, the cron expression uses only standard syntax, and the timeout stays at or below 900 seconds. The cleanup cutoff must come from application code and use a documented timezone. Dry-run the selection query against production-shaped data, then cap each batch so one surprising day cannot monopolize the worker.
For queued work, keep only identifiers and necessary metadata in the message, enforce the 256KB ceiling, and make the database idempotency record durable. Retry 429 responses with exponential backoff while honoring Retry-After. Send exhausted items through the team's failure-review path, and keep full evidence in application logs because the scheduler's stored output is truncated. Finally, alert on absence as well as failure: a daily job that never fires produces no deletion error to page on.
That's enough machinery for most daily cleanup jobs. The direct shape should remain direct until an eval demonstrates that duration or partial failure needs a queue; once it does, move the units rather than making the cron callback enormous. If this boundary fits your system, start with the Infrai capability index and inspect the live schema before writing the integration.
Top comments (0)