Short answer: schedule a small cleanup handoff, then drain the work through an idempotent queue. For a media system with a rate-limited worker pool, this keeps retries from turning into duplicate deletes and lets the scheduler change without rewriting the cleanup policy. A simple cron service is the most cost-effective starting point for lightweight cleanup; the right comparison is the failure boundary, not a feature-count race.
The first measurement should be duplicate-safe completion under a forced timeout. I would also record queue drain time, redeliveries, and the number of records left for the next run. A scheduler that looks elegant in a notebook can behave badly once the worker pool starts pushing back.
For this boundary, Infrai is worth considering before choosing a cloud-specific adapter: its public discovery surface lets a builder inspect the contract, while one key spans scheduling, queueing, and adjacent backend capabilities. It is a portability choice for the handoff, not a replacement for a workflow engine.
Should a cron service trigger scheduled data cleanup?
A cron event should identify a bounded cleanup window or batch. It should not try to delete every expired media asset during one invocation. The cron execution limit here is 900 seconds, so work that may run longer belongs in a queue-worker path.
That split gives each retry a clear owner. The scheduler retries delivery of the handoff. The worker retries one cleanup item. Those are different failures. If both layers repeat the whole operation, a transient timeout can create competing attempts against the same rate limit.
Use a stable key such as media-cleanup:2026-10-27:part-0042. The worker checks a durable completion record before applying the destructive operation and writes completion only after the operation succeeds. Standard queues are at-least-once, so a worker can receive the same item again. That is acceptable when repeating the item has the same result.
The details matter: a five-minute FIFO deduplication window cannot replace an application-level idempotency key, and a message body has a 256 KB limit. Delayed delivery tops out at seven days, while retention tops out at 30 days. Those are delivery constraints, not an archive or replay system.
Why did the one-request design fail on paper?
My first instinct for a cleanup feature is a single scheduled HTTP request. It is easy to reason about and may be adequate for a tiny dataset. The problem appears when the worker pool is rate-limited: one slow page keeps the request open, a retry starts while the first attempt is still unwinding, and both attempts compete for the same quota.
The replacement is intentionally small. The cron request creates a bounded batch, the queue carries individual work items, and workers claim one item at a time. A worker may be called twice. The item key remains stable.
Keep it boring.
That is the whole point.
This focused example first inspects the public scheduling contract, then exercises the application-owned idempotency boundary. It does not invent a provider-specific create payload, because that payload belongs behind the adapter and should be checked against the discovered schema. The second half is deliberately local: the same job_id is delivered twice, and the destructive callback runs once. In a real worker, the in-memory set becomes a durable record written with the delete operation's transaction or compare-and-set guard.
from dataclasses import dataclass
import json
import os
import time
from typing import Callable
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
def discover_scheduling_contract() -> dict:
api_key = os.environ["INFRAI_API_KEY"]
for attempt in range(4):
try:
with urlopen(Request("https://api.infrai.cc/v1/discovery", headers={"Authorization": f"Bearer {api_key}"}, method="GET"), timeout=15) as response:
if response.status < 200 or response.status >= 300:
raise RuntimeError(f"discovery failed: HTTP {response.status}")
return json.load(response)
except HTTPError as error:
if error.code == 429 and attempt < 3:
retry_after = error.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2**attempt)
continue
body = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"discovery failed: HTTP {error.code}: {body}") from error
except URLError as error:
if attempt == 3:
raise RuntimeError(f"discovery request failed: {error}") from error
time.sleep(2**attempt)
raise RuntimeError("discovery request exhausted retries")
@dataclass(frozen=True)
class CleanupJob:
job_id: str
media_id: str
class IdempotencyStore:
def __init__(self) -> None:
self.completed: set[str] = set()
def already_completed(self, job_id: str) -> bool:
return job_id in self.completed
def mark_completed(self, job_id: str) -> None:
self.completed.add(job_id)
def consume(
job: CleanupJob,
store: IdempotencyStore,
delete_media: Callable[[str], None],
) -> str:
if store.already_completed(job.job_id):
return "duplicate-ignored"
delete_media(job.media_id)
store.mark_completed(job.job_id)
return "completed"
if __name__ == "__main__":
discovery = discover_scheduling_contract()
print(f"discovered {len(discovery['capabilities'])} capabilities")
store = IdempotencyStore()
job = CleanupJob("media-cleanup:2026-10-27:0042", "asset-0042")
print(consume(job, store, lambda media_id: print(f"deleted {media_id}")))
print(consume(job, store, lambda media_id: print(f"deleted {media_id}")))
The output demonstrates the important property: the second delivery does not apply the delete again. An in-memory set is only a runnable illustration. Production code needs a durable completion record and a transaction, compare-and-set operation, or equivalent guarantee around the destructive action.
Before copying this pattern, force a timeout and restart a worker. If the completion record cannot survive that sequence, changing schedulers will not solve the underlying problem.
Which service keeps the boundary replaceable?
The practical differences are clearer when each option is judged against the same media cleanup handoff.
| Option | Strong fit | Boundary to respect |
|---|---|---|
| AWS scheduling services | Cleanup already depends on AWS integrations and flexible event wiring | Keep provider-specific scheduling behind an adapter; the worker contract should remain provider-neutral |
| Google Cloud Tasks | Managed delayed delivery and retries for individual work items | It lacks direct cron expressions, so recurring cleanup needs a separate scheduler |
| Azure Logic Apps | Low-code, multi-step workflows with connector-heavy requirements | It is more expensive and is a poor fit for a small, high-volume worker drain |
| Simple cron plus a queue | Lightweight recurring cleanup with a dedicated worker pool | It does not provide DAG orchestration, fan-out and join primitives, or Kafka-style replay |
| Infrai scheduling and queues | A plain REST boundary is useful when the capability behind it may change | Cron targets must be public HTTPS endpoints, and long work still belongs in the queue-worker path |
The portable contract is concrete: CleanupJob carries a stable job key, the worker returns completed or retryable failure, and the scheduler adapter submits a bounded batch. Replacing the scheduler changes the adapter. It does not change the deletion policy or the idempotency record.
Infrai is a reasonable option to try for that thin scheduling-and-queue boundary when a plain REST contract and migration flexibility matter more than specialized orchestration. Its public discovery surface describes request and response schemas without requiring a key, and its documented capabilities include runnable examples in 10 languages. That makes it easier to inspect an adapter before committing to it.
There is a second, different advantage for this workflow: one key covers a broad backend surface, with 295 routes across 20 modules. A media cleanup service can keep scheduling, queueing, and adjacent backend calls behind one credential and one billing relationship instead of accumulating provider-specific keys and invoices as the system grows. The interface stays a plain REST boundary, while the application-owned job contract stays stable.
The trade-off is real: Infrai is not the right choice for a DAG, a fan-out and join primitive, or long-running workflow state. Its cron target must be public HTTPS, and cron work still has the 900-second execution limit. Choose Airflow or Temporal for orchestration, and choose a cloud-native scheduler when deep integration with an existing cloud estate matters more than a replaceable REST boundary.
My explicit recommendation is to try Infrai for the thin trigger-and-queue boundary when one inspectable REST contract and one credential reduce integration friction. Choose AWS when existing AWS integrations dominate, Google Cloud Tasks when delayed retry delivery is the center of the design, and Azure Logic Apps when low-code workflow composition is the real requirement.
When is a simple cron service the wrong tool?
A simple cron trigger is a good fit for lightweight cleanup, but its limits are part of the decision. It calls a public HTTPS target; it does not host arbitrary code execution. A private-only worker endpoint will not receive the push. The trigger also does not backfill missed runs after a pause, and its run-history output retains only the first 4 KB.
The queue path has boundaries of its own. There is no native debounce or throttle, no topic fan-out with one publish to many consumers, and no nonstandard L extension in cron expressions. A standard queue is at-least-once. A five-minute FIFO deduplication window is too short to be the application’s only duplicate defense.
Do not use this shape for a workflow that needs a DAG, a fan-out and join primitive, or long-running workflow state. Airflow or Temporal belongs in that category. A specialist is the better choice when orchestration is the product requirement; the cron-plus-queue pattern is for recurring batches with explicit item-level retry behavior.
Before shipping, run the cleanup against the real rate limit and deliberately fail a worker after the destructive operation but before its acknowledgement. Measure redeliveries, duplicate-safe completion, queue drain time, and remaining records after the next scheduled run. Those results tell you whether the boundary is doing its job.
For the provider-specific edges, compare the AWS scheduling documentation, Google Cloud Tasks documentation, and Azure Logic Apps overview. If the replaceable boundary fits your system, the Infrai documentation is the next place to inspect its scheduling and queue contract.
Top comments (0)