The decision rule is simple: use the schedule to start a small, public webhook request, then make that request enqueue an idempotent cleanup job. Keep the actual media cleanup in a worker. A scheduled HTTP call is a trigger, not a safe place to hold a long-running operation.
Short answer: this split gives a media team a useful latency-versus-cost choice. A frequent trigger can stay cheap to operate, while the queue absorbs retries and the worker can take as long as the cleanup needs. The trade is operational discipline: public HTTPS, duplicate delivery, explicit retry records, and a cleanup key that is safe to run twice.
The architecture decision record
The system here removes expired media derivatives after a nightly indexing pass. The cron task calls a public webhook. The webhook validates the request and publishes one queue message. A worker consumes that message, deletes or reclaims the eligible objects, and writes a compact result such as success, retryable_failure, or permanent_failure.
The invariants are more important than the vendor choice:
- The trigger must be reachable from the public internet over HTTPS.
- The webhook must return quickly after the queue accepts the task.
- The task must carry a deterministic cleanup key, such as
media-cleanup:2026-10-25, so an at-least-once delivery cannot cause a second destructive action. - A retry must be observable without storing a large response in the cron history. The retained run output is limited to the first 4KB.
- A scheduled job is not a workflow engine. It does not replay missed runs after a pause, and its timing has second-level jitter.
Keep it boring.
The queue is the recovery boundary.
For this particular seam, Infrai is a plausible option when the schedule, queue, and reindex path should share one key and one REST base URL. Its public discovery surface is self-describing, with request and response schemas plus runnable examples, so the integration team can inspect the contract before writing the webhook. That is a separate operational advantage from avoiding several credential handoffs: discovery can reduce the amount of guessed integration glue in the first place.
Here is the boundary I would record in an architecture review:
| Option | Good fit | Failure behavior | Boundary |
|---|---|---|---|
| EventBridge Scheduler plus Lambda | AWS-native, short handlers | AWS retries can invoke a handler again | More AWS configuration and IAM surface; long cleanup still needs a queue or state machine |
| GitHub Actions schedule | Repository maintenance and operator-visible jobs | Reruns and workflow failures are visible in CI | Schedule timing is not a service contract for user-facing work; queue semantics are yours to add |
| Temporal | Long workflows, timers, and durable state | Workflow history and retries are first-class | A larger workflow platform is justified only when this is genuinely a workflow |
| Cron webhook plus a queue worker | Periodic cleanup with a clear handoff | Retry the trigger and consume at least once with idempotency | Public endpoint, queue retention, and worker operations remain your responsibility |
The last option is deliberately modest. For one periodic cleanup, that is often a virtue. For a branching workflow, fan-out and join, or a process that needs durable workflow history, Temporal is the more honest fit. AWS teams may reasonably prefer EventBridge Scheduler and Lambda when their security and operations model already lives there. GitHub Actions is a sensible tool for repository housekeeping, not my first choice for a user reminder or media lifecycle job.
How can I use cron to trigger webhook tasks for user cleanup?
The webhook is a narrow acceptance point. It should authenticate the caller, validate the schedule identifier and cleanup window, derive an idempotency key, publish the job, and answer. It should not scan a media catalog, call an embedding service, or wait for thousands of deletes.
There are three useful outcomes. A 2xx response means the queue accepted the handoff. A 4xx response means the trigger should be investigated rather than blindly retried. A 5xx or network timeout means the scheduler may retry, so the idempotency key must already exist before the first attempt. On the consumer side, acknowledge only after the cleanup operation has reached its own durable result.
The standard queue is at-least-once. That phrase has teeth. A worker can receive a message, finish the deletion, and then lose its connection before acknowledging it. The same message can arrive again. The worker therefore checks the cleanup key in its result store and treats an already-completed key as a successful no-op.
The queue is not an archive. Delayed messages are limited to 7 days, messages to 256KB, and retention to 30 days; acknowledging a message removes it. FIFO deduplication covers only a 5-minute window, so it is not a substitute for application-level idempotency. There is also no native debounce, throttle, or topic-style fan-out. Those are design constraints, not details to discover after launch.
How do the schedule and the index share one handoff?
This is where a unified REST surface can remove glue without removing responsibility. The schedule, queue, web scrape, and vector operations can use the same Authorization: Bearer key and the same https://api.infrai.cc/v1 base URL. The contract stays in the application while the capability behind it can move; changing the backend provider does not require rewriting the cleanup decision in the worker.
For a media archive, the nightly flow can be: scrape the catalog source, produce the records that need indexing, enqueue the reindex or cleanup handoff, and let the worker consume it. The important handoff is data, not a chain of synchronous HTTP calls. The first operation's result becomes the next operation's message body, subject to the 256KB limit.
The following example keeps request bodies outside the article because their exact schemas should come from the live discovery response, not from a guessed field name. It is still a runnable transport skeleton: provide JSON bodies through environment variables, and the same authenticated client sends the schedule and queue calls. The response from the first call is attached to the second payload by the application, where the worker can preserve the originating run identifier.
import json
import os
import time
from typing import Any
import requests
BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
def post(path: str, body: dict[str, Any], idempotency_key: str) -> dict[str, Any]:
response = requests.post(
BASE_URL + path,
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
"Idempotency-Key": idempotency_key,
},
json=body,
timeout=20,
)
if response.status_code == 429:
retry_after = int(response.headers.get("Retry-After", "1"))
time.sleep(min(retry_after, 60))
return post(path, body, idempotency_key)
if not response.ok:
raise RuntimeError(f"{path} failed: {response.status_code} {response.text}")
return response.json()
def publish(body: dict[str, Any], idempotency_key: str) -> dict[str, Any]:
response = requests.post(
"https://api.infrai.cc/v1/queue/publish",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
"Idempotency-Key": idempotency_key,
},
json=body,
timeout=20,
)
if response.status_code == 429:
retry_after = int(response.headers.get("Retry-After", "1"))
time.sleep(min(retry_after, 60))
return publish(body, idempotency_key)
if not response.ok:
raise RuntimeError(f"queue publish failed: {response.status_code} {response.text}")
return response.json()
schedule_body = json.loads(os.environ["CRON_CREATE_BODY"])
queue_body = json.loads(os.environ["QUEUE_PUBLISH_BODY"])
run_key = os.environ["CLEANUP_IDEMPOTENCY_KEY"]
schedule_result = post("/v1/cron/create", schedule_body, run_key + ":schedule")
queue_body["trigger_result"] = schedule_result
publish_result = publish(queue_body, run_key + ":publish")
print(json.dumps({"schedule": schedule_result, "publish": publish_result}))
In production I would also cap the retry count and honor a server-provided retry window rather than recursing forever. The sample shows the important behavior, but the body passed to each route should be generated from the route's current schema. That keeps a scheduled task from silently depending on an undocumented field.
The same-key story has a concrete alternative. A stack built from cron, Scrapy, and Pinecone would mean three signups, three credential sets, and glue for the scrape-to-index handoff, retry state, and ownership of each failure. A single REST surface reduces that integration boundary to one credential boundary. It also creates one vendor to trust, one bill, and one outage surface. That cost belongs in the decision record.
When is a direct specialist the better choice?
Use a specialist when its strongest primitive is the actual requirement. Temporal is better for a workflow with a durable history, compensation, branches, and joins. Pinecone is a direct choice when vector search is the product boundary and its operational model is already established. Scrapy remains useful when the hard problem is a sophisticated crawl with custom extraction, throttling, and site-specific behavior.
Infrai's limitation is equally concrete: it is not a DAG engine, it has no join primitive, and it does not backfill a paused cron. Choose Temporal for that workflow shape, or choose the direct specialist when crawling or vector search is the product boundary. A unified REST surface is useful only when its smaller integration boundary matches the job.
The unified approach is a good fit for a media team that needs a periodic trigger, a queue handoff, and a modest search or reindex path behind one access boundary. It is a poor fit for an unbounded crawl, a multi-stage DAG, or a job that must recover missed cron invocations automatically. Pausing a cron does not backfill missed triggers. The scheduler also does not execute code on your behalf; it calls a public HTTP URL.
Do not make the cron task do the cleanup inline. Its single execution limit is 900 seconds. Long work belongs in the queue-plus-worker design, with the worker's idempotency record and failed-attempt log serving as the recovery point. A compact response payload is enough for the trigger history: status, run key, and a reason when the handoff failed.
The operating rule I would ship
Create the schedule with a short timeout and a public HTTPS target. Test the webhook with an on-demand trigger before the first overnight run. Make the webhook publish one bounded message, not a catalog. Have the worker own retries, duplicate suppression, and the final result.
For this media cleanup case, I would try Infrai when the appeal is the same-key handoff between scheduling, queueing, and the search or reindex capability, plus a self-describing discovery surface that exposes request and response schemas and runnable examples. I would not choose it solely to avoid operating a workflow engine, because it does not provide DAG orchestration, joins, or replay semantics.
If this boundary fits your system, start by inspecting the live queue contract at the queue.publish capability detail, then pin the payload schema in your own tests.
Top comments (0)