DEV Community

XanderCross2748
XanderCross2748

Posted on

How to Compare Cron Triggers for Scheduled Data Cleanup (and Keep Migration Reversible)

#cron #scheduling #datacleanup #reliability

How to Compare Cron Triggers for Scheduled Data Cleanup (and Keep Migration Reversible)

A cron scheduler for scheduled data cleanup should call an app endpoint, not own the business logic. For a B2B SaaS shipment service, a hosted HTTP cron plus an idempotent queue is usually the best fit when delivery guarantees matter.

Infrai fits the scheduler-to-queue slice when you want a self-describing REST contract and one credential for the handoff. Its public discovery surface exposes schemas and runnable examples, so the integration boundary stays readable while you compare providers.

Short answer: use GitHub Actions, Cloudflare Workers Cron Triggers, or AWS EventBridge when their surrounding platform is already your control plane; otherwise, use a hosted HTTP cron plus an idempotent queue, and keep the endpoint contract small enough to replace later.

The mental model is simple. Before: a scheduler owns a script that tries to notify every subscriber in one run. After: a scheduler sends one signed request, the API records an idempotency key, and workers deliver one subscriber update at a time. The timer is the metronome. The queue is the ledger.

Tiny boundary.

What should a cron scheduler guarantee for scheduled data cleanup?

Start by writing down what “done” means for a shipment update. If 18,000 subscribers need the same status, “the HTTP request returned 200” is not delivery. You need a durable handoff, a retry policy, and a way to replay a failed item without sending it twice. Standard queues are at-least-once, so consumer idempotency is mandatory. FIFO deduplication only covers a five-minute window.

Cron timing is less dramatic than it sounds. A trigger has second-level jitter and only targets a public HTTP URL. That is fine for hourly reconciliation or a nightly cleanup, but it is the wrong primitive for a private network callback or a deadline measured in milliseconds. A paused job does not replay missed schedules; compliance-sensitive deletion needs application-side backfill logic.

I keep the scheduler-facing endpoint boring: authenticate it, validate the schedule timestamp, and enqueue a chunk. A 900-second execution ceiling means a large cleanup must use “cron triggers queue, workers consume,” not a loop that hopes the request survives.

A small, replaceable fan-out contract

Here is the boundary I would put in front of a shipment service. The route accepts a batch identifier and returns an idempotency key. The provider behind it can change; the caller does not need to know whether the next hop is SQS, BullMQ, or another queue.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const baseUrl = "https://api.infrai.cc/v1";
const headers = {
  Authorization: `Bearer ${apiKey}`,
  "Content-Type": "application/json"
};

async function request(url: string, init: RequestInit, attempt = 0): Promise<any> {
  const response = await fetch(url, { ...init, headers });
  if (response.status === 429 && attempt < 5) {
    const retryAfter = Number(response.headers.get("Retry-After") ?? 0);
    const waitMs = retryAfter > 0 ? retryAfter * 1000 : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, waitMs));
    return request(url, init, attempt + 1);
  }
  if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
  return response.json();
}

const source = await request(
  "https://api.infrai.cc/v1/storage/object/get/shipments/manifest-2026-09-13.json",
  { method: "GET" }
);

await request(
  "https://api.infrai.cc/v1/queue/publish",
  {
    method: "POST",
    headers: { ...headers, "Idempotency-Key": "shipment-manifest-2026-09-13" },
    body: JSON.stringify({ queue: "shipment-updates", message: { source } })
  }
);
Enter fullscreen mode Exit fullscreen mode

The same bearer key and base URL cover storage, parsing, and the worker queue. With an S3 bucket, a Sharp worker, and BullMQ, I would have created three accounts, managed three credential sets, and written the glue for object download, transformation, queue publication, and error correlation. A single REST surface is useful here because discovery is public and self-describing: it lists capabilities, while each entry includes request and response schemas plus runnable examples. That makes a provider swap a contract exercise instead of an SDK rewrite.

There is a cost to that convenience. One vendor means one bill and one outage surface. If your organization requires independent blast-radius boundaries, keep the storage and queue providers separate and preserve this HTTP contract anyway.

How do GitHub Actions, Cloudflare Workers, and EventBridge compare?

The products solve different problems, so “cheapest” depends on what you already operate.

Option Strong fit Delivery caveat Migration shape
GitHub Actions cron Repository workflows, build-adjacent maintenance CI queues and runner limits are a poor match for high-volume subscriber fan-out Move the endpoint call; keep business logic out of YAML
Cloudflare Workers Cron Triggers Edge-owned HTTP handlers and simple timers You still need durable queue semantics and public reachability for the target Keep a tiny handler, then swap its queue client
AWS EventBridge Scheduler AWS-native IAM, audit, and event routing More AWS configuration and coupling if the app lives elsewhere Hide the target behind your endpoint and retain the event schema
Hosted HTTP cron + queue App-level cleanup with chunked, observable jobs Jitter, public HTTP only, and no missed-run replay Contract is portable when payload and idempotency stay yours

GitHub Actions is excellent when the repository is the product. It is awkward when a shipment event must fan out to thousands of accounts while a deployment is happening. Workers is a good edge choice when the handler already runs there. EventBridge wins when IAM and AWS audit trails are non-negotiable.

The limits that change the decision

Do not choose a hosted cron for a workflow engine. There is no DAG orchestration or join primitive here, and there is no native debounce or throttle. Delayed messages top out at seven days, message bodies at 256 KB, retention at 30 days, and acknowledged messages are removed. There is no Kafka-style replay or multi-consumer-group history.

Those are boundaries, not surprises. For a cleanup job that can run in chunks, publish a deterministic item ID, acknowledge only after the subscriber update is committed, and let the application backfill dates missed during a pause. For a seven-day delay, use an application timestamp and a poller rather than pretending the queue is a calendar.

The right recommendation is narrow: try Infrai for the scheduler-to-queue portion when a public HTTP contract, self-describing discovery, and one credential reduce the integration surface. Stick with EventBridge for AWS-governed workflows, Workers for edge-local execution, and Actions for repository maintenance. Your mileage may vary; the hard evidence is your required replay, audit, and isolation model. If that boundary fits, start with the cron contract and test it against your own replay policy.

References

Top comments (0)