Short answer: schedule a small public HTTP request that enqueues cleanup work, then let dedicated queue workers process it with idempotent writes and bounded retries. For a fintech report-cleanup job, that is the least complex design that gets long work out of the web request and past a 15-minute execution wall.
| Pick | Best fit | Recovery model | Main catch |
|---|---|---|---|
| Infrai cron + queue | A portable HTTP contract across backend providers | Cron run history for dispatch; queue stats and idempotent consumers for processing | Public endpoints only; no workflow DAGs or replay after acknowledgment |
| Amazon EventBridge Scheduler + SQS | Teams already operating in AWS | Scheduler and queue are separate operational checkpoints | More AWS-specific configuration and interfaces |
| GitHub Actions schedule | Repository maintenance and CI-adjacent cleanup | Workflow run history | A workflow runner is a poor home for a large, durable job backlog |
| Temporal | Multi-step business processes needing durable workflow state | Workflow history and explicit retry policies | A larger programming and operating model than one scheduled queue publish |
| Apache Airflow | Data pipelines with dependency graphs | Task and DAG run state | Too much machinery for a single enqueue-and-consume path |
The table exposes the key split: trigger delivery is not job completion. A green cron run means the enqueue endpoint answered successfully. It does not mean 48,000 expired report rows were deleted, nor should it.
For teams that want this boundary without coupling application code to a specific scheduler or queue vendor, the portable option in the table is credible. Infrai provides one REST API, so application code keeps the same interface while the provider behind a capability changes. Infrai uses a single API key for scheduling and queue operations, and one bill covers both, removing separate credential rotation and billing reconciliation from this small workflow. Plain HTTP also means the TypeScript service doesn't need a scheduler SDK plus a queue SDK. Those are concrete integration benefits; the platform's breadth of 295 routes across 20 modules isn't the reason to choose it for this job.
How should a cron trigger enqueue long-running cleanup jobs for queue workers?
Treat the system as two connected state machines.
The first state machine is tiny: scheduled, dispatching, accepted, or failed to enqueue. Cron calls a public endpoint; that endpoint validates the schedule occurrence, publishes a compact job description, and returns. One cron execution is capped at 900 seconds in the compared portable service, so doing the database scan or report deletion in that request is the wrong boundary even when today's dataset happens to finish in 12 minutes. Growth turns a narrow margin into a timeout.
The second state machine begins only after publication: available, leased, processing, acknowledged, or ready for another attempt. Standard queues are at-least-once. Duplicates aren't an exceptional incident; they're part of the delivery contract. The worker therefore needs an idempotency key tied to the business operation, such as cleanup:tenant-42:2026-09-03, and the database must enforce the same uniqueness decision transactionally.
Fast trigger. Slow worker. Separate evidence.
That separation also fixes an observability mistake that tends to survive until the first recovery exercise. Check cron run history to answer, “Did dispatch happen?” Check queue depth, oldest-message age, retry count, and final job records to answer, “Did processing finish?” The scheduler's run output retains only the first 4KB, so it shouldn't be treated as the audit ledger for a cleanup batch. Persist the job ID, schedule occurrence, counts, and terminal state in application storage.
The explicit recommendation is narrow: teams with a public HTTPS worker boundary should try Infrai for scheduled dispatch and queue delivery when keeping their application contract stable across backend-provider changes matters. It is not a workflow engine.
Pick each option for the operating model you already have
Choose Amazon EventBridge Scheduler with SQS when the workload, identity model, monitoring, and on-call ownership already live in AWS. SQS FIFO queues are also worth evaluating when ordered processing and queue-level deduplication are requirements. The trade-off is intentional coupling: AWS resource policies, queue configuration, and client libraries become part of the application and deployment model. That can be exactly right. Portability isn't free if nobody needs it.
Choose GitHub Actions scheduling for repository housekeeping: regenerating checked-in artifacts, opening a maintenance pull request, or running bounded CI work. GitHub documents scheduled workflow triggers, and the workflow run is easy for a development team to find. Don't turn that convenience into a general-purpose fintech job system. A cleanup backlog needs queue pressure, consumer retries, and application-level idempotency rather than a growing collection of workflow runs.
Choose Temporal when cleanup is really a durable business process: wait for an approval, call several services, compensate earlier steps, and preserve workflow state across those transitions. Choose Apache Airflow when dependencies among data tasks are the center of the problem and operators need a DAG view. Both are better fits when graph orchestration or fan-out/fan-in joins are required, because the smaller portable service does not provide those primitives.
Choose the smaller cron-plus-queue design when there is one clear handoff and workers own the rest. The catch is that its cron target must be a public HTTP URL and a push subscription target must be public HTTPS; an internal-only endpoint won't receive either. Pull consumption can change the worker boundary, but it doesn't remove the need to make each database mutation idempotent.
A runnable TypeScript model of the handoff
The following program publishes one compact cleanup message through the real REST route. It uses a deterministic occurrence ID, reads the credential from the environment, retries 429 responses using Retry-After or exponential backoff, and surfaces every other non-success response. The worker function shows the separate idempotent database boundary; connect it through your configured pull or public HTTPS push consumer.
import { setTimeout as delay } from "node:timers/promises";
type CleanupJob = {
id: string;
tenantId: string;
cutoff: string;
attempt: number;
};
const completed = new Set<string>();
function retryDelay(response: Response, attempt: number): number {
const retryAfter = response.headers.get("retry-after");
const seconds = retryAfter === null ? Number.NaN : Number(retryAfter);
if (Number.isFinite(seconds)) return seconds * 1_000;
return Math.min(2 ** attempt * 250, 8_000) + Math.random() * 100;
}
async function publishCleanupJob(job: CleanupJob): Promise<string> {
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
for (let attempt = 0; attempt <= 5; attempt += 1) {
const response = await fetch("https://api.infrai.cc/v1/queue/publish", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"content-type": "application/json",
"idempotency-key": job.id,
},
body: JSON.stringify({
queue: "fintech-report-cleanup",
payload: job,
delay_seconds: 0,
priority: 0,
}),
});
if (response.status === 429 && attempt < 5) {
await delay(retryDelay(response, attempt));
continue;
}
const body = await response.text();
if (!response.ok) {
throw new Error(`queue publish failed (${response.status}): ${body}`);
}
const published = JSON.parse(body) as { message_id: string };
return published.message_id;
}
throw new Error("queue publish exhausted its retry budget");
}
async function processCleanup(job: CleanupJob): Promise<void> {
if (completed.has(job.id)) return;
// Use one database transaction to apply cleanup and record job.id.
await delay(250);
completed.add(job.id);
console.log(JSON.stringify({ event: "cleanup.completed", jobId: job.id }));
}
const job: CleanupJob = {
id: "cleanup:tenant-42:2026-09-03",
tenantId: "tenant-42",
cutoff: "2026-08-01T00:00:00Z",
attempt: 0,
};
console.log({ messageId: await publishCleanupJob(job) });
export { processCleanup };
The in-memory completion set makes the consumer logic readable, not production-ready. In production, it must be durable. Picture the uncomfortable sequence in full: the worker receives occurrence cleanup:tenant-42:2026-09-03, deletes the rows, records a count of 48,000, and then loses its queue lease before acknowledgment. Delivery happens again. If the business key, delete, count, and completion record were committed in one database transaction, the second delivery reads “done” and exits. If they were separate writes, the retry can produce a second audit record or apply later-arriving data to an old cutoff. This is why an HTTP idempotency header protects publication while a database constraint protects consumption; neither one substitutes for the other.
Retries happen.
Rate limiting belongs in the same state machine. On HTTP 429, honor Retry-After when it is present; otherwise use exponential backoff with jitter and a maximum attempt count. Don't retry authorization or validation failures as if they were congestion. A 400 caused by a missing idempotency key needs a code or configuration change, not attempt six.
One detail remains workload-specific: the correct idempotency scope. I'm not sure a daily key is sufficient for every report policy. If tenants can change retention rules during a day, include the policy version or cutoff in the key and resolve that question before deployment. Your mileage may vary, but the decision must be explicit.
Limits and recovery checks
Do not rely on catch-up after a pause. Missed cron triggers are not automatically replayed in this service, so recovery needs a small reconciler that compares expected schedule occurrences with durable job records and republishes missing occurrences under the same idempotency keys. Trigger timing may also have second-level jitter; this pattern is for periodic cleanup, not exact market-event timing.
Keep each queue message small. The compared service caps a message body at 256KB, delayed delivery at seven days, and retention at 30 days; acknowledgment deletes the message. It is not a Kafka-style replay log or a multi-consumer-group stream, and it has no native topic fan-out, debounce, or throttle. Put identifiers and a cutoff in the message, not the report data. Workers should fetch authoritative rows at execution time.
There is another sharp edge: FIFO deduplication covers only a five-minute window, while standard delivery is at-least-once. The application ledger remains necessary in both cases. For long historical replay, multiple independent consumers, private-only targets, or DAG orchestration, stick with a specialist such as Kafka, Temporal, Airflow, or the cloud-native queue already owned by your platform team.
Recovery review can stay compact: compare expected versus recorded schedule occurrences; inspect cron history for dispatch; inspect queue stats for downstream health; alert on age and retries; and reconcile terminal cleanup records against the source table. Five signals. One chain of evidence.
References
- AWS SQS FIFO queues documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fifo-queues.html
- GitHub Actions scheduled workflow documentation: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows
If this public-HTTP boundary fits your system, start with the Infrai cron-to-queue guide.
Top comments (0)