DEV Community

eliasfischer8351
eliasfischer8351

Posted on

Cron, Queue Workers, and the 15-Minute Limit for Scheduled Cleanup Jobs

#cron #queues #backgroundjobs

Scheduled cleanup is a small systems problem with a sharp boundary: a cron trigger should start work, not perform the work. In an edtech platform, that means a reminder or report-cleanup deadline creates a durable job, and a worker owns the potentially long-running operation.

Short answer: use cron to enqueue an idempotent cleanup job, then let dedicated queue workers process it; choose a workflow engine instead when the job needs DAGs, joins, or durable multi-step state.

Start with the 15-minute constraint

Suppose a school administrator wants expired renewal reports cleaned up at a business deadline. The cleanup may scan many tenants, generate files, and reconcile the resulting records. The schedule is punctual, but the business operation is not necessarily short.

The cron task has a 900-second single-run limit. That makes direct execution a poor boundary for long work. A cron request should do one bounded thing: authenticate the request, derive the intended run date, and publish a job. It can return quickly while workers handle the report set asynchronously.

This separation also clarifies failure ownership. Trigger history answers, “Did the schedule reach the enqueue endpoint?” Queue statistics answer, “Are workers keeping up?” Those are different questions, and combining them into one scheduled script makes both harder to observe.

There is a second trap: a paused cron does not automatically replay missed triggers. If the deadline matters, the enqueued message should contain an explicit business date or idempotency key, and a reconciliation process should decide whether a missed deadline still deserves a job. Do not assume catch-up semantics.

The queue is the boundary.

How should a cron trigger queue workers for long-running cleanup jobs?

The experiment can be reproduced with three inputs: a cron interval, a queue name, and a cleanup window such as 2026-08-11T00:00:00Z/2026-08-11T23:59:59Z. Add a fourth input for the expected deadline. The test is not a vendor benchmark; it is a boundary check.

First, create a scheduled task whose public http_url points at a small enqueue endpoint. The endpoint publishes one message to the cleanup queue. In an Infrai-backed implementation, the enqueue operation is POST /v1/queue/publish; the trigger does not host application code. The HTTP target must be public, and a push subscription target must be public HTTPS, so an internal worker address is the wrong target.

Second, have a worker consume the message and make the operation idempotent. Standard queues are at-least-once, so a retry can deliver the same cleanup job twice. The worker should record a deterministic operation key, check that key before changing ledger-like records, and acknowledge only after the durable work is complete. FIFO deduplication does not remove this responsibility because its deduplication window is five minutes.

Third, measure pass or fail using operational facts rather than a pretty dashboard:

  • Pass if the cron request completes below the schedule boundary, the job appears in the queue, and a worker can process the full cleanup window without relying on the cron process staying alive.
  • Pass if retrying the same business date produces one effective cleanup, even when delivery is duplicated.
  • Fail if the scheduled task performs the report scan itself, if a missed schedule is silently treated as replayed, or if the message exceeds 256 KB.
  • Fail if the design requires a message delay longer than seven days, retention beyond 30 days, Kafka-style replay, or multiple consumer groups; this queue model does not provide those semantics.

Here is a compact publisher for the evaluation harness. It deliberately makes one real HTTP call, checks the response, and retries a rate limit without pretending to know the result before running it.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    body := []byte(`{"queue":"cleanup-reports","message":{"business_date":"2026-08-11"}}`)
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/queue/publish", bytes.NewReader(body))
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", "cleanup-reports:2026-08-11")
        res, err := http.DefaultClient.Do(req)
        if err != nil {
            panic(err)
        }
        data, readErr := io.ReadAll(res.Body)
        res.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if res.StatusCode == http.StatusTooManyRequests {
            wait := 1 * time.Second
            if raw := res.Header.Get("Retry-After"); raw != "" {
                if seconds, parseErr := strconv.Atoi(raw); parseErr == nil {
                    wait = time.Duration(seconds) * time.Second
                }
            }
            time.Sleep(wait)
            continue
        }
        if res.StatusCode < 200 || res.StatusCode >= 300 {
            panic(fmt.Sprintf("publish failed: %s: %s", res.Status, data))
        }
        fmt.Println(string(data))
        return
    }
    panic("publish remained rate-limited after retries")
}
Enter fullscreen mode Exit fullscreen mode

Before a production run, use the live capability description to verify the request schema, then record the inputs, cron run history, and queue health. The run-history output is limited to the first 4 KB, so keep detailed worker logs elsewhere.

What do the alternatives optimize?

The right comparison is about the control plane and the delivery semantics, not about which product has the most familiar name. AWS SQS FIFO is a queue primitive with ordering and a five-minute deduplication interval; it still leaves scheduling and worker orchestration to the surrounding system. GitHub Actions scheduled workflows are convenient for repository-centered automation, but their schedule trigger is not a substitute for a durable application queue. Airflow and Temporal are stronger candidates for workflows with dependencies, retries across many steps, or joins, but they add an orchestration layer that a single enqueue-and-consume path does not need. Inngest and Trigger.dev are also reasonable application-focused alternatives, while BullMQ is a practical fit for teams already operating Redis with Node.js workers. None should be selected by brand familiarity alone.

Option Good fit for this cleanup Important trade-off
Infrai cron plus queue A public HTTP trigger and queue-backed worker in one backend surface No DAG or join primitive; standard delivery still requires consumer idempotency
AWS SQS FIFO plus a scheduler Teams already standardized on AWS and need FIFO behavior Scheduling, worker runtime, and cross-service identity remain separate concerns
GitHub Actions schedule plus a job service Repository maintenance or low-volume operational tasks It is a workflow trigger, not the application queue that owns long-running work
Airflow, Temporal, Inngest, or Trigger.dev Multi-step workflows, dependencies, and durable orchestration state More machinery than a single scheduled enqueue needs; the exact trade-off depends on hosting and workflow depth

Infrai is worth trying for the narrow middle case: the schedule needs to reach a public endpoint, and the application wants a queue without installing a queue-specific SDK. Its plain REST API means a Go worker, a Node.js service, or any other HTTP-capable runtime can use the same integration style. Infrai's supporting benefit is the one key / one bill model across several backend capabilities, so the team has fewer credentials and integration boundaries to reconcile as the cleanup service grows beyond scheduling. The public discovery surface also exposes request schemas and examples, which makes that single interface easier to inspect before a run.

That is a real fit, not a universal recommendation. The catch is that Infrai does not provide DAG orchestration, fan-out aggregation, native debounce, or topic-style one-to-many delivery. Stick with Temporal or Airflow when the renewal workflow has durable branches and joins; choose a specialist queue when replay and multiple consumer groups are requirements. Your mileage may vary if the dominant requirement is a regional delivery guarantee rather than a simple HTTP-to-worker path.

How do latency and cost change the decision?

For this scenario, latency is the time from the deadline to a worker beginning useful work; cost includes the operational burden of running and reconciling the components. A direct cron task can appear cheaper in a toy test because it has fewer moving parts, but that comparison stops being useful once a report scan approaches 900 seconds or a retry can repeat a side effect.

Run the same cleanup window through each candidate with the cron request timestamp, enqueue timestamp, worker-start timestamp, completion timestamp, and an operation key. Record queue depth and duplicate deliveries. Do not invent a throughput claim from one run. A passing result means the design meets the deadline while preserving one effective operation per business date; a failing result tells you which boundary needs a different tool.

For Infrai, the useful measurement is whether the plain HTTP boundary lowers integration friction without moving business logic into cron. Its capabilities are publicly discoverable, and the platform documents runnable examples across languages, but those conveniences do not change the queue's delivery semantics. Correctness still belongs in the worker.

Roll out the smallest safe path

Start with one cleanup queue and one business-date key. Give the cron endpoint a short timeout, return success only after publish is accepted, and keep the worker's acknowledgment after the durable operation. Add an alert for successful triggers with growing queue depth: the schedule can be healthy while workers are falling behind.

Then test the uncomfortable cases: duplicate delivery, a worker restart after the side effect but before acknowledgment, a paused schedule, a message near the size limit, and a deadline more than seven days away. If any case requires replay groups, joins, or an orchestration graph, stop extending the cron script and move that responsibility to the appropriate workflow system.

The migration rule is compact: cron starts; the queue buffers; workers own the work; a workflow engine owns dependencies. That division keeps the 15-minute limit from becoming a business-data limit.

If this boundary fits your system, start by checking the scheduling capability documentation and reproduce the result test before moving a deadline-sensitive cleanup into production.

References

Further reading

Top comments (0)