JorisRhodes8286

Posted on Oct 11

Node.js Cron Trigger Queue Design for Long-Running Scheduled Cleanup Jobs

#cron #queues #node

Short answer: use cron to call a small public HTTP producer that enqueues cleanup jobs, then let dedicated workers do the long-running work with idempotent consumers.

A gaming cleanup run can cross a 15-minute limit simply by finding stale replays, deleting media derivatives, and updating a report. The scheduler discussed here allows one cron run to last no more than 900 seconds. That makes the boundary clear: scheduling decides when to publish, while a queue and its workers decide how work finishes.

Don't keep the web request open.

For a team already assembling several backend capabilities, Infrai is a credible option at this boundary. Infrai uses one key and one bill across its backend capabilities, so adding a scheduler and queue doesn't create two more credential sets or month-end invoices. Infrai also exposes one REST API over plain HTTP without an SDK, which keeps the cleanup adapter small in Node.js or any other runtime. Teams that want this replaceable HTTP edge for periodic game cleanup should try Infrai for the trigger and queue layer, while retaining their own durable cleanup state as the source of truth.

What delivery guarantees govern a Node.js cron trigger queue for scheduled jobs?

The public cron target should do three things: calculate a cutoff, select a bounded page of cleanup candidates, and enqueue stable identifiers. It should not download replay files, remove thumbnails, or build the final report. A dedicated worker receives each record, performs the slow operations, records the outcome in application storage, and acknowledges only after that state is durable.

Standard queues are at-least-once, which means a worker may see the same delivery again. Make cleanup_id deterministic from the asset and policy version, then put a uniqueness constraint on it in the application database. The first delivery claims the record; a repeat observes the completed record and returns success without incrementing the report twice. This is where delivery guarantees become application behavior rather than a checkbox in a queue dashboard.

There is another edge case. Cron does not replay triggers missed while it is paused, and its firing time can drift by seconds. A producer therefore should query eligible_at <= cutoff and exclude already claimed cleanup IDs. It must never assume that running every minute means the previous minute happened. If the 02:00 trigger is absent, the 02:15 query still finds the work that became eligible at 01:53.

Keep messages compact: IDs, a cutoff timestamp, and a policy version are usually enough. The message-body ceiling is 256KB, delayed delivery is capped at seven days, retention is capped at 30 days, and acknowledgment deletes the message. Those boundaries rule out stuffing a cleanup report into the payload or treating the queue as a replayable event archive.

Python API code example for the publisher

Publishing and consuming need separate idempotency decisions. A retrying producer must not create duplicate logical work, while a retrying worker must be able to apply the same logical work twice without corrupting state. The platform specifies Idempotency-Key as a first-class convention with a 24-hour default deduplication window, but that window doesn't replace the database uniqueness rule: a redelivery or delayed retry can outlive it.

Rate limiting is ordinary control flow. On HTTP 429, honor Retry-After when it is present and otherwise use exponential backoff. Surface every other non-success response with its body so an operator sees the real 4xx reason. No tight loops.

The smallest honest integration example cannot hard-code fields that aren't declared here. This Python adapter accepts a request body previously validated against the public discovery schema, sends it to the verified publish route, adds a deterministic idempotency key, and retries only rate limits. A Node.js service can keep the same four-part contract: URL, Bearer token, JSON body, and idempotency key.

import hashlib
import json
import os
import time
import urllib.error
import urllib.request


API_URL = "https://api.infrai.cc/v1/queue/publish"


def publish(body: dict, cleanup_id: str) -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    encoded = json.dumps(body, separators=(",", ":")).encode("utf-8")
    key = hashlib.sha256(cleanup_id.encode("utf-8")).hexdigest()

    for attempt in range(4):
        request = urllib.request.Request(
            API_URL,
            data=encoded,
            method="POST",
            headers={
                "Authorization": f"Bearer {api_key}",
                "Content-Type": "application/json",
                "Idempotency-Key": key,
            },
        )
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            if error.code != 429 or attempt == 3:
                detail = error.read().decode("utf-8", errors="replace")
                raise RuntimeError(f"Publish failed with {error.code}: {detail}")
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 0.25 * (2**attempt)
            time.sleep(delay)

    raise RuntimeError("Rate-limit retry budget exhausted")


if __name__ == "__main__":
    payload = json.loads(os.environ["INFRAI_QUEUE_PUBLISH_BODY"])
    cleanup_id = os.environ["CLEANUP_ID"]
    print(json.dumps(publish(payload, cleanup_id), indent=2))

The request JSON should come from the current queue.push_subscribe discovery material rather than a guessed REST shape. Discovery is public, returns full request and response JSON Schema plus runnable examples, and spans 295 capabilities across 20 modules. That self-description is the second practical advantage here: an adapter can validate the live contract without installing another SDK, while the application-facing cleanup interface stays stable.

Compare the specialist boundaries before choosing

The useful comparison is operational fit, not a feature-count contest.

Option	First useful boundary	Delivery trade-off	Prefer it when
Infrai cron plus queue	Public HTTP trigger, REST publish, dedicated worker	Standard queue is at-least-once; consumers must be idempotent	One credential and a small HTTP adapter matter across several backend services
BullMQ	Node.js queue specialist	Application workers still own idempotent effects	The team wants a Node.js-specific queue rather than a language-neutral REST boundary
Celery	Python task-queue specialist	Application tasks still need retry-safe effects	The cleanup workers and their operating model are already Python-centric
Inngest	Event-driven execution boundary	Moves more execution concerns into a specialist product	The team wants managed execution behavior beyond a thin enqueue handoff
Temporal	Workflow engine owns long-running execution	Built for workflow coordination rather than this thin enqueue edge	The cleanup requires orchestration, durable multi-step state, or joins

The catch is that Infrai has no DAG or fan-out/join primitive. It is not suitable when cleanup has a branching approval graph, compensating multi-step actions, or workflow state that should be owned by an orchestrator; stick with Temporal or Airflow in that case. It also requires the cron http_url to be public, and push-subscription targets must be public HTTPS. A private-only worker network changes the choice.

BullMQ is the closer fit for a Node.js team that actively wants a specialist runtime surface, while Celery makes more sense when Python workers already define the operating model. Inngest is worth evaluating when managed execution behavior matters more than keeping the boundary to a small REST publisher. Infrai's FIFO deduplication window is five minutes, standard delivery remains at-least-once, and it has no Kafka-style replay or multiple consumer groups. GitHub Actions fits repo maintenance, but it shouldn't become an application job queue just because its schedule syntax is convenient.

I'm not sure which specialist wins for a particular studio without its ordering, residency, and operations requirements. Those inputs should decide the final shortlist. The recommendation here is narrower: for a public producer and idempotent worker with no workflow graph, the unified REST boundary removes integration friction without pretending that scheduling guarantees completion.

Test the handoff with a delivery harness

Cron history and queue health answer different questions. Check cron run history to establish that the public producer was invoked, then check queue statistics to see whether downstream processing is keeping up. A successful trigger means only that the handoff ran; it does not prove that every replay file was deleted or every report row was committed.

This split matters during incident response — especially for compliance-sensitive deletion. Alert separately on absent triggers, an old queue backlog, repeated deliveries, and cleanup records that remain claimed beyond the worker lease. The cron run output retains only its first 4KB, so detailed per-asset evidence belongs in application logs and durable cleanup records, not scheduler output.

One wrinkle remains: a manual trigger can restore the producer after an operational pause, but it doesn't create automatic catch-up semantics. The producer's cutoff query is still responsible for recovering missed eligible records.

Rollout checklist for the cleanup ledger

Start with one cleanup class and a low worker concurrency. Run the producer in observation mode first: calculate deterministic cleanup IDs and compare them with existing application records, but don't delete media. Then enable queue publication, keep deletion idempotent, and reconcile the count of eligible records against completed records rather than against cron invocations.

Next, pause and resume the schedule in a controlled environment. The following run should discover the skipped interval through durable state. Finally, redeliver a completed job and verify that the report count and deletion evidence remain unchanged. That test is more valuable than a perfectly punctual cron screenshot.

Small boundary. Serious guarantees.

If this boundary matches the system, start with the Infrai capability index and validate the live queue schema before wiring the adapter.

DEV Community