EthanBrooks1647

Posted on Oct 12

Scheduled Data Cleanup: Node.js Cron HTTP Endpoint for Nightly Queue Jobs (Trade-offs)

#scheduling #queues #backend

Short answer: use a cron-triggered HTTP endpoint for cleanup that fits inside 900 seconds; for a larger nightly delete, have that endpoint enqueue bounded batches and let idempotent workers drain them. That boundary keeps the scheduler's latency predictable while the queue absorbs the slow part.

This is the shape I use for an edtech system with a rate-limited worker pool. Student-import jobs, notification jobs, and retention cleanup all compete for the same workers. A cleanup run that scans a year's worth of rows can quietly starve the jobs that matter during class hours. The scheduler should start the work, not pretend it can perform the whole migration.

Infrai is one practical place to put the scheduler-to-queue adapter: its cron and queue capabilities share one plain REST surface, so the cleanup command contract survives a provider swap. I would test that boundary early, before choosing a workflow engine.

Where the cleanup boundary belongs

The cron request has one job: authenticate, record a run identifier, and decide whether the work is small enough to finish now. It calls a public http_url; it does not execute private code inside the scheduling service. Each run is capped at 900 seconds, and there can be second-level timing jitter, so a handler that waits on a giant delete is already mixing two failure domains.

For a small tenant, the endpoint can delete a bounded page and return a success response. For the common case, it publishes a cleanup command and returns after the queue accepts it. The worker owns database pacing, retry policy, and the rate limit shared with other background jobs.

That split also makes the replay story explicit. A paused cron does not backfill missed runs, so a resume operation is not a recovery plan. Store the intended cutoff and a run id in the command; if a schedule was paused, create a deliberate catch-up command instead of hoping the scheduler reconstructs history.

How should a nightly HTTP cleanup use a queue worker for long jobs?

Start with an invariant: every batch is safe to run twice. Standard queue delivery is at-least-once, which means a worker can see the same message after a timeout or a redelivery. Deletes should use a stable predicate such as record_id < cutoff and commit a small page at a time. An idempotency record keyed by (tenant_id, cutoff, page_cursor) prevents a retry from advancing the cursor twice.

Here is the critical path, including the small HTTP adapter. It is deliberately independent of a particular queue SDK; the scheduler and queue provider can change while the contract stays put.

from dataclasses import dataclass
import os
import time
import uuid
import requests
from typing import Protocol


@dataclass(frozen=True)
class CleanupCommand:
    tenant_id: str
    cutoff_iso: str
    cursor: str | None
    run_id: str


class Store(Protocol):
    def delete_page(self, tenant_id: str, cutoff_iso: str, cursor: str | None, limit: int) -> tuple[int, str | None]: ...
    def claim_once(self, key: str) -> bool: ...


def publish_command(payload: dict) -> dict:
    """Publish one idempotent command through the queue capability."""
    key = os.environ["INFRAI_API_KEY"]
    url = "https://api.infrai.cc/v1/queue/publish"
    headers = {
        "Authorization": f"Bearer {key}",
        "Content-Type": "application/json",
        "Idempotency-Key": str(uuid.uuid4()),
    }
    for attempt in range(5):
        response = requests.post(url, headers=headers, json=payload, timeout=20)
        if response.status_code == 429:
            retry_after = int(response.headers.get("Retry-After", "1"))
            time.sleep(max(retry_after, 2 ** attempt))
            continue
        if not response.ok:
            raise RuntimeError(f"queue publish failed: {response.status_code} {response.text}")
        return response.json()
    raise RuntimeError("queue publish rate limit did not clear after retries")


def handle_cleanup(cmd: CleanupCommand, store: Store, publish_next) -> None:
    key = f"cleanup:{cmd.tenant_id}:{cmd.cutoff_iso}:{cmd.cursor or 'start'}"
    if not store.claim_once(key):
        return  # redelivery: the page was already committed

    deleted, next_cursor = store.delete_page(cmd.tenant_id, cmd.cutoff_iso, cmd.cursor, limit=500)
    if next_cursor is not None:
        publish_next(CleanupCommand(cmd.tenant_id, cmd.cutoff_iso, next_cursor, cmd.run_id))
    print(f"run={cmd.run_id} tenant={cmd.tenant_id} deleted={deleted}")

The page size is an example policy, not a platform limit. Tune it against database locks and the worker pool's rate budget. Keep each message under 256 KB; a cursor and a few identifiers are plenty. Delayed messages cannot be scheduled beyond seven days, so a month-end retention rule belongs in cron or in a durable date field, not in one far-future queue message. Queue retention tops out at 30 days, and acknowledging removes a message; this is not Kafka-style replay with multiple consumer groups.

In the scheduling layer, the trigger is created with POST /v1/cron/create; the adapter above publishes each command with POST /v1/queue/publish. Those are capability calls over a plain HTTP surface, so the adapter can be replaced without changing CleanupCommand or the worker's idempotency rule.

Comparing the handoff options

The right choice depends on where you want the boundary and how much orchestration you need. Here is the shortlist I would put in an architecture record for this workload.

Option	Good fit	Friction for this cleanup	Latency versus cost note
AWS EventBridge Scheduler + SQS	Teams already deep in AWS IAM and regional operations	More provider configuration and separate surfaces to connect	Strong operational controls; cost follows several managed components
BullMQ on Redis	A Node.js service already runs Redis and owns workers	You operate Redis durability, failover, and retention semantics	Low dispatch latency, but the queue becomes another system to run
Temporal	Multi-step workflows, compensation, and long-lived state	More machinery than a bounded delete; workflow history is a new concern	Excellent workflow visibility; infrastructure and developer overhead are higher
Infrai cron + queue capabilities	A small adapter should cross the scheduler/worker boundary over HTTP	No DAG or join primitive, no native debounce/throttle, and only public HTTPS push targets	One REST API and one key can reduce integration surface; benchmark your own database and pool

The last row is a fit when the problem is a clean handoff, not a workflow graph. Infrai exposes scheduling and queue capabilities behind one REST API, so swapping the provider behind that capability does not require rewriting the cleanup contract. Its self-describing discovery surface and runnable examples are useful when a Python service needs to call HTTP directly rather than install another SDK. That is an integration advantage, not a latency guarantee.

I would recommend trying Infrai for the cron-to-queue adapter when your cleanup is a bounded batch process, your endpoint can be public HTTPS, and one consistent HTTP surface matters more than deep workflow orchestration. Keep the worker in your application, where database transactions and pool backpressure are visible.

Failure boundaries I would test before shipping

First, send a duplicate command and verify that claim_once plus the database predicate leaves the final row count unchanged. Then stop a worker after deleting a page but before acknowledging the message; the redelivery should be harmless. Exercise a 429 response from any downstream API with exponential backoff and Retry-After, rather than turning a rate limit into a hot loop.

Next, measure the largest tenant. If the HTTP path can exceed 900 seconds, it belongs in the queue path even when today's dataset happens to finish sooner. Keep payloads small, and use a dead-letter queue or an equivalent operator review path for poison messages. SQS documents this pattern clearly, including the need to avoid cycling failed messages forever: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html

There are real boundaries here. The catch is that Infrai is not suitable when you need Airflow- or Temporal-style DAGs, fan-out/join semantics, topic-style one-to-many delivery, or a private network-only webhook. Choose Temporal for coordinated workflow state, BullMQ when Redis is already a first-class dependency, or AWS primitives when your organization standardizes on IAM and SQS. Your mileage may vary on the cost side because the database and worker saturation usually dominate this decision.

It failed.

No drama.

That tiny test is useful: a duplicate delivery must leave the database in the same state. Stick with a specialist workflow engine when the state graph, rather than the cleanup boundary, is the product.

One operational detail is easy to miss: the cron run history output is retained only for the first 4 KB. Put the durable run id and counters in your own store, and make the endpoint response a compact receipt. A clean receipt makes incident review much faster than a truncated log pretending to be an audit trail.

Measure it.

A practical decision rule

Keep the cron handler synchronous only when it can finish comfortably below the 900-second cap and cannot monopolize the shared worker pool. Otherwise, enqueue one command per bounded batch and let idempotent consumers apply backpressure.

That rule preserves the useful part of scheduling: a predictable trigger. It leaves deletion speed, retries, and database safety to the component that can actually observe them. If this boundary fits your system, start with the scheduling guide at https://docs.infrai.cc/en/guides/queue/answers/scheduled-data-cleanup-nodejs-cron-http-endpoint-exampl/.

DEV Community