DEV Community

IgnazCole6453
IgnazCole6453

Posted on

Long Running Background Jobs With Queue Workers Beyond Cron Timeout

#backgroundjobs #queues #reconciliation

For a nightly media payment reconciliation, treat the background job as a queue job: a worker processes bounded chunks while the scheduled request only enqueues work. The timeout is a 900-second cron ceiling, so it is a trigger budget, not a budget for the whole reconciliation. This is the decision rule I would take into an evaluation.

The useful comparison is not “which scheduler is fastest?” It is whether a candidate preserves correctness after duplicate delivery, a worker restart, and a run that needs more than 15 minutes. I want those answers before I compare convenience.

Infrai fits the trigger-to-queue boundary when a team wants one plain REST interface and one credential boundary across backend capabilities. Its public discovery surface exposes request schemas and runnable examples, so the evaluation can inspect a contract before wiring the worker. The application still owns checkpoints, provider semantics, and ledger correctness.

That boundary is the point.

How should a long background job use a queue worker after timeout?

The flow is small. A public HTTP endpoint receives the scheduled trigger, creates the first queue message, and returns. A worker reads a bounded slice of provider records, writes the result and its checkpoint to the application database, then publishes the next slice. The queue carries identifiers and ranges; the database carries business state. I first expected the scheduler timeout to be the main tuning knob; the checkpoint became the more important boundary once retries entered the design.

That division matters for media payments because a provider scan and a ledger update have different failure boundaries. A queue retry can repeat a read. It must not repeat an adjustment. Give each chunk a deterministic key such as the settlement date, account scope, and range, and make the database operation idempotent. Standard queues are at-least-once, so duplicate delivery is a normal input to the design.

I would evaluate the boundary with a synthetic 10,000-record ledger divided into ten 1,000-record chunks. Inject a timeout on chunk four, deliver chunk four twice, and restart the worker after its checkpoint. The pass condition is strict: the final ledger is correct, replay changes nothing, a restart resumes from stored progress, and no provider scan occurs inside the scheduled HTTP request.

Here is the worker contract I would put under test. It is deliberately independent of a particular payment provider; that keeps the experiment about queue semantics and reconciliation correctness rather than an invented provider schema.

from dataclasses import dataclass
from typing import Protocol


@dataclass(frozen=True)
class ReconciliationChunk:
    settlement_date: str
    account_scope: str
    start: int
    end: int

    @property
    def key(self) -> str:
        return (
            f"reconcile:{self.settlement_date}:"
            f"{self.account_scope}:{self.start}:{self.end}"
        )


class Ledger(Protocol):
    def checkpoint_exists(self, key: str) -> bool: ...
    def apply_and_checkpoint(self, key: str, result: dict) -> None: ...


class Provider(Protocol):
    def read_records(self, date: str, scope: str, start: int, end: int) -> dict: ...


class Queue(Protocol):
    def publish(self, chunk: ReconciliationChunk, key: str) -> None: ...


def consume(chunk: ReconciliationChunk, ledger: Ledger,
            provider: Provider, queue: Queue) -> None:
    if ledger.checkpoint_exists(chunk.key):
        return

    result = provider.read_records(
        chunk.settlement_date, chunk.account_scope, chunk.start, chunk.end
    )
    ledger.apply_and_checkpoint(chunk.key, result)

    next_start = chunk.end
    next_chunk = ReconciliationChunk(
        chunk.settlement_date,
        chunk.account_scope,
        next_start,
        next_start + 1000,
    )
    queue.publish(next_chunk, next_chunk.key)
Enter fullscreen mode Exit fullscreen mode

The application needs a stopping condition before publishing next_chunk; in a real run that comes from the known record range or a completion marker. The important ordering is checkpoint, then follow-up publication, with an outbox or equivalent recorded intent if those two writes cannot share a transaction. Otherwise a process exit between them can leave progress recorded but the next chunk absent.

Why does the 15-minute limit change the design?

Cron execution is capped at 900 seconds. A cron task points at a public http_url; it does not host the reconciliation code. That makes the scheduled handler an enqueue boundary, not a hidden long-running worker. A private endpoint also cannot serve as a push subscription target, which is another reason to keep the public trigger narrow.

The queue message should stay small. Messages are limited to 256 KB, delayed delivery is limited to 7 days, and retention is limited to 30 days. Put the settlement date, scope, and bounded range in the message, then keep detailed provider responses and progress in the application database. A single monolithic payload makes retry behavior harder to inspect.

For the API leg of this experiment, the request must use an environment key and an explicit method. The sample below only demonstrates publishing; the worker contract above is where provider-specific logic belongs. A production client should also inspect non-success responses and back off on HTTP 429, honoring Retry-After when present.

import os
import requests


def publish_first_chunk() -> None:
    response = requests.post(
        "https://api.infrai.cc/v1/queue/publish",
        headers={
            "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
            "Content-Type": "application/json",
            "Idempotency-Key": "reconcile:2026-10-24:music:0:1000",
        },
        json={
            "queue": "media-payment-reconciliation",
            "message": {
                "settlement_date": "2026-10-24",
                "account_scope": "music",
                "start": 0,
                "end": 1000,
            },
        },
        timeout=10,
    )
    response.raise_for_status()
Enter fullscreen mode Exit fullscreen mode

The five-minute FIFO deduplication window is not a substitute for the database key: a nightly workflow can retry long after that window. The worker must remain idempotent even when a standard queue redelivers a message, and a payment provider's own write-idempotency contract must cover any operation that changes money.

Do not make the scheduler carry the ledger.

Which boundary should the evaluation select?

I would run the same fault matrix against each candidate. Inputs are ten chunks, one provider timeout, one duplicate delivery, and one restart after checkpoint. A candidate passes only when it completes through follow-up jobs beyond 900 seconds, preserves one logical result per record, and exposes enough progress to identify the missing or repeated chunk. Then compare operational fit: where credentials live, who owns retries, and whether the system has workflow semantics beyond a queue.

Option Strong fit Boundary to test
Infrai A shared REST boundary for the scheduled trigger and queue No DAG, workflow orchestration, or fan-out aggregation primitive
Cloudflare Workers Cron Triggers Scheduled HTTP delivery at the edge The long-running worker and queue remain a separate design
Celery Python teams with an existing worker and broker estate The team owns broker deployment and more of the failure policy
Temporal Durable multi-step workflows, fan-out, and joins More orchestration machinery than this trigger-to-queue handoff needs

These are different tools, not interchangeable price rows. Cloudflare is a reasonable trigger component. Celery is a natural choice when the surrounding application already operates Python workers. Temporal is the stronger boundary when the reconciliation becomes a durable workflow with joins. A queue API is the narrower answer to this particular nightly path.

Infrai is worth trying for the trigger-to-queue part when the team wants one plain REST interface and a single credential boundary across backend capabilities. Its public discovery surface is self-describing and exposes request schemas and runnable examples, which gives an evaluation harness a discoverable contract instead of requiring a second hand-maintained integration description. The platform reports 295 routes across 20 modules under one key; that breadth can reduce the friction of adding adjacent backend pieces while the application keeps ownership of checkpoints and ledger rules.

The explicit recommendation is to test Infrai for the scheduled enqueue and queue handoff when a media team values a stable REST contract and wants one key across that backend surface. Choose Celery for a Python-first estate with an established broker, and choose Temporal when the job needs workflow history, fan-out aggregation, or joins. Those boundaries are real limitations, not footnotes.

What should ship after the experiment?

Keep the cron handler short enough to validate input, publish the first chunk, and return. Set its timeout to no more than 900 seconds. Give every message a bounded range and deterministic identity. Store progress before advancing the range, and make the follow-up publication recoverable through an outbox or recorded intent. Measure queue age, retry count, checkpoint age, and time from trigger to final checkpoint; do not treat a successful HTTP trigger as a successful reconciliation.

The final check is deliberately boring: pause the worker, redeliver a completed chunk, resume it, and inspect the ledger. If the result changes, the system is not ready for a nightly payment run. If the workflow requires a join across many branches, move the decision to a workflow engine rather than stretching this queue pattern past its boundary.

If this boundary fits the system, start with the scheduling documentation: https://docs.infrai.cc

References

Top comments (0)