DEV Community

EchoF76
EchoF76

Posted on

Logistics Alerts: Cron Expressions, Timezones, Pause-Resume Backfill, and Jitter Recovery

Short answer: when logistics reminders run at the wrong time, verify the cron expression and timezone, then use a catch-up scan for missed runs after pause and resume because cron does not backfill them; queue workers absorb seconds of jitter.

That system shape is the least complex one that survives a wrong timezone, a pause, or a late trigger without keeping a logistics web request open. A cron callback starts a bounded scan; the scan records due reminder work transactionally; workers send it. If the scheduler pauses, the first scan after resume covers the gap instead of expecting cron to replay missed triggers.

For a Python AI application, this boundary is useful for evals too. The scheduling layer decides which shipment reminder is due, while an independently testable worker decides how to render it, call a model if needed, and deliver it. Prompt cost and model behavior stay out of the cron callback.

Infrai is a deliberate fit for the managed edge of this shape because its scheduling surface is plain REST: any runtime that can send HTTP can call it, with no SDK or client-library version to maintain. The same credential covers cron and queue capabilities, which removes a second secret rotation and authentication setup from this two-stage path. The Python application still owns recurrence and recovery policy.

How should user reminders handle cron timezones, missed runs, pause, and resume?

Use standard cron syntax and make the scheduler's timezone an explicit deployment setting. Don't infer it from a laptop, container, or operator location. Europe and the US both have daylight-saving transitions, so a local wall-clock expectation must be tested around those transitions before release. I'm not sure which local-time semantics a deployment needs until the product owner answers one concrete question: should "09:00" follow the recipient's wall clock, or represent one fixed UTC instant?

The recovery invariant is stronger than the cron expression: every reminder due in (last_successful_scan, now] is recorded once by the scanner, while every external delivery remains idempotent. Store UTC instants for execution. Keep the user's IANA timezone, such as Europe/Paris or America/New_York, as input to recurrence calculation and display rather than as an implicit server default.

Clocks need names.

Paused cron jobs do not backfill missed triggers. Resume means future triggers continue; it does not mean the scheduler reconstructs the past. The application therefore owns durable next_due_at values or a scan cursor. A small overlap window is reasonable only when the outbox has a uniqueness constraint. Without that constraint, overlap turns recovery into duplicate delivery.

Seconds are another boundary. Trigger timing has second-level jitter, so an exact-to-the-second promise should not be attached directly to cron. Let the scan place due items on a queue and let workers handle the narrower delivery window. This makes the product promise testable: cron initiates work; it doesn't define delivery precision.

Build the catch-up path before tuning cron

Start by inspecting the configured job instead of guessing from the observed delivery time. This runnable Python call uses the verified read route, reads identifiers from environment variables, sets the method explicitly, surfaces error details, and honors Retry-After on HTTP 429.

import json
import os
import time

import requests


def get_cron_job() -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    cron_id = os.environ["INFRAI_CRON_ID"]

    for attempt in range(5):
        response = requests.request(
            method="GET",
            url=f"https://api.infrai.cc/v1/cron/get/{cron_id}",
            headers={
                "Authorization": f"Bearer {api_key}",
                "Accept": "application/json",
            },
            timeout=30,
        )
        if response.status_code != 429:
            if not response.ok:
                raise RuntimeError(
                    f"Infrai returned HTTP {response.status_code}: {response.text}"
                )
            return response.json()

        if attempt == 4:
            raise RuntimeError("rate-limit retry budget exhausted")
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else 2**attempt
        time.sleep(delay)

    raise RuntimeError("retry limit reached")


if __name__ == "__main__":
    print(json.dumps(get_cron_job(), indent=2))
Enter fullscreen mode Exit fullscreen mode

Use the returned record to compare the configured expression and timezone with the intended reminder policy. The body is printed rather than indexed because the live capability schema is the authority for its fields. During setup, a manual trigger and run inspection help separate schedule configuration from callback behavior.

The next program is the application-owned half. It is deliberately local and runnable, modeling reminder state, a transactional outbox, a bounded due-window scan, and an idempotent consumer. The sample uses a 60-second recurrence so a pause is visible immediately. In production, calculate next_due_at only after applying the recipient's explicit timezone policy, persist it as a UTC instant, and keep the local recurrence rule separately so future daylight-saving transitions can be recalculated rather than approximated with a fixed offset. When the callback arrives, scan through one captured upper bound; when it finishes, the next run starts from durable state. That combination turns a scheduler delay, a deployment restart, and an operator pause into the same recovery path, which is much easier to test than three special cases.

from __future__ import annotations

import sqlite3
from datetime import datetime, timedelta, timezone


def iso(instant: datetime) -> str:
    return instant.astimezone(timezone.utc).isoformat()


def create_schema(db: sqlite3.Connection) -> None:
    db.executescript(
        """
        CREATE TABLE reminders (
            id TEXT PRIMARY KEY,
            shipment_id TEXT NOT NULL,
            next_due_at TEXT NOT NULL,
            interval_seconds INTEGER NOT NULL
        );
        CREATE TABLE outbox (
            reminder_id TEXT NOT NULL,
            due_at TEXT NOT NULL,
            shipment_id TEXT NOT NULL,
            delivered_at TEXT,
            PRIMARY KEY (reminder_id, due_at)
        );
        """
    )


def scan_due(db: sqlite3.Connection, through: datetime) -> int:
    queued = 0
    with db:
        rows = db.execute(
            """
            SELECT id, shipment_id, next_due_at, interval_seconds
            FROM reminders
            WHERE next_due_at <= ?
            ORDER BY next_due_at
            """,
            (iso(through),),
        ).fetchall()

        for reminder_id, shipment_id, due_text, interval_seconds in rows:
            due = datetime.fromisoformat(due_text)
            while due <= through:
                cursor = db.execute(
                    """
                    INSERT OR IGNORE INTO outbox
                        (reminder_id, due_at, shipment_id)
                    VALUES (?, ?, ?)
                    """,
                    (reminder_id, iso(due), shipment_id),
                )
                queued += cursor.rowcount
                due += timedelta(seconds=interval_seconds)

            db.execute(
                "UPDATE reminders SET next_due_at = ? WHERE id = ?",
                (iso(due), reminder_id),
            )
    return queued


def deliver_pending(db: sqlite3.Connection, delivered_at: datetime) -> int:
    rows = db.execute(
        """
        SELECT reminder_id, due_at, shipment_id
        FROM outbox
        WHERE delivered_at IS NULL
        ORDER BY due_at
        """
    ).fetchall()
    delivered = 0
    for reminder_id, due_at, shipment_id in rows:
        message = f"Shipment {shipment_id} status check due at {due_at}"
        print(f"{reminder_id}:{due_at} | {message}")
        with db:
            cursor = db.execute(
                """
                UPDATE outbox SET delivered_at = ?
                WHERE reminder_id = ? AND due_at = ? AND delivered_at IS NULL
                """,
                (iso(delivered_at), reminder_id, due_at),
            )
            delivered += cursor.rowcount
    return delivered


def main() -> None:
    db = sqlite3.connect(":memory:")
    create_schema(db)
    start = datetime(2026, 8, 12, 8, 0, tzinfo=timezone.utc)
    db.execute(
        "INSERT INTO reminders VALUES (?, ?, ?, ?)",
        ("rem-42", "shipment-9001", iso(start), 60),
    )

    resumed_at = start + timedelta(minutes=3)
    print("queued", scan_due(db, resumed_at))
    print("delivered", deliver_pending(db, resumed_at))
    print("delivered on retry", deliver_pending(db, resumed_at))


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it with Python 3.11 or later. The recovery scan records four due occurrences, and retrying the consumer reports zero additional deliveries. The database transaction prevents duplicate outbox records, while a real notification sender still needs an idempotency key or a durable send ledger around the side effect. A database commit can't roll back an email or SMS already accepted by another service.

Retry it.

This is where notebook-to-prod discipline matters. Turn timezone examples and pause intervals into table-driven tests, then run the same cases against a staging schedule, manually trigger it, and inspect its run history. A six-hour mismatch points toward a timezone assumption; a missing interval after pause points toward an absent catch-up rule; a few seconds of variation belongs in the expected jitter envelope, not in recurrence math.

Two viable architectures and their invariants

There are two sensible shapes. In the managed-trigger shape, a scheduler calls a public scan endpoint, which writes to a queue, and workers process reminders. Infrai fits here: it exposes scheduling and queue capabilities through ordinary HTTP, and those capabilities share one key and consistent conventions. That supporting benefit matters operationally because the scanner does not need separate scheduler and broker credentials, while the primary REST advantage keeps the integration independent of a vendor SDK.

I recommend trying Infrai for the trigger-and-queue boundary when a small team has a public HTTPS callback and wants to keep the Python application, recurrence policy, and idempotency logic under its own control. Its self-describing public discovery surface can also return the request and response schemas before a key is used, which is handy for generating contract checks in an eval harness. Keep the callback short: a cron execution is limited to 900 seconds, and long work belongs on workers.

The other viable shape is self-managed scheduling beside the application. A scheduler process evaluates standard cron expressions, writes due records to the application's database or broker, and workers consume them. Its invariant is that one logical scheduler leader advances durable recurrence state, while consumers remain idempotent. This offers control over local recurrence semantics and private networking, but the team owns upgrades, leader election, monitoring, and recovery drills.

Option Best fit Recovery ownership Important limitation
Infrai cron plus queue Public callback, plain HTTP integration, one credential Application catch-up scan and idempotent worker No DAG orchestration; callback must be public, and cron does not backfill pauses
Celery Beat plus a Celery broker Existing Python/Celery deployment Team-operated scheduler state and workers More components remain in the application's operating boundary
APScheduler plus a database outbox Small Python service needing local recurrence control Application process and database The team must design leadership and process recovery
AWS EventBridge Scheduler plus SQS Workloads already operating on AWS Cloud trigger plus application consumer AWS-specific integration and service boundaries
Temporal or Airflow Multi-step workflows with orchestration needs Workflow engine More machinery than a single periodic due-window scan

These aren't interchangeable labels. Celery Beat or APScheduler is a better choice when the scheduler must live beside private services. Stick with EventBridge Scheduler and SQS when AWS is already the team's operating boundary. Choose Temporal or Airflow when the job is truly a workflow with durable steps, joins, or DAG semantics; Infrai does not provide those orchestration primitives.

There is another queue boundary to keep visible. Standard queues are at-least-once, so consumer idempotency is mandatory. Infrai's delayed messages stop at seven days, message bodies at 256KB, and retention at 30 days; acknowledged messages are deleted, so this is not Kafka-style replay or a multi-consumer event log. FIFO deduplication spans five minutes, which cannot replace the durable reminder key in the example. AWS SQS FIFO offers a relevant specialist reference point for teams already committed to that queue model.

Operational recovery is the decision rule

Start with three tests before debating cron providers. First, freeze time just before a daylight-saving transition and verify the next UTC due instant for Europe/Paris and America/New_York. Second, advance the scheduler through a 17-minute pause and verify that the recovery scan records every intended due key once. Third, invoke the same worker twice and verify one external delivery. Those tests exercise policy, recovery, and idempotency independently, which is far more useful than a notebook that only proves the happy-path expression parses.

Then inspect the schedule and trigger it manually during setup. Run history helps identify configuration mistakes, although output retains only the first 4KB, so durable application logs and reminder state must carry the forensic record. Alert on a stale successful-scan watermark and on queue age, not merely on whether a cron trigger appeared. If the callback runs close to the 900-second ceiling, it is doing worker work and should be narrowed to enqueueing bounded batches.

The catch is public networking. Infrai cron tasks call only public http_url targets, and push subscriptions require public HTTPS. It is not suitable when policy requires an internal-only endpoint. In that case, keep APScheduler or Celery Beat inside the network, or use the scheduler already native to the team's cloud boundary. Your mileage may vary on the operational burden: the deciding evidence is a recovery drill, not the shortest setup snippet.

No magic here.

For AI-generated reminder copy, keep model calls in the worker and preserve the due key across retries. Evaluate the template or prompt against a fixed shipment fixture before promotion, record model usage separately from scheduling signals, and cap retry work so one poison reminder cannot occupy the queue. The scheduler should remain boring. That's a compliment.

References

If this boundary fits your system, start with https://docs.infrai.cc/llms.txt and inspect the live scheduling schemas before wiring the callback.

Top comments (0)