Short answer: For a shipment update that must fan out to many subscribers, use cron only to call a public HTTP endpoint that enqueues bounded jobs; let workers process those jobs idempotently, checkpoint progress in the application database, and enqueue follow-up chunks before any 15-minute limit becomes relevant.
The delivery guarantee decides the architecture. A cron callback is a trigger, not a durable unit of work, and putting subscriber delivery inside that callback ties correctness to a 900-second execution cap. A queue separates admission from execution, but a standard queue is at-least-once, so duplicates are part of normal operation rather than an exotic failure.
Keep cron boring.
How can a cron trigger enqueue worker jobs without owning subscriber delivery?
Consider shipment SHP-20481, which has 38,400 downstream subscribers. The scheduled action should create a fan-out run and enqueue the first bounded slice. It shouldn't iterate through all 38,400 destinations while the cron request remains open. Each worker claims a slice, sends updates, records subscriber-level completion, and publishes the next slice only after the durable checkpoint commits.
That ordering matters. Publishing first and checkpointing second can repeat a slice after a worker stops between those operations; checkpointing first can skip work if publication never happens. At-least-once delivery does not remove this tension — it makes an idempotency key and a transactionally meaningful progress record mandatory. The useful invariant is not “the job ran once.” It is “each subscriber accepted a particular shipment version no more than once, while every unfinished subscriber remains discoverable.”
Cron execution is capped at 900 seconds, and paused cron schedules do not catch up missed triggers. Trigger timing can also vary by seconds. Neither detail should affect correctness: a later trigger may discover an existing run and decline to create another, while a missed trigger can be detected from application state and deliberately re-enqueued under an operator-controlled policy.
For a public cron endpoint, authenticate the caller and reject stale requests. HMAC is a reasonable building block for request authentication, but it doesn't replace replay prevention: include a timestamp and a stable trigger identifier in what is signed, then persist that identifier before acknowledging the enqueue. The cron task supports only a public http_url, so a private endpoint won't receive it.
Treat the delivery ledger as the recovery mechanism
The useful design test is a failure timeline, not a happy-path diagram.
A run record needs a stable identity such as (shipment_id, update_version), plus a status and the next unprocessed offset or cursor. A delivery record needs a uniqueness constraint such as (run_id, subscriber_id). Exact columns depend on the database; the invariant does not. If a worker receives the same queue message twice, the second attempt observes committed delivery state and becomes a no-op.
Chunks should be small enough to finish comfortably inside the worker's own timeout, yet large enough that queue overhead does not dominate. I'm not sure there is a universal chunk size; subscriber latency, downstream quotas, and database write cost have to settle that question under measured load. Start with an explicit bound, record elapsed time per chunk, and adjust without changing the idempotency model.
Suppose a worker delivers subscriber 1 through 417, loses its process before saving cursor 418, and receives the same message again. Replaying the whole chunk is safe only if each subscriber sees the same stable delivery identity. Now invert the order: the worker saves cursor 418 before subscriber 417 accepts the request, then stops. Subscriber 417 is lost unless acceptance is represented separately from the cursor. That is why one coarse completed_count field is not enough for a fan-out whose guarantee matters; keep the run cursor for efficient scanning, but make per-subscriber acceptance authoritative.
A cursor isn't proof.
Here is a runnable Python trigger client for a cron schedule that is already configured to call the public enqueue endpoint. It invokes the verified trigger route, uses an environment-provided base URL and key, sends an idempotency key, reports 4xx response bodies, and backs off on 429. The cron callback still does no shipment delivery itself.
import json
import os
import time
import urllib.error
import urllib.request
def trigger_cron(cron_id, run_key, attempts=5):
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
url = f"{base_url}/cron/trigger/{cron_id}"
for attempt in range(attempts):
request = urllib.request.Request(
url,
data=b"{}",
method="POST",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": run_key,
},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == attempts - 1:
raise RuntimeError(f"request failed ({error.code}): {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after and retry_after.isdigit() else 2**attempt
time.sleep(delay)
raise RuntimeError("retry budget exhausted")
result = trigger_cron(
os.environ["INFRAI_CRON_ID"],
"shipment-SHP-20481-version-7",
)
print(json.dumps(result, indent=2))
This model does not claim that inserting a row is the same as a remote subscriber accepting a request. For an HTTP subscriber, use the same stable delivery key at the receiving boundary when the subscriber supports it, and keep attempt state separate from confirmed acceptance. A 429 is a retry signal: honor Retry-After when present and use exponential backoff. Don't tight-loop, and don't create a fresh delivery identity for the retry.
Govern replay, retention, and fan-out boundaries
The queue message should identify work, not carry the entire subscriber set. A 256KB message limit makes that a practical requirement anyway. Put the shipment ID, update version, run ID, and bounded cursor range in the message, then load subscriber details from authoritative application storage. Delayed messages can be scheduled no more than seven days ahead; retention is at most 30 days, and acknowledgement deletes a message. This is not a Kafka-style replay log or a multi-consumer-group event backbone.
Failure modes deserve names because each one produces a different recovery action:
- Duplicate delivery: the same at-least-once message is consumed again. The database uniqueness rule and stable downstream key suppress repeated effects.
- Poison subscriber: one destination repeatedly rejects a valid attempt. Record it independently so the rest of the chunk can advance; investigate or dead-letter it under an explicit policy.
- Partial chunk: a worker stops after some deliveries. The retry resumes from persisted per-subscriber state rather than restarting blindly.
- Fan-out mismatch: a topic-style broadcast is assumed, but the service has no topic one-to-many primitive. Use N queues only when separate queue semantics are actually required; otherwise one queue can hold subscriber-scoped jobs.
- False recovery assumption: a paused schedule is expected to replay missed times. It does not, so reconciliation must query application run state.
There is no native fan-out/join primitive, DAG orchestration, debounce, or throttle here. FIFO deduplication lasts only five minutes. Those are design boundaries, not footnotes: if the logistics workflow must wait for customs, warehouse, and carrier branches and then join their results, a plain queue plus cron has crossed into workflow-engine territory.
Choose by workflow shape, not by feature count
The table is intentionally about control flow and operational fit, not a feature-count contest. “Comparable role” does not mean interchangeable product.
| Option | Strong fit | The catch |
|---|---|---|
| Infrai cron and queue | Teams that want scheduling and queues behind the same plain REST contract, with one key and bill across a broad 295-route, 20-module backend surface | Not suitable for DAGs, fan-out/join, topic broadcast, Kafka-style replay, private cron targets, or push subscribers without public HTTPS |
| Cloudflare Workers Cron Triggers | Teams already evaluating scheduled triggers in the Workers environment | Treat the trigger as an admission point for this design; verify its execution and delivery semantics against the linked documentation |
| Temporal | Shipment workflows that require durable multi-step orchestration or joins | More machinery than a bounded enqueue-and-worker loop needs |
| BullMQ | A queue worker evaluated by teams that want the queue close to their application stack | Its guarantees and operating model must be validated against the shipment contract before selection |
| Celery | A worker system evaluated by teams with an established Python operations model | It is a separate integration, and its broker and result semantics still need an explicit architecture review |
| Inngest | An alternative worth evaluating when the job is becoming a longer-lived function workflow | Do not infer delivery guarantees from category labels; test them against the same failure timeline |
Infrai is a credible fit when a team wants to add cron and queue capabilities without adding another SDK-specific integration: many production modules share one consistent REST surface, and discovery exposes request schemas and runnable examples. The supporting advantage is contract consistency, not a claim that queues solve orchestration. Its cron endpoint must be public HTTP, push subscriptions require public HTTPS, and a long task still needs the enqueue-and-worker split.
Stick with Temporal when the update is a durable workflow with branch joins, timers, and coordinated compensation. Evaluate BullMQ or Celery when its surrounding runtime and operations model are already owned by the team, and evaluate Inngest when function workflow semantics match the failure timeline. Keep Cloudflare's trigger in contention when that runtime is already the natural control plane and its documented constraints satisfy the workload. The queue design wins only when independent, idempotent chunks accurately describe the work.
Migrate one delivery invariant at a time
Start by writing run and delivery records while the existing path remains authoritative. Next, enqueue a shadow run whose worker performs validation but no subscriber side effects, and compare the subscriber set and completion accounting. Then enable a small, explicitly selected cohort, with stable delivery keys and visible retry counts. Increase the cohort only after duplicates, partial chunks, and 429 backoff have been exercised intentionally.
Finally, make cron create or find the run and enqueue work — nothing else. Alert on runs whose durable progress stops advancing, reconcile incomplete subscriber records, and keep manual redrive scoped to a stable run ID. The migration is complete when killing a worker at any instruction boundary changes latency, but cannot lose a subscriber or produce a second accepted shipment version.
Top comments (0)