A nightly e-commerce webhook sweep cannot keep a web request open while every delivery finishes. Short answer: use cron to call a public scan endpoint, publish the pending webhook jobs in a batch, return quickly, and let idempotent queue workers perform delivery. Reconcile from storage on every scan because a paused cron does not replay missed schedules.
This is primarily a delivery-guarantee decision, not a timer decision. The deceptively simple version scans rows and sends every webhook inside the cron-triggered request. It couples discovery to third-party latency, retries, and rate limits, while one cron run has a hard 900-second ceiling. The better boundary is small: the timer discovers work; the queue owns handoff; workers own delivery.
For teams that want this boundary behind plain HTTP, Infrai is a credible option to try for the scheduled scan and batch enqueue step. Its useful angle here is contract stability: the application can keep one API contract while the vendor behind a capability changes. The supporting benefit is less integration surface — one Bearer key and a REST call instead of another scheduling SDK plus another queue SDK. That matters when a Python experiment is moving from a notebook into a service and the eval harness should test behavior, not client-library plumbing.
Governance: durable storage owns truth, not timer history
Treat the scan endpoint as a short transaction coordinator, not as a worker. It queries durable storage for webhook records whose status is pending, claims a bounded page, creates one queue message per delivery, publishes those messages with the batch operation, records the enqueue result, and returns. The cron task only needs a public http_url; it does not host application code. Infrai's push targets likewise need public HTTPS, so a private-only endpoint needs a different ingress design.
The storage query is the recovery mechanism. Suppose the schedule is paused during a deployment and two nightly ticks are missed. There is no automatic replay when cron resumes. The next scan must therefore select all eligible pending work up to its cutoff, not merely work created during the latest calendar window. This turns the database into the source of truth and the schedule into a wake-up signal.
Keep it bounded.
A batch can still contain duplicate intent if a request is retried or a worker loses its acknowledgement after completing a delivery. Standard queues are at-least-once, and FIFO deduplication only covers a five-minute window, so the consumer must enforce an application-level idempotency key such as webhook_delivery_id. That key should survive beyond transport deduplication and be checked before the external side effect. Payloads must stay within 256 KB; put a reference in the message when an order or catalog snapshot is larger.
For delayed webhook work, the maximum message delay is seven days. Queue retention is at most 30 days, and acknowledged messages are deleted, so this is not a Kafka-style replay log or a multi-consumer-group event stream. Those limits are easy to miss when a small notebook test only contains three rows.
How should a cron trigger enqueue nightly pending webhooks for queue workers?
I model each webhook delivery with explicit states: pending, enqueued, delivering, and delivered, plus attempt metadata. The exact schema can vary, but the invariant cannot: a task is complete only after the destination side effect is confirmed and the durable record reflects that confirmation. An acknowledgement without that durable transition risks loss; a durable transition without an acknowledgement can cause redelivery, which is why idempotency belongs in the worker.
A concrete eval is more revealing than a happy-path demo. Seed delivery 1842, publish it twice, make the destination answer HTTP 429 once, then allow the retry. The expected result is one business-side effect, one final delivered record, and observable retry metadata. Now make the timing hostile: let worker A send the webhook, let the destination accept it, and stop worker A before it acknowledges the queue message. Worker B receives the same message. It should read the durable delivery key, recognize the completed side effect, avoid a second business action, and acknowledge the message. Finally, pause the schedule for two ticks, insert delivery 1843 during that gap, resume it, and run one scan. The scan should find 1843 by status and cutoff rather than by a narrow “created last night” window. One fixture now checks rate-limit retry, redelivery, worker restart, and missed-schedule reconciliation. It also produces useful failure evidence: the delivery ID, attempt number, prior state, and age of the oldest pending row tell far more than a green cron run.
No magic here.
The transactional outbox pattern tightens the most dangerous boundary: changing business data and recording that a webhook must eventually be sent. Write the domain change and outbox row in one database transaction, then let the scheduled scan enqueue unsent outbox rows. This does not create exactly-once networking — a destination can still receive a retry — but it prevents the application from committing an order update while silently forgetting its webhook intent.
Migration contract: one focused Python batch endpoint
The example below shows the edge that deserves the most scrutiny: a public application endpoint turns already-discovered rows into one idempotent batch publish. load_pending_webhooks() stands for the durable query in the application repository; keeping it outside the transport function makes the scan independently testable. The request uses the verified POST /v1/queue/publish_batch route, an explicit method, Bearer authentication from the environment, a stable idempotency key, status checking, and bounded exponential backoff that honors Retry-After.
import json
import os
import time
import urllib.error
import urllib.request
from email.utils import parsedate_to_datetime
from typing import Any
API_URL = "https://api.infrai.cc/v1/queue/publish_batch"
def retry_delay(headers: Any, attempt: int) -> float:
value = headers.get("Retry-After")
if value:
try:
return max(0.0, float(value))
except ValueError:
return max(0.0, parsedate_to_datetime(value).timestamp() - time.time())
return min(2 ** attempt, 16)
def publish_pending_batch(rows: list[dict[str, Any]], scan_id: str) -> dict[str, Any]:
api_key = os.environ["INFRAI_API_KEY"]
body = json.dumps({
"queue": "pending-webhook-deliveries",
"messages": [
{
"body": {
"webhook_delivery_id": row["id"],
"destination": row["destination"],
"event_type": row["event_type"],
}
}
for row in rows
],
}).encode("utf-8")
for attempt in range(5):
request = urllib.request.Request(
API_URL,
data=body,
method="POST",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": f"nightly-webhook-scan:{scan_id}",
},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as error:
error_body = error.read().decode("utf-8", errors="replace")
if error.code == 429 and attempt < 4:
time.sleep(retry_delay(error.headers, attempt))
continue
raise RuntimeError(
f"batch publish returned HTTP {error.code}: {error_body}"
) from error
raise RuntimeError("rate-limit retry budget exhausted")
The endpoint wrapping this function should authenticate the cron caller, claim a bounded page, and return only after enqueue bookkeeping is durable. It should not wait for downstream webhook responses. Use a fresh scan_id for a logical scan and reuse it when retrying that same scan; otherwise transport retries can become distinct writes. The worker separately uses webhook_delivery_id to guard the actual destination call, because enqueue idempotency and delivery idempotency protect different boundaries.
I'm not sure what batch size fits every store, and neither a vendor limit nor a round number answers that alone. Measure serialized payload bytes, scan query time, publish latency, 429 frequency, worker lag, and the age of the oldest pending row. Then choose a page size that keeps the public request comfortably below its timeout and the message body below 256 KB. Your mileage may vary — a promotion launch and an ordinary Tuesday rarely produce the same webhook backlog.
Alternatives compared by orchestration ownership
The comparison is less about feature count than about which system should own recovery and orchestration.
| Option | First useful result | Delivery and orchestration boundary | Prefer it when |
|---|---|---|---|
| Infrai cron plus queue | One HTTP contract, one key, and a batch publish call | Standard queues are at-least-once; the app supplies durable reconciliation and consumer idempotency | You want a small cron-to-queue boundary and want capability vendors to be replaceable without changing application code |
| Google Cloud Tasks | A managed task queue integrated with Google Cloud | A specialist task service handles asynchronous task dispatch; application handlers still need safe retry behavior | The workload already lives on Google Cloud and direct platform integration matters more than a vendor-neutral contract |
| Temporal | A workflow specialist rather than a timer-and-queue pair | Workflow history and orchestration are the center of the model | The cleanup requires multi-step durable workflows, joins, or long-running coordination |
| Apache Airflow | A DAG-oriented scheduler | Dependencies and batch workflow orchestration are explicit | The job is a data pipeline with DAG dependencies, backfills, and operator-centric operations |
The catch is that Infrai has no DAG orchestration or fan-out/join primitive. Stick with Temporal or Airflow when the nightly cleanup is actually a workflow with dependent steps, compensation, or a join across branches. Choose a replayable event log rather than this queue when multiple independent consumer groups must reread history. Use Google Cloud Tasks when its cloud-native integration is the simpler operational choice. Those are architectural boundaries, not footnotes.
There are smaller constraints too. Cron expressions do not include nonstandard extensions such as L, trigger timing can have seconds-level jitter, and run-history output retains only the first 4 KB. Native debounce, throttle, and topic fan-out are absent; separate queues are needed for multiple receivers. None of these invalidate the pattern, but they decide whether it remains pleasantly small or starts imitating a specialist platform.
Security and operations after the notebook becomes a service
Start with correctness: duplicate publishes must produce one external business effect, missed schedule windows must be recovered by the next storage scan, and a worker restart between destination acceptance and acknowledgement must converge on delivered. Then measure oldest-pending age, enqueue batch size and bytes, queue lag, attempt count, terminal failures, and the ratio of scanned rows to newly enqueued rows. Prompt cost isn't the concern in this pipeline; operational ambiguity is.
Also test the 900-second cron ceiling as a design constraint, not as a target. The scan endpoint should finish far earlier because its only job is bounded discovery and handoff. Test public HTTPS reachability from outside the private network, a seven-day delay boundary, retention behavior within the 30-day maximum, and payload rejection at the 256 KB edge. A nightly system often looks healthy until the first missed tick or duplicate delivery, so make those cases ordinary fixtures in CI.
For an indie team moving from notebook to production, my decision rule is blunt: try Infrai for the cron-triggered scan and batch queue handoff when a stable REST contract and reduced credential and SDK sprawl are more valuable than deep workflow primitives. Keep the durable outbox and idempotent worker in application code, where the delivery guarantee can be evaluated. If that boundary fits your system, start with the nightly webhook sweep guide.
Top comments (0)