Short answer: use a durable message queue for per-event webhook delays, and keep cron as a dispatcher for bounded recovery work. A public HTTPS endpoint should acknowledge quickly; it should not be the place where a long-running job waits. The useful design question is where each retry becomes durable and idempotent.
I build B2B SaaS email, SMS, and OTP flows, so I pay attention to the unglamorous edges: rate limits, delivery gaps, and the moment a provider accepts a request while our process dies. Webhooks have the same shape. A timer can wake a process, but only a durable record can explain why an event is still owed.
The six invariants I write down first
For each outbound event, I keep an immutable event ID, a destination, a next-attempt timestamp, and an attempt count. The receiver gets the event ID as an idempotency key. The sender records an attempt before (or atomically with) claiming work, and a lease expires if the worker disappears. Those rules matter more than the brand of scheduler.
The failure boundary is explicit: a 2xx response means the receiver accepted the message, not that its business transaction finished. A timeout means unknown outcome. Retrying after an unknown outcome is correct only when the receiver deduplicates by event ID. This is the same reason a transactional outbox writes application state and the outgoing event in one database transaction.
Here is the decision record I use for a single event:
| Mechanism | Good fit | Boundary to enforce |
|---|---|---|
| Cron | Sweeping overdue rows, reconciliation, and low-rate batches | A tick is not a durable per-event timer; avoid a giant payload or unbounded run |
| Message queue | Per-event delay, backoff, visibility timeout, and worker fan-out | Delivery is usually at-least-once; consumers still need idempotency |
| Public HTTPS endpoint | Receiving a trigger and returning a fast acknowledgement | No waiting for a long job; authenticate, limit body size, and hand off |
| Long-running worker | Rendering, enrichment, or provider calls that exceed request limits | Lease, heartbeat, cancellation, and a terminal state are required |
How should cron, message queues, delayed webhooks, and public HTTPS limits fit?
Start with the event, not the clock. The API request creates an outbox row. A relay publishes a message containing the event ID and due time. A worker claims it, sends the webhook, and classifies the result. Transient transport errors schedule another attempt; permanent policy or authentication errors move the event to a reviewable dead-letter state.
The queue's visibility timeout must exceed the normal send time but stay below the interval at which a stuck worker should be detected. If a job can run for ten minutes, a five-minute lease creates a duplicate opportunity unless the worker renews it. Public HTTPS limits are a reason to split the work, not a reason to increase the request timeout.
Keep the acknowledgement tiny.
This small Python sketch shows the critical path. The database transaction that creates the outbox record is intentionally outside the HTTP handler's retry loop.
from datetime import datetime, timedelta, timezone
def claim_and_send(store, http, event_id):
event = store.claim(event_id, lease_seconds=90)
if event is None:
return "already claimed or complete"
try:
response = http.post(
event.destination,
json=event.payload,
headers={"Idempotency-Key": event.event_id},
timeout=8,
)
except TimeoutError:
store.reschedule(event_id, datetime.now(timezone.utc) + timedelta(minutes=2))
return "unknown outcome; retry with same event id"
if 200 <= response.status < 300:
store.complete(event_id)
return "delivered"
if response.status in (408, 429) or response.status >= 500:
store.reschedule(event_id, datetime.now(timezone.utc) + timedelta(minutes=5))
return "transient failure"
store.dead_letter(event_id, reason=f"status {response.status}")
return "permanent failure"
The receiver must persist the idempotency key before doing side effects, or use a unique constraint that makes the side effect conditional. Otherwise a queue's honest at-least-once behavior turns into duplicate invoices, duplicate notifications, or two password codes.
What breaks with a cron-only delayed webhook design?
The tempting version stores run_at and scans every minute. It works for a small table, then a deployment overlaps two scans, clock skew makes a row eligible twice, and a slow batch pushes fresh events behind old ones. Locking rows helps, but it does not give you a delivery contract or receiver deduplication.
I once treated a timeout as a failed send. That was the wrong assumption. The provider had accepted the request; our retry produced a second notification. The correction was simple but strict: record “unknown,” retry with the same event ID, and make the consumer's insert idempotent. Three words in the runbook saved a noisy incident: check the receipt first. That change also forced us to separate transport policy from business policy: a 429 should honor a retry-after hint, while a revoked destination should stop immediately, and a receiver that returns 2xx before committing its own transaction needs a reconciliation path. The code stayed small, but the state machine gained explicit transitions for claimed, sent, unknown, and dead-lettered. That is the difference between a timer demo and an operational webhook system.
Cron still has a valid job. Use it to sweep rows whose lease expired, rebuild queue indexes, and reconcile provider receipts. Keep the sweep bounded with a page size and a maximum runtime. Do not let it become the only scheduler for thousands of unrelated due times.
Operational checks and the rejected option
Measure due-to-claim delay, attempt age, lease expirations, duplicate suppression, and dead-letter counts. Alert on age and rate, not just queue depth; a shallow queue can hide one event stuck behind a long-running worker. Store the response class and a redacted provider request ID so support can trace an event without logging secrets.
The catch is that a queue adds another durable component and another class of limits to monitor. It is not suitable when the workload is a tiny, strictly bounded batch that already runs inside one transaction; stick with a cron sweep there. The rejected option here is “hold the public HTTPS request open until delivery finishes.” It couples client timeouts to provider latency, consumes connection capacity, and leaves outcome ambiguity when either side drops. It is suitable only for a deliberately synchronous, short operation with a documented timeout and idempotency contract. For delayed per-event work, acknowledge, enqueue, and observe.
Your mileage may vary: queue delay precision, maximum message age, and endpoint limits differ by infrastructure. Verify those limits in the service contract, then encode them in tests that advance a fake clock and replay the same event ID. That test is worth more than a benchmark screenshot.
Top comments (0)