Short answer: treat the hold window as a business deadline, then use a durable delayed queue whose delivery handler is idempotent. For a logistics SaaS, the queue is allowed to deliver the expiration webhook more than once; the reservation state transition is not. That choice matters more than whether the delay is implemented by a timer, a broker, or a database poller.
I learned to start with the reservation record, not the scheduler. A shipment slot held for 15 minutes needs an explicit expires_at, an immutable event id, and a state transition that can be safely retried. If the scheduler loses a process restart, the database still tells us what is late. If a webhook receiver answers slowly, the delivery attempt can be retried without creating a second release of the same slot.
What fails when a delayed webhook is treated like a timer?
A timer is an in-memory promise. It disappears during a deploy, and a fleet of replicas can create duplicate timers unless ownership is carefully coordinated. Cron improves visibility but introduces polling gaps and clock skew. A delayed queue gives the task a durable home, yet it does not make delivery exactly once. The receiver can acknowledge a request after committing work, before committing work, or not at all. Each ordering produces a different retry decision.
It failed.
The failure I watch for is a 200 response that arrives after the network already dropped the connection. The sender records a timeout and retries; the receiver has already released the reservation. That is a normal distributed-systems outcome, not evidence that the queue is broken. Store an idempotency key with the state change, and make a duplicate request return the already-known result.
A practical payload carries reservation_id, expires_at, attempt, and event_id. The worker loads the reservation in a transaction, checks that its status is still held, and changes it to expired only when the deadline has passed. A later attempt sees expired and exits successfully. Three words: repeatable by design.
Measure it.
How should a SaaS team schedule webhook retries after a delay on a public HTTPS endpoint?
Separate scheduling from delivery. The scheduler creates one task with a due time. A worker claims due tasks with a lease, sends the HTTPS request, and records the outcome. The lease prevents two workers from sending at the same instant, while the idempotency key handles the race that still occurs after a lease expires. Keep the public endpoint boring: authenticate the request, validate the event schema, persist it, and acknowledge quickly.
Retry policy should classify responses, not blindly count failures. HTTP 429 means the recipient is asking the client to slow down; MDN documents Retry-After as the signal to honor when present. A 401 or 403 usually needs operator action, while a 410 may mean the endpoint was intentionally removed. Timeouts and selected 5xx responses can be retried with exponential backoff and jitter. Cap the attempts, then move the task to a reviewable dead-letter state.
Here is the core decision in Python-like service code. The interface is intentionally generic so the same rules work with a hosted broker, a database-backed queue, or a self-managed worker.
def handle_expiration(task, db, http):
reservation = db.get_reservation(task["reservation_id"])
if reservation is None:
return "discarded"
if reservation.status == "expired":
return "already_applied"
if reservation.status != "held" or now_utc() < reservation.expires_at:
return "not_due"
with db.transaction() as tx:
changed = tx.expire_once(
reservation.id,
idempotency_key=task["event_id"],
)
if not changed:
return "already_applied"
response = http.post(
task["callback_url"],
json={"event_id": task["event_id"], "type": "reservation.expired"},
headers={"Idempotency-Key": task["event_id"]},
timeout_seconds=5,
)
return classify_delivery(response)
The transaction boundary is deliberate. If the callback fails, the local expiration remains true and the delivery task can retry. If your business rule requires the partner to confirm before releasing inventory, model that as a separate state machine; do not hide it inside a retry loop.
Which queue semantics and observability signals matter?
Look for explicit acknowledgement, visibility timeout or lease behavior, delayed delivery, and a dead-letter path. RabbitMQ's consumer acknowledgement guidance is a useful baseline: acknowledgement confirms that the consumer has taken responsibility, but it does not erase the need for duplicate-safe handlers. Whatever queue you pick, document what happens when a worker dies after the side effect and before the acknowledgement.
| Mechanism | Good fit | Main risk to control |
|---|---|---|
| Database poller | Small tenant count, strong audit trail | Polling lag and lock contention |
| Broker with delayed delivery | Bursty workloads, many consumers | Lease expiry and duplicate sends |
| Managed scheduler | Small operations team | Limited replay or residency controls |
Metrics should tell a story an on-call engineer can act on: due tasks, age of the oldest task, delivery latency, retry counts by status class, 429 rate, lease expirations, and dead-letter volume. Log the event id and reservation id together. Sample request bodies carefully; logistics payloads can contain addresses and phone numbers, so compliance rules apply to logs as much as to storage.
Run failure drills with a fake public endpoint: delay responses, close connections after reading the body, return 429 with Retry-After, and replay the same event ten times. I once assumed a five-minute backoff was enough; a partner's rate limit window made that assumption wrong. Your mileage may vary, so measure the receiver's contract instead of guessing.
When is a simpler mechanism the right trade-off?
A database poller is often suitable for a small tenant count and modest throughput. It is easy to inspect and can recover missed deadlines by querying expires_at. A broker is a better fit when you need high fan-out, independent consumers, or queue-level backpressure. A managed scheduler can reduce operational work, but it may constrain delivery controls, replay tooling, or data residency.
The catch is operational shape. A queue adds credentials, worker deployment, dashboards, and a reconciliation job. It is not suitable when the only requirement is a handful of internal reminders and a single durable database already exists. Stick with a transactional outbox plus polling when auditability and simple recovery matter more than burst throughput. Choose a broker when the cost of coordinating workers yourself exceeds the cost of another moving part.
A rollout that protects inventory
Start by writing the state machine and idempotency tests before introducing delayed delivery. Backfill expires_at for existing holds, emit outbox events in the same transaction as reservation writes, and run a reconciler that finds held rows past their deadline. Shadow the new worker against a test endpoint, compare duplicate and late-delivery counts, then enable one warehouse at a time.
Keep the old expiration path until the new path has survived deploys, clock changes, and a controlled worker kill. The success criterion is not zero retries. It is zero incorrect inventory transitions, with every retry explainable from the event and lease history.
Top comments (0)