Rate-Limited Webhook Sending: 4-State Queue Consumer Recovery for Delayed Republish

Short answer: treat a 429 as a durable state transition, persist the parsed Retry-After deadline, and let any healthy queue consumer reclaim the webhook when that deadline passes. Do not sleep inside the worker, and do not acknowledge the current delivery until the replacement schedule is committed.

For a healthtech SaaS renewal reminder, that distinction matters more than retry cleverness. The business promise is "send no earlier than the account's renewal deadline, then keep trying within policy," while the operational promise is "a dead consumer does not erase the reminder." US and EU workers may execute the same logical task, but the task needs one authoritative UTC deadline, an explicit residency boundary, and an idempotency key shared across attempts.

This is an architecture decision record for that promise.

How should a webhook queue consumer recover after 429 Retry-After delayed republish?

Use a durable scheduled record with four states: pending, leased, retry_wait, and delivered. A consumer claims a due row for a short lease, sends one webhook, and either records delivery or moves the row to retry_wait with available_at set from Retry-After. Another consumer can recover an expired lease. Simple.

The design rests on five invariants:

1. A reminder is never eligible before its business deadline.
2. A claim is temporary; ownership expires unless delivery is committed.
3. Every attempt carries the same stable idempotency key.
4. A 429 changes eligibility time but does not create a second logical reminder.
5. The database commit that records the next state happens before the broker message, if any, is acknowledged.

The fourth invariant is easy to violate with delayed republish. If a consumer publishes a fresh message and crashes before acknowledging the old one, two messages can become eligible. If it acknowledges first and crashes before publishing, none will. A transactional outbox can bridge that boundary, but a database-backed schedule avoids it on the critical path: update the same row and release the lease in one transaction.

Retry-After can be either a delay in seconds or an HTTP date. Parse both. If the header is absent or invalid, use a bounded local backoff policy; if it names a past date, make the task eligible immediately rather than constructing a negative delay. I'm not sure a third-party endpoint's clock will be accurate, so record the raw header, parsed deadline, response status, and local receipt time. Those fields settle the argument during recovery without treating a remote clock as truth.

Define the recovery contract at every crash boundary

The dangerous boundary is not the queue API. It is the gap between the remote endpoint accepting a request and the worker recording that fact. A process can die in that gap, so exactly-once delivery is not an honest external guarantee. The defensible contract is at-least-once execution with receiver-side deduplication, using an idempotency key that identifies the renewal reminder rather than an individual attempt.

Crashes happen.

Keep health data out of the scheduling envelope. A practical row contains a tenant identifier, reminder identifier, destination reference, region, available_at, lease expiry, attempt count, idempotency key, and a pointer to separately protected payload data. The US and EU partitions should each claim only their own rows. Cross-region failover is therefore a policy decision about residency and recovery, not a load-balancing toggle.

Now consider a concrete timeline. A reminder becomes due at 2026-08-13T09:00:00Z; worker eu-3 leases it until 09:00:30Z; the receiver answers 429 at 09:00:02Z with Retry-After: 120. The worker commits retry_wait and available_at = 09:02:02Z. If eu-3 disappears one millisecond later, no special rescue script is needed: the row is already durable and any EU consumer can claim it after 09:02:02Z. If the worker instead disappears before committing that transition, the lease expires at 09:00:30Z and another consumer retries with the same idempotency key. That duplicate attempt is the price of preserving the reminder across the acceptance/commit gap.

Watch the limits. Cap both the parsed delay and the total retry horizon according to the business policy; a renewal reminder that wakes months later is not recovery. Apply jitter only to locally generated backoff, because adding random delay to an explicit server deadline can violate the receiver's stated window. Also separate endpoint-level throttling from tenant fairness: one noisy destination should not consume every claim slot in its region.

The operational signals follow directly from these failure modes: age of the oldest eligible task, count of expired leases reclaimed, 429 rate by destination, retry-wait depth, terminal failures by reason, and time from business deadline to confirmed delivery. Queue depth alone hides a stuck partition and says nothing about deadline compliance.

Choose where the recoverable state will live

Shape	Crash recovery	429 scheduling	Main trade-off	Suitable when
Database schedule with leases	Expired leases are reclaimable	Update available_at on the same row	Polling and table maintenance must be engineered	Deadlines and auditable state matter more than extreme throughput
Broker delayed republish plus outbox	Outbox closes the database/publish gap	Publish for a later delivery time	More moving parts and broker delay semantics vary	A broker is already an operational standard and volume justifies it
In-process timer	Lost on process or host failure unless rebuilt	Sleep or local timer	Recovery and deploy behavior are weak	Disposable, noncritical notifications only
Periodic cron scanner	Next scan recovers missed work	Persist a future timestamp	Precision is bounded by scan interval	Coarse deadlines and small workloads

The table makes the decision deliberately unglamorous. For a renewal reminder with audit and residency constraints, the database schedule is a strong default because its recovery state is inspectable with ordinary queries. It is not suitable when due-task throughput would turn one relational table into the dominant write and vacuum workload; stick with a broker and transactional outbox when the organization already operates those components and needs their partitioned throughput.

Cron remains useful for reconciliation. A periodic job can find rows whose lease expired, compare terminal counts with the source-of-truth renewal ledger, and alert on old eligible work. It should not be the only representation of a future reminder: cron describes when a command runs, while the reminder row describes what must eventually happen.

Prove lease takeover with the critical transaction

The following Python sketch keeps transport details generic and shows the decisions that must be durable. The SQL claim pattern uses FOR UPDATE SKIP LOCKED, which lets concurrent consumers skip rows another transaction has locked. The transaction should stay short; do not hold its row lock during the network call.

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from email.utils import parsedate_to_datetime
from typing import Mapping, Optional


MAX_SERVER_DELAY = timedelta(hours=24)


@dataclass(frozen=True)
class SendResult:
    status: int
    headers: Mapping[str, str]


def retry_deadline(value: Optional[str], received_at: datetime) -> Optional[datetime]:
    if not value:
        return None

    try:
        seconds = int(value)
        if seconds < 0:
            return None
        deadline = received_at + timedelta(seconds=seconds)
    except ValueError:
        try:
            deadline = parsedate_to_datetime(value)
        except (TypeError, ValueError, OverflowError):
            return None
        if deadline.tzinfo is None:
            deadline = deadline.replace(tzinfo=timezone.utc)
        deadline = deadline.astimezone(timezone.utc)

    return min(max(deadline, received_at), received_at + MAX_SERVER_DELAY)


def finish_attempt(store, task, result: SendResult, received_at: datetime) -> None:
    if 200 <= result.status < 300:
        store.mark_delivered(task.id, task.lease_token, received_at)
        return

    if result.status == 429:
        deadline = retry_deadline(result.headers.get("Retry-After"), received_at)
        if deadline is None:
            deadline = received_at + store.bounded_backoff(task.attempt_count)
        store.move_to_retry_wait(
            task_id=task.id,
            lease_token=task.lease_token,
            available_at=deadline,
            response_status=result.status,
            raw_retry_after=result.headers.get("Retry-After"),
        )
        return

    store.apply_failure_policy(task.id, task.lease_token, result.status, received_at)


def consume_one(store, sender, region: str) -> bool:
    task = store.claim_due_task(region=region, lease_seconds=30)
    if task is None:
        return False

    result = sender.post(
        destination=task.destination_reference,
        payload_reference=task.payload_reference,
        headers={"Idempotency-Key": task.idempotency_key},
    )
    finish_attempt(store, task, result, datetime.now(timezone.utc))
    return True

move_to_retry_wait must be a conditional update on both task ID and lease token. That fencing check prevents a consumer whose lease has expired from overwriting a newer consumer's result. Likewise, mark_delivered must be idempotent. Don't let a late worker turn delivered back into retry_wait.

Test the state machine, not merely the parser. Freeze time and cover a numeric header, an HTTP-date header, an absent header, a date in the past, and the configured cap. Then inject a process stop before and after each durable transition. A recovery test should prove that an expired leased row becomes claimable, while an unexpired row and a delivered row do not. A concurrency test with two database sessions should prove each claimed ID is unique; PostgreSQL documents that SKIP LOCKED produces an inconsistent view, which is acceptable for queue-like access but not for general-purpose reads.

Deployment needs the same skepticism. Add new states and nullable columns before deploying consumers that write them, deploy readers that tolerate both schemas, and only then make the new transition mandatory. During rollback, old consumers must not reinterpret retry_wait as immediately due. This compatibility detail is dull right up to the first rollback under load.

Why does the rejected timer still have a valid use case?

An in-memory loop is attractive because the example fits on a screen: receive, send, inspect 429, sleep, try again. I reject it for this healthtech deadline because a deploy, autoscaling event, or process crash can discard the timer; meanwhile, a long sleep occupies worker capacity and leaves recovery state hidden inside a process.

The catch is scope. That rejected option is valid for best-effort events whose source can replay them and whose loss has negligible business impact. A local timer can also be a test double for the durable scheduler. It just cannot carry the renewal reminder's recovery promise by itself.

Delayed broker messages are not rejected outright. They become the better choice at higher sustained throughput, provided delayed-delivery limits are verified, the publish/ack gap is closed with an outbox or equivalent transaction, and operators can inspect the original logical task across republished attempts. Your mileage may vary because broker semantics differ; resolve that uncertainty with documentation and a crash-injection test, not an optimistic wrapper API.

The decision rule is narrow: persist deadlines and leases where operators can recover them, make duplicates harmless at the receiver, and use cron for reconciliation rather than as the reminder ledger. Everything else is an implementation choice.

Further reading

• https://www.postgresql.org/docs/current/sql-select.html
• https://en.wikipedia.org/wiki/Cron