Rate-Limited Webhook Queue: Consumer Lease Timing After 429s in US and EU SaaS

Short answer: treat 429 Retry-After as a scheduling decision, not as a reason to hold a worker open: persist one delivery record with a stable idempotency key, move its next-attempt time forward, release the queue slot, and let a regional consumer claim it again only when it is due.

For a healthtech SaaS sending outbound webhooks from US and EU regions, that design gives the on-call team something it can reason about. Duplicate suppression lives in durable state, recipient backpressure doesn't consume active worker capacity, and the latency-versus-cost decision becomes an explicit service objective instead of a collection of sleep calls. The destination may still receive a repeated request after an ambiguous network outcome, so the contract must also require idempotent handling at the receiver.

Don't start with cron. Start with the delivery state machine.

Define the duplicate-delivery failure boundary

Each webhook delivery needs a durable identity and four transitions: ready, leased, delayed, and terminal. A ready row may be claimed by one consumer. A leased row has a short ownership deadline. A 429 response returns the row to delayed with next_attempt_at derived from Retry-After. A successful response makes it terminal; a permanent rejection also becomes terminal, but with a distinct outcome for audit and support work.

The stable delivery identity matters more than the queue brand. Put it in an Idempotency-Key header and keep the same value for every attempt. A consumer must never generate a fresh delivery ID during republish, because doing so turns one logical notification into several unrelated notifications from the receiver's point of view. This doesn't promise exactly-once transport. It creates an at-least-once delivery contract with a practical duplicate boundary: the receiver can store the key and return its previous result when the same logical delivery appears again.

For healthtech payloads, keep the queue envelope deliberately small. It can carry the delivery ID, tenant ID, destination reference, region, attempt count, and due time; the worker can resolve the request body from controlled storage at send time. That separation limits how much application data gets copied through retry infrastructure and makes deletion or access-policy changes easier to apply in one place.

Define the latency objective in terms the scheduler can enforce. One useful policy is: "95% of accepted deliveries begin their first attempt inside the normal dispatch window, and delayed attempts begin within a bounded scheduler lag after their due time." The actual windows are local choices, not universal constants. I'm not sure a single target can serve both urgent clinical workflow notifications and routine data synchronization; separate classes are warranted when the consequence of delay differs.

Then capacity-plan the worst credible retry wave. If a destination that normally accepts a large share of regional traffic starts rate limiting, delayed work must not crowd out first attempts for unrelated tenants. Partition or fairly schedule by tenant and destination, cap concurrent sends per destination, and reserve consumer capacity for never-attempted work. Otherwise one noisy endpoint converts its own limit into everyone else's latency incident. This is failure containment: the lease boundary must prevent a destination's backpressure from becoming a regional backlog.

How should a rate-limited webhook queue consumer handle 429 Retry-After?

The consumer should acknowledge the current queue lease only after the new due time is durable. It reads Retry-After, accepts either an integer delay in seconds or an HTTP date, clamps the result to operator-defined minimum and maximum delays, records the response class and next attempt, and releases the lease. If the header is absent or invalid, it applies a configured backoff with jitter. The important ordering is persist, then acknowledge. Reversing those operations creates a loss window.

No sleeping worker.

Holding a process open until the retry time looks simple in a Node.js example, yet the economics are poor under sustained throttling: memory, connection slots, deployment drain time, and queue leases all grow with the number of delayed deliveries. A durable due-time index lets consumers spend compute only on eligible work. This is the central latency-versus-cost trade: polling more often reduces scheduler lag but raises idle database and worker activity; polling less often costs less but adds delay even after the receiver is ready.

Use FOR UPDATE SKIP LOCKED when a relational database is the queue and several consumers claim work concurrently. The query can lock eligible rows while allowing other consumers to move past already locked rows. Keep the transaction short: claim rows and commit, then perform network I/O outside the transaction. A network call inside the lock transaction couples destination latency to database contention, which is exactly the sort of hidden capacity multiplier that wakes an SRE at the wrong hour.

The following Go code shows the scheduling core. It is intentionally queue-neutral; Store can sit behind a relational due-time table or another durable delayed-delivery mechanism. The sample doesn't pretend that a local timer is durable.

package delivery

import (
    "context"
    "errors"
    "io"
    "net/http"
    "strconv"
    "strings"
    "time"
)

type Job struct {
    ID             string
    IdempotencyKey string
    Destination    string
    Body           io.Reader
    Attempt        int
}

type Store interface {
    MarkDelivered(ctx context.Context, id string, deliveredAt time.Time) error
    Delay(ctx context.Context, id string, nextAttempt time.Time, reason string) error
    MarkPermanentFailure(ctx context.Context, id string, status int) error
}

type Consumer struct {
    Client       *http.Client
    Store        Store
    Clock        func() time.Time
    MinRetry     time.Duration
    MaxRetry     time.Duration
    FallbackWait func(attempt int) time.Duration
}

func (c *Consumer) Send(ctx context.Context, job Job) error {
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, job.Destination, job.Body)
    if err != nil {
        return err
    }
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", job.IdempotencyKey)

    resp, err := c.Client.Do(req)
    if err != nil {
        return err // Leave lease recovery to the durable queue policy.
    }
    defer resp.Body.Close()
    _, _ = io.Copy(io.Discard, resp.Body)

    now := c.Clock().UTC()
    switch {
    case resp.StatusCode >= 200 && resp.StatusCode < 300:
        return c.Store.MarkDelivered(ctx, job.ID, now)
    case resp.StatusCode == http.StatusTooManyRequests:
        wait := retryDelay(resp.Header.Get("Retry-After"), now)
        if wait <= 0 {
            wait = c.FallbackWait(job.Attempt)
        }
        wait = clamp(wait, c.MinRetry, c.MaxRetry)
        return c.Store.Delay(ctx, job.ID, now.Add(wait), "rate_limited")
    case resp.StatusCode >= 400 && resp.StatusCode < 500:
        return c.Store.MarkPermanentFailure(ctx, job.ID, resp.StatusCode)
    default:
        return errors.New("retryable delivery response")
    }
}

func retryDelay(value string, now time.Time) time.Duration {
    value = strings.TrimSpace(value)
    if seconds, err := strconv.ParseInt(value, 10, 64); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    if deadline, err := http.ParseTime(value); err == nil {
        return deadline.Sub(now)
    }
    return 0
}

func clamp(value, minimum, maximum time.Duration) time.Duration {
    if value < minimum {
        return minimum
    }
    if value > maximum {
        return maximum
    }
    return value
}

The network-error branch is deliberately different from the 429 branch. A connection can fail after the receiver accepted the body but before the sender observed the response. The sender cannot infer success or failure from that ambiguity, so it should let the durable lease expire or record a retry without changing the idempotency key. A second request is possible. Duplicate side effects are not, provided the receiving contract honors that key.

The database claim can stay compact:

const claimDue = `
WITH candidates AS (
    SELECT id
    FROM webhook_deliveries
    WHERE region = $1
      AND state IN ('ready', 'delayed')
      AND next_attempt_at <= $2
    ORDER BY next_attempt_at, id
    FOR UPDATE SKIP LOCKED
    LIMIT $3
)
UPDATE webhook_deliveries AS d
SET state = 'leased',
    lease_until = $2 + $4::interval,
    leased_by = $5
FROM candidates
WHERE d.id = candidates.id
RETURNING d.id, d.destination_ref, d.idempotency_key, d.attempt;
`

Run one scheduler per data region and make region part of both the claim predicate and the record's immutable routing metadata. US work stays in the US scheduler; EU work stays in the EU scheduler. Cross-region failover is not a free reliability switch because it changes where queued metadata and payload access occur. If cross-region processing is allowed, make that an explicit system contract and test it. If it isn't, regional capacity and recovery plans must stand on their own.

Migrate scheduler ownership without changing delivery identity

Lease behavior creates a portable contract: any scheduler that preserves the delivery ID, due time, attempt count, and regional routing can take ownership without changing what the receiver sees. That matters because the first implementation need not be permanent. A relational due-time queue is often the smallest initial operational surface when the application already depends on that database and the delivery volume fits its capacity envelope. A managed delayed queue moves scheduling operations out of the application database but introduces a service contract and migration boundary. A self-hosted broker gives the team more control while assigning it patching, capacity, backup, and on-call work.

The catch is that none removes operational responsibility; each places it somewhere different.

Mechanism	Failure ownership	Latency and cost pressure	Not suitable when
Relational due-time table	Poll cadence and indexes are directly controlled	Reuses an existing dependency, but retry waves compete with application database headroom	Delivery load can exhaust the database capacity reserved for core transactions
Managed delayed queue	Delay and redelivery follow the service contract	Less queue machinery to operate; usage and lock-in must be budgeted	Required delay semantics, regional placement, or migration options don't meet the contract
Self-hosted broker	Fine-grained control depends on the chosen broker	Infrastructure cost is visible, and the team owns upgrades and incidents	The team cannot staff broker operations without weakening its SLOs

Stick with the relational option when workload tests show comfortable headroom and the team values a single transactional record of state changes. Choose a managed queue when reduced on-call load is worth the external dependency and its delay semantics meet the required bound. Choose self-hosting when control or an existing operations capability justifies it. I would reject any selection memo that names a preferred mechanism but leaves retry-wave capacity, recovery ownership, and engineer interruption time blank.

Cron has a narrower role here. It is useful for periodic reconciliation, such as finding expired leases or checking terminal records against retention policy, but a cron expression is not the per-delivery source of truth. Individual Retry-After deadlines belong in durable delivery rows or messages, where they can vary per destination and per attempt.

Prove regional isolation before rollout

Before deployment, run the consumer against a deterministic receiver that returns 429 with an integer Retry-After, 429 with an HTTP date, 429 with a malformed header, a permanent 4xx, and a success. Add a case where the receiver records the idempotency key and closes the connection before the sender observes a response. The next attempt must reuse the key, and the test receiver must apply the side effect once.

Measure queue age by class and region, not just aggregate throughput. The minimum useful signals are first-attempt age, due-but-unclaimed age, delayed count, claim rate, response class, attempts per logical delivery, expired leases, and destination concurrency. Alert on user-visible symptoms tied to the service objective. A raw queue depth alarm is weak because a large population scheduled for tomorrow isn't late, while ten due records stuck for an hour may be an incident.

Deployment should begin with one region or a small tenant cohort while the old sender remains available. Compare the new path's first-attempt latency, scheduler lag, duplicate-key observations, and permanent-failure classifications. Don't dual-send the same logical webhook through both paths; shadow reads and state comparisons are safer than doubling externally visible requests.

Rollback is a state transition, not a binary redeploy. Stop new claims in the new consumer, wait for its short leases to expire, verify that no sender still owns them, and enable claims in the previous consumer against the same durable records. Preserve next_attempt_at, attempt count, region, and idempotency key. If rollback regenerates envelopes, resets due times, or loses keys, it has discarded the very controls that make the procedure safe.

Finally, rehearse saturation. Inject enough rate-limited destinations to fill the planned delayed-work envelope, then verify that fresh work for an unrelated destination remains inside its latency objective. This is where a plausible design becomes an operable one — or fails quickly enough to be redesigned before production.

Further reading

• Cron overview
• PostgreSQL SELECT, including FOR UPDATE and SKIP LOCKED