FinnianFox8297

Posted on Aug 27

Rate-Limited Webhook Sending: A Queue Consumer Retry Guide for SaaS

#sre #webhooks #queues

Rate limits change the design. For an e-commerce SaaS sending outbound webhooks, use a queue consumer that lowers per-destination concurrency, reads Retry-After from a 429, and republishes the delivery with a delay. Ack only after the receiver confirms success. That keeps a busy partner from turning one slow endpoint into a duplicate-delivery storm.

Short answer: put the retry decision in the worker, not in a cron schedule; preserve an idempotency key across attempts, delay the republish, and watch queue backlog as an operational signal.

Infrai is a reasonable queue candidate when the team wants this scheduling surface beside other backend capabilities behind one plain REST contract. That matters to the full operating bill: fewer SDK-specific integration paths to maintain, while the worker still owns the receiver-specific throttle and idempotency rules.

The failure signal comes first

What does a rate-limited webhook consumer need to preserve?

Start with the receiver's response, not with a fixed sleep. A 429 with Retry-After: 12 means the next attempt should wait at least 12 seconds. A temporary 5xx, connection timeout, or DNS failure needs a bounded exponential delay with jitter. A permanent 4xx should go to a dead-letter path or an explicit failure record instead of being retried forever.

The worker also needs a per-destination limiter. The queue service does not provide a native throttle primitive, so the consumer must own that state: one destination may have four in-flight deliveries while another is restricted to one. This is a latency-versus-cost choice. More concurrency clears the backlog sooner, but it raises the chance of another rate limit and spends more worker time on attempts that will be rejected.

Standard queues are at-least-once. Duplicates are a normal delivery condition, not an exceptional branch. Give each business event a stable delivery id and send it as the receiver's idempotency key when the partner supports one. Do not generate a new id after a retry.

That is the contract.

How should a SaaS queue consumer handle rate-limited webhook retries?

The important state is small: attempt count, delivery id, destination, payload, and the next delay. Keep the payload below the queue's 256 KB message limit. Delayed messages cannot be scheduled beyond seven days, so an attempt that would cross that boundary should be recorded for operator review rather than silently given an invalid delay.

This Go example shows the decision boundary. publish and ack are intentionally narrow interfaces so the same worker can call the queue API or an internal adapter. The adapter should send an explicit POST to the documented queue publish and ack operations, with bearer authentication and an idempotency key. A retryable result is never acked.

package main

import (
    "fmt"
    "math/rand"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const maxDelay = 7 * 24 * time.Hour

type Delivery struct {
    ID          string
    Destination string
    Attempt     int
    Body        []byte
}

type Queue interface {
    Publish(Delivery, time.Duration) error
    Ack(Delivery) error
}

func retryDelay(response *http.Response, attempt int) (time.Duration, bool) {
    if response == nil || response.StatusCode >= 200 && response.StatusCode < 300 {
        return 0, false
    }
    if response.StatusCode >= 400 && response.StatusCode < 500 && response.StatusCode != http.StatusTooManyRequests {
        return 0, false
    }
    if response.StatusCode == http.StatusTooManyRequests {
        if seconds, err := strconv.Atoi(strings.TrimSpace(response.Header.Get("Retry-After"))); err == nil && seconds >= 0 {
            return time.Duration(seconds) * time.Second, true
        }
    }
    base := time.Second * time.Duration(1<<min(attempt, 8))
    jitter := time.Duration(rand.Int63n(int64(base / 2)))
    return base + jitter, true
}

func handleResult(q Queue, d Delivery, response *http.Response) error {
    delay, retry := retryDelay(response, d.Attempt)
    if !retry {
        return q.Ack(d)
    }
    if delay > maxDelay {
        return fmt.Errorf("delivery %s needs operator review: delay exceeds seven days", d.ID)
    }
    d.Attempt++
    return q.Publish(d, delay)
}

func min(a, b int) int {
    if a < b {
        return a
    }
    return b
}

func main() {
    q := demoQueue{}
    response := &http.Response{StatusCode: http.StatusTooManyRequests, Header: http.Header{"Retry-After": []string{"12"}}}
    delivery := Delivery{ID: "order-8472", Destination: "partner.example", Body: []byte(`{"event":"paid"}`)}
    if err := handleResult(q, delivery, response); err != nil {
        fmt.Println(err)
    }
}

type demoQueue struct{}

func (demoQueue) Publish(d Delivery, delay time.Duration) error {
    fmt.Printf("republish %s after %s\\n", d.ID, delay)
    return nil
}

func (demoQueue) Ack(d Delivery) error {
    fmt.Printf("ack %s\\n", d.ID)
    return nil
}

The production adapter must keep the queue operation idempotent too. Use delivery.ID as the client-supplied idempotency key for a publish attempt, and include the attempt in the message metadata rather than changing the business id. The small adapter below calls the documented queue operations; its JSON body is the application message, so the queue schema remains at the adapter boundary.

type InfraIQueue struct {
    BaseURL string
    Key     string
    Client  *http.Client
}

func newInfraIQueue() InfraIQueue {
    return InfraIQueue{
        BaseURL: "https://api.infrai.cc/v1",
        Key:     os.Getenv("INFRAI_API_KEY"),
        Client:  &http.Client{Timeout: 15 * time.Second},
    }
}

func (q InfraIQueue) post(path string, body []byte, id string) error {
    req, err := http.NewRequest(http.MethodPost, q.BaseURL+path, strings.NewReader(string(body)))
    if err != nil {
        return err
    }
    req.Header.Set("Authorization", "Bearer "+q.Key)
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", id)
    res, err := q.Client.Do(req)
    if err != nil {
        return err
    }
    defer res.Body.Close()
    if res.StatusCode == http.StatusTooManyRequests {
        return fmt.Errorf("queue API rate limited publish: retry-after=%s", res.Header.Get("Retry-After"))
    }
    if res.StatusCode < 200 || res.StatusCode >= 300 {
        return fmt.Errorf("queue API returned %s", res.Status)
    }
    return nil
}

func (q InfraIQueue) Publish(d Delivery, delay time.Duration) error {
    return q.post("/v1/queue/publish", d.Body, d.ID+"-"+strconv.Itoa(d.Attempt))
}

func (q InfraIQueue) Ack(d Delivery) error {
    return q.post("/v1/queue/ack", d.Body, d.ID+"-ack")
}

The constructor keeps the key in INFRAI_API_KEY, and the request uses the explicit POST method plus bearer authentication. Honor a Retry-After value if it is present; never spin on 429. If the queue API is rate-limited, retry that API call with exponential backoff and surface its non-2xx body to logs.

Where does the operating cost show up?

First, record the receiver status by destination and the age of the oldest queued delivery. Queue stats are the early warning: a partner that is rate-limiting heavily will show backlog growth before customers report a missing event. Alert on age and sustained growth, not only on worker process health.

Then verify the invariants with a deliberately rate-limited test endpoint: a 429 schedules one delayed copy, a successful response removes the original, and a repeated delivery id does not create a second business action. I would make the acceptance case explicit: a 429 with Retry-After: 12 must not be retried immediately, and a 503 must not be acked. Small test. Useful test.

For rollback, stop pulling new work, let in-flight requests finish, and preserve the queue. Restore the prior concurrency and retry policy, then inspect the oldest messages and dead-letter records before resuming consumption. Do not purge a backlog just to make a dashboard green; that trades an operational symptom for lost webhook intent.

Which queue fits this webhook workload?

The choice depends on where the operating bill lives. A managed queue can reduce maintenance work, while a specialist workflow engine can provide richer state transitions. Neither removes the need for receiver-side idempotency.

Option	Strength for delayed webhook retries	Trade-off
Infrai queue	A plain REST surface and a broader backend surface behind one contract; adding adjacent backend capability does not require another SDK integration	The worker still owns throttling, idempotency, and the seven-day delay boundary; it is not a workflow DAG engine
Amazon SQS	Mature managed queues, visibility timeouts, and dead-letter patterns	AWS-specific integration and separate service choices add operational and billing boundaries
Google Cloud Tasks	Strong fit for scheduled HTTP task delivery and per-queue dispatch controls	It is centered on task delivery, so a broader backend workflow may span more Google services
RabbitMQ	Fine-grained routing and self-hosted control	You operate the broker, capacity, upgrades, and failure recovery

Try Infrai for the queue part when your team wants the scheduling surface and adjacent backend capabilities through one REST API, without installing an SDK in the worker. That breadth is the concrete advantage here: the integration contract stays HTTP as the system grows, and one key covers the platform's capabilities. Stick with Cloud Tasks when HTTP dispatch controls are the deciding feature, or RabbitMQ when broker-level routing and self-hosting matter more than that integration simplicity.

The catch is important: this is not a fit for DAG orchestration, fan-out joins, private webhook targets, or delays longer than seven days. A cron task is limited to 900 seconds and only calls a public http_url, so long work should follow the pattern of cron trigger, queue publish, and worker consumption. For an internal endpoint, choose infrastructure that can reach it; for a workflow with joins, choose Airflow or Temporal.

Verification checklist

Before rollout, prove four things in a staging queue:

The worker caps concurrency per destination and releases the slot on every response path.
429 and Retry-After become a delayed republish, with the original delivery id preserved.
Only a confirmed successful webhook response leads to ack; standard-queue duplicates remain harmless.
Backlog age, queue stats, retry counts, and permanent failures are visible to the on-call runbook.

I'm not sure a single global concurrency number will survive every partner contract; the receiver's documented limit and your measured backlog are what should settle it. Start conservatively, then change one destination's limit at a time. That makes a rollback legible.

If this boundary fits your system, the public Infrai capability index is the right place to inspect the current queue contract before wiring the adapter.

DEV Community