DEV Community

FairchildBlake8483
FairchildBlake8483

Posted on

Configure Queue Backpressure for Rate-Limited Webhook Processing in 2026 (HTTP-Based)

#sre #webhooks #queues #ratelimiting

Use the HTTP cron task as a producer, then let a durable queue meter webhook work at the consumer. The queue should carry an idempotency key, a delivery deadline, and enough attempt metadata to make retry decisions without holding the web request open. For a marketplace cleanup job, that means the scheduler can finish quickly while workers remove stale reservation state at a rate the downstream service can tolerate.

The important rule is this: retry transport failures and explicit throttling responses, but make the cleanup operation safe to repeat before you increase concurrency. A queue limits pressure; it does not make a non-idempotent webhook safe.

The example uses a periodic cleanup of expired marketplace reservations.

To configure the queue, set the rate-limiting policy at the worker boundary and keep the HTTP-based cron task responsible only for discovery and enqueueing.

What failure is the queue meant to contain?

A periodic cleanup usually has a lumpy workload. One run might find 40 expired marketplace reservations; another might find 4,000 after a delayed reconciliation. Sending every webhook from the HTTP request couples the scheduler's timeout to the downstream service's capacity. It also creates an unpleasant failure shape: the request times out, the caller retries, and the same cleanup batch is submitted twice.

I start by separating three clocks:

Clock Owner Decision
Schedule interval cron task When to discover work
Visibility deadline queue and worker When an unfinished message may be retried
Retry backoff worker How quickly a failed delivery returns

Those clocks should not be silently substituted for one another. A five-minute schedule is not a five-minute retry delay, and a worker timeout is not proof that the downstream webhook was never accepted. The last case is where duplicate deliveries are born.

The producer should enqueue one message per logical cleanup action, rather than one giant message for the whole scan. Include a stable key such as reservation-cleanup:{reservation_id}:{expiry_epoch}. The exact key is an application decision, but it must be derived from the business action, not from a random request ID.

How should I configure a queue so HTTP-based webhook processing stays rate-limited?

A worker should acquire capacity before making the HTTP call. A token bucket, a fixed interval between sends, or a concurrency-plus-rate policy can work; the choice depends on whether the remote limit is expressed as requests per second, concurrent requests, or both. Keep the limiter in the worker layer so a queue backlog remains visible instead of turning the scheduler into a hidden rate-limiting mechanism.

I initially treated the queue's retry count as the important safety setting. The harder problem is the acknowledgement boundary: a timeout can hide a successful remote write, so retry policy and idempotency have to be designed together.

When the endpoint returns 429 Too Many Requests, use the server's Retry-After value when it is present. HTTP defines that field as either a delay in seconds or an HTTP date, so parsing must handle both forms. For a transient 5xx, a network timeout, or a connection failure, use bounded exponential backoff with jitter. Do not retry validation failures or permanent authorization failures as if they were capacity problems.

Here is the shape I want reviewers to see in code. The transport is generic; the policy is explicit.

package worker

import (
    "context"
    "fmt"
    "math/rand"
    "net/http"
    "strconv"
    "strings"
    "time"
)

func retryDelay(resp *http.Response, attempt int) (time.Duration, bool) {
    if resp != nil && resp.StatusCode == http.StatusTooManyRequests {
        if value := resp.Header.Get("Retry-After"); value != "" {
            if seconds, err := strconv.Atoi(strings.TrimSpace(value)); err == nil && seconds >= 0 {
                return time.Duration(seconds) * time.Second, true
            }
            if when, err := http.ParseTime(value); err == nil {
                delay := time.Until(when)
                if delay < 0 {
                    delay = 0
                }
                return delay, true
            }
        }
        return backoff(attempt), true
    }

    if resp == nil || resp.StatusCode >= 500 {
        return backoff(attempt), true
    }
    return 0, false
}

func backoff(attempt int) time.Duration {
    if attempt < 1 {
        attempt = 1
    }
    base := time.Duration(1<<min(attempt-1, 5)) * time.Second
    return base + time.Duration(rand.Int63n(int64(time.Second)))
}

func min(a, b int) int {
    if a < b {
        return a
    }
    return b
}

func deliver(ctx context.Context, client *http.Client, endpoint string, body []byte, key string) error {
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, strings.NewReader(string(body)))
    if err != nil {
        return err
    }
    req.Header.Set("Idempotency-Key", key)
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("webhook status %s", resp.Status)
    }
    return nil
}
Enter fullscreen mode Exit fullscreen mode

This snippet deliberately leaves queue acknowledgement out. A worker acknowledges only after the business operation has reached its durable success state. If the process dies after the remote service accepts the request but before acknowledgement, the message can be delivered again. That is an expected at-least-once delivery consequence, not a reason to pretend the queue provides exactly-once effects.

What makes cleanup idempotent under duplicate delivery?

The receiver needs a durable record keyed by the action identifier. On receipt, it should either apply the cleanup and record completion in one transactional boundary, or use a state transition that makes a repeated cleanup harmless. For a reservation, active -> expired is safer than "subtract one inventory unit" because a second request sees an already-expired record and has no second decrement to perform.

The outbox or inbox record needs a retention policy longer than the maximum retry horizon. Otherwise a very late duplicate can look new. Store the key, status, timestamps, and a compact result; do not put an unbounded response body in the queue message. This is an operational decision worth writing down because changing retention changes the duplicate risk.

A dead-letter queue is the quarantine path for messages that exhaust the retry policy. It should preserve the original message and enough attributes to identify the failure, while an operator-facing replay process should require an explicit decision. Automatic replay of every dead letter is how a remote outage becomes a sustained retry storm. The queue documentation for dead-letter queues describes this separation of repeatedly unsuccessful messages from the main queue.

How do I verify the runbook before production?

Test the boundaries, not only the happy path. A useful harness returns 429 with a delay, closes the connection after accepting a request, returns a permanent 400, and holds a request past the client timeout. Then assert four things: the producer finishes independently of delivery time; the limiter caps sends; retries respect the selected policy; and duplicate keys do not duplicate the cleanup effect.

During rollout, watch queue age, visible message count, retry count by status class, dead-letter count, worker concurrency, and downstream response latency. Queue depth alone is late evidence. A rising age with a flat enqueue rate says capacity or retry policy is wrong; a rising 429 rate says the worker is exceeding the remote budget even if the queue is shrinking.

Rollback should be boring: stop consumers or reduce their permitted rate, keep the producer from adding new work if discovery itself is unsafe, and preserve queued messages for later inspection. Do not delete the queue to clear a page. After the downstream limit recovers, replay a small, known set of dead letters and compare idempotency records before restoring normal concurrency.

The trade-off is real. A queue adds persistence, worker operations, and replay decisions; it is a poor fit for a tiny action that must complete synchronously before the caller can continue. For a bursty periodic cleanup, keeping the HTTP request open is the worse boundary because it couples discovery to downstream capacity.

The decision is therefore less about which queue has the most knobs. It is about whether the complete path has a bounded retry policy, observable backpressure, and a receiver that can safely see the same action twice. For periodic marketplace cleanup, those properties matter more than keeping the original HTTP request alive.

References

Top comments (0)