DEV Community

EllsworthPierce7528
EllsworthPierce7528

Posted on

Fintech Cleanup Sweeps: Webhook Queue Rate Limits, Delayed Retries, and a DLQ

#go #queues #webhooks #sre

The trade-off that settles this is not price per message. It's whether the queue hands you delayed jobs, per-destination rate limiting, and a DLQ you don't have to rebuild in application code — pick the one whose retry semantics you can make idempotent cheaply, because retry amplification, not the invoice, is what gets a periodic cleanup paged at 03:00.

Everything below is that one decision, taken apart.

Cleanup that outlives the HTTP request

The job is unglamorous. A fintech ledger carries card authorization holds that expire on a schedule; once a hold passes its expiry, the money has to be released in the ledger and the merchant has to be told over a webhook so their own reconciliation lines up. US and EU tenants live in separate regions for residency reasons, so it is two sweeps with two queues, not one global one. Every merchant endpoint publishes a limit — 5 requests per second is a normal number — and some of them enforce it with a hard 429 rather than a queue of their own. So the sweep is not "release N rows"; it is "release N rows at a pace someone else controls, exactly once, in two jurisdictions, without a human watching."

Running that inside an HTTP handler is the original sin.

The shape of that failure is always the same: someone wires the sweep to an admin endpoint, the gateway cuts the connection at its 60-second limit, the caller (or the load balancer, or an impatient operator) retries, and now two sweeps walk the same rows. Half the holds get released twice, the merchant gets two webhooks with different payload ids, and their reconciliation opens a ticket that lands back on you. Missed jobs are annoying. Duplicate deliveries are the ones that generate finance-side work, and they are the reason the idempotency question outranks the pricing question in this design.

The trigger is not the reliability boundary either. Scheduled CI triggers are convenient — GitHub's own documentation says scheduled workflows can be delayed during periods of high load, so treat the tick as a hint, not a guarantee. Whatever fires the sweep, the sweep itself has to be safe to run twice, late, or concurrently with its own previous run.

What a rate-limited sweep actually costs

Per-message pricing is the smallest line in this system, and reasoning about it first is how teams end up with a cheap queue and an expensive on-call rotation.

Do the arithmetic on amplification instead. Take 50,000 expired holds in a nightly window, a 2% slice of destinations answering 429, and a worker that retries five times on a fixed one-second backoff with no jitter: that slice turns 1,000 deliveries into 6,000 requests, all aimed at the endpoints that were already asking you to slow down, all in the same second because fixed backoff synchronizes retries. The queue charge for those extra messages is rounding error. The costs that actually show up are the merchant's rate limiter escalating to longer bans, your egress and log volume (per-attempt structured logs are what you need in a postmortem, and they are billed per line), and the engineer who has to decide at 03:00 whether the DLQ contains 400 poison payloads or one broken endpoint.

There is a deployment cost too. A queue with server-side pacing means the limit is a config value you can change without a deploy; a queue without it means the limiter lives in your worker, and every limit change is a code change, a rollout, and a chance to get it wrong under pressure.

Should a rate-limited webhook queue own delayed jobs, retries, and the DLQ?

Compare the candidates on where each mechanism lives, not on the sticker. Caps and defaults move, so I re-read these pages before I commit a design to them.

Option Delayed jobs Pacing per destination Failure landing zone
Amazon SQS (standard) DelaySeconds up to 15 minutes consumer-side; you write the limiter redrive policy to a DLQ after maxReceiveCount
RabbitMQ / CloudAMQP delayed-message-exchange plugin, or TTL plus dead-letter routing consumer-side, via prefetch and your own token bucket dead-letter exchange; quorum queues add a delivery-limit
Google Cloud Tasks scheduleTime up to 30 days out queue-level: max dispatches per second and max concurrent dispatches retry config only — no dead-letter queue, so you record the terminal failure yourself
Upstash QStash delay on publish, HTTP push model per-endpoint controls, no long-lived consumers inspectable DLQ you can re-publish from

Read that table as four different answers to one question: who holds the pacing state. Cloud Tasks puts it in the queue and gives you nothing to catch terminal failures; SQS and RabbitMQ put it in your worker and give you a real landing zone for poison messages. Neither arrangement is wrong. But the one you choose decides how much of the next section you have to build yourself, and that is the real bill.

For a two-region sweep the practical constraint is usually duller than any feature matrix: a queue you can run in both us-east and eu-central with the same worker binary, so the EU deployment isn't a fork of the US one.

Idempotency keys make redelivery boring

At-least-once delivery is the contract every one of these systems actually offers. The design goal is not zero duplicates in flight — it is that a duplicate does nothing the second time.

That starts before the queue. The sweep plans work and hands out keys; it never releases a hold itself.

// sweep.go — the sweep only plans work. Nothing is released in here.
type Hold struct {
    ID        string
    AccountID string
    Region    string // "us" or "eu": separate queues, same binary
    ExpiresAt time.Time
}

// Stable across every redelivery: no attempt counter, no run id, no wall clock.
func idemKey(h Hold) string {
    sum := sha256.Sum256([]byte("hold.released|" + h.ID + "|" + h.ExpiresAt.UTC().Format(time.RFC3339)))
    return hex.EncodeToString(sum[:])
}

func planSweep(ctx context.Context, db *sql.DB, q Queue, now time.Time) (int, error) {
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return 0, err
    }
    defer tx.Rollback()

    rows, err := tx.QueryContext(ctx, `
        SELECT id, account_id, region, expires_at
          FROM auth_holds
         WHERE state = 'active' AND expires_at < $1
         ORDER BY expires_at
         LIMIT 500
           FOR UPDATE SKIP LOCKED`, now)
    if err != nil {
        return 0, err
    }
    defer rows.Close()

    n := 0
    for rows.Next() {
        var h Hold
        if err := rows.Scan(&h.ID, &h.AccountID, &h.Region, &h.ExpiresAt); err != nil {
            return n, err
        }
        // Spread the batch instead of firing 500 messages at one endpoint.
        delay := time.Duration(n/25) * time.Second
        if err := q.Enqueue(ctx, h.Region, Message{Key: idemKey(h), HoldID: h.ID}, delay); err != nil {
            return n, err
        }
        n++
    }
    if err := rows.Err(); err != nil {
        return n, err
    }
    return n, tx.Commit()
}
Enter fullscreen mode Exit fullscreen mode

Enqueue-then-commit is a dual write, and I left it that way on purpose: if the process dies between the last Enqueue and Commit, the next tick re-plans the same holds and the queue receives the same keys again. That's survivable precisely because each key comes from the hold and its expiry rather than from the run. A transactional outbox removes the dual write entirely and is the better shape once the sweep matters enough to justify the extra table — the reason to know both is that the key discipline is what makes either one safe.

The worker side does one attempt and reports one of three outcomes. Retry policy is data, not if statements scattered through delivery code.

// worker.go — one attempt, three outcomes. The queue decides what happens next.
type Outcome int

const (
    Done Outcome = iota
    RetryLater
    DeadLetter
)

func attempt(ctx context.Context, d Delivery, lim *rate.Limiter, secret []byte, hc *http.Client) (Outcome, time.Duration, error) {
    if err := lim.Wait(ctx); err != nil { // one bucket per endpoint, never one global bucket
        return RetryLater, 5 * time.Second, err
    }

    ts := strconv.FormatInt(time.Now().Unix(), 10)
    mac := hmac.New(sha256.New, secret) // HMAC-SHA256, RFC 2104
    mac.Write([]byte(ts))
    mac.Write([]byte("."))
    mac.Write(d.Body)

    req, err := http.NewRequestWithContext(ctx, http.MethodPost, d.Endpoint, bytes.NewReader(d.Body))
    if err != nil {
        return DeadLetter, 0, err // a malformed endpoint will not get better on attempt 4
    }
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", d.Key)
    req.Header.Set("X-Hold-Signature", ts+"."+hex.EncodeToString(mac.Sum(nil)))

    resp, err := hc.Do(req)
    if err != nil {
        return RetryLater, backoff(d.Attempt), err
    }
    defer resp.Body.Close()
    io.Copy(io.Discard, resp.Body) // drain, so the connection gets reused

    switch {
    case resp.StatusCode == http.StatusTooManyRequests:
        // The receiver's own limiter outranks ours.
        return RetryLater, retryAfter(resp.Header.Get("Retry-After"), backoff(d.Attempt)), nil
    case resp.StatusCode >= 500:
        return RetryLater, backoff(d.Attempt), nil
    case resp.StatusCode >= 400:
        return DeadLetter, 0, fmt.Errorf("rejected: %d", resp.StatusCode)
    default:
        return Done, 0, nil
    }
}
Enter fullscreen mode Exit fullscreen mode

Three details in there carry most of the reliability. backoff must include jitter, or every retry from a batch lands in the same millisecond and re-triggers the limit that caused it. Retry-After is a documented HTTP header and receivers that send it are telling you the exact answer — honouring it is cheaper than guessing. And the Idempotency-Key header only helps if the receiver stores it; when a merchant ignores it, the fallback is that your own release step writes to a ledger table with a unique constraint on the key, so the second delivery is a no-op insert and the webhook is just noise.

Testing this is less work than it sounds. A fake endpoint that returns 429 for the first two calls, then 200, then 500 forever, exercises every branch above in a unit test, and a counter of distinct Idempotency-Key values versus total requests is the single metric worth alerting on. If that ratio drifts above 1.0 in normal operation, your retry policy is louder than your idempotency guarantees.

Where this advice stops working

The catch is scale in the other direction. If the sweep touches a few hundred rows in one region and no external endpoint is rate limiting you, a queue is a second system to operate for no benefit — a cron-triggered job with an advisory lock, a LIMIT clause, and an in-process limiter is honestly fine, and it is less to page on.

Ordering is the other boundary. Standard queues are not suitable when releases for one account must land in order; that pushes you toward FIFO-style grouping or an ordered log, and both cost throughput. If the cleanup grows human approvals, compensation steps, or multi-day waits, a message queue isn't the right tool at all — that's workflow-engine territory, and bolting state machines onto retry policies is how the state machine ends up in your DLQ.

I'm not sure there's a universal answer on where the limiter belongs. Queue-side pacing has fewer moving parts, worker-side pacing gives you finer control per destination and works identically in both regions; your mileage may vary with how many endpoints you deliver to. What doesn't vary is the ordering of the decision: settle idempotency first, then retry policy, then the DLQ, and only then look at what any of it costs per million messages.

Sources

Top comments (0)