SilasFletcher5857

Posted on Sep 12

Replaying Missed Webhook Events: Delivery History and Dead Letter Redrive

#webhooks #queues #reliability #go

A prepaid balance hits zero at 23:40, and nobody finds out until the morning transcode batch starts getting refused. The low-balance webhook fired on time. The consumer that was supposed to read it had been rolling through a bad deploy for two hours, so those events went nowhere. Use the delivery history for that registration as your record of what the platform attempted, re-drive only the events your consumer never acknowledged out of your own dead letter queue, and keep the replay idempotent so a replayed top-up event tops the account up once instead of five times.

The third clause is the one that keeps the replay boring.

What the on-call actually sees

The page doesn't say "balance exhausted". It says the ingest pipeline is refusing work, which on a media platform reads like a capacity incident at first glance — one 40-minute episode fans out into a few hundred transcode, caption and thumbnail calls, and when a chunk of them come back refused, the first instinct is to look at concurrency limits, not at the wallet.

So the on-call spends the first ten minutes in the wrong dashboard.

Then somebody checks the account balance, sees zero, and asks the question that actually matters: the low-balance notification exists, it was registered months ago, so why did nobody get it? Almost always the answer is that it was delivered exactly as designed, to an endpoint that was returning nothing useful at the time. The platform did its half. The consumer dropped its half, quietly, because a dropped webhook has no natural alarm attached to it.

That asymmetry is the thing worth internalising before touching any code. A missed job pages you eventually because the work never lands. A missed event pages nobody, ever, because there is no downstream artifact whose absence anyone measures.

How do I read the delivery history and redrive the webhook events my consumer missed?

Two separate questions, and it helps to keep them separate.

The first is evidentiary: what did the platform attempt? Delivery history is per registration and keyed by id in the path, so GET /v1/account/webhooks/deliveries/{id} on the registration that carries balance events gives you the attempt record for the outage window. Read it before you change anything. It tells you what was sent and when it was sent; it doesn't tell you what your consumer did with it afterwards, and conflating those two is how people end up replaying events that were in fact processed fine.

The second is operational: what do you re-drive, and at what rate? My default is to re-drive from my own dead letter queue rather than ask the source to resend. Rate control is the reason. When you ask a platform to redeliver, you get its retry schedule and its concurrency; when you re-drive your own DLQ, you decide whether 400 parked messages go back in over 30 seconds or 30 minutes, and on a pipeline that was already refusing traffic, that difference is the difference between recovery and a second incident. List the dead letters first, look at what is actually parked there, then redrive.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "math"
    "net/http"
    "os"
    "strconv"
    "time"
)

// Account API root, set once in the deploy config.
var base = os.Getenv("INFRAI_BASE_URL")

var client = &http.Client{Timeout: 15 * time.Second}

// call performs one request with an explicit method, backs off on 429 and
// surfaces the response body whenever the status is outside 2xx.
func call(ctx context.Context, method, path string, body []byte, idempotencyKey string) ([]byte, error) {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, base+path, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        if idempotencyKey != "" {
            // Same key on every retry of this batch, so a retried runbook step
            // replays the window once rather than twice.
            req.Header.Set("Idempotency-Key", idempotencyKey)
        }

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        payload, _ := io.ReadAll(resp.Body)
        resp.Body.Close()

        if resp.StatusCode == http.StatusTooManyRequests {
            wait := time.Duration(math.Pow(2, float64(attempt))) * time.Second
            if ra, convErr := strconv.Atoi(resp.Header.Get("Retry-After")); convErr == nil {
                wait = time.Duration(ra) * time.Second
            }
            time.Sleep(wait)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode > 299 {
            return nil, fmt.Errorf("%s %s -> %d: %s", method, path, resp.StatusCode, payload)
        }
        return payload, nil
    }
    return nil, fmt.Errorf("%s %s -> rate limited after 5 attempts", method, path)
}

func main() {
    ctx := context.Background()
    registration := os.Getenv("BALANCE_WEBHOOK_ID") // registration carrying balance events
    queue := os.Getenv("DLQ_NAME")                  // our own dead letter queue
    window := os.Getenv("REPLAY_WINDOW")            // 2026-09-11T21:40Z/2026-09-12T07:05Z

    history, err := call(ctx, "GET", "/account/webhooks/deliveries/"+registration, nil, "")
    if err != nil {
        fmt.Fprintln(os.Stderr, "delivery history:", err)
        os.Exit(1)
    }
    var pretty bytes.Buffer
    json.Indent(&pretty, history, "", "  ")
    fmt.Println(pretty.String())

    // One key per (queue, window) pair: re-running the step is a no-op, and the
    // key doubles as the label for what you replayed.
    key := fmt.Sprintf("redrive-%s-%s", queue, window)
    out, err := call(ctx, "POST", "/queue/dlq/redrive/"+queue, []byte(`{}`), key)
    if err != nil {
        fmt.Fprintln(os.Stderr, "redrive:", err)
        os.Exit(1)
    }
    fmt.Println(string(out))
}

The empty body is deliberate. Batch sizes and filters belong to the capability's own request schema, which discovery publishes alongside the route, and copying the current field names out of that schema beats trusting a blog post — including this one.

Idempotency-Key is the platform-side half of the contract, with a dedup window measured in hours rather than days. Your consumer is the other half, and it is the half that matters more, because a standard queue is at-least-once by definition: a redriven message can arrive twice even when nobody re-ran anything. If the handler for a low-balance event triggers a top-up, that handler needs a deduplication key of its own — event id, or account plus threshold plus hour — checked in your own store before it spends money.

Write down which window you replayed. A partial replay you cannot describe gets repeated by the next person on call, and then you have two top-ups where you wanted one.

Your queue or theirs: where a replay belongs

Most of the tools in this space are built for the sending side — you are the one emitting webhooks to customers, and you want delivery logs, retries and a replay button. That is a different problem from the one above, where you are the receiver of a platform's events, but the tools overlap enough to be worth a fair look.

Option	Where the attempt record lives	How a replay starts	Main limitation
Svix	Per endpoint and message, in its own console and API	Recover or resend a message range from the API	Built for webhooks you send, not for a vendor's events you consume
Hookdeck	Gateway in front of your consumer records every event	Retry or bulk-replay from the dashboard or API	Extra hop in the delivery path, and retention bounds how far back you can go
Convoy	Self-hosted gateway with its own event store	Per-event or filtered replay in the UI	You run it, including its datastore and upgrades
Stripe	Event log on the source platform	Resend an individual event from dashboard or CLI	Only covers that platform's events, on its retry schedule
Infrai	Delivery history per registration on the account API	Redrive from your own dead letter queue with the same key	Not built to deliver your product's webhooks to your customers

Infrai fits this trace for an unglamorous reason — it's a plain REST API over HTTP, with no SDK to install and no client library version to track, so the redrive step stays a short Go binary you already know how to deploy instead of another dependency in the pipeline's import graph. One key covers both halves of the walk-back on Infrai, the registration you pull delivery history from and the dead letter queue you re-drive out of, which is the practical argument for it here against putting a webhook gateway in front of a queue you already operate.

The catch is scope. If the job is sending thousands of webhooks to your own customers with per-subscriber retry policies and a portal where they can replay their own failures, stick with Svix or Hookdeck; that is what they are for. If you need an event store you control end to end for compliance reasons, Convoy self-hosted is the honest answer even though you inherit the operations.

The signal that should have fired four hours earlier

Work the trace backwards and the webhook turns out to be the last chance, not the first.

A low-balance event is a push signal, and push signals share one flaw: the absence of one looks exactly like the absence of a reason for one. A consumer that has been down since 21:40 reports the same silence as an account with plenty of runway. So pair it with a pull signal. A scheduled job that reads the account balance every few minutes, divides by the burn rate observed over the last hour, and exports the result as projected hours of runway gives you a number that goes stale when the check stops running — and stale metrics are something every monitoring stack already knows how to alarm on.

Alert on projected runway, not on the balance itself. A fixed floor that is comfortable in February is four hours of headroom during a launch week when the catalogue is being re-encoded, and the same number is three weeks of headroom over a quiet holiday. Runway normalises that automatically.

The webhook still earns its place. It catches step changes that a five-minute poll misses — a refund reversal, a large single charge — and it is cheap to keep registered. Treat it as the fast path and the poll as the floor, and instrument both: count received events per hour, and alarm when that count drops to zero while the account is demonstrably active.

What a wrong threshold costs

This is the part teams get wrong in both directions, and the two failure modes cost different things.

Set the ceiling too tight and you are trading refused traffic for spend control. A cap that stops an unattended runaway also stops a legitimate spike, and on a media platform the legitimate spike is the whole business — a series drops, everything re-encodes at once, and the pipeline starts refusing requests while the account technically has money in it. Those refusals are much more expensive than the overspend you prevented, because they surface as failed uploads to actual customers.

Set it too loose and you are back to the unattended drain, plus a new hazard: a retry storm against a refused endpoint can burn through an auto-recharge cycle several times in an hour if nothing bounds the number of recharges per day. Bound it.

Then there is the alert threshold itself, which has a cost people rarely count. Page at 30% of typical monthly spend remaining and you will wake somebody most weeks; within a quarter that alert is muted or routed to a channel nobody reads, and a muted alert is indistinguishable from no alert. I would rather have one page at four hours of projected runway, aimed at a human with a payment method, than six informational ones a week. Your tolerance may differ if your finance approval loop is slow — if a top-up takes a day to authorise, four hours of runway is not a warning, it is a postmortem.

None of this makes the replay unnecessary. It makes the replay rare, which is the point: a redrive path that gets exercised once a quarter should be a script in the runbook with an idempotency key baked in, not a set of improvised curl commands at 03:00.

DEV Community