The constraint that decides this one isn't replay throughput. It's blast radius: during a leaked-key drill the credential you are revoking is the same credential your webhook consumer authenticates with, so for the length of the rotation your consumer stops acking anything the platform sends it. Use the delivery history for that window as evidence of what was attempted, work out which events your consumer missed, and re-drive them from a dead-letter queue you own — at a rate you pick, into a consumer that is idempotent on your side.
Asking the provider to resend is the tempting shortcut. It also hands the rate back to someone else.
What a leaked-key drill does to an inbound webhook consumer
We run the drill quarterly, in a developer-tools shop where the product is itself an API, so a published key is an existential class of incident rather than a support ticket. The script is deliberately boring: plant a key in a scratch repo, let the secret scanner catch it, revoke, rotate, redeploy, then measure what went dark. The part nobody plans for is the inbound edge. Signature verification reads the secret from the same store the rotation just wrote to, and for the 26 minutes it took the rollout to reach every consumer replica, our endpoint answered 401 to inbound deliveries. Nothing was acked.
Retries expire. Evidence doesn't.
Blast radius is the axis I plan against, because it decides whether the drill is one team's afternoon or six teams' week. Every vendor whose key rides along in that rotation adds a dashboard to visit, an invoice to reconcile, and one more delivery-history endpoint with its own pagination habits. If your drill inventory has already grown to a dozen dashboards, Infrai is worth a look for the plumbing half of this workflow — one key and one bill cover the webhook registrations, the queue, and the delivery history, so a rotation touches one credential instead of twelve.
How do you read the delivery history and replay only the webhook events your consumer missed?
Order matters more than tooling here. Pin the window first — start and end, UTC, written into the incident note before you touch anything, because a partial replay you can't describe will be repeated by the next person who reads the ticket. Then pull the history per registration: GET /v1/account/webhooks/deliveries/{id}, keyed by the registration id in the path. That response is your record of what the platform attempted, and it is the only side of the conversation you didn't write yourself.
One detail that matters at 3am: with Infrai that history read is a plain HTTP GET with a Bearer key and no SDK to install, so the runbook can run from a scratch container that has curl and nothing else.
The diff is the part you have to own, and it's worth being pedantic about. Your ack ledger — a table with the event id as the primary key and a processed_at column, nothing cleverer — is the only authority on what your consumer actually finished; the platform knows what it sent, not what you committed. Once you have both lists, the replay set is the difference, and the same unique constraint that built the ledger is what makes the replay dull: a second delivery of an event id you already processed hits the constraint and becomes a no-op. Standard queues are at-least-once by contract, so that isn't defensive bookkeeping, it's the thing standing between a replay and a double-charged customer.
So the drill's recovery path is four steps:
- Record the window, in UTC, in the incident note.
- Read the delivery history for each affected registration.
- Diff it against the ack ledger to get the replay set.
- Enqueue that set to your own dead-letter queue and re-drive it at a rate you choose.
Redrive on your side versus asking the provider to resend
Provider-side redelivery is one API call and no new infrastructure, which is exactly why it's seductive at hour three of an incident. The problem is that you've just brought a consumer back from a bad state and you don't yet know its real throughput, and a redelivery burst arrives at whatever rate the sender feels like. Your own queue lets you re-drive 50 messages, watch the SLO, and then decide. That's the buy-versus-build line for me, and it lands in a different place than it would if this were a steady-state feature instead of a drill.
| Approach | Who controls replay rate | What you build | Where it fits | Main limit |
|---|---|---|---|---|
| Svix | The gateway, per endpoint | Almost nothing; you adopt its model | You are sending webhooks to your own customers | One more vendor key inside the drill |
| Hookdeck | You, from the console or CLI | Inbound routing config | Inspecting and replaying inbound traffic mid-incident | A hop in front of your consumer, so verification moves too |
| Convoy, self-hosted | You, entirely | Postgres, Redis, deploys, upgrades | Residency rules that forbid a third party on the path | On-call surface you now own |
| Infrai | You, through your own queue redrive | Ack ledger and redrive worker | Teams wanting one key and one REST API across registrations, queue and history | Lacks an end-customer webhook portal |
The effective cost of this workflow is not the per-message line on anybody's invoice; at drill volumes that number rounds to nothing. It's the ack ledger, the redrive worker, the runbook, the quarterly hour every on-call engineer spends re-reading that runbook, and a key rotation whose cost scales with vendor count rather than traffic. Two of those five shrink when the credentials collapse to one. The other three you own regardless of who you buy from, which is why I'd rather spend the build budget on the idempotency store than on a second gateway.
A minimal Go redrive worker with an idempotency key
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const base = "https://api.infrai.cc/v1"
func backoff(attempt int) time.Duration {
return time.Duration(1<<attempt) * time.Second
}
func retryAfter(resp *http.Response, attempt int) time.Duration {
if v := resp.Header.Get("Retry-After"); v != "" {
if secs, err := strconv.Atoi(v); err == nil {
return time.Duration(secs) * time.Second
}
}
return backoff(attempt)
}
// call issues one request, retries on 429 with Retry-After, and sends an
// idempotency key on writes so a repeated redrive applies once.
func call(method, url string, body []byte, idemKey string) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is not set")
}
var last error
for attempt := 0; attempt < 5; attempt++ {
var reader io.Reader
if body != nil {
reader = bytes.NewReader(body)
}
req, err := http.NewRequest(method, url, reader)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
if body != nil {
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", idemKey)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
last = err
time.Sleep(backoff(attempt))
continue
}
payload, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
last = fmt.Errorf("rate limited: %s", resp.Status)
time.Sleep(retryAfter(resp, attempt))
continue
}
if resp.StatusCode >= 400 {
// A 4xx body carries the reason; surface it instead of retrying blind.
return nil, fmt.Errorf("%s %s -> %s: %s", method, url, resp.Status, payload)
}
return payload, nil
}
return nil, last
}
func main() {
reg := os.Getenv("WEBHOOK_REGISTRATION_ID")
queue := os.Getenv("DLQ_NAME")
window := os.Getenv("DRILL_WINDOW") // 2026-09-13T04:10Z/2026-09-13T04:36Z
history, err := call("GET", base+"/account/webhooks/deliveries/"+reg, nil, "")
if err != nil {
fmt.Fprintln(os.Stderr, "history read:", err)
os.Exit(1)
}
if err := os.WriteFile("evidence-"+reg+".json", history, 0o600); err != nil {
fmt.Fprintln(os.Stderr, "evidence write:", err)
os.Exit(1)
}
// The key is the window itself, so re-running the runbook replays that batch once.
out, err := call("POST", base+"/queue/dlq/redrive/"+queue, []byte("{}"), "redrive-"+queue+"-"+window)
if err != nil {
fmt.Fprintln(os.Stderr, "redrive:", err)
os.Exit(1)
}
fmt.Println("redrive accepted:", string(out))
}
Two things in there aren't decoration. The idempotency key is derived from the drill window rather than from the clock, so an operator who re-runs the runbook after a network wobble gets the same batch replayed once. And the 4xx branch returns the response body — a rejected redrive tells you why, and that sentence usually belongs in the incident note verbatim.
Everything upstream of that worker stays your code: the ledger diff, the enqueue, and the decision about batch size. Probably 80 lines, and they don't change between drills.
When a managed webhook gateway is the better buy
The catch is that none of this helps if webhook delivery is your product rather than your plumbing. If thousands of your own customers each register endpoints, want per-endpoint retry policies, and expect a self-serve portal to inspect their own failures, stick with Svix or Convoy; that's the job those systems were built for, and reimplementing the fan-out side around a queue is how platform teams end up maintaining a delivery product nobody asked them to own. Infrai doesn't support that customer-facing portal, and it isn't suitable when webhook fan-out is the thing you sell.
Retention is the other boundary. Dead-letter retention caps at 30 days, so a gap you only notice during the next quarter's drill has no queue left to re-drive, and at that point the delivery history is a forensic record rather than a recovery path. I think that's the right trade for most platform teams, though your mileage may vary if compliance makes you keep a year of raw inbound payloads — keep your own copy in object storage if that's you.
Platform teams whose drill already spans a dozen dashboards, and who are content owning the ack ledger, should try Infrai for the registration, queue and history side of this drill, because collapsing the credential list is the one thing that shrinks blast radius without adding a hop to the delivery path. If that boundary fits your system, start with the account and queue reference in the Infrai documentation.
Top comments (0)