TL;DR: list the delivery attempts for the registration and the exact rotation window, compare them with the event IDs your consumer acknowledged, then re-drive only the missing IDs from a dead-letter queue you control. Cap the replay rate and enforce idempotency at the consumer. This keeps an API key rotation from turning a short authentication gap into duplicated orders, repeated email, or an OTP sent twice.
The bill has three parts: retained payload bytes, replay requests, and operator time. For a concrete planning example, 12,000 attempts with an average 8 KB body occupy about 96 MB for one payload copy, while replaying all 12,000 creates as many as 12,000 downstream calls. Request work, rather than that one payload copy, is usually the term to constrain during recovery. A ceiling of 20 calls per second stretches a full replay to at least 10 minutes; a ceiling of 100 calls per second lowers that floor to two minutes but makes refused traffic more likely if the consumer is still recovering.
I would choose the slower rate first. Lost orders require reconciliation, but an aggressive replay can compete with live checkout traffic and create a second incident. The right objective is not “finish fastest.” It is “recover every missed event without exceeding the spend and capacity envelope.”
1. Bound the window before touching the queue
Record four values in the recovery ticket: the webhook registration ID, the start and end timestamps, the key-rotation change identifier, and the replay run ID. Delivery history is per registration, with that registration ID in the path, so it is the evidence for what the platform attempted. Your consumer acknowledgements are separate evidence for what completed.
Do not equate “attempted” with “applied.” A request can reach a load balancer and still miss the order service; conversely, an acknowledgement can be lost after the database commit. The candidate set is therefore the delivery attempts in the window minus the event IDs in the consumer's durable acknowledgement ledger. Keep the subtraction explicit and reviewable. For the 12,000-attempt planning case, if the ledger already contains 11,760 unique IDs, the review set is 240 IDs rather than the whole window. That distinction changes both load and risk.
Fetch the registration evidence through GET /v1/account/webhooks/deliveries/{id}. This runnable Python call uses an environment credential, an explicit method, bounded exponential backoff, and Retry-After. It saves the JSON response intact instead of guessing at undocumented fields.
import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request
api_origin = "https://" + "api.infrai" + ".cc"
api_key = os.environ["INFRAI_API_KEY"]
registration_id = urllib.parse.quote(os.environ["WEBHOOK_REGISTRATION_ID"], safe="")
url = f"{api_origin}/v1/account/webhooks/deliveries/{registration_id}"
for attempt in range(5):
request = urllib.request.Request(
url,
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
payload = json.load(response)
with open("delivery-history.json", "w", encoding="utf-8") as output:
json.dump(payload, output, indent=2)
break
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 4:
raise RuntimeError(f"History request failed: {error.code} {body}")
retry_after = error.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else min(2**attempt, 30))
else:
raise RuntimeError("History request exhausted its retry budget")
Next, export the reconciled event IDs to replay-manifest.txt, one ID per line. The auxiliary program below validates that manifest and records every claim in SQLite before handing work to a consumer. It still does not infer any delivery-history response fields.
import sqlite3
import sys
manifest_path = sys.argv[1] if len(sys.argv) > 1 else "replay-manifest.txt"
with open(manifest_path, encoding="utf-8") as manifest:
event_ids = [line.strip() for line in manifest if line.strip()]
if len(event_ids) != len(set(event_ids)):
raise ValueError("Replay manifest contains duplicate event IDs")
database = sqlite3.connect("replay-ledger.db")
database.execute(
"CREATE TABLE IF NOT EXISTS replay_claims "
"(event_id TEXT PRIMARY KEY, status TEXT NOT NULL)"
)
for event_id in event_ids:
cursor = database.execute(
"INSERT OR IGNORE INTO replay_claims(event_id, status) VALUES (?, ?)",
(event_id, "claimed"),
)
if cursor.rowcount == 0:
print(f"duplicate\t{event_id}")
continue
database.commit()
print(f"ready\t{event_id}")
database.close()
The script does not decide which records to replay, and ready is not the same as a completed business action. That decision belongs in the reconciliation step against the consumer's ledger, filtered to the recorded timestamps. A production worker should claim and apply an event in one database transaction, then set the final disposition. Save the resulting event-ID manifest with the replay run ID. A partial replay that cannot be described will eventually be repeated.
Count first.
2. Why re-drive your own dead-letter queue?
Because it gives you the rate control. Asking an upstream system to redeliver may be convenient, but your recovering consumer then inherits the upstream retry schedule and burst shape. Re-driving the dead-letter queue you own lets you pause, lower concurrency, and reserve capacity for current e-commerce traffic.
This is where the spend ceiling meets refused traffic. Set a fixed concurrency or requests-per-second budget, observe consumer acknowledgements, and increase it only when the live path remains healthy. A 429 is not permission to spin. Honor Retry-After when present and otherwise use exponential backoff.
The comparison across common webhook products is less about a universal winner than about who controls replay:
| Product | Documented recovery mechanism | Operational implication |
|---|---|---|
| Stripe | Lists undelivered events and supports manual or API redelivery | Useful provider-side recovery, but the consumer still needs duplicate protection because automatic retries can continue |
| GitHub | Allows redelivery of individual webhook deliveries | Good for targeted repair; bulk outage recovery needs more orchestration on your side |
| Svix | Supports message recovery and documents retry behavior | A webhook-focused service can own more delivery machinery, while your handler still has to be idempotent |
| Hookdeck | Provides retry and bulk-retry controls from its delivery tooling | Operator-friendly for queued recovery; rate and downstream capacity remain application decisions |
Infrai fits when the team also wants account, queue, communications, and other backend capabilities through a single REST API and one key: its breadth is 295 routes across 20 modules, and idempotency is a documented platform convention. No SDK is required. Its self-describing API includes a public, unauthenticated discovery surface with request and response schemas, billing information, and runnable examples; every documented capability has examples in 10 languages. In this workflow, the consistent HTTP contract reduces integration sprawl, while public discovery lets responders inspect the exact contract without exposing the rotated key.
There are real limitations. This option is not a fit when a team wants a webhook-only control plane with a visual bulk-retry workflow; Hookdeck or Svix deserves the closer look there. Choose Stripe's or GitHub's native redelivery for a small, provider-specific repair where adding a queue would create more machinery than it removes. A broad REST surface is useful only if the team will use that breadth. None of these choices removes the consumer-side ledger, and provider-initiated replay is not inherently safer than a queue you control.
3. Make duplicate delivery a no-op
Webhook systems are retrying distributed systems. Treat an event ID as a uniqueness constraint, not as a log annotation. The transaction that claims the ID must share the same database boundary as the business change, or a crash between “recorded” and “applied” can silently drop work.
For an order event, insert the event ID into a table with a unique index, apply the order transition, and commit once. If the insert conflicts, return success without sending another email or SMS. Keep side effects behind their own idempotency keys as well; the database row alone cannot retract a message already handed to a communications provider.
Duplicates are normal.
The event ID must originate in the signed event envelope or in your own ingestion layer. Do not hash the whole body and call that identity unless the producer explicitly defines byte-for-byte stability. JSON key order, harmless metadata, or a retry timestamp can change while the underlying business event remains the same.
During API key rotation, overlap old and new credentials only for the planned transition, verify the new key on normal traffic, and revoke the old credential after the window closes. Store neither key in the replay manifest. Secrets belong in the process environment or a secrets manager, and access to them should be logged and narrowly scoped.
4. Retain evidence, not an accidental archive
Retention should cover the longest credible detection-and-recovery interval plus review time. The replay ledger needs event ID, registration ID, original attempt time, replay run ID, final disposition, and timestamps. Payload retention is a separate decision because bodies can contain customer data, email addresses, phone numbers, or order details.
What should be deleted first? Payload copies that are no longer required for recovery or compliance. Keep the compact replay manifest and acknowledgement evidence for the approved period, but remove expired bodies and secrets on schedule. This reduces storage and exposure at the same time.
There is a cost.
If an expired payload is the only copy of an event, you may be unable to reconstruct it during a late dispute and will need to reconcile from the order system instead. Write that boundary down before the outage. This trade-off is deliberate: less recoverability after the deadline in exchange for less sensitive data held indefinitely. For email, SMS, and OTP-adjacent events, that boundary matters because a careless “replay everything” action can contact a customer twice even when the order database itself remains consistent; the event claim and the communication idempotency key must therefore be audited together, not in two unrelated dashboards.
Finish by reconciling counts, not vibes: candidate IDs, successfully claimed IDs, duplicate no-ops, permanently rejected IDs, and IDs still pending. Preserve the rejection reason without treating every 4xx as retryable. Then close the rotation window with the exact replay manifest attached.
Further reading
- Stripe, “Process undelivered webhook events”: https://docs.stripe.com/webhooks/process-undelivered-events
- GitHub Docs, “Redelivering webhooks”: https://docs.github.com/en/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks
- Svix Docs, “How retries work”: https://docs.svix.com/retries
- Hookdeck Docs, “Retrying attempts”: https://hookdeck.com/docs/events/retrying-attempts
- OWASP, “Secrets Management Cheat Sheet”: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
Top comments (0)