Short answer: recover missed account-platform webhooks from an immutable delivery record, then redrive only events whose authorization, identity, and ordering checks still pass. Keep the provider-facing replay operation separate from your consumer-owned dead-letter queue (DLQ). That separation gives an incident commander a readable audit trail and lets the application retry without asking the provider to guess your business state.
The hard constraint is access auditability. During an outage, an operator must be able to answer four questions for every event: who requested recovery, which original delivery was selected, what payload was sent, and what happened after the retry. A useful design makes those answers derivable from retained records rather than from a chat transcript or a mutable dashboard.
What must be true before a replay is safe?
Start with an event identity that survives retries. A provider delivery identifier is useful, but it is not always a business idempotency key. Store both: the immutable delivery id and a deterministic event id derived from the signed payload or an explicitly documented event field. Keep the raw bytes needed for signature verification, plus a parsed envelope for indexing. Parsing alone is not enough; a re-serialized JSON document can differ in whitespace or key order and invalidate a signature.
The next invariant is authorization. A replay endpoint should require a narrowly scoped credential, record the actor, and bind the request to an incident or change ticket. OWASP recommends controlled secret storage, rotation, and avoiding secrets in source code or logs. Apply that rule to replay credentials as well as ordinary API keys. A redrive token in a shell history is an audit gap.
The third invariant is bounded selection. Never make an operator paste an unbounded time range into a replay job. Accept an explicit set of delivery ids, or a query with a maximum count and a reviewable filter. I use a dry-run that returns the candidate count and a digest of selected ids before any message is emitted. The digest is compact evidence that the approved set and the executed set were the same.
The fourth invariant is a duplicate-safe consumer. At-least-once delivery means a retry can arrive after the original request eventually succeeds. The consumer should record the event id and outcome in one transaction with its domain change, or use an equivalent idempotency mechanism. Exactly-once behavior cannot be inferred from a successful HTTP response.
That is the audit boundary.
Delivery history is an audit log, not a mailbox
A delivery record should be append-only from the operator's perspective. A practical record contains: event id, delivery id, tenant id, received timestamp, payload hash, signature verification result, attempt number, response status, response timestamp, actor or system principal, and a retention class. Keep a pointer to encrypted payload storage when the payload contains personal or contractual data. The index remains searchable without exposing the body to every on-call user.
Retention needs arithmetic. Suppose an account platform receives 1,000,000 events per day and the retained envelope plus metadata averages 2 KB. Thirty days is about 60 GB before replication, indexes, and encryption overhead (1,000,000 x 2 KB x 30). That number is not a price argument; it is a capacity and access-control argument. Longer retention may help an audit, but it also widens the set of records that must be protected and eventually deleted. Set retention by contractual and incident requirements, then document the exception path.
Telemetry has the same shape. Sampling successful deliveries can control byte volume, but sampling failures makes incident reconstruction probabilistic. I keep every failure transition and sample repetitive success traces after the delivery record has been durably written. Labels such as tenant id, endpoint URL, and exception text have high cardinality; placing them in a metric label can create an index larger than the event data. Put those values in structured logs with access controls, and keep metric dimensions bounded to status class, region, and retry outcome.
A delivery history query should show the chain without pretending that a provider's status is your consumer's state. For example, 202 Accepted means the receiving HTTP server accepted the request for processing; it does not prove that the account mutation committed. Record both the transport result and the application result, with separate timestamps.
How can I replay missed platform webhook events safely?
Use a two-stage command: select, then execute. The selection response is signed or stored with the incident record. Execution reads that exact selection, checks that each event is still within retention and authorization policy, and places work onto your queue with an idempotency key. It should not call the provider for each item in a tight loop; a provider outage is the reason for recovery, and a burst can turn a consumer incident into a rate-limit incident.
A generic interface can be small and still auditable:
curl -sS -X POST https://events.example.invalid \
-H 'Authorization: Bearer ${RECOVERY_TOKEN}' \
-H 'Content-Type: application/json' \
--data '{"delivery_ids":["d_01","d_02"],"reason":"incident-8472","dry_run":true}'
curl -sS -X POST https://queue.example.invalid \
-H 'Authorization: Bearer ${RECOVERY_TOKEN}' \
-H 'Content-Type: application/json' \
--data '{"selection_id":"sel_9f2","rate_per_second":25,"require_digest":"sha256:..."}'
The rate is a control surface, not a magic constant. Start below the consumer's observed sustainable throughput, watch queue age and downstream error rate, and increase in measured steps. Keep a circuit breaker that pauses redrive when authorization checks fail, duplicate rate rises above the expected baseline, or downstream latency crosses the incident threshold. A pause must be resumable from the last acknowledged item; restarting from the beginning is how duplicate storms happen.
Pause first.
Ordering deserves an explicit policy. If account events are causally ordered, partition by account id and preserve sequence numbers within each partition. If the provider exposes no sequence, do not invent one from arrival time; mark the stream as unordered and make the consumer reconcile state. A replay that restores an old update after a newer update can be worse than a missed event, so the reconciliation path is part of recovery, not an optional cleanup.
Your own DLQ should capture terminal consumer failures with the original delivery id, attempt history, and a machine-readable failure class. Keep the DLQ payload immutable and attach operator annotations as separate records. Redriving from the DLQ then becomes a local queue operation with the same audit controls as provider replay. The two paths can converge on one idempotent work handler, but they should retain different provenance fields: provider_replay and consumer_redrive.
This design has limits. It is not suitable for a provider that exposes no durable delivery record unless an earlier capture layer exists. It is also a poor fit for consumers with irreversible side effects; manual reconciliation is safer than automatic redrive. That trade-off favors a slower, review-heavy process even when the queue is growing, and teams should choose that boundary deliberately.
Which evidence should an incident review require?
An incident review can be concise if the data model did the work. Export the approved selection digest, actor identity, authorization decision, count attempted, count accepted, count rejected, and final consumer outcomes. Include the retention policy version and the code version of the redrive worker. Hashes provide tamper evidence, but they do not replace access controls or key rotation.
I also compare the four counts rather than trusting a single dashboard number:
| Count | Meaning |
|---|---|
| Selected | Deliveries approved by the bounded query |
| Enqueued | Items written to the recovery queue |
| Applied | Events whose idempotent handler committed |
| Quarantined | Items held for policy, ordering, or data review |
A gap between selected and enqueued points to the recovery service. A gap between enqueued and applied belongs to consumer processing. Quarantined items should never disappear into a generic retry counter; they need a named owner and a next decision.
Access logs should answer who viewed payload bytes, not only who started a job. Separate metadata access from body access, and redact credentials, authorization headers, and unnecessary personal fields before indexing. OWASP's guidance is useful here because replay tooling concentrates privileged secrets and sensitive payloads in one workflow.
A compact rollout for an account platform
Ship the record schema and idempotency check before shipping the replay button. In a staging environment, inject a timeout after the consumer commits but before the HTTP response, then verify that a retry produces one domain change and two transport attempts. Inject an expired signature and confirm that the event is quarantined without exposing its payload to the general operator role. Finally, run a bounded selection against a fixture set and compare the stored digest with the execution request.
For production, gate the feature behind an incident role, require a reason, and default to dry-run. Keep a small canary batch, observe queue age and downstream saturation, and expand only after the canary reaches a terminal state. The rollback is a pause plus a queue drain policy; deleting delivery history removes the very evidence needed to explain the incident.
The durable decision rule is simple: replay only from records you can authenticate, authorize, bound, and reconcile. If any one of those properties is missing, preserve the event for investigation and fix the control before increasing throughput.
Top comments (0)