Short answer: keep an append-only delivery history, replay only named event IDs, and redrive through an idempotent worker that records tenant attribution before it touches billing. A small manifest and a dry run are more valuable than a large retry button.
| Situation | Pick this path | Why |
|---|---|---|
| One missed lease event | Replay its event ID | The blast radius stays visible |
| A burst of 20–50 failed deliveries | Build a bounded manifest from history | Operators can review tenants before enqueueing |
| The consumer rejects valid retries | Pause replay and fix the contract | Repeating a permanent failure only grows the queue |
| Billing ownership is unclear | Reconcile first | Attribution matters more than throughput |
The workflow below treats provider history as evidence, not as a control plane. Your service owns the decision to replay, the audit trail, and the final accounting check.
What should a property team record before replaying missed webhook events?
Start at ingress. Write the provider event ID, tenant ID, received timestamp, attempt number, HTTP status, body hash, and consumer version to durable storage before placing the message on a queue. Keep the original body encrypted and restrict access; the OWASP Secrets Management Cheat Sheet describes the same rotation, access-review, and audit expectations that apply to replay credentials.
In a tabletop run, imagine 32 lease-update events from a 14-minute outage window. The manifest should show 32 unique IDs but perhaps only 11 tenant IDs, because several buildings generated multiple updates. For each row, resolve the tenant-to-account mapping as it existed at receipt time, then record the mapping that will be used for replay. Compare those two values before enqueueing. A mismatch is not a reason to silently choose the newest account: hold the row for reconciliation, attach the operator's reason, and leave the event out of the batch. After the dry run, store the manifest checksum with the replay request. That single checksum lets a reviewer prove that the set inspected is the set processed, even if a tenant changes ownership while the worker is draining the queue.
Give every operator action a replay_id, a reason, an identity, and a bounded filter. A filter can be a set of event IDs or a narrow time interval with a maximum count. Log that request before work starts. If nobody can answer “which tenants will this touch?”, the filter is too broad.
A green delivery status does not prove that a lease change reached the ledger. An HTTP 2xx can be emitted before a database commit, and a process restart can leave the account unchanged. Acknowledge only after the transaction commits, then make the handler idempotent on the event ID.
That detail is easy to miss.
How can you replay missed platform webhook events, read delivery history, and redrive your own dead letter queue?
Use four explicit stages: discover, plan, enqueue, verify. Discovery is read-only. Planning creates a manifest with event IDs, tenant IDs, source timestamps, and the expected destination account. Enqueueing attaches a replay marker, a rate limit, and a deduplication key. Verification compares applied state with the manifest and separates success, duplicate, and permanent-failure outcomes.
The provider adapter should be the only code that knows a vendor’s history interface. The rest of the application can depend on a tiny boundary like this:
type Delivery = {
eventId: string;
tenantId: string;
body: unknown;
attempts: number;
lastStatus: number | null;
};
type ReplayPlan = {
replayId: string;
deliveries: Delivery[];
};
export async function buildReplayPlan(
eventIds: string[],
history: { getByEventId(id: string): Promise<Delivery | null> },
): Promise<ReplayPlan> {
const deliveries = (await Promise.all(
eventIds.map((id) => history.getByEventId(id)),
)).filter((delivery): delivery is Delivery => delivery !== null);
if (deliveries.length === 0) throw new Error("Replay set is empty");
return { replayId: crypto.randomUUID(), deliveries };
}
The worker sends Idempotency-Key: webhook:{event_id} to the application boundary and persists each attempt. A duplicate is a valid result. For ordering-sensitive lease changes, partition by tenant_id and process each partition serially while allowing parallel work across tenants.
Before enqueueing, run the manifest against a staging consumer or a dry-run validator. Check that the tenant still maps to the same account, that the schema version is accepted, and that the event is inside the contractual recovery window. A moved tenant is not a reason to guess; it is a reconciliation case.
When should a replay enter an owned dead letter queue?
Classify failures before choosing a retry count. Timeouts and 503 responses are usually transient. Schema validation, an unknown tenant, or a missing authorization grant is usually permanent until data or configuration changes. Exponential backoff with jitter limits pressure on the consumer, but it cannot repair a bad payload.
After the retry budget is exhausted, put the original event, error class, attempt timestamps, consumer version, and replay_id in your DLQ. Encrypt the message store. Restrict redrive permissions. Expire records according to the retention policy, and alert on message age as well as queue growth.
For a redrive, require a new redrive_id and a reviewed reason. Re-check tenant mapping at execution time. Then emit a completion record linking that redrive to every outcome. Support can now explain one tenant’s invoice without searching several worker logs.
Keep provider delivery and application recovery separate. The provider history answers “what was sent and when?” Your DLQ answers “what did our code reject, and what changed before we tried again?” Mixing those ledgers makes a duplicate look like a missing event.
Limits and practical decision checks
Replay cannot reconstruct an event that was never retained. If the history window is shorter than the recovery promise, add a durable archive or schedule a domain-level reconciliation job. Your mileage may vary when a provider exposes aggregate metrics but no per-event body; capture an ingress record in your own system before queueing.
This pattern is not suitable when the downstream action is inherently non-idempotent and cannot accept an event key. Use a transactional outbox, a ledger, or a human approval step in that case. Stick with a managed replay facility when your team cannot staff access reviews, retention, paging, and reconciliation; owning a DLQ buys control and also creates operational work.
I start every incident review with two numbers: the count of unique event IDs and the count of affected tenant IDs. They are rarely the same. A replay that reports only HTTP 200 hides that difference, while a manifest makes it testable.
The decision rule is plain: redrive only what you can name, attribute, and verify.
Top comments (0)