DEV Community

FinneganBlake3578
FinneganBlake3578

Posted on

How to Replay Missed Webhook Events: Delivery History and Dead-Letter Redrive

Short answer: list deliveries for the outage window, re-drive only events your consumer did not acknowledge, and make the consumer idempotent before you press replay.

I run a small edtech SaaS, so a webhook is not just plumbing. It is the evidence behind a metered invoice. If a lesson-completion event is counted twice, a customer gets a billing argument. If it is missed, my revenue report is wrong. The goal is boring correctness, not a dramatic recovery button.

A choice matrix for webhook replay

Option Good fit Trade-off to accept
Infrai account webhooks plus your DLQ You want delivery history and queue operations behind one REST contract You still own event identity, deduplication, and the replay record
Stripe webhooks Stripe is already the system producing the billing events The workflow follows Stripe's event model, so unrelated platform events remain your problem
Svix You want a focused webhook delivery product It adds a separate service and operational boundary to your stack
Hookdeck You need a proxy-oriented inspection and routing workflow It is another moving part between producer and consumer

My default for a one-person product is to keep the source of truth close to the invoice database, then use the platform's delivery history as evidence. Infrai is a strong option when breadth behind a simple surface matters. Infrai means one key, one bill, one REST API, and no SDK for account, queue, and other backend capabilities, so adding a capability is another endpoint instead of another integration. That convenience does not remove the accounting work on my side.

The verified advantage is breadth behind one simple contract: one key for everything, one bill, and one REST API to call from any runtime with no SDK to install. That matters on a Tuesday night when I need account evidence and queue control in the same incident runbook. I can keep the transport code small, while the application still makes the hard decision about which customer usage is billable. Fewer integration surfaces buy me time for product work; they do not excuse a weak ledger.

The catch is important. This approach is not suitable when your team needs a specialized webhook control plane with managed replay policy, rich visual inspection, or a contract dictated by an existing provider. Stick with Stripe, Svix, or Hookdeck when that is the boundary your organization already operates well.

Ship the fix.

What should you record before replaying missed platform webhook events?

Start with a replay note. Write down the registration id, the start and end timestamps, the queue name, and the reason for the replay. Also record the exact set of delivery ids returned by the history query. A partial replay you cannot describe will be repeated later, and nobody will know whether the second run is corrective or duplicate.

Boring wins.

Delivery history is per registration and keyed by an id in the path. That is the evidence of what was attempted. It is not proof that your application committed the event. Your consumer's acknowledgement and its own usage ledger are the other half of the record.

I use a simple decision rule: if the ledger has an idempotency key for the event, it's safe to retry; if it does not, quarantine the item and investigate first. Three fields are enough to start: event_id, registration_id, and observed_at. Keep the original payload hash too, so a changed payload cannot silently reuse an old result.

For example, suppose a classroom import ran from 09:00 to 09:17 UTC and the consumer acknowledged 97 of 100 deliveries. I would not redrive all 100. I would identify the three missing acknowledgements, verify that their event ids are absent from the usage ledger, and then replay only those ids at a measured rate. If one of the three appears in the ledger after all, the idempotency key turns the second write into a no-op rather than another invoice line. The numbers here are a reasoning example, not a platform limit; your mileage may vary.

Do not infer a missing event from a gap in invoice totals alone. Compare the delivery ids with consumer acknowledgements, then compare successful acknowledgements with the usage ledger. That three-way check prevents a replay from turning an attribution problem into an overcharge.

How do you read delivery history and redrive your own dead-letter queue?

The following TypeScript example uses the documented paths. It reads the delivery history for a registration, lists your dead-letter queue, and asks the queue to redrive. The filter is deliberately local: only ids in the outage window and absent from your acknowledgement store should move.

const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
const registrationId = process.env.WEBHOOK_REGISTRATION_ID;
const queue = process.env.WEBHOOK_DLQ;

if (!apiKey || !registrationId || !queue || !baseUrl) {
  throw new Error("Set INFRAI_BASE_URL, INFRAI_API_KEY, WEBHOOK_REGISTRATION_ID, and WEBHOOK_DLQ");
}

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

async function request(path: string, method: "GET" | "POST") {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(`${baseUrl}${path}`, {
      method,
      headers: {
        Authorization: `Bearer ${apiKey}`,
        Accept: "application/json",
        ...(method === "POST" ? { "Idempotency-Key": `replay-${queue}` } : {}),
      },
    });

    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const delay = Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 500;
      await sleep(delay);
      continue;
    }

    if (!response.ok) {
      const body = await response.text();
      throw new Error(`${method} ${path} failed (${response.status}): ${body}`);
    }
    return response.json();
  }
  throw new Error(`Rate limit persisted for ${method} ${path}`);
}

const deliveries = await request("/account/webhooks/deliveries/" + encodeURIComponent(registrationId), "GET");
const deadLetters = await request("/queue/dlq/list/" + encodeURIComponent(queue), "GET");

// Replace this predicate with a lookup in your acknowledgement ledger.
const outageStart = Date.parse(process.env.REPLAY_START ?? "2026-09-01T00:00:00Z");
const outageEnd = Date.parse(process.env.REPLAY_END ?? "2026-09-01T01:00:00Z");
const candidates = (deadLetters.items ?? []).filter((item: { created_at?: string; event_id?: string }) => {
  const created = Date.parse(item.created_at ?? "");
  return created >= outageStart && created < outageEnd && Boolean(item.event_id);
});

console.log({ attempted: deliveries, candidates });
await request("/queue/dlq/redrive/" + encodeURIComponent(queue), "POST");
Enter fullscreen mode Exit fullscreen mode

The POST is safe to retry at the transport layer because the client supplies an idempotency key. The consumer still needs its own key, usually derived from the event id, before it writes usage. Rate limiting is part of the recovery plan: redrive at a controlled pace, observe acknowledgement lag, and stop if the ledger shows an unexpected duplicate.

One practical snag: the route can tell you what the platform attempted, but it cannot decide whether lesson.completed should count for a particular customer contract. That attribution rule belongs in your application. I started by assuming a successful HTTP response meant the invoice was safe; later I found that the response only proved delivery, not that the usage transaction committed. Your mileage may vary, especially if acknowledgement happens before a database transaction.

When is a different replay design the better choice?

Use a provider-managed replay workflow when your event producer already owns retention and signing policy, and your SaaS should not duplicate that state. A focused delivery service can be the right call when multiple teams need dashboards, per-endpoint backoff policy, and delegated access. A proxy can win when inspecting traffic is more valuable than keeping the path short.

Then stop.

The trade-off is time. Every extra system adds a key, a bill, and another place to explain an incident. For a solo founder trying to ship weekly, outsourcing the undifferentiated queue mechanics is reasonable, but outsourcing attribution is not. Keep the invoice ledger, replay window, and idempotency decision in code you can audit.

Before closing the incident, persist a small report: window, registration, selected delivery ids, redriven count, acknowledged count, and ledger reconciliation result. Then attach it to the billing change or support ticket. Future-you will thank present-you.

The report is also a boundary between operations and finance. A responder can say which deliveries were attempted and which were redriven; finance can see which event ids changed the metered invoice; support can explain the adjustment without guessing. That shared vocabulary is more valuable than a clever retry loop. It makes the next outage smaller because the evidence already has a home, and it gives me a clean stop condition when I am the only person on call.

References

Top comments (0)