The crash window that turns webhook retries into duplicate side effects

A webhook handler can return 200 OK, log "processed", and still create the same charge, email, or entitlement twice.

The bug usually lives in a small crash window between acceptance and completion.

A provider only knows whether your endpoint accepted a delivery. It does not know whether your business operation completed exactly once. If the connection drops, your process crashes, or the response times out, the provider may retry. That retry is correct behavior. The duplicate side effect is our bug.

The dangerous sequence

A common handler looks harmless:

async function handleWebhook(event: ProviderEvent) {
  const seen = await db.events.find(event.id);
  if (seen) return;

  await sendWelcomeEmail(event.customerId);
  await db.events.insert({ id: event.id, status: "done" });
}

Now place a crash after sendWelcomeEmail succeeds but before the insert commits.

The provider sees no successful response and retries. The database still says the event is new, so the handler sends a second email. Replace email with createShipment, issueRefund, or grantCredits and the incident becomes expensive.

This is also why a check-then-insert guard is not enough. Two workers can receive the same event at nearly the same time. Both check, both see nothing, and both execute the effect before either inserts the marker.

Acceptance is not completion

Separate the inbound transport decision from the business-work decision.

1. Verify the signature and parse the event.
2. Atomically record the provider event ID under a unique constraint.
3. Return a success response once durable acceptance is complete.
4. Process the durable record asynchronously.

The unique insert decides which delivery owns the work. A duplicate delivery becomes a no-op at the database boundary rather than after an external side effect.

async function acceptWebhook(raw: Buffer, headers: Headers) {
  const event = verifyAndParse(raw, headers);

  const inserted = await db.transaction(async tx => {
    return tx.inbox.insertIfAbsent({
      provider: "stripe",
      eventId: event.id,
      payload: event,
      state: "pending",
      attempts: 0,
    });
  });

  return { status: 200, duplicate: !inserted };
}

The database needs a unique key such as (provider, event_id). Application-level checks without a constraint still race.

Make completion durable too

The worker processing the inbox record needs explicit states. pending, leased, completed, and terminal_failure are more useful than a single boolean.

A lease lets a crashed worker's job become eligible again. Store lease_until, increment attempts, and let another worker claim the row only after the lease expires. Do not hold a database transaction open while calling a remote provider.

If processing must update local state and enqueue another action, use a transactional outbox:

await db.transaction(async tx => {
  await tx.accounts.applyEntitlement(accountId, plan);
  await tx.outbox.insert({
    kind: "send_receipt",
    key: `receipt:${event.id}`,
    payload: { accountId, plan },
  });
  await tx.inbox.markCompleted(event.id);
});

The local state change, outbox entry, and completion marker now commit together. A separate dispatcher sends the receipt.

Stable idempotency keys cross the final boundary

An outbox makes local intent durable, but the dispatcher can still crash after the remote provider accepts a request and before local completion is recorded.

When the remote API supports idempotency, send a stable key derived from the logical operation, not from the attempt:

await emailProvider.send(message, {
  idempotencyKey: outbox.key,
});

Every retry of the same logical operation must reuse the same key. A random key per attempt defeats the mechanism.

When the remote API does not support idempotency, classify the operation honestly. Some actions are naturally safe to repeat. Some can be reconciled by querying the remote system. Some remain ambiguous and require a manual review state instead of blind retries.

Retry classification matters

A retry policy should distinguish:

• transient failures: timeouts, connection resets, 429, and selected 5xx responses;
• permanent failures: invalid payloads, authentication failures, and most 4xx responses;
• ambiguous outcomes: the request may have succeeded, but the response was lost.

Use bounded exponential backoff with jitter. Honor Retry-After. After the attempt budget is exhausted, move the operation to a visible terminal state with enough context to investigate. Infinite retry is not resilience; it is a quiet denial-of-service against your own queue and the provider.

Reconciliation closes the gap

Even a strong design needs a repair loop. Periodically compare local records with the provider's authoritative state. Find accepted events stuck in leased, outbox records with ambiguous outcomes, and local entitlements that disagree with payment state.

Reconciliation turns an invisible inconsistency into a queued repair or an explicit alert.

Test the crash points, not only the happy path

A useful failure-path suite kills the process at each boundary:

1. before the inbox insert;
2. after the insert but before the response;
3. after claiming work but before the side effect;
4. after the side effect but before completion is recorded;
5. during outbox dispatch;
6. while two workers race on the same event.

Then replay the same provider event and assert the business invariant: one shipment, one entitlement transition, one refund, or one message.

Exactly-once delivery is rarely available end to end. What you can build is durable acceptance, atomic local transitions, stable idempotency at remote boundaries, bounded retries, and reconciliation. That combination makes at-least-once delivery safe enough for real systems.

A concrete implementation example

I applied the same contract in Posnic/POS PR #476, a focused webhook retry and dead-letter contribution now under maintainer review. I also isolated a separate lost sync-response race in Activepieces: a fast worker can publish before the API registers its listener, causing a false timeout after a successful run. The patch adds deterministic tests for timeout, 400, 500, success after retry, and permanent failure. It also verifies that retries reuse the same delivery ID, stop at a fixed attempt budget, persist only safe error categories, and strip shop/time details from dead-letter records.

That distinction is worth testing explicitly: a transport timeout and a 500 are candidates for bounded retry; most 4xx responses are permanent until the request changes. Treating them all alike either drops recoverable work or hammers a request that can never succeed.

Use this 7-question preflight checklist to review an integration. If one provider boundary is already failing, FlowPatch offers a $99 fixed-scope failure-path diagnostic with a risk matrix, retry/state rules, patch plan, and failure-path tests.