Reliable Webhooks in TypeScript: Idempotency, Deduplication and Failure Recovery

Webhook providers deliver at least once, not exactly once. That single fact explains most webhook bugs: duplicate emails, double-credited balances, orders marked paid twice. Here is a pattern for handling deliveries safely with a Postgres table and a small amount of TypeScript, plus the failure cases it is designed around.

What the provider actually promises

Read the delivery semantics of the provider you integrate with. Stripe's documentation says endpoints may receive the same event more than once and recommends logging processed event IDs to skip repeats (Stripe webhooks docs). GitHub allows redelivery of a delivery, and each delivery carries a unique ID in a header (GitHub: handling webhook deliveries). The details differ, but the shape is the same: retries happen, ordering is not guaranteed, and the receiver has to cope.

So the receiver needs three properties:

1. Authenticity. Only accept payloads you can verify.
2. Idempotency. Processing the same event twice has the same effect as processing it once.
3. Recoverability. If processing fails halfway, you can retry without corrupting state.

Step 1: Verify the signature against the raw body

Signature verification must use the exact bytes the provider signed. The classic mistake is parsing JSON first and re-serializing it, which changes whitespace and key order.

import crypto from "node:crypto";

export function verifySignature(rawBody: Buffer, header: string, secret: string) {
  const expected = crypto.createHmac("sha256", secret).update(rawBody).digest("hex");
  const a = Buffer.from(expected);
  const b = Buffer.from(header);
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

Header formats vary by provider (some include a timestamp, some a prefix such as sha256=), so adapt this to the provider's spec. Use a constant-time comparison, and reject stale timestamps if the provider supplies one.

Step 2: Record the event before you act on it

The core of deduplication is a table with a unique constraint on the provider's event ID.

CREATE TABLE webhook_events (
  provider      text        NOT NULL,
  event_id      text        NOT NULL,
  event_type    text        NOT NULL,
  payload       jsonb       NOT NULL,
  status        text        NOT NULL DEFAULT 'received',  -- received | processing | processed | failed
  attempts      int         NOT NULL DEFAULT 0,
  last_error    text,
  received_at   timestamptz NOT NULL DEFAULT now(),
  processed_at  timestamptz,
  PRIMARY KEY (provider, event_id)
);

Insert with ON CONFLICT DO NOTHING (PostgreSQL INSERT docs). The insert itself is the deduplication check, and it stays correct under concurrent deliveries because the database enforces uniqueness, not your application code.

const inserted = await db.query(
  `INSERT INTO webhook_events (provider, event_id, event_type, payload)
   VALUES ($1, $2, $3, $4)
   ON CONFLICT (provider, event_id) DO NOTHING
   RETURNING event_id`,
  [provider, event.id, event.type, event]
);

if (inserted.rowCount === 0) {
  // Seen before. Acknowledge so the provider stops retrying.
  return res.status(200).end();
}

Notice what this does not do: it does not check "have I processed this?" before inserting. A read-then-write check has a race window. Two simultaneous deliveries can both read "not seen" and both proceed.

Step 3: Separate acknowledging from processing

Providers typically time out slow endpoints and retry. If your handler does heavy work inline, a slow downstream call can trigger a retry while the first attempt is still running.

A safer shape is: verify, store, return 2xx quickly, process asynchronously. A queue is the usual tool, but a simple worker that polls webhook_events for status = 'received' is enough for an early-stage product and has fewer moving parts.

Claim work with a row lock so two workers do not take the same event:

UPDATE webhook_events
SET status = 'processing', attempts = attempts + 1
WHERE (provider, event_id) = (
  SELECT provider, event_id FROM webhook_events
  WHERE status = 'received'
  ORDER BY received_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

Step 4: Make the business effect idempotent too

Deduplicating the event is not enough, because the same real-world action can arrive under different event IDs (a provider retry that creates a new event, or a manual replay). Where a handler changes money, entitlements, or state, anchor the effect to a natural key.

For example, record a payment against the provider's payment ID with a unique constraint, rather than incrementing a balance:

await db.query(
  `INSERT INTO payments (provider_payment_id, user_id, amount_cents)
   VALUES ($1, $2, $3)
   ON CONFLICT (provider_payment_id) DO NOTHING`,
  [payment.id, userId, payment.amount]
);

Compute balances from rows, or update them in the same transaction as the insert. The rule of thumb: an operation that is naturally "set to X" is safer than one that is "add 1."

Do the state change and the status = 'processed' update in a single transaction. If the process dies between them, you will either redo the whole thing (safe, because of the natural key) or none of it.

Step 5: Plan for out-of-order delivery

Do not assume invoice.paid arrives after invoice.created. Two defenses work well:

• Fetch current state when it matters. Treat the webhook as a notification and read the object from the provider's API before acting, so a late, old event cannot overwrite newer state.
• Use version or timestamp guards. Only apply an update if the incoming event is newer than what you have stored.

Also decide what happens when an event references something you have not seen yet. Parking it as failed with a retry is better than dropping it.

Step 6: Give failures an explicit state

An event that fails permanently should not disappear into logs. With the table above, a retry policy is a few lines:

async function handle(row: WebhookEvent) {
  try {
    await process(row);
    await markProcessed(row);
  } catch (err) {
    const exhausted = row.attempts >= MAX_ATTEMPTS;
    await markFailed(row, String(err), exhausted ? "failed" : "received");
  }
}

Retries with backoff for transient errors, a terminal failed state after the limit, and an alert on that state. A failed row with the original payload and last error is something a person can inspect and replay after fixing the bug. That recovery path is the reason to store the payload at all.

Failure scenarios worth testing

Write tests for these specifically. They are the ones that break in production:

• The same event delivered twice, sequentially and concurrently.
• A handler that throws after a partial side effect.
• A process crash between processing and marking processed.
• An older event arriving after a newer one.
• A valid payload with an invalid signature, and a valid signature on a modified body.

Limits of this approach

A Postgres table is a reasonable default for a young product, but it is not a universal answer. High event volume may justify a proper queue, and retention needs a plan so the events table does not grow without bound. If handlers call external APIs that are not idempotent, you also need idempotency keys on those outbound calls. The IETF has an in-progress draft for an Idempotency-Key header that describes the idea (draft-ietf-httpapi-idempotency-key-header).

Summary

Verify the raw body, record the event ID under a unique constraint, acknowledge fast, process separately, anchor effects to natural keys, and give failures a visible state. None of it is clever, and that is the point: webhook code should be boring enough that retries stop being scary.

---

I'm Muhammad Abdullah, a full-stack engineer working as AbdullahBuilt. I build SaaS products, integrations and AI-assisted workflows.

Disclosure: this post was drafted with AI assistance and edited and reviewed by me.