DEV Community

JC TechLabs
JC TechLabs

Posted on

Webhooks Arrive Twice, Late and Out of Order: Designing Idempotent Handlers

Every webhook provider promises at-least-once delivery, which means your handler will see the same event more than once. Design for that on day one, or a payment gets marked paid twice, a confirmation email goes out three times, or an order ships before the refund lands.

Payment gateways, CRMs, e-commerce platforms and messaging APIs all retry when your endpoint is slow or returns a non-2xx status. Network timeouts mean the provider often cannot tell whether you processed the event, so it sends it again. Some providers also let an operator replay events from a dashboard. And because events are delivered over independent HTTP requests, a payment.captured can arrive before the payment.authorized that logically preceded it.

None of that is a bug in the provider. It is the contract. This post walks through a handler design in four steps, Verify, Record, Process and Publish, that stays correct under duplicates, delays, replays and reordering. The examples are a reference scenario using TypeScript and PostgreSQL, not a description of any client system. A Python/FastAPI implementation with tests follows in a companion post.

The failure modes you are designing for

Write these down before writing code, because each one maps to a specific control:

Failure mode What goes wrong without a control Control
Forged or tampered request Anyone who finds the URL can mark orders paid Signature check on the raw body
Captured request replayed later A valid old request is resent by an attacker Timestamp tolerance plus dedupe
Provider retry after timeout Same event processed twice Unique constraint on event ID
Slow handler Provider times out, retries, load grows Store first, process asynchronously
Crash between DB write and side effect Email sent but DB not updated, or the reverse Transactional outbox
Out-of-order delivery Older state overwrites newer state State precedence, not arrival order
Operator replay of a week of events Mass duplicate side effects All of the above, tested

Verify: authenticate the exact bytes you received

Most providers sign each delivery with an HMAC (commonly SHA-256) over the request body, sometimes combined with a timestamp, and send the result in a header. Three details decide whether verification actually works.

Use the raw body. If your framework parses JSON and you re-serialize it before hashing, key order or whitespace can change and valid signatures fail. Worse, teams then "fix" it by disabling the check. Capture the raw bytes first:

import crypto from "node:crypto";
import express from "express";

const app = express();

app.post(
  "/webhooks/payments",
  express.raw({ type: "application/json" }), // req.body is a Buffer
  async (req, res) => {
    const ok = verifySignature(req.body, req.header("x-signature"), secrets());
    if (!ok) return res.sendStatus(401);
    // ...Record step below
  }
);
Enter fullscreen mode Exit fullscreen mode

Compare in constant time, and check the timestamp. A plain string comparison can leak timing information. A timestamp inside the signed payload lets you reject captured requests replayed hours later.

function verifySignature(body: Buffer, header: string | undefined, keys: string[], toleranceSec = 300) {
  if (!header) return false;
  const parts = Object.fromEntries(header.split(",").map((p) => p.split("=", 2)));
  const ts = Number(parts.t);
  if (!Number.isFinite(ts) || Math.abs(Date.now() / 1000 - ts) > toleranceSec) return false;

  const signed = Buffer.concat([Buffer.from(`${ts}.`), body]);
  const given = Buffer.from(String(parts.v1 ?? ""), "hex");
  return keys.some((key) => {
    const expected = crypto.createHmac("sha256", key).update(signed).digest();
    return given.length === expected.length && crypto.timingSafeEqual(given, expected);
  });
}
Enter fullscreen mode Exit fullscreen mode

The t=...,v1=... header format here is illustrative. Follow your provider's documented scheme exactly; some sign the body only and some include the timestamp.

Support more than one secret. Accepting a list of keys lets you rotate the signing secret without dropping deliveries during the switch.

Signature verification proves who sent the request. It does not prove the event is new. That is the next step's job.

Record: let the database enforce "exactly once"

The single most effective idempotency control is a unique constraint. Application-level checks like "look up the event, insert if missing" race under concurrent retries. A constraint does not.

CREATE TABLE webhook_inbox (
  id          bigserial PRIMARY KEY,
  provider    text        NOT NULL,
  event_id    text        NOT NULL,
  event_type  text        NOT NULL,
  payload     jsonb       NOT NULL,
  status      text        NOT NULL DEFAULT 'pending',
  attempts    int         NOT NULL DEFAULT 0,
  received_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (provider, event_id)
);
Enter fullscreen mode Exit fullscreen mode
INSERT INTO webhook_inbox (provider, event_id, event_type, payload)
VALUES ($1, $2, $3, $4)
ON CONFLICT (provider, event_id) DO NOTHING
RETURNING id;
Enter fullscreen mode Exit fullscreen mode

If a row comes back, this is a new event. If nothing comes back, it is a duplicate, and the right response is still 200. Returning an error for a duplicate only makes the provider retry again.

Choose the dedupe key carefully. Use the provider's event ID, not the object ID: a single payment legitimately produces several events. If a provider gives no event ID, derive a key from fields that identify the delivery (object ID, event type and the provider's timestamp), and document that choice.

This pattern is often called an inbox: every inbound event lands in one durable table before anything else happens.

Process: acknowledge fast, work in the background

After the insert, return 200 immediately. Do the real work in a worker that reads from the inbox. This keeps your endpoint fast, so the provider stops retrying, and it means a bug in business logic never loses an event: the row is still there to reprocess.

A worker can claim rows safely with FOR UPDATE SKIP LOCKED, so several workers never pick the same event:

SELECT id, event_type, payload
FROM webhook_inbox
WHERE status = 'pending'
ORDER BY id
LIMIT 1
FOR UPDATE SKIP LOCKED;
Enter fullscreen mode Exit fullscreen mode

Two rules make processing idempotent even when the inbox dedupe is bypassed, for example by a manual replay:

  1. Business writes are idempotent too. Upsert the payment by its provider payment ID instead of inserting a new row per event.
  2. State moves by precedence, not arrival order. Define a rank for each status and only move forward:
INSERT INTO payments (provider_payment_id, order_id, status, status_rank, provider_updated_at)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (provider_payment_id) DO UPDATE
SET status = EXCLUDED.status,
    status_rank = EXCLUDED.status_rank,
    provider_updated_at = EXCLUDED.provider_updated_at
WHERE payments.status_rank < EXCLUDED.status_rank
RETURNING provider_payment_id;
Enter fullscreen mode Exit fullscreen mode

If captured arrives before authorized, the late authorized event matches no row in the WHERE clause and changes nothing. For objects without a clear state machine, compare the provider's own updated_at or version field instead. When in doubt, fetch the current object from the provider's API and treat the webhook as a notification that something changed, not as the source of truth.

On failure, increment attempts, record the error and schedule a retry with backoff. After a fixed number of attempts, mark the row dead and alert a person. A dead row is a visible, queryable problem; a swallowed exception is not.

Publish: side effects go through an outbox

Sending an email, calling a fulfilment API or posting to a CRM from inside the handler creates the classic dual-write problem. If the process crashes after the email but before the commit, the retry sends the email again. If it crashes after the commit but before the email, the customer never hears back.

The fix is a transactional outbox: in the same database transaction as the business write, insert a row describing the side effect.

INSERT INTO outbox (topic, dedupe_key, payload)
VALUES ('payment.captured', 'payment:' || $1 || ':captured', $2)
ON CONFLICT (dedupe_key) DO NOTHING;
Enter fullscreen mode Exit fullscreen mode

A separate dispatcher reads the outbox and performs the side effect, marking rows sent. The dispatcher can still deliver twice after a crash, so pass the dedupe_key downstream as an idempotency key wherever the receiving API supports one. The guarantee you get is at-least-once delivery with deduplication at each hop, which in practice is what "exactly once" means.

Replay safety: test it, do not assume it

The real test of this design is a replay. Build one into your test suite and your runbook:

  • Send the same signed event twice concurrently; assert one inbox row and one business change.
  • Send captured then authorized; assert the final status is captured.
  • Make the handler throw once; assert the event retries and succeeds with no partial writes.
  • Replay a batch of already-processed events; assert zero new outbox rows.
  • Send a correctly signed event with a timestamp outside tolerance; assert 401.

If all five pass, an operator can press "resend all" in a provider dashboard without anyone holding their breath.

Checklist

  • [ ] Signature verified on raw bytes, constant-time compare, timestamp tolerance, rotatable secrets
  • [ ] Inbox table with a unique constraint on provider event ID
  • [ ] Duplicates return 200
  • [ ] Endpoint stores and acknowledges; a worker processes
  • [ ] Business writes are upserts keyed on the provider object ID
  • [ ] State changes by precedence or provider version, never arrival order
  • [ ] Side effects written to an outbox in the same transaction
  • [ ] Retries with backoff, a dead state and an alert
  • [ ] Replay tests for duplicates, reordering, failure and stale signatures

We build payment and business integrations with these controls in place from the first release: JC TechLabs API development and system integration.

Top comments (5)

Collapse
 
holdalarms profile image
HoldAlarms •

Nice table. One row I'd add: the event that never arrives at all. Dedupe and the outbox cover "too many" and "out of order", but if your endpoint is down longer than the provider's retry window (Stripe, for example, retries for up to three days in live mode), the event is gone from your side and nothing in the handler notices.

Two cheap controls for that:

  • a periodic reconciliation job that lists recent events from the provider's API (Stripe's /v1/events returns the last 30 days) and feeds any missing IDs through your Record step, which the unique constraint already makes safe to repeat
  • an alert on silence: if an event type you normally see every hour hasn't arrived in N hours, page someone, because an endpoint that returns 200 to zero traffic looks healthy on every other dashboard
Collapse
 
challan116ux profile image
challan116-ux •

Verify, Record, Process, Publish is a good spine — the step most handlers get wrong is Record, because they record after processing and then a crash replays the side effect. The out-of-order case deserves the emphasis you give it: in shipment work an 856 status arriving before the order it references is normal traffic, not corruption, and the handler has to park it and reconcile later instead of rejecting it. Ordering is a property you reconstruct, not one the transport owes you.

Collapse
 
holdalarms profile image
HoldAlarms •

One detail worth making explicit in the inbox/outbox pattern: acknowledge only after the durable inbox insert commits. If that write fails, return a retryable error; once it commits, a duplicate can safely get 200. Also keep dedupe records longer than the provider’s replay window, or an old replay can become “new” after cleanup. A replay runbook and an alert on dead-letter age help close that gap.

Collapse
 
delayrun profile image
delay.run •

Those replay tests at the end are what most teams skip, and they're the only way to know the precedence rule really holds. One case I'd add to the list: two different events for the same payment picked up by two workers at the same moment. SKIP LOCKED keeps them off the same inbox row, but not off the same payment, so the status_rank condition in the upsert is doing all the work there. Do you lock on the payment ID as well, or trust the WHERE clause to settle it?

Collapse
 
packetexchange profile image
Packet Exchange •

Your state precedence row is the one most people skip, and it bites hardest in messaging. Speaking as Packet Exchange, a carrier exchange with an SMS and voice API, delivery receipts are a good stress test, since a delivered status can land before the sent one.

One addition: some upstreams omit or reuse event IDs, so the unique constraint ends up on message ID plus status.

How would you key the dedupe table when a provider gives no stable event ID?