Every webhook provider promises at-least-once delivery, which means your handler will see the same event more than once. Design for that on day one, or a payment gets marked paid twice, a confirmation email goes out three times, or an order ships before the refund lands.
Payment gateways, CRMs, e-commerce platforms and messaging APIs all retry when your endpoint is slow or returns a non-2xx status. Network timeouts mean the provider often cannot tell whether you processed the event, so it sends it again. Some providers also let an operator replay events from a dashboard. And because events are delivered over independent HTTP requests, a payment.captured can arrive before the payment.authorized that logically preceded it.
None of that is a bug in the provider. It is the contract. This post walks through a handler design in four steps, Verify, Record, Process and Publish, that stays correct under duplicates, delays, replays and reordering. The examples are a reference scenario using TypeScript and PostgreSQL, not a description of any client system. A Python/FastAPI implementation with tests follows in a companion post.
The failure modes you are designing for
Write these down before writing code, because each one maps to a specific control:
| Failure mode | What goes wrong without a control | Control |
|---|---|---|
| Forged or tampered request | Anyone who finds the URL can mark orders paid | Signature check on the raw body |
| Captured request replayed later | A valid old request is resent by an attacker | Timestamp tolerance plus dedupe |
| Provider retry after timeout | Same event processed twice | Unique constraint on event ID |
| Slow handler | Provider times out, retries, load grows | Store first, process asynchronously |
| Crash between DB write and side effect | Email sent but DB not updated, or the reverse | Transactional outbox |
| Out-of-order delivery | Older state overwrites newer state | State precedence, not arrival order |
| Operator replay of a week of events | Mass duplicate side effects | All of the above, tested |
Verify: authenticate the exact bytes you received
Most providers sign each delivery with an HMAC (commonly SHA-256) over the request body, sometimes combined with a timestamp, and send the result in a header. Three details decide whether verification actually works.
Use the raw body. If your framework parses JSON and you re-serialize it before hashing, key order or whitespace can change and valid signatures fail. Worse, teams then "fix" it by disabling the check. Capture the raw bytes first:
import crypto from "node:crypto";
import express from "express";
const app = express();
app.post(
"/webhooks/payments",
express.raw({ type: "application/json" }), // req.body is a Buffer
async (req, res) => {
const ok = verifySignature(req.body, req.header("x-signature"), secrets());
if (!ok) return res.sendStatus(401);
// ...Record step below
}
);
Compare in constant time, and check the timestamp. A plain string comparison can leak timing information. A timestamp inside the signed payload lets you reject captured requests replayed hours later.
function verifySignature(body: Buffer, header: string | undefined, keys: string[], toleranceSec = 300) {
if (!header) return false;
const parts = Object.fromEntries(header.split(",").map((p) => p.split("=", 2)));
const ts = Number(parts.t);
if (!Number.isFinite(ts) || Math.abs(Date.now() / 1000 - ts) > toleranceSec) return false;
const signed = Buffer.concat([Buffer.from(`${ts}.`), body]);
const given = Buffer.from(String(parts.v1 ?? ""), "hex");
return keys.some((key) => {
const expected = crypto.createHmac("sha256", key).update(signed).digest();
return given.length === expected.length && crypto.timingSafeEqual(given, expected);
});
}
The t=...,v1=... header format here is illustrative. Follow your provider's documented scheme exactly; some sign the body only and some include the timestamp.
Support more than one secret. Accepting a list of keys lets you rotate the signing secret without dropping deliveries during the switch.
Signature verification proves who sent the request. It does not prove the event is new. That is the next step's job.
Record: let the database enforce "exactly once"
The single most effective idempotency control is a unique constraint. Application-level checks like "look up the event, insert if missing" race under concurrent retries. A constraint does not.
CREATE TABLE webhook_inbox (
id bigserial PRIMARY KEY,
provider text NOT NULL,
event_id text NOT NULL,
event_type text NOT NULL,
payload jsonb NOT NULL,
status text NOT NULL DEFAULT 'pending',
attempts int NOT NULL DEFAULT 0,
received_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (provider, event_id)
);
INSERT INTO webhook_inbox (provider, event_id, event_type, payload)
VALUES ($1, $2, $3, $4)
ON CONFLICT (provider, event_id) DO NOTHING
RETURNING id;
If a row comes back, this is a new event. If nothing comes back, it is a duplicate, and the right response is still 200. Returning an error for a duplicate only makes the provider retry again.
Choose the dedupe key carefully. Use the provider's event ID, not the object ID: a single payment legitimately produces several events. If a provider gives no event ID, derive a key from fields that identify the delivery (object ID, event type and the provider's timestamp), and document that choice.
This pattern is often called an inbox: every inbound event lands in one durable table before anything else happens.
Process: acknowledge fast, work in the background
After the insert, return 200 immediately. Do the real work in a worker that reads from the inbox. This keeps your endpoint fast, so the provider stops retrying, and it means a bug in business logic never loses an event: the row is still there to reprocess.
A worker can claim rows safely with FOR UPDATE SKIP LOCKED, so several workers never pick the same event:
SELECT id, event_type, payload
FROM webhook_inbox
WHERE status = 'pending'
ORDER BY id
LIMIT 1
FOR UPDATE SKIP LOCKED;
Two rules make processing idempotent even when the inbox dedupe is bypassed, for example by a manual replay:
- Business writes are idempotent too. Upsert the payment by its provider payment ID instead of inserting a new row per event.
- State moves by precedence, not arrival order. Define a rank for each status and only move forward:
INSERT INTO payments (provider_payment_id, order_id, status, status_rank, provider_updated_at)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (provider_payment_id) DO UPDATE
SET status = EXCLUDED.status,
status_rank = EXCLUDED.status_rank,
provider_updated_at = EXCLUDED.provider_updated_at
WHERE payments.status_rank < EXCLUDED.status_rank
RETURNING provider_payment_id;
If captured arrives before authorized, the late authorized event matches no row in the WHERE clause and changes nothing. For objects without a clear state machine, compare the provider's own updated_at or version field instead. When in doubt, fetch the current object from the provider's API and treat the webhook as a notification that something changed, not as the source of truth.
On failure, increment attempts, record the error and schedule a retry with backoff. After a fixed number of attempts, mark the row dead and alert a person. A dead row is a visible, queryable problem; a swallowed exception is not.
Publish: side effects go through an outbox
Sending an email, calling a fulfilment API or posting to a CRM from inside the handler creates the classic dual-write problem. If the process crashes after the email but before the commit, the retry sends the email again. If it crashes after the commit but before the email, the customer never hears back.
The fix is a transactional outbox: in the same database transaction as the business write, insert a row describing the side effect.
INSERT INTO outbox (topic, dedupe_key, payload)
VALUES ('payment.captured', 'payment:' || $1 || ':captured', $2)
ON CONFLICT (dedupe_key) DO NOTHING;
A separate dispatcher reads the outbox and performs the side effect, marking rows sent. The dispatcher can still deliver twice after a crash, so pass the dedupe_key downstream as an idempotency key wherever the receiving API supports one. The guarantee you get is at-least-once delivery with deduplication at each hop, which in practice is what "exactly once" means.
Replay safety: test it, do not assume it
The real test of this design is a replay. Build one into your test suite and your runbook:
- Send the same signed event twice concurrently; assert one inbox row and one business change.
- Send
capturedthenauthorized; assert the final status iscaptured. - Make the handler throw once; assert the event retries and succeeds with no partial writes.
- Replay a batch of already-processed events; assert zero new outbox rows.
- Send a correctly signed event with a timestamp outside tolerance; assert
401.
If all five pass, an operator can press "resend all" in a provider dashboard without anyone holding their breath.
Checklist
- [ ] Signature verified on raw bytes, constant-time compare, timestamp tolerance, rotatable secrets
- [ ] Inbox table with a unique constraint on provider event ID
- [ ] Duplicates return
200 - [ ] Endpoint stores and acknowledges; a worker processes
- [ ] Business writes are upserts keyed on the provider object ID
- [ ] State changes by precedence or provider version, never arrival order
- [ ] Side effects written to an outbox in the same transaction
- [ ] Retries with backoff, a
deadstate and an alert - [ ] Replay tests for duplicates, reordering, failure and stale signatures
We build payment and business integrations with these controls in place from the first release: JC TechLabs API development and system integration.
Top comments (5)
Nice table. One row I'd add: the event that never arrives at all. Dedupe and the outbox cover "too many" and "out of order", but if your endpoint is down longer than the provider's retry window (Stripe, for example, retries for up to three days in live mode), the event is gone from your side and nothing in the handler notices.
Two cheap controls for that:
Verify, Record, Process, Publish is a good spine — the step most handlers get wrong is Record, because they record after processing and then a crash replays the side effect. The out-of-order case deserves the emphasis you give it: in shipment work an 856 status arriving before the order it references is normal traffic, not corruption, and the handler has to park it and reconcile later instead of rejecting it. Ordering is a property you reconstruct, not one the transport owes you.
One detail worth making explicit in the inbox/outbox pattern: acknowledge only after the durable inbox insert commits. If that write fails, return a retryable error; once it commits, a duplicate can safely get 200. Also keep dedupe records longer than the provider’s replay window, or an old replay can become “new” after cleanup. A replay runbook and an alert on dead-letter age help close that gap.
Those replay tests at the end are what most teams skip, and they're the only way to know the precedence rule really holds. One case I'd add to the list: two different events for the same payment picked up by two workers at the same moment. SKIP LOCKED keeps them off the same inbox row, but not off the same payment, so the status_rank condition in the upsert is doing all the work there. Do you lock on the payment ID as well, or trust the WHERE clause to settle it?
Your state precedence row is the one most people skip, and it bites hardest in messaging. Speaking as Packet Exchange, a carrier exchange with an SMS and voice API, delivery receipts are a good stress test, since a delivered status can land before the sent one.
One addition: some upstreams omit or reuse event IDs, so the unique constraint ends up on message ID plus status.
How would you key the dedupe table when a provider gives no stable event ID?