Testing webhook handlers against duplicates, races and out-of-order delivery

Most webhook handlers get tested once: send one event, check for a 200, move on. Production doesn't send one event. Stripe's docs say so. Under Handle duplicate events: "Webhook endpoints might occasionally receive the same event more than once." Under event ordering: "Stripe doesn't guarantee the delivery of events in the order that they're generated." And in live mode Stripe retries failed deliveries "for up to three days with an exponential back off."

So the handler you need is one that gives the right business result when the same event arrives twice, when two copies arrive at the same moment, and when a refund shows up before the purchase it refunds. A 200 status tells you none of that.

This post builds a small handler, shows how each of those cases breaks it, and then shows how to test for them in a normal test run.

The delivery cases worth testing

Case	What happens in production	What breaks
Sequential duplicate	A retry after a timeout, or the same event sent twice	Credits granted twice
Concurrent duplicate	Two copies hit two instances at once	"Check then insert" dedupe races and both copies pass
Out of order	A refund arrives before the purchase	The refund is lost, the purchase is granted
Forged signature	Anyone can POST to your URL	Unverified events change state
Missing signature	A health checker or scanner hits the URL	Handler crashes with a 500
Stale timestamp	A captured request is replayed later	Old event accepted
Malformed JSON	Truncated body	Unhandled JSON.parse returns a 500, so the provider retries

The last four are about input checking. The first three are where most real bugs are, so most of this post is about them.

The race in "check then insert"

Here's a handler that looks correct. It verifies the signature, parses the body safely, and skips events it has seen:

if (db.processed.has(event.id)) return new Response('duplicate');
await grantCredits(user, 100);      // a database call
await db.markProcessed(event.id);   // another database call

Send the event twice, one after the other, and it works. The second request sees the id and stops.

Send two copies at the same time and both requests run the has() check before either one reaches markProcessed(). Both see "not processed", and both grant credits. Any await between the check and the write opens this window, and with a real database there's always one.

The fix is to make the claim and the check one atomic step, before the work:

INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;
-- no row returned means another request already claimed it

The Redis version is SET webhook:evt_123 1 NX EX 259200. If the work goes into a queue, a BullMQ job with jobId set to the event id does the same thing, as long as the completed job isn't removed straight away.

Out of order: record facts, don't do arithmetic

Suppose the handler adds 100 credits on checkout.session.completed and subtracts 100 on charge.refunded, never going below zero. If the refund arrives first, it subtracts from zero and does nothing. Then the purchase arrives and adds 100. The customer ends up refunded and still holding credits.

The fix is to store what you know about the object (paid, refunded) and work out the result from that, so the order of arrival doesn't matter. Another option is to fetch the current object from the API when an event arrives and act on that instead of the payload. Stripe's docs mention using the API to retrieve objects for the same reason.

Testing all of it in one run

You could write each of these tests by hand. Each one needs a correctly signed request (Stripe uses an HMAC-SHA256 over timestamp.body in a t=...,v1=... header), a way to reset your state, and a check on business state afterwards rather than on status codes.

I packaged that as webhook-crash-lab. You give it a handler (any (req: Request) => Response function, or a URL), a secret, an event, and two functions: one to reset your state and one to read it. It sends the seven cases above (eight scenarios, because a valid single delivery is the baseline), and compares your state to what you said it should be.

Here's a complete, runnable file. The handler does the two fixes from above: it claims the event id before any await, and it records facts per checkout session instead of adding and subtracting.

// crash-test.mjs
import { runCrashLab, toText, verifySignature } from 'webhook-crash-lab';

const SECRET = 'whsec_test_secret';
const db = { processed: new Set(), sessions: new Map(), ledger: [] };

const sessionOf = (event) => event.data.object.session ?? event.data.object.id;
const credits = () =>
  [...db.sessions.values()].filter((s) => s.paid && !s.refunded).length * 100;

async function handler(req) {
  const raw = await req.text();
  if (!verifySignature(raw, req.headers.get('stripe-signature'), SECRET)) {
    return new Response('bad signature', { status: 400 });
  }
  let event;
  try {
    event = JSON.parse(raw);
  } catch {
    return new Response('invalid json', { status: 400 });
  }

  // Claim the event id before any await.
  if (db.processed.has(event.id)) return new Response('duplicate');
  db.processed.add(event.id);

  // Record facts about the session instead of adding and subtracting.
  const s = db.sessions.get(sessionOf(event)) ?? { paid: false, refunded: false };
  if (event.type === 'checkout.session.completed') s.paid = true;
  if (event.type === 'charge.refunded') s.refunded = true;
  db.sessions.set(sessionOf(event), s);
  db.ledger.push(event.id);

  return new Response('ok');
}

const checkout = { id: 'evt_1', type: 'checkout.session.completed', data: { object: { id: 'cs_1' } } };
const refund = { id: 'evt_2', type: 'charge.refunded', data: { object: { id: 'ch_1', session: 'cs_1' } } };

const result = await runCrashLab({
  target: handler,
  secret: SECRET,
  event: checkout,
  adapter: {
    reset: () => {
      db.processed.clear();
      db.sessions.clear();
      db.ledger = [];
    },
    observe: () => ({ credits: credits(), ledgerEntries: db.ledger.length }),
  },
  expect: { once: { credits: 100, ledgerEntries: 1 } },
  sequence: { events: [checkout, refund], expected: { credits: 0 } },
});

console.log(toText(result));
process.exitCode = result.summary.fail > 0 ? 1 : 0;

npm i -D webhook-crash-lab
node crash-test.mjs

webhook-crash-lab: checkout.session.completed

  PASS  single-valid           One valid delivery
  PASS  duplicate-sequential   Same event delivered again (retry)
  PASS  duplicate-concurrent   Same event delivered in parallel
  PASS  out-of-order           Related events arrive out of order
  PASS  invalid-signature      Forged signature
  PASS  missing-signature      No signature header
  PASS  stale-timestamp        Old, replayed delivery
  PASS  malformed-json         Broken JSON body

  8 passed, 0 failed, 0 not tested

Now break it. Replace the claim-and-record block with the "check then insert" version and plain arithmetic (add credits: 0 to db, reset it in reset(), and have observe() return db.credits):

if (db.processed.has(event.id)) return new Response('duplicate');
await new Promise((r) => setTimeout(r, 5)); // stands in for a DB round trip
if (event.type === 'checkout.session.completed') db.credits += 100;
if (event.type === 'charge.refunded') db.credits = Math.max(0, db.credits - 100);
db.ledger.push(event.id);
db.processed.add(event.id);

The run now reports:

  PASS  single-valid           One valid delivery
  PASS  duplicate-sequential   Same event delivered again (retry)
  FAIL  duplicate-concurrent   Same event delivered in parallel
        business state: The effect was applied more than once or not at all: credits: expected 100, got 500; ledgerEntries: expected 1, got 5.
        Why: "Check then insert" dedupe races: two requests both see "not processed" and the effect is granted twice.
        Fix: Claim the event id atomically (INSERT ... ON CONFLICT DO NOTHING, Redis SET NX, BullMQ jobId = event.id) before doing the work.
  FAIL  out-of-order           Related events arrive out of order
        business state: Final state is wrong after out-of-order delivery: credits: expected 0, got 100.
        Why: Stripe does not guarantee event order. A refund that arrives first must not be lost.
        Fix: Record late-arriving facts (tombstones / status with timestamps) or re-fetch the object from the API before applying.
  ...
  6 passed, 2 failed, 0 not tested

Note that the sequential duplicate still passes. That's why a single "send it twice" test isn't enough.

The concurrent scenario sends 5 copies in parallel. The out-of-order scenario sends your sequence in reverse and checks the final state against sequence.expected.

A few details that matter for using it in CI:

• If you leave out the adapter, expect.once or sequence, those checks are reported as "not tested", not passed.
• Each delivery has a timeout (10 seconds by default), so a handler that hangs fails the scenario instead of hanging the run.
• npx webhook-crash-lab init writes a config file, and npx webhook-crash-lab run exits with code 1 when anything fails, so it can be a CI step.
• For GitHub webhooks, set provider: 'github' to sign with X-Hub-Signature-256. GitHub signatures have no timestamp, so the stale-timestamp case is reported as not tested.
• Point reset() at a test database. It's going to wipe whatever it touches.

To see a broken handler and a fixed one side by side without writing anything:

npx webhook-crash-lab demo

Letting a coding agent run the loop

A lot of webhook handlers are now written by coding agents, and they tend to pass the single-event test and stop. There's a companion MCP server, webhook-crash-lab-mcp, that gives an agent four tools: run_crash_lab, run_demo, explain_report and list_scenarios. Failures come back as structured data with the failed checks, the reason and a suggested fix, so the agent can fix the handler and run it again.

claude mcp add webhook-crash-lab -- npx -y webhook-crash-lab-mcp

Any MCP client that can start a stdio server works the same way, with npx -y webhook-crash-lab-mcp as the command.

What this kind of testing doesn't cover

It tests the invariants you write down. If your observe() doesn't read the table where the bug is, the run won't see it. It also doesn't test Stripe's own behaviour or your production infrastructure. Two instances behind a load balancer can still race on things that a single test process can't reproduce, such as an in-memory dedupe set, which only works in one process. Use a database constraint or Redis for the real claim, as in the SQL above. The in-memory Set in the example is fine for a test because Node runs the check and the add() without anything in between.

Checklist

• Verify the signature on the raw body, before parsing. Use your provider's library where one exists. Stripe's libraries reject timestamps older than 5 minutes by default.
• Return 400, not 500, for missing signatures and bad JSON.
• Claim the event id atomically, before doing the work.
• Store facts about objects instead of applying deltas, or re-fetch the object.
• Test for business state, not status codes, and include a concurrent duplicate and a reversed sequence.

---

Disclosure: I'm Krrish Kumar, and I maintain webhook-crash-lab and webhook-crash-lab-mcp.

Comments

[challan116-ux]

The concurrent duplicate is the case that exposes the most handlers, because check-then-insert reads as correct in a code review and fails only when two copies truly arrive at the same moment — which is exactly what a retry after a timeout produces in production. The test has to force that simultaneity deliberately; sequential tests will pass a racy handler every time. The out-of-order refund-before-purchase case deserves the same treatment: a handler that discards what it cannot apply yet has not handled the event, it has lost it, and only a test that delivers events backwards will show the difference.