The Alert

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The 43-Millisecond Heist: Slaying a Race Condition That Charged Customers Twice

The Alert

At 3:14 a.m., Sentry pinged me with the alert nobody ever wants to see: DuplicateChargeDetected. A customer had been charged twice for the same order — and the two charges were 43 milliseconds apart.

Digging through six months of payment logs, I found 17 more duplicate charges. Almost all of them clustered around midnight and noon UTC. The bug had a schedule.

The Symptom

Our payment provider retries webhooks aggressively: no 200 within 30 seconds, and the event fires again. Our handler looked perfectly reasonable:

// BEFORE: check, then act
router.post("/webhooks/payments", async (req, res) => {
  const event = req.body;

  const existing = await db.payments.findOne({ where: { eventId: event.id } });
  if (existing) return res.status(200).end(); // "already processed"

  await chargeCustomer(event);                // 💸 charges the card
  await db.payments.create({ eventId: event.id });
  res.status(200).end();
});

Check-then-act. A textbook TOCTOU race. For a year it never failed — because retries never overlapped. Then we moved to multiple instances behind a load balancer. Two identical webhooks hit two different pods in the same millisecond. Both checked, both saw nothing, both charged.

The midnight/noon pattern? That's when the provider runs batch reconciliation and replays unacknowledged events in parallel bursts.

Sentry's breadcrumb timeline is what cracked it: two identical request spans, 43 ms apart, told the whole story at a glance.

The Fix

You can't fix a race with a faster check. You fix it by letting the database be the referee — claim first, charge second:

// AFTER: the database arbitrates the race
router.post("/webhooks/payments", async (req, res) => {
  const event = req.body;
  let claim;

  try {
    // UNIQUE(eventId) — only one insert can win.
    claim = await db.processedEvents.create({ eventId: event.id, status: "processing" });
  } catch (err) {
    if (isDuplicateKey(err)) return res.status(200).end(); // another worker won
    throw err;
  }

  try {
    await chargeCustomer(event, { idempotencyKey: event.id }); // provider-side safety net
    await claim.update({ status: "done" });
  } catch (err) {
    await claim.destroy(); // release the claim so a retry can proceed
    Sentry.captureException(err);
    return res.status(500).end();
  }

  res.status(200).end();
});

Defense in depth: a unique-constraint claim on our side, plus an idempotency key on the provider side.

Before / After

	Before	After
Delivery model assumed	Exactly-once	At-least-once
Arbitration	Application check	Database unique index
Duplicates in 6 months	17	0 (3 months, incl. staged retry storms)

What I Learned

1. "It worked for a year" ≠ "it's correct." Concurrency bugs simply wait for your traffic to grow.
2. Never trust webhooks to arrive once. Design handlers to be idempotent by construction.
3. Refund first, fix second. All 17 customers were refunded with apology notes before we shipped anything.

The Win

What I'm proudest of isn't the fix — it's the regression test. I wrote a chaos test that fires the same webhook at ten concurrent workers and asserts exactly one charge. It runs in CI on every pull request. The bug didn't just die; it can never come back.

Harmony restored to the codebase. Sleep restored to me. 🐛🔨