Every developer has one bug that lives rent-free in their head years later. This is one of mine: a payment system that worked flawlessly in every test, every staging deploy, every demo — and then silently double-charged a handful of customers, every single day, for eleven days straight, before anyone noticed.
The worst part wasn't the bug itself. It was how confidently wrong every early theory about it was.
The Symptom
Customer support started flagging a pattern: a small number of users were being charged twice for the same order. Not consistently. Not for every user. Just a handful, seemingly at random, scattered across different days.
The first assumption was the obvious one — a double-click on the checkout button, or a retry logic bug in the frontend triggering two requests.
// Suspect #1: naive double-submit prevention
async function handleCheckout() {
setLoading(true);
await submitOrder(cart);
setLoading(false);
}
The theory made sense. setLoading should disable the button during the request. Except the logs showed something stranger: the duplicate charges weren't happening milliseconds apart, like a double-click would cause. They were happening exactly 24 hours apart, down to the second.
That single detail killed the double-click theory instantly, and pointed somewhere far weirder: something running on a schedule.
The Real Culprit: A Scheduled Job Nobody Suspected
Buried in the backend was a nightly reconciliation job — something that ran at midnight server time to sync pending orders with the payment provider and retry any that hadn't received a webhook confirmation yet.
async function reconcilePendingOrders() {
const pending = await db.orders.findMany({ status: "pending" });
for (const order of pending) {
const confirmed = await paymentProvider.checkStatus(order.paymentId);
if (!confirmed) {
// Assume the original charge failed silently — retry it
await paymentProvider.charge(order.amount, order.customerId);
}
}
}
This looked completely reasonable on its own. The bug wasn't in this function — it was in what "pending" actually meant, and a race condition nobody had considered between this job and the webhook handler running at almost the exact same moment.
The Race Condition Hiding Underneath
The payment provider's webhook, confirming a successful charge, sometimes arrived at just the wrong moment — right as the reconciliation job was reading the order's status.
// Webhook handler — runs whenever the payment provider confirms a charge
async function handlePaymentWebhook(event) {
const order = await db.orders.findById(event.orderId);
await db.orders.update(order.id, { status: "confirmed" });
}
If the webhook fired between the moment the reconciliation job read an order's status and the moment it decided to retry the charge, the job would act on stale data — treating an order as still pending, even though it had just been confirmed a few hundred milliseconds earlier.
00:00:00.000 Reconciliation job reads order → status: "pending"
00:00:00.150 Webhook arrives → updates order → status: "confirmed"
00:00:00.300 Reconciliation job, still working off stale data, charges again
This is a textbook check-then-act race condition, and it only had a realistic chance of occurring during the narrow window when both processes touched the same order within milliseconds of each other — which is exactly why it looked "random" from the outside, but was completely deterministic given the right timing.
Why It Took So Long to Reproduce
Nobody could reproduce it on demand, because the bug required two independent systems — a cron job and a webhook — to collide within a tiny timing window, and almost every local test environment ran these processes sequentially rather than concurrently. The bug was real, consistent, and explainable — it just refused to show up anywhere except production traffic at real scale.
The Fix: Make the Check-Then-Act Atomic
The actual fix had nothing to do with retries, webhooks, or timing tricks. It was about eliminating the race condition at the database level, using an atomic conditional update instead of a separate read-then-write.
async function reconcilePendingOrders() {
const pending = await db.orders.findMany({ status: "pending" });
for (const order of pending) {
const confirmed = await paymentProvider.checkStatus(order.paymentId);
if (!confirmed) {
// Atomically claim the order for retry — fails silently if another
// process already changed its status in the meantime
const claimed = await db.orders.updateMany({
where: { id: order.id, status: "pending" },
data: { status: "retrying" },
});
if (claimed.count === 0) {
// Someone else changed the status first — safe to skip
continue;
}
await paymentProvider.charge(order.amount, order.customerId);
}
}
}
The key change: the where clause requires status: "pending" at the moment of the update, not just at the moment of the earlier read. If the webhook had already flipped the status to "confirmed" in between, claimed.count comes back as zero, and the retry never fires. The database itself becomes the single source of truth for "did anything change since I last looked," instead of trusting a stale in-memory read.
The Lesson That Actually Stuck
The instinct after a bug like this is to add more logging, more monitoring, more alerting around the symptom. All useful, but secondary. The real lesson was architectural: any time two independent processes can read and act on the same piece of state, "check then act" needs to become "atomically claim, then act." That single principle prevents an entire category of race conditions that no amount of retry logic or logging would have caught, because the bug wasn't in the logic — it was in the assumption that reading and acting on data could safely happen as two separate steps.
Tools That Would Have Caught This Faster
In hindsight, proper distributed tracing across the webhook handler and the cron job would have surfaced the millisecond-level overlap almost immediately, instead of requiring days of log archaeology across two unrelated systems. If your stack doesn't already have this kind of visibility, it's worth browsing a categorized debugging and observability tools hub that breaks down tracing, logging, and race-condition detection tools by use case, rather than piecing together your own from scratch mid-incident.
Skip the Expensive Tier Until You Need It
A lot of observability platforms charge steep monthly fees for distributed tracing and alerting that smaller teams genuinely don't need at their current scale. Before committing budget to an enterprise tier, it's worth checking a free and open-source alternative to paid debugging and monitoring tools, since several open-source tracing and logging projects now cover exactly this kind of race-condition visibility without a subscription attached.
If this reminded you of a bug that took way longer to find than it should have, share it with whoever was on call that week.
Top comments (0)