We didn't have an incident. Nothing was on fire. We just decided to sit down and audit our own payments and refunds code the way we'd audit someone else's — assuming it was wrong somewhere, and trying to prove it.
It was wrong in three places. Here's what we found.
Bug 1: cancelling an order quietly leaked everything
Checkout deducts stock, credits used gift card balance against the order, and records coupon usage — all inside one transaction, which is correct, because checkout really is one atomic operation.
The problem showed up later. If that order got cancelled before it was ever paid — which happens constantly, people abandon pending COD orders, staff cancel stale carts — none of that got reversed. No restock. No gift card credit given back. No coupon usage released. The inventory number just silently drifted further from reality every time this happened, forever, with nothing in the logs to tell you it was happening.
The fix was two parts. First, cancelling an order that's still unpaid now triggers the reverse of checkout — restock, gift card credit, coupon release — but only when payment status is still pending/processing/failed, never for something already paid (that goes through a real refund instead, which already had its own restock logic). Second, we added a background job that auto-cancels orders still sitting unpaid after 48 hours, because the bug wasn't really "cancel doesn't restock" — it was "nothing was watching abandoned orders at all."
Bug 2: our refund idempotency key wasn't actually durable
We had an idempotency key on refunds specifically to stop a retried request from double-refunding a customer. Looked fine in the code. The problem was where it lived: generated in memory when the refund object was created, and only written to the database after the actual gateway call to Stripe/PayPal already succeeded.
That ordering is the whole bug. If the process died between "gateway confirms the refund" and "we save the row with that key," a retry wouldn't recognize anything — it would generate a brand new key and refund the customer a second time. The lock we already had prevented two simultaneous refund requests from racing each other, but this was a sequential retry-after-crash problem, which a lock does nothing for.
Fixed by restructuring into two real phases: create the refund row as "processing" and commit that to the database before calling the gateway. On any retry, we look for an existing "processing" row for the same payment/amount/reason and resume that one instead of creating a fresh row with a fresh key. The crash-signature and the resume-signature are now the same thing.
Bug 3: return refunds never touched the actual payment gateway
This one's embarrassing in retrospect. The RMA (return merchandise) refund flow marked the order "refunded" in the dashboard and generated something that looked like a gateway refund ID — but it was fabricated, a formatted string, not a real response from Stripe or PayPal. For every single return, regardless of how the customer originally paid.
So a customer returns something, support marks the return complete, the dashboard says refunded — and the customer's actual card or PayPal account was never touched. The money just sits there until someone notices the complaint.
The fix was almost insultingly simple once found: delete the fabrication, and route RMA refunds through the exact same refund command real admin-initiated refunds use — the one that actually calls Stripe or PayPal and has the idempotency fix from bug 2 baked in. One code path instead of two.
Why we're posting this instead of staying quiet about it
"We tested it" almost always means "we tested the paths we expected someone to take." The expensive bugs live in the paths that only show up under real-world weirdness — abandoned carts, process crashes mid-request, a return flow nobody stress-tested because it's not the exciting part of the system. We'd rather find those ourselves and talk about it than have a customer find one for us.
If you maintain a payments system of your own — what's the equivalent story you've got? Genuinely curious what shape these bugs take in other people's systems.
Top comments (1)
the refund idempotency key is the bug that hurts least the day it happens and most three months later. a key that survives a crash mid-refund is the difference between a retry that no-ops and a retry that double-pays, and the only test that catches it is killing the process at each step of the refund flow, not running the happy path twice. bug 3 has a quieter cousin too: marking refunded in your own ledger without the gateway ever acknowledging means your books and Stripe's books diverge in a direction reconciliation won't flag, because each side assumes the other recorded it. posting this instead of staying quiet is the right call, and the audit-summary format makes it reusable for anyone running a payments stack.