A double-charged customer at 6 PM on a Saturday taught me more about POS software development than any architecture diagram ever did
I still remember the Slack message. "Customer says they were charged twice. Register 3. She's standing there right now."
We pulled up the logs. Sure enough — one tap, two charges, $47.98 taken twice from the same card in under two seconds. The cashier hadn't double-tapped. The customer hadn't done anything wrong. Our payment flow had simply retried a request that had already succeeded, and processed it again.
That single incident sent me down a six-week rabbit hole that ended with me rebuilding our entire payment flow from the ground up — this time, built around idempotency from day one. If you're doing any kind of POS software development, this is the failure mode that will eventually find you, whether you're ready for it or not.
Here's what broke, why it broke, and what I built instead.
The Problem Nobody Warns You About in POS Software Development
When people talk about building point-of-sale systems, the conversation usually centers on UI speed, inventory sync, or hardware compatibility. Almost nobody talks about the thing that actually causes the scariest bugs: network unreliability at the exact moment money changes hands.
A POS terminal isn't like a typical web app. It's often running on unstable Wi-Fi in the back of a retail store, communicating with a payment processor over a connection that can drop mid-request. And here's the catch — when a payment request times out, you genuinely don't know if the charge went through or not.
Our original system handled this the "obvious" way: if a request timed out or failed, retry it. Simple. Intuitive. Completely wrong.
Because sometimes the original request had succeeded — the customer's bank had already approved and processed the charge — and our system just hadn't received the confirmation in time. So it retried. And the payment processor, having no way to know this was a duplicate, dutifully charged the card again.
Multiply that across thousands of transactions a day, across flaky store Wi-Fi, and you get exactly the kind of Saturday-evening Slack message I described above.
What Was Actually Breaking
Once I started digging, the double-charge wasn't an isolated bug — it was a symptom of a deeper architectural flaw. Here's what I found:
1. Retries had no memory.
Every retry was treated as a brand-new transaction. The system had no concept of "this is the same purchase attempt as before" — it just fired the request again.
2. Network timeouts were being treated as failures.
A timeout doesn't mean "it failed." It means "I don't know what happened." We were conflating the two, and reacting to uncertainty as if it were a definite negative result.
3. There was no single source of truth for transaction state.
Different parts of the system — the terminal, the local cache, the backend — each had their own idea of whether a payment had gone through, and they weren't always in sync.
4. Partial failures were invisible.
If a payment succeeded but the confirmation write to our database failed, the system had no way to reconcile that gap. From the customer's perspective, they were charged. From our system's perspective, nothing had happened.
This is the uncomfortable truth about POS software development: most of the complexity isn't in the happy path. It's in the countless ways a transaction can be interrupted halfway through — and whether your system knows how to recover from that gracefully.
Rebuilding Around Idempotency
Idempotency, in plain terms, means: no matter how many times you send the same request, it only ever takes effect once.
Here's how I rebuilt the payment flow around that principle.
1. Every transaction gets a unique idempotency key — generated once, reused forever
The moment a cashier initiates a payment, the terminal generates a unique key tied to that specific purchase attempt. This key doesn't change no matter how many times the request gets retried. It's generated client-side, before the network call ever happens, so even a total connection failure doesn't prevent it from existing.
2. The payment processor deduplicates using that key
Most modern processors support idempotency keys natively — you pass the key along with the charge request, and if the processor sees the same key again, it returns the result of the original request instead of processing a new charge. This alone would have prevented our double-charge incident entirely.
3. Timeouts trigger a status check, not a retry
This was the biggest mental shift for me. Instead of treating a timeout as "try again," the new flow treats it as "go find out what actually happened." Before retrying anything, the system queries the transaction status using the idempotency key. Only if that status comes back as genuinely failed does it attempt the charge again.
4. Transaction state lives in one place, not three
I consolidated transaction state into a single authoritative store, with the terminal and backend both treating it as the source of truth rather than keeping their own separate versions. Every other part of the system reads from — and only from — that source.
5. Reconciliation runs as a background safety net
Even with all of the above, I added a reconciliation job that periodically cross-checks completed payments against processor records. If anything drifts out of sync — a charge that succeeded but wasn't recorded, for example — it gets flagged automatically instead of surfacing as a customer complaint three days later.
What Changed After the Rebuild
The difference wasn't subtle. In the six months after launch, duplicate-charge incidents went from a recurring weekly headache to effectively zero. Support tickets related to "why was I charged twice" dropped off almost entirely. And just as importantly, our team stopped dreading flaky store internet — because the system was finally designed to handle uncertainty instead of pretending it didn't exist.
The bigger lesson, though, wasn't really about idempotency keys or API design. It was this: in POS software development, the network is not your friend, and money is not forgiving of ambiguity. Any part of the system that assumes a request either "worked" or "didn't work" — with no room for "I'm not sure yet" — is a system waiting to double-charge someone at 6 PM on a Saturday.
If you're building or maintaining a point-of-sale system and you haven't specifically designed for idempotency, I'd treat it as one of the highest-leverage fixes you can make. It's not glamorous. Nobody's going to compliment your UI for it. But it's the difference between a payment flow customers trust and one that eventually embarrasses you in front of a cashier, a customer, and a very unhappy Slack channel.
Top comments (0)