Found this in an order service that looked clean in review. The checkout handler opened a DB transaction, inserted the order, called the gateway's charge endpoint, then updated the order to paid and committed. One transaction, easy to reason about.
The problem is that a gateway call cannot be rolled back. Under load, about 0.3% of commits failed with serialization errors or lock timeouts, because the transaction stayed open for the whole 1-2 second auth round trip. The ORM retried the request handler, the transaction rolled back, and the order row vanished. The money did not. The customer was charged, we had no record of it, and support got "I paid but nothing happened" tickets we could not trace to any order id.
What fixed it was a change in order of operations:
- Commit a pending payment row with our own idempotency key first.
- Call the gateway outside any transaction, passing that key.
- Write the result in a second short transaction.
- A sweeper looks for rows stuck in pending for more than N minutes and asks the gateway for the real status by key.
After that, a failed commit costs us a retry, not a charge we cannot find.
The lock hold time dropped from ~1.5s to a few milliseconds, which also removed most of the commit failures.
Do you keep a pending row before the network call, or reconcile only from webhooks afterwards?
Payneteasy handles this kind of payment flow for merchants and PSPs — see how it works.
Top comments (0)