The 200 OK that meant the payment had failed

Over one weekend we marked one thousand four hundred orders as paid that had not been paid. Nothing in our logs looked wrong. The gateway responded quickly, our error rate was zero, and the first anyone knew was on Monday when finance reconciled the settlement report against our order table and found a gap with five figures in it.

The gateway returns HTTP 200 for a declined card. The body says so, in a field called result with the value REJECTED and a response code of 51 for insufficient funds. Our shared HTTP client treated any 2xx as success and only parsed the body on non-2xx. That client had been written for a different integration, where the transport status and the business outcome genuinely did line up, and then reused everywhere because it was the one that existed. For most of our partners it was fine. For this one it meant a decline and an approval were the same event.

I do not think the gateway is wrong, either. The request was received, understood and processed correctly. That is what 200 means. Whether the money moved is a different question and a different layer, and expecting a transport code to answer it is our mistake, not theirs.

Every integration now has its own response decoder that must classify a reply into success, retryable or terminal by looking at the body. The default branch throws: an unrecognised result string is an incident, not an assumed success. We wrote contract tests from real recorded responses including the ugly ones, declines, partial captures and the one where the partner returns 200 with an empty body during their maintenance window.

The part that matters most is that it is checked twice. A daily reconciliation job pulls the partner's settlement file and compares it line by line with what our system believes. Any order we call paid that they do not is an alert the next morning. Money integrations get reconciliation because agreement between two systems is something you verify, not something you assume from a status code.

Transport success and business success are two different facts. Do not let one library answer for both.

– Sergey Shinder

Comments

[Kiell Tampubolon]

This is the webhook story that never gets old: transport success and business success are different signals, and only the boring one shows up in your dashboard. The pattern that saved me: treat every webhook as a claim, verify the signature, then reconcile against the provider API before marking anything paid. A 200 OK only means your doorbell works, not that the money actually arrived. Learned that the hard way when my dashboard showed green while a provider was quietly failing to recieve callbacks.

[Siya]

The daily reconciliation job is the part that stands out, most teams (including in conversations I've had on this) stop at "parse the body correctly," but that still assumes your code knows every terminal result value the vendor can send. Comparing against their settlement file catches drift even when your decoder doesn't recognize a new terminal state yet. Curious how you decided reconciliation should be daily rather than tighter, was that a deliberate tradeoff against how fast a gap like the 1400-order one could compound, or mostly about what the partner's settlement file cadence allowed?