Idempotency

I found out the hard way that a timeout is not a failure.

We had a payout that went through on the provider's side. Their response never reached us. Our retry logic did exactly what we'd told it to do, and the customer received the money twice.

I remember sitting with that for a while, because the code was correct. Every line of it did what it was supposed to. The wrong assumption was upstream of the code.

We'd generated the idempotency key at the request boundary, the HTTP call gets a UUID, the provider dedupes on it. That works until your own service retries at a higher layer and mints a fresh key for the same transfer. Two keys, one intention, and no way for the provider to know they're related.
The fix was to move the key to the transfer boundary. One key per business intention, carried down through every retry beneath it.

The second part took me longer to accept: trust your own state over the provider's response. A timeout isn't a failure, it's an unknown, and you reconcile it asynchronously instead of deciding synchronously on a response that never arrived.

Most payment bugs I've seen since aren't in the happy path. They're in the gap between "it failed" and "I don't know."
How does your system tell those apart?