Exactly-once money movement: surviving retries and duplicate webhooks

Most of my work has been on systems where money moves, and the bugs I learned to fear were never the loud ones. They were quiet. A customer charged twice. A repayment that went through but never got recorded. You rarely see these in a demo. You see them days later, in reconciliation, when the numbers stop agreeing and someone has to work out why.

Two ordinary things cause most of them. Operations fail halfway, so something retries them. And a payment provider only promises to deliver its callback at least once, which in practice means sometimes twice, sometimes minutes late, sometimes out of order. Put those together and the obvious version of a payment handler will, eventually, do the wrong thing.

The first fix everyone reaches for is to check before acting:

if (!alreadyPaid(orderId)) {
  charge(orderId)
  markPaid(orderId)
}

It looks right. It isn't, the moment there is more than one worker or a webhook gets redelivered, and in production there always is. Two callbacks for the same order arrive together, both ask "already paid?", both hear no, both charge. The gap between the check and the write is where the double charge lives. In-memory locks don't help, because they don't reach across processes.

What worked for me was to stop being careful in application code and let the database settle it. Claim a unique key for the operation before you touch the money, and let a unique constraint reject the duplicate.

create table idempotency_keys (
  key         text primary key,      -- e.g. the provider reference
  status      text not null,         -- PENDING | DONE
  response    jsonb,
  created_at  timestamptz default now()
);

const claimed = insertIfAbsent(key)
if (!claimed) return storedResponseFor(key)   // replay, do not re-charge

const result = charge(...)
finalize(key, result)                          // status = DONE, store response
return result

Now two callbacks racing on the same key collide on the primary key. One wins and does the work; the other reads back the stored result and returns it. There's no window between the check and the act, because the check is the act.

For a status change the same idea is even smaller. Make the update conditional:

update payments
   set status = 'PAID'
 where id = $1
   and status in ('PENDING', 'PROCESSING');

If no row changed, someone already moved it, and you quietly do nothing.

A few things I got wrong before I got them right. The key has to be stable across retries, the provider's transaction reference, not a timestamp. Store the response, not just the fact it happened, so a duplicate gets the same answer the first call did. And keep a reconciliation job as a backstop, because idempotency can't help with the callback you never received.

I came to engineering from law, and the habit that carried over is looking for the edge case that bites later. With money, that edge case is almost always right here: two callbacks in the same millisecond, a retry mid-write. It isn't the interesting part of the work. It's most of the work.