A contractor emailed to say she had been paid twice. She was not complaining, which somehow made it worse.
That was nine days after the run. The batch held 3,412 items, every one reconciled, and my dashboard said 3,412 sent. It also said 3,414 payments had actually left, and nobody had ever thought to compare those two numbers because there had never been a reason to.
Two items. €1,284 between them. The interesting part is not the bug. It is that every individual piece of the system behaved exactly the way its documentation said it would.
The provider did nothing wrong
It posts a webhook when a payout item becomes payable, and waits five seconds for my service to answer. If no answer arrives, it assumes the message was lost and posts again. That is a sensible design and it is written plainly in their docs.
My handler's p99 that afternoon was 6.2 seconds.
So the provider waited its five seconds, heard nothing, and redelivered. The second delivery landed 41 milliseconds after the first by my own log timestamps, while the original was still running. Both of them went all the way through, and both of them sent a payment.
If you are integrating anything that pays people in batches, the three things worth reading before you write a line are idempotency, item-level errors and webhook behaviour. Between them they decide whether a retry is safe, and I had assumed rather than checked.
The shape of the mistake
My handler read the payout item, saw its status was still pending, sent the payment, then wrote the status back as sent.
Read that as one request and it is obviously fine. Read it as two requests arriving 41 milliseconds apart and the problem is immediate: both of them read, both of them saw pending, because neither had written anything yet. The gap between reading a value and acting on it was nearly six seconds wide, and I had left it completely unguarded.
This is check-then-act. I could have named it in an interview. Nobody caught it in review either, because in the single-request reading it looks correct, and the single-request reading is the one your eye does.
The reason it survived testing is duller still. A duplicate that arrives while you are still processing the first one is genuinely hard to produce by hand, so nobody ever had. I now replay exactly that case with a webhook simulator set to deliver twice before anything of mine goes near production.
Why deduplicating on the event ID did not save me
I did have deduplication. That is the part that stung.
There was a table of processed event IDs, and the handler wrote to it once it had finished. It changed nothing, for a reason that took an embarrassing while to see: the deduplication check was itself a read followed by an act. Same race, one table over. Both requests looked, both found nothing, both carried on.
There is a second reason it was the wrong key anyway. An event ID identifies a delivery. What I actually needed to guarantee was one payment per payout item, and those are not the same promise. Key on the delivery and you are protected against the provider sending the same envelope twice. Key on the thing that matters to the business and you are protected against everything that means pay this person, however it arrives.
The fix was a rule, not more code
The database is the only part of the system that can settle an argument between two requests arriving at the same instant. So I let it.
Now there is a small table whose only job is to record that a payout item has been claimed, and the item's identifier is its primary key. Before anything sends a payment, it tries to insert a claim. Exactly one of two simultaneous attempts can succeed, because the database will not accept the same key twice. The winner sends the payment. The loser is told the row already exists, does nothing, and returns.
No lock table. No coordination service. No clever application logic. The uniqueness rule is the entire mechanism and everything around it is bookkeeping.
The second half was making the handler fast enough that the provider stops retrying at all. It now acknowledges the message immediately and does the actual work on a queue, where the provider is not holding a stopwatch. My p99 went from 6.2 seconds to about 40 milliseconds and the retries stopped.
What it costs
Claiming an item before sending means a crash between the claim and the send leaves that item claimed and unpaid. It will not retry. The claim already exists, so every later delivery correctly declines to act, which is precisely what I asked the system to do. So I run a sweeper that releases claims older than fifteen minutes with nothing to show for them. That sweeper is now something I maintain forever, and it has a failure mode of its own, because one that is too eager reintroduces the original bug from the other direction. It is a far smaller problem than paying people twice. It is not nothing, though, and anyone telling you the idempotent version is strictly better has never operated one.
Acknowledging immediately has the same shape. I traded the provider's retry logic, which was free and well tested, for my own queue's retry logic, which is now my problem.
I also cannot fully explain why only two of the 3,412 duplicated. The slow window touched perhaps forty items and the rest presumably lost the race in the harmless direction. I never reproduced the exact interleaving outside a load test, and I have stopped trying.
The part that was never an engineering problem
I found this because a contractor was honest. That is not a control.
The check that would have caught it, comparing items marked sent against payments actually made, is not difficult. It existed nowhere in my stack because those two numbers lived in different systems owned by different teams, and no single person had ever been uncomfortable enough to go and put them side by side. If you need to argue for the time to fix something like this, someone measured a month of that kind of gap and found more than 40 per cent of the handling time sitting inside the exceptions rather than the ordinary work. That number is a better argument than any of mine.
There is also a version of this with no concurrency in it at all, where the file was already wrong before any of my code ran and
a spreadsheet had eaten the leading zeros. Different failure, same lesson about trusting an upstream you did not write.
Go and compare the number of items you marked sent last month against the number of payments your provider actually made. Those two figures should be identical, and until you have looked, you do not know that they are.
Figures and identifiers in this post are illustrative.
How did you find yours? Mine told me herself, nine days late.
Payout Engineering at Gruv
Top comments (0)