At the end of March, an accounts payable clerk at a supplier called Wei, the finance director of a 500-person equipment manufacturer, to ask a small question. Why had she received two purchase orders for the same forty-foot container — 4471 and 4472 — same date, same amount, same twelve line items?
Wei pulled both records. Both were approved. Both had been created in our platform fifteen seconds apart, at 14:07:11 and 14:07:26. No human being had touched anything in between.
The integration log explained it in one line. Their ERP had pushed the order to us and waited thirty seconds for an answer. We took longer than thirty seconds — the insert itself was fine, but an automation attached to it was not — so the ERP gave up and tried again. We had already committed the first order. The second call created a second one.
Everyone involved did exactly what they had been told to do.
A timeout is not a failure
A timeout is a sentence about the caller, not about the work. It says: I stopped waiting. It does not say the request never arrived, and it does not say the other side did not commit. There are three states behind a timeout — never arrived, arrived and committed, arrived and still running — and two of the three mean the work exists.
Our platform collapses all three into one word. Then it acts on the word: it treats the absence of an answer as an answer, and lets the caller's retry through as if it were a brand-new intention.
That single conflation is where the duplicates come from. Not a race, not a bug in a queue. A system that cannot distinguish "I don't know" from "no".
Our transaction ends at our own wall
The documentation is unusually honest about atomicity: one automation is one transaction, and if it throws or is told to stop, everything it did to the data tables rolls back.
That guarantee is real, and it is narrower than it sounds. It covers our tables. It does not cover the ERP on the other end of an HTTP request, and it cannot, because we are not the one holding that lock.
So the rollback is asymmetric. We undo our half; their half stays. A purchase order can be cancelled in our database and alive in theirs. And because rollback feels like safety, it also licenses the retry — we undo, we try again, and the second attempt lands on top of a first attempt that was never undone.
There is a second asymmetry underneath. The script runtime is synchronous — no promises, no timers, no background workers — so a retry cannot be scheduled for later. It happens inside the same transaction, holding the same locks. A retry is not a courtesy; it is a second write inside the critical section.
The identifier arrives after the act
Look at what the write API asks for. An API key, a table identifier, and the row. That is all. The record's id is issued by the platform after it decides to accept the row.
This ordering is the whole problem. Idempotency requires the caller to name the intent before the side effect, and the server to remember the name. Here the only name a write has is the one the write receives. At the moment of intent there is nothing to name. So when the client retries, it has no way to say "this is the same thing I asked for twenty seconds ago", and we have no way to agree.
The response does carry a request identifier, useful for support tickets and useless for deduplication: it is minted per request, so the retry gets a new one. A receipt for a different transaction.
The same hole exists on the human path: a double tap on Submit, a refresh on a slow page, a proxy replaying a POST. All produce a second row, because nothing in the write path claims sameness.
The customer's workaround is also a race
Wei's team did what competent teams do. They wrote a script that runs before the insert: query the table for a row with the same supplier, the same amount, and the same delivery week; if one exists, skip. It worked for a while, and they were proud of it.
It is a check followed by a write, with no lock in between. Two retries a few hundred milliseconds apart can both query, both find nothing, both insert — exactly what the original bug did. Except now the duplicate comes from our customer's code, so it becomes their ticket, not ours.
The pattern repeats: the platform withholds the guarantee, the customer reimplements it badly, and the failure moves somewhere we no longer have to look.
A unique index is not idempotency
The other usual answer is to put a unique index on the order number. It is the one place in the platform where a second write is physically refused, and it does catch 4472 in many cases.
But refusal is not idempotency. Idempotency means the second attempt returns the first result — the same 200, the same body, the same id. A unique constraint returns an error. And an error on the second attempt is indistinguishable, from the client's side, from a genuine conflict: someone else legitimately used that number, or the row failed validation for an unrelated reason.
The correct client behaviour on an error is to escalate. So a person re-keys the order by hand, slightly differently — a different supplier contact, a corrected delivery date — and now the supplier has three versions of one container and a phone call to make.
We turned one duplicate into one duplicate plus one manual entry, and filed it under data quality.
What the fix actually costs
The mechanics are not exotic. A key table with a unique constraint on scope and key. Insert the key before the outbound call. If the insert collides, the caller is a retry: return the stored response, or "in progress", but never do the work twice. Store the response in the same transaction as the business row. On failure, mark the key failed with a reason, and decide explicitly whether the key may be reused.
Every part of that is expensive in a way a demo never shows.
The key has to come from the caller, and our API has no field for it. Adding one is small; getting every integration, mobile client, and middleware vendor to send a stable one is a year of conversations.
The in-between state becomes visible. A key that is received but not done is a pending record, and pending needs a sweeper, a timeout, an owner, and a screen. The honest label for that screen is "we are not sure the other side got it", which is the sentence product teams avoid printing.
And the key has to be derived from business meaning — supplier, period, revision — which means a human can derive it wrongly. A wrong key does not create a duplicate; it silently swallows a genuine second order. That failure is quieter than the one we set out to fix, which is why most teams stop halfway and call the half a feature.
The bill also arrives late. Nothing alerts at 14:07:26. Both rows pass every rule and look like ordinary work. The duplicate surfaces three weeks later, in a payment run, when someone compares a bank file to a supplier statement.
The uncomfortable conclusion
We did not ship a duplicate order. We shipped a word. "Failed" is what our integration layer says when it does not know, and everything downstream — the rollback, the retry, the green checkmark on the second attempt — behaves as if the word were true.
A timeout is an absence of information. We treat it as information, and then act on it automatically, at the exact moment a human would have picked up the phone.
The uncomfortable part is not that we lack an idempotency key. Most platforms do. It is that we built an impressive transactional story — one automation, one transaction, all or nothing — and then drew the boundary at our own wall, where all-or-nothing quietly becomes our-half-or-nothing.
Fifteen seconds. Two orders, both correct, both approved, one of them imaginary.
We call the second one a duplicate. From the supplier's side, it is simply the second order we asked for.
Top comments (0)