DEV Community

Cover image for My exactly-once refund paid twice, and 373 passing tests could never have caught it
Vinicius Pereira
Vinicius Pereira

Posted on

My exactly-once refund paid twice, and 373 passing tests could never have caught it

#ai #python #testing #langchain

I spent a week building a repo about making LangGraph agents trustworthy in production. Routing measured against labelled fixtures instead of vibes. A human gate before anything irreversible. Durable resume, so a process that dies mid-refund comes back and does not pay the customer twice.

373 tests, all green. CI green on four Python versions. Then I ran an adversarial pass over it, whose only job was to break my claims rather than confirm them, and the first thing it broke was the one I had written the repo to prove.

The claim

The centre of the whole thing is one sentence from my README:

INSERT OR IGNORE then read. The claim and the check are one statement, so two attempts cannot both believe they are first.

Standard idempotency. Every effectful action gets a key, the key is inserted once, and whoever loses the race replays the stored result instead of running the handler again. My tests hammered it: eight concurrent processes on one key, one handler execution, seven replays. Crash injected at every boundary in the run, resumed from disk, effect count unchanged.

The bug

The key was built like this:

action_id = digest({
    "ticket": ticket_id,
    "tool": call.name,
    "args": call.args,      # <- what the model typed
})
Enter fullscreen mode Exit fullscreen mode

call.args is raw model output. A language model asked for a $45.00 refund can write that amount in more than one way, and all of them are valid JSON that my schema happily accepts:

    4500  ->  action_id 1270c45d1612d5d1
  "4500"  ->  action_id 8f0f6363fea8adcf
  4500.0  ->  action_id b78a82d722dde95d

distinct idempotency keys for one identical $45.00 refund: 3
Enter fullscreen mode Exit fullscreen mode

Three keys. Three refunds. Same customer, same invoice, same amount.

If a queue retries a ticket, or a customer submits twice, or the same conversation is driven again for any of the boring reasons production does that, and the model spells the number differently on the second pass, my exactly-once guarantee quietly becomes at-least-once. On money.

Why my tests were structurally incapable of finding it

This is the part I keep thinking about.

The test suite runs offline against a deterministic stand-in: a keyword classifier that, given the same ticket, emits byte-identical JSON every single time. That is a deliberate design choice and mostly a good one. It makes the suite fast, free, and reproducible, with no API key and no network.

It also means the second delivery of a ticket always produced exactly the same string as the first. The suite could never observe the failure, because the only component capable of producing the failure had been replaced with one that cannot.

My tests were not weak. They were blind by construction, in a way that green output cannot show you. Every assertion I had written was true. The thing I never asserted was that two equivalent calls collapse to one key, because with a deterministic generator, equivalent and identical are the same word.

The fix is one line of intent: derive the key from the arguments after the tool's own schema has parsed them, not from whatever the model typed.

parsed = spec.validate_args(call.args).model_dump(mode="json")
action_id = digest({
    "ticket": ticket_id,
    "tool": call.name,
    "args": parsed,          # parsed, not typed
})
Enter fullscreen mode Exit fullscreen mode

Now 4500, "4500", 4500.0, " 4500 " and "+4500" all produce one key, while seven genuinely different refunds still produce seven. And there is a test that drives the same ticket twice with different spellings and asserts a single handler execution, so it stays fixed.

The second thing it broke: a metric that was measuring the wrong noun

While it was in there, the same pass killed a number I had been quoting proudly.

My report said tool choice 21/22 correct (95.5%). It compared the name of the tool the model picked against the expected name. Nothing anywhere compared the arguments.

So this ticket:

We were charged $90.00 on INV-10032 but only $45.00 of that was valid. Refund the difference.

with the model proposing issue_refund(invoice_ref="INV-10032", amount_cents=9000) scored as perfectly correct. Right tool, double the money. A version proposing $9,000 instead of $45 also scored perfectly correct.

"Chose the right action" and "chose the right tool name" are not the same claim, and I had been publishing the second one under the first one's label.

The fix added expected arguments to every fixture and a separate metric that counts them. Which is why my headline numbers got worse:

before after
routing accuracy 92.0% 90.2%
tool metric 21/22 name only 21/23 counting arguments
in-doubt stops 8 24
crash boundaries swept 176 208

Nothing regressed. The measurements stopped flattering me. The in-doubt count tripled because the audit also found two windows inside the effect where a crash could land and my crash-point list did not know they existed, which meant a line in my report was true by accident rather than by design.

What the repo does now

The full run in 21 seconds: routing scored, an irreversible refund held at the human gate, and the effect count staying at one across a crash

The whole thing in one pass. What follows is the same three acts, one at a time.

Routing measured against 51 labelled fixtures, and a parser that turns untrusted output into a named escalation instead of an exception

Routing is scored per branch with a confusion matrix, and the parser never raises: output it cannot trust becomes an escalation carrying one of nine named reasons, recorded in state.

An irreversible refund held at the human gate, with must-stop and must-not-stop both measured

Irreversible tools stop for a human, and the interrupt is measured in both directions. Missing a required stop is scored at threshold zero, because there is no acceptable rate for shipping a refund nobody approved. Stopping when you did not need to is scored separately with a real budget, because approval fatigue is how a gate stops meaning anything: interrupt people often enough for nothing and they start clicking approve without reading.

The same run killed mid-effect and resumed: the handler execution count stays at one

And the part that took the longest to get honest. The process can be killed at any of 208 points in a run and resumed from disk. 112 of the reachable boundaries come back byte-identical with the effect count unchanged. The other 24 land in a window that genuinely cannot be closed, between claiming the effect and recording its result, and there the system stops and says it does not know rather than guessing. A human reconciles it with the ledger in front of them.

Zero duplicates. Zero mismatches. A CI gate fails the build when any of those numbers moves.

GitHub logo vinimabreu / langgraph-production

A LangGraph support agent with the reliability layer around it: routing measured against labelled fixtures, a human gate before irreversible tools, durable resume with exactly-once effects, and a CI gate that fails when a number moves.

langgraph-production

CI Python License: MIT

A LangGraph agent with the reliability layer that decides whether it can ship.

The routing scorecard, the human gate before an irreversible refund, and a crash that leaves the effect count at one

Building the graph is the easy half. The half that decides whether it goes live is everything around it: what happens when the process dies between two nodes whether the refund gets paid twice on the way back up, whether an approval somebody gave on Tuesday can authorise a different action on Thursday, and whether anyone can say out loud how often the router is right.

This repo answers those four questions with code and with numbers, on a support desk for a fictional product. All data is synthetic.

The report this repo exists to produce

Real output from python -m langgraph_production.evaluation on this repository, not an illustration:

========================================================================
SUPPORT GRAPH EVALUATION
========================================================================
  fixtures      answer 13, escalate 15, tool 23  (total 51)
ROUTING
  accuracy      46/51 = 90.2%
  macro F1      0.907
  tool choice   22/23 correct (95.7%) on
…
View on GitHub

The takeaway I did not expect

I went in expecting the audit to find sloppy corners. It found the opposite: my careful parts were fine, and the failure was hiding in the seam between two decisions that were each individually correct.

Using a deterministic stand-in for tests: correct.
Keying idempotency off the tool call: correct.

Together they produce a system that is provably exactly-once against a generator that cannot vary, and at-least-once against the one you actually ship with.

So the question I now ask about any test suite, including yours: what can this suite not observe, by construction? Not what did I forget to assert. What has been designed out of the room. If you replace the non-deterministic component with a deterministic one for testing, every bug that only exists because of non-determinism is invisible to you, and it will stay invisible while the dashboard stays green.

Green tests are evidence about the world you built for them. It is worth being explicit about how much of the real one you left out.

Code, numbers, and the harness that produces them: github.com/vinimabreu/langgraph-production

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

Top comments (0)