Most of what breaks in a payment system is not the payment path. It is the check you built to watch the payment path — a check that was green every time you looked, and would have been green no matter what happened.
Over a few months we collected six incidents with the same shape. Different mechanisms, same outcome: a signal that carried zero information while looking perfectly healthy.
What they share is worth stating up front:
A signal is worth exactly what its ability to come back negative is worth. If you cannot describe the observation that would make it fail, it is not a measurement — it is decoration that happens to print red when things are bad and green the rest of the time.
1. The exit code that was always zero
Our documented first-run script ended with something like this:
if npx minia2a-cli trial x402-time; then
CALL_OK=1
fi
A developer whose free trials were spent ran it and got, in green: ✓ That was a real signed call, answered by the live gateway. The line above, in red, said the call had been refused. The script reported a refused call as a success.
The cause was not the wording. The CLI exited 0 when the gateway refused — and also 0 on a genuine success. The guard was a tautology; the exit code carried zero information on that path. It looked more reliable than a log line, which is exactly why it survived review.
The fix was to stop asking a proxy and read the producer: capture the CLI output, strip ANSI, take the three-digit code off its own Status: line, and treat only 200 as success.
2. The field that can hold only one value
An endpoint returned trialExhausted: true for callers who had never made a call. That reads like a fact about the caller. It was a constant.
A subtler version: our stats endpoint published creditsRemaining as 152,180,000 next to creditsIssued as 332,219. A "remaining" 458× larger than its "issued" is not a remainder. Reading the handler explained it — both fields were populated from the same variable, SUM(credits) FROM agents, after the meaning of "remaining" was deliberately repointed to a different quantity. The value was intentional; the name never moved with it, and the result was a public JSON object that contradicted itself.
None of this throws an error. A structurally-pinned field returns 200 and a well-formed body. If you are consuming an API, the useful question is not "is this field present?" but "what value could this field hold that would tell me something?"
3. The tuple with no labels
A refusal body read:
Free trial calls exhausted for wallet 0x0ab5… (0/5 total). Pay per call via x402.
(0/5) parses as "0 remaining out of 5" and equally as "0 used out of 5". The word exhausted pushes a reader toward the second — so the sentence announces that 0 of 5 calls were used, immediately after saying they are all gone. A CLI printed this verbatim to developers, which is how we found it.
Two numbers with no names are not a measurement. The repair is one line: state the semantics in the sentence (5 of 5 used) rather than relying on a reader to infer them from position.
A related trap: the proposed fix swapped a hardcoded
0for a variable that is structurally zero in that branch. That looks like an improvement and changes nothing. A fix whose output cannot change cannot be verified from outside — which makes "we fixed it" unfalsifiable. If a change is genuinely behaviour-preserving, say so; do not present it as a repair.
4. The timestamp that looked authoritative
We keep a read-only reference copy of a service's source on the machine it runs on. Before quoting line numbers from it, we checked whether it was stale. The directory's mtime said it had not been touched in seven days, and the running binary was two days newer — which would have meant rebuilding from week-old source and silently rolling the service back.
The files told the opposite story. Their mtimes were two minutes before the binary was built. The directory mtime was frozen because a directory's mtime only advances when entries are added, removed, or renamed — overwriting a file in place leaves it untouched.
A timestamp is not evidence of freshness; it is evidence of one specific filesystem event. We were one command away from rebuilding a live service on the strength of a number that looked more reliable than the ones next to it.
5. The word list that matched the fix's own description
After fixing incident 1, we checked the old claim was gone by grepping for it. Hit count: 2.
The sentence was still in the file — now inside a conditional branch, so it no longer printed on the refusal path. A search for a string tells you the string exists. It cannot tell you whether the behaviour changed, and a fix that is about behaviour will often leave the old string in place. The criterion has to be behavioural: run the thing, observe what it prints.
The same mistake has a mirror image on the fixing side: a rewrite rule pinned to an old literal value can only ever fire once. Rules that encode today's numbers rot silently, and they rot in the direction of looking fine.
6. The probe pinned to a status code
A liveness probe tested endpoints for 404 and treated everything else as alive. Then a server began answering a dead route with 403, and the probe went blind — reporting health for endpoints that no longer existed. Blindness and health produced identical output.
Status codes are the producer's choice and can change for reasons that have nothing to do with your question. Where the body carries a machine-readable field — a retryable flag, an error code, an accepts array — key on that instead. And write down the negative case explicitly: if the probe cannot distinguish upstream failed from resource genuinely absent, it will eventually report one as the other.
The question that catches all six
For every check you write, ask: what would this print if the thing it watches were broken?
If the answer is "the same thing it prints now", you have found an always-green check.
That is the whole test. It requires no tooling, and it catches every incident above.
Two corollaries we now apply by default:
Prefer the producer's own machine-readable field to any proxy. An exit code, a status code, or a human sentence are all summaries someone else wrote for a different purpose. If the producer gives you a field, read the field.
Make the negative case reachable in a test. Verify by running the failing branch for real — an exhausted wallet, an absent binary, a real refusal — rather than by inspecting the code that handles it. Reading the code tells you what you intended. Only running it tells you what happens.
These incidents come from operating an x402 pay-per-call marketplace. The patterns are general, and none of them are specific to payments.
Top comments (0)