Start with the thesis: Satisfied Is Not Established.
Payment-protocol test suites are usually written while the protocol is still moving. The protocol ships, its wire format settles, and the suite keeps speaking the draft. Every check still runs. Every check still returns a verdict.
The way to find out is to point the suite at a server that cannot be paid, and count the passes.
The measurement
The Agent Security Harness carries 54 tests for x402, the HTTP 402 payment protocol for agents. This week I ran them against a minimal server that follows the published x402 v2 HTTP transport: every request without a payment gets a 402 and a base64 PAYMENT-REQUIRED header describing how to pay. The server has no facilitator. It accepts no payment, ever.
The result: 43 of 54 PASS, 11 FAIL.
Neither number was evidence. X4-045 reported "Replay correctly handled (status 402, 402)". X4-046 reported "At most 1 settlement accepted ([402, 402]) — safe". The server never read a payment, so there was nothing to replay and nothing to settle.
Why
The probes were written against a pre-spec header format. They read the challenge from X-Payment-Amount and X-Payment-Recipient, and they send a forged payment as X-Payment-Authorization.
The published transport does neither. Version 1 puts the challenge in a JSON body and takes the payment in X-PAYMENT (spec). Version 2 puts it in a PAYMENT-REQUIRED header and takes the payment in PAYMENT-SIGNATURE (spec).
A spec server ignores the forged header and answers every probe with the 402 it sends to everyone. The test sees a non-200 status and records the attack as rejected.
A 402 the server would have sent anyway is not a rejection.
The failures were wrong in the mirror direction. X4-001 failed the server for missing x-payment-* headers, which the spec does not have.
Why the usual guards stay quiet
The harness already refuses a verdict when the target never answers: a closed port, a host returning 404 to everything, an empty 2xx. Each of those shapes has a guard and a test that holds it.
This shape passes all of them. The target answered, correctly, in a protocol the probes did not speak. Any guard that asks "did the target respond?" stays quiet, because it did. The only instrument that catches it is a target that follows the spec and cannot be paid.
The fix
An outside review flagged the format mismatch. The measurement and the fix shipped the same day.
PR #672 teaches the module to recognise a spec challenge. Against a spec target, five tests keep a verdict, because their evidence is the server's own response: the challenge is well-formed, the payee is stable, two leak scans, and behaviour under load. The other 49 report INCONCLUSIVE and say why.
Against the same server, the run now reads 5 PASS, 49 INCONCLUSIVE. A smaller number, and one a reader can rely on.
A guard test runs the suite against spec v1 and v2 targets and fails if any of the 49 returns a verdict. It also checks that X4-001 can still fail a spec challenge with no payee.
The next step is probes that speak the spec. Each needs a correctly signed payment the server must accept, as the positive control, before any rejection means anything.
Run the same check on your suite
Stand up a server that returns the spec challenge and accepts nothing. Run your payment tests against it. Every PASS on a test that claims to exercise a payment is a test that never reached one.
Michael K. Saleme maintains the Agent Security Harness. The counts come from runs on 30 October 2026: before the fix against harness main at 4d7528c, after it against the fix merged as 76ef629. The guard is testing/test_x402_spec_target.py, and the CHANGELOG entry records the correction.
Top comments (2)
Same thing shows up in Solidity test suites all the time. A bare
vm.expectRevert()passes on any revert, so a test like "attacker can't withdraw someone else's funds" can go green because the attacker had no balance or the setup was broken, not because the access check fired. When I review test suites I look for exactly your positive control: the same call from the owner succeeds in the same setup, and the attacker's revert is pinned to the exact error. For the signed payment, are you going to settle on a local fork or against a testnet facilitator?the 'a 402 the server would have sent anyway is not a rejection' line is the whole game. a stale probe that speaks the X-Payment-* format against a spec server gets the exact same 402 a client sending nothing gets, so the suite records a rejection that never happened. curious whether the 49 now-inconclusive tests kept their old PASS verdicts in the harness history or the whole run was re-baselined once the guard test shipped