Start with the thesis: Satisfied Is Not Established.
Payment-protocol test suites are usually written while the protocol is still moving. The protocol ships, its wire format settles, and the suite keeps speaking the draft. Every check still runs. Every check still returns a verdict.
The way to find out is to point the suite at a server that cannot be paid, and count the passes.
The measurement
The Agent Security Harness carries 54 tests for x402, the HTTP 402 payment protocol for agents. This week I ran them against a minimal server that follows the published x402 v2 HTTP transport: every request without a payment gets a 402 and a base64 PAYMENT-REQUIRED header describing how to pay. The server has no facilitator. It accepts no payment, ever.
The result: 43 of 54 PASS, 11 FAIL.
Neither number was evidence. X4-045 reported "Replay correctly handled (status 402, 402)". X4-046 reported "At most 1 settlement accepted ([402, 402]) — safe". The server never read a payment, so there was nothing to replay and nothing to settle.
Why
The probes were written against a pre-spec header format. They read the challenge from X-Payment-Amount and X-Payment-Recipient, and they send a forged payment as X-Payment-Authorization.
The published transport does neither. Version 1 puts the challenge in a JSON body and takes the payment in X-PAYMENT (spec). Version 2 puts it in a PAYMENT-REQUIRED header and takes the payment in PAYMENT-SIGNATURE (spec).
A spec server ignores the forged header and answers every probe with the 402 it sends to everyone. The test sees a non-200 status and records the attack as rejected.
A 402 the server would have sent anyway is not a rejection.
The failures were wrong in the mirror direction. X4-001 failed the server for missing x-payment-* headers, which the spec does not have.
Why the usual guards stay quiet
The harness already refuses a verdict when the target never answers: a closed port, a host returning 404 to everything, an empty 2xx. Each of those shapes has a guard and a test that holds it.
This shape passes all of them. The target answered, correctly, in a protocol the probes did not speak. Any guard that asks "did the target respond?" stays quiet, because it did. The only instrument that catches it is a target that follows the spec and cannot be paid.
The fix
An outside review flagged the format mismatch. The measurement and the fix shipped the same day.
PR #672 teaches the module to recognise a spec challenge. Against a spec target, five tests keep a verdict, because their evidence is the server's own response: the challenge is well-formed, the payee is stable, two leak scans, and behaviour under load. The other 49 report INCONCLUSIVE and say why.
Against the same server, the run now reads 5 PASS, 49 INCONCLUSIVE. A smaller number, and one a reader can rely on.
A guard test runs the suite against spec v1 and v2 targets and fails if any of the 49 returns a verdict. It also checks that X4-001 can still fail a spec challenge with no payee.
The next step is probes that speak the spec. Each needs a correctly signed payment the server must accept, as the positive control, before any rejection means anything.
Run the same check on your suite
Stand up a server that returns the spec challenge and accepts nothing. Run your payment tests against it. Every PASS on a test that claims to exercise a payment is a test that never reached one.
Michael K. Saleme maintains the Agent Security Harness. The counts come from runs on 30 October 2026: before the fix against harness main at 4d7528c, after it against the fix merged as 76ef629. The guard is testing/test_x402_spec_target.py, and the CHANGELOG entry records the correction.
Top comments (15)
Same thing shows up in Solidity test suites all the time. A bare
vm.expectRevert()passes on any revert, so a test like "attacker can't withdraw someone else's funds" can go green because the attacker had no balance or the setup was broken, not because the access check fired. When I review test suites I look for exactly your positive control: the same call from the owner succeeds in the same setup, and the attacker's revert is pinned to the exact error. For the signed payment, are you going to settle on a local fork or against a testnet facilitator?The plan is a testnet facilitator on Base Sepolia, not built yet. A local fork proves the signature and the transfer are valid, but the thing under test is whether the server hands the payment to something that verifies it before serving the resource. A fork with no facilitator in the loop skips exactly that hop.
Your expectRevert point maps one to one. Pinning the revert to the exact error is what turns "it failed" into "the check I meant fired." The x402 equivalent is reading the rejection reason the facilitator returns, not just the 402.
Once the facilitator is in the loop I'd add two payments that are signed fine but wrong, one 1 unit under the price and one to a different payTo. The facilitator only checks the payload against the requirements the server sends it. The stock server compares the client's
acceptedwith its own list first, but a server that just forwardsacceptedlets the client set its own price, and the facilitator still says valid.The facilitator is in the loop now: the official x402 Python SDK (2.26.0) with the public x402.org facilitator on Base Sepolia.
Your first two cases were already in it. One unit under the price came back 402
invalid_exact_evm_payload_authorization_value_mismatch. A differenttocame back 402invalid_exact_evm_recipient_mismatch.Your second point was the better one, and the suite could not see it. Every probe echoed the server's own requirement in
accepted, so the facilitator always compared a wrong payment with the right price.So we built your case today. Two probes rewrite
acceptedto the client's own price or recipient and sign to match, and we ran them against two servers.The official server refused both before verify: 402
No matching payment requirements.The second server was the same one with that match replaced by "use the client's
accepted". The public facilitator verified and settled both payments: 9999 of 10000, and a full payment to an address the server never offered. Both are on chain.A facilitator can only check a payment against the terms it is handed. Whether those are the server's terms is the server's job.
Every official SDK does that match. The spec text does not require it.
Run record, with tx hashes and what it does not show: github.com/msaleme/red-team-blue-t...
assetwould be the third one for that pair. The client deploys its own token withtransferWithAuthorization, puts it inaccepted.assetwith the right amount and the rightpayTo, and signs to match. Reading eip3009.ts, the facilitator takes the asset from the requirements and only checks the address has bytecode, and the EIP-712 name and version come fromextra, so those are the client's too. Amount and recipient look perfect in the logs and the server got paid in something worth nothing.It also means the facilitator's wallet calls a contract the client picked. I don't know if x402.org keeps its own asset list on top of the stock code, that's what a probe would show. Thanks for putting all this on chain, really appreciate it.
You called it. Built the probe today as X4-069 and ran it against both servers.
I deployed a plain EIP-3009 token on Base Sepolia, put it in
accepted.assetwith its own name and version inextra, and signed over its domain. Amount andpayTowere the server's own.The official Python server refused it at the match: 402
No matching payment requirements.With the match removed, the x402.org testnet facilitator verified and settled it. Its wallet called my contract, and 10000 of a worthless token went to
payTo. In the logs it reads as a correct payment.So your open question has an answer for the testnet endpoint: no asset list. Only the server's match checks the token. I haven't tested mainnet and won't claim it.
It's now the third test vector on the spec issue, with your name on it: github.com/x402-foundation/x402/is...
Run record and tx: github.com/msaleme/red-team-blue-t...
Thanks for running it, and for the credit.
On your facilitator question I'd say yes. The match protects the server, but it's the facilitator's wallet paying gas to call a contract nobody looked at past "has bytecode".
eip3009.tsalready has this rule for one case: a factory that isn't ineip6492AllowedFactoriesgets refused before the wallet calls it, and the x402.org facilitator keeps that list, ported from the CDP one. An asset is the same kind of thing, a contract that wallet calls because a request named it. So a per-network asset list would follow a pattern they already ship.Agreed, and the Python SDK makes the case for you. ExactEvmSchemeConfig has eip6492_allowed_factories, empty by default, and its docstring says facilitators "must explicitly list every factory they trust to prevent arbitrary transaction injection." The asset is the same shape: a contract the facilitator's wallet calls because a request named it, with nothing but a bytecode check in front of it.
That belongs in its own issue, on the facilitator side; #4161 is about the server. Do you want to open it? I'll add the run record and the settlement tx. If you'd rather I open it, I'll credit you as the source.
Go for it, you've got the run record and the settlement tx, it reads better coming from you. Thanks for the credit.
You're credited as the source, with a link to your comment. It proposes a per-network asset list next to eip6492AllowedFactories, unset by default so nothing breaks, and notes that DEFAULT_ASSETS already holds a starting list the facilitator never reads. If the maintainers like the shape, I'll send the Python PR.
We ran into this exact gap. A test that only checks "402, then 200 after paying" tells you almost nothing about the server. Our fix was a negative set: send a payment to the wrong recipient, an expired authorization and a reused nonce, and expect a refusal every time.
Same three in the run, all refused by the facilitator: wrong recipient (
invalid_exact_evm_recipient_mismatch), expiredvalidBefore(invalid_exact_evm_payload_authorization_valid_before), and replay of the accepted payment (invalid_exact_evm_nonce_already_used). We also sent one valid payment five times in parallel, and it settled exactly once. bloqarl's comment above names the case none of these reach.Five parallel copies of one payment settling exactly once is the result I care about most here.
I will read bloqarl's comment properly before I say anything about that case.
the 'a 402 the server would have sent anyway is not a rejection' line is the whole game. a stale probe that speaks the X-Payment-* format against a spec server gets the exact same 402 a client sending nothing gets, so the suite records a rejection that never happened. curious whether the 49 now-inconclusive tests kept their old PASS verdicts in the harness history or the whole run was re-baselined once the guard test shipped
Nothing was rewritten. A report from before the fix still says PASS, and it stays that way: editing old results would hide the defect instead of recording it. The CHANGELOG carries a correction entry that says which results to discount: any x402 run against a spec server before the fix.
New runs use the new rule, so the baseline moves forward from the fix rather than backward over history.