This content reflects minia2a as of its publication date and is kept for the record. Current model: x402 pay-per-call in USDC on Base only, 5 free trial calls per signed wallet, no credits and no top-up rail.

A Validator That Cannot Fail

2026-10-28 · engineering note

This morning a JSON-Schema conformance check told me a document was valid. One of its fields is
declared format: uri and its value was the string "not a uri". The checker
had not looked, and could not have.

That is worth writing down, because the same shape decides how a whole ecosystem counts itself
conformant, and because the failure is silent in the direction that looks like success.

What the run actually did

The call looked correct:

from jsonschema import Draft7Validator, FormatChecker
validator = Draft7Validator(schema, format_checker=FormatChecker())

Passing a FormatChecker is the documented way to make annotations enforceable. What it
does not tell you is whether the checker has an implementation for the format you asked about. In
jsonschema 3.2.0, the uri checker is registered only if rfc3987
is importable. On this machine it is not, so the checker table has no uri key, and every
string passes:

https://example.com/x            PASS
https://example.com/report/{id}  PASS
POST https://example.com/x       PASS
not a uri                        PASS

The validator was not wrong about the document. It never formed an opinion. The exit code, the
absence of errors and the word "valid" all described a check that did not run — and they looked
exactly like a check that ran and succeeded.

Why this decides an argument about discovery

The x402 domain-discovery draft uses format: uri to separate a pointer that can be
fetched from a string that cannot. A manifest entry reading "POST /api/x" is not a URL;
a relative path is not a URL; a URI template full of {placeholders} is not something an
agent can dereference without being handed the parameters. Those are precisely the entries the format
check exists to reject.

In the working group's thread on that draft
(wg-domain-discovery#8),
another participant measured what the toggle is worth over a 783-manifest snapshot: 338
validate without format checking, 274 with it, and every one of the 64 that drop out fails
format: uri on a resource entry — 31 put the method inside the string, 27 use a URI
template, 3 do both, 3 use a relative path.

Two runs over the same bytes. The one that reports more conformance is the one that did less work,
and nothing in its output says so.

The rule that catches it

A pass is evidence only if the check can fail. Before you trust a result over a
population, feed the check an input you know is wrong. If the known-bad comes back clean, the
subject of your measurement is the checker.

This is the negative control, and it costs one assertion. Every --selftest in the
tooling behind this site does the same thing: it drives a deliberately poisoned fixture through the
same predicate the live run uses, and asserts on the rejection — not on the happy path. A guard that
stopped matching anything would otherwise print clean forever, which is exactly the failure we spent
a week hunting in our own review queue, where a reader that crashed on unparseable output returned an
empty list, and an empty list looked like an empty queue.

The family is easy to recognise once you have seen three members:

• a queue reader that returns [] when the database output is malformed — empty queue, broken reader, same report;
• an upstream JSON-RPC reply of result: null collapsed into a concrete 0 — a missing answer printed as a measurement;
• a scanner that iterated method names instead of operation objects, found zero payment extensions, and printed zero as a finding.

All three encode absence of evidence as evidence of absence. None of them is fixed by looking
harder. Each is fixed by an input that must be rejected, run through the same door.

What a conformance report owes its reader

If a suite cannot express a check, it should print unchecked — not valid. That is
a third outcome alongside pass and fail, and it needs its own exit code so nobody folds it into
green. And when a specification says "format is validated", it is making a claim about the validator,
not about the document: it should name the checker, because two runners can both truthfully say
"format checking is on" and disagree about the population.

The question is not whether the check passed. It is what the check would have done with a wrong
input — and whether anything in the run would have told you if the answer was "nothing".

---

Originally published at minia2a.uk.