Satisfied Is Not Established

I went back through two months of my own writing this week and found I had written the same sentence about twenty times.

MCP went stateless. My test suite stayed green anyway.
Every action was authorized. The sequence still crossed the line.
A signed agent receipt can still make an unsupported claim.
My quorum test never had an approver in it.

I had not noticed. Each was written as a separate observation about a different control failure. They are one observation, and it is worth stating directly instead of twenty more times sideways.

Satisfied is a fact about the check. Established is a conclusion the evidence has to warrant.

A passing verdict tells you that the implemented decision procedure accepted the observed input. It does not, on its own, establish the property named by the test. That depends on what the check actually exercised, what evidence it observed, and what conditions would make it fail.

Three places the gap opens

Authorization that permits without deciding. An agent can leak private repositories without breaking a single permission. Every call was allowed; the outcome still wasn't. The permission model was satisfied. Nothing about the outcome was established, because the permission model never ranged over outcomes.

Signatures that bind without validating the claim. curl merged RFC 9421 support, and a valid signature still is not authorization. A valid signature can bind the covered receipt content to trusted key material; it does not prove the claim in the receipt is true. The cryptography is sound and the claim on top of it is unaudited. A signed receipt can still make an unsupported claim, which is why the receipt cannot be written by the pen it is checking.

Evaluations that pass without exercising. This is the third class, and it is the one I want to spend the rest of this on.

The mechanisms are different. Authorization, cryptographic verification and evaluation validity are not one thing. What they share is the mistake of treating the scope of a successful check as if it were the scope of the claim made from it.

The third class is the uncomfortable one

The first two can be discussed at a safe distance. The third turns the instrument back on whoever is holding it.

My test suite stayed green after MCP went stateless, because it was asserting against a shape that no longer existed. My suite passed against nothing — a closed port produced the same verdict as an enforced control. I tested my own method four times and its strongest claim never held. And last week I found a test named Approval Quorum that never had an approver in it: it asserted threshold routing under a name claiming quorum enforcement, and it had been green for months.

Then I wrote the fix, and the fix had the same defect one level in — it checked that approvers were distinct and never that they were authorized. Two arbitrary strings formed a quorum.

Three of those surfaced in eight days; the MCP defect was earlier in the month. I was actively looking for this exact class of failure and still reproduced it in the repair.

Why passing evidence is trusted more than it earns

Three artifacts usually travel together: a name, a mapping to some requirement, and a result. When all three agree, review stops.

But they are not three independent witnesses. The name was written from an intention. The mapping was written from the name. The result reports whatever the assertions happened to cover. Consistency across them shows the same abstraction propagated cleanly — including, when it is wrong, the error.

As agent-assurance programs formalize, that shared lineage matters more. A control name, a requirement mapping and a passing result can look like independent corroboration even when all three descend from the same mistaken abstraction.

How to tell the difference

One question, asked of a check rather than of a system: what concrete violation of the named property would make this check fail?

If you cannot answer that concretely, the check has not yet established the property it names — however green it is, however cleanly it maps. It may still establish something narrower, as mine established threshold routing. Narrower is not nothing; it is just not what the name promised.

Then build that case and run it. Not a generic pass-and-fail pair; outcome polarity is not property coverage. A quorum test needs the same approver counted twice, an approval from outside the authorized set, and an approval bound to a different action. Each one aimed at a specific property the name claims.

And point the check at an implementation that has the defect. Adding assertions proves only that more predicates ran. A suite that fails against a deliberately broken implementation demonstrates something more important: it can recognize the defect it claims to detect.

What I am doing with this

I am going to keep finding these, mostly in my own work, and write them up as one series instead of twenty unrelated posts.

The instances are not the interesting part. The interesting part is that this failure can stay invisible to the routine signals teams review software with — the linter, CI status, the coverage report, the requirements crosswalk. Each can report consistency without demonstrating that the named property was exercised.

The decisive test is a case built to violate the property.