DEV Community: Michael "Mike" K. Saleme

Satisfied Is Not Established

Michael "Mike" K. Saleme — Sat, 05 Sep 2026 18:04:36 +0000

I went back through two months of my own writing this week and found I had written the same sentence about twenty times.

MCP went stateless. My test suite stayed green anyway.
Every action was authorized. The sequence still crossed the line.
A signed agent receipt can still make an unsupported claim.
My quorum test never had an approver in it.

I had not noticed. Each was written as a separate observation about a different control failure. They are one observation, and it is worth stating directly instead of twenty more times sideways.

Satisfied is a fact about the check. Established is a conclusion the evidence has to warrant.

A passing verdict tells you that the implemented decision procedure accepted the observed input. It does not, on its own, establish the property named by the test. That depends on what the check actually exercised, what evidence it observed, and what conditions would make it fail.

Three places the gap opens

Authorization that permits without deciding. An agent can leak private repositories without breaking a single permission. Every call was allowed; the outcome still wasn't. The permission model was satisfied. Nothing about the outcome was established, because the permission model never ranged over outcomes.

Signatures that bind without validating the claim. curl merged RFC 9421 support, and a valid signature still is not authorization. A valid signature can bind the covered receipt content to trusted key material; it does not prove the claim in the receipt is true. The cryptography is sound and the claim on top of it is unaudited. A signed receipt can still make an unsupported claim, which is why the receipt cannot be written by the pen it is checking.

Evaluations that pass without exercising. This is the third class, and it is the one I want to spend the rest of this on.

The mechanisms are different. Authorization, cryptographic verification and evaluation validity are not one thing. What they share is the mistake of treating the scope of a successful check as if it were the scope of the claim made from it.

The third class is the uncomfortable one

The first two can be discussed at a safe distance. The third turns the instrument back on whoever is holding it.

My test suite stayed green after MCP went stateless, because it was asserting against a shape that no longer existed. My suite passed against nothing — a closed port produced the same verdict as an enforced control. I tested my own method four times and its strongest claim never held. And last week I found a test named Approval Quorum that never had an approver in it: it asserted threshold routing under a name claiming quorum enforcement, and it had been green for months.

Then I wrote the fix, and the fix had the same defect one level in — it checked that approvers were distinct and never that they were authorized. Two arbitrary strings formed a quorum.

Three of those surfaced in eight days; the MCP defect was earlier in the month. I was actively looking for this exact class of failure and still reproduced it in the repair.

Why passing evidence is trusted more than it earns

Three artifacts usually travel together: a name, a mapping to some requirement, and a result. When all three agree, review stops.

But they are not three independent witnesses. The name was written from an intention. The mapping was written from the name. The result reports whatever the assertions happened to cover. Consistency across them shows the same abstraction propagated cleanly — including, when it is wrong, the error.

As agent-assurance programs formalize, that shared lineage matters more. A control name, a requirement mapping and a passing result can look like independent corroboration even when all three descend from the same mistaken abstraction.

How to tell the difference

One question, asked of a check rather than of a system: what concrete violation of the named property would make this check fail?

If you cannot answer that concretely, the check has not yet established the property it names — however green it is, however cleanly it maps. It may still establish something narrower, as mine established threshold routing. Narrower is not nothing; it is just not what the name promised.

Then build that case and run it. Not a generic pass-and-fail pair; outcome polarity is not property coverage. A quorum test needs the same approver counted twice, an approval from outside the authorized set, and an approval bound to a different action. Each one aimed at a specific property the name claims.

And point the check at an implementation that has the defect. Adding assertions proves only that more predicates ran. A suite that fails against a deliberately broken implementation demonstrates something more important: it can recognize the defect it claims to detect.

What I am doing with this

I am going to keep finding these, mostly in my own work, and write them up as one series instead of twenty unrelated posts.

The instances are not the interesting part. The interesting part is that this failure can stay invisible to the routine signals teams review software with — the linter, CI status, the coverage report, the requirements crosswalk. Each can report consistency without demonstrating that the named property was exercised.

The decisive test is a case built to violate the property.

My Quorum Test Never Had an Approver in It

Michael "Mike" K. Saleme — Sat, 05 Sep 2026 15:28:57 +0000

Start with the thesis: Satisfied Is Not Established.

Last week I wrote about tests that pass because nothing was there — a suite pointed at a closed port, reporting that a control held when nothing had been contacted.

This is the opposite failure, and I think it is the more common one.

Everything was there. The test called a working reference implementation and produced a true verdict about threshold routing. It just reported that verdict as evidence of quorum.

The test

def test_fb_013_approval_quorum(self) -> None:
    """FB-013: Above-threshold spend requires manual approval (MEDIUM)."""
    engine = PolicyEngine(SpendPolicy(
        allowlist=frozenset({"0xM"}), per_tx_cap=100_000_000,
        auto_sign_threshold=1_000_000))
    decision, reason = engine.evaluate("0xM", 5_000_000, self._now())
    self._finish(
        test_id="FB-013", name="Approval Quorum Above Threshold",
        control="policy_approval_quorum",
        ref="Fireblocks Policy Engine — approver quorum above auto-sign threshold",
        model_pass=(decision == "require_approval"),
        ...)

First, what this is not. PolicyEngine here is my own reference model of Fireblocks-style policy routing, living in my harness. The ref= string names the real product because that is the behaviour the model imitates. Nothing below is a defect in Fireblocks, or in any shipped product. The defective code is mine.

Now read the assertion, then read the name.

There is no approver set anywhere in this test. No identities, so nothing for distinctness to be checked against. No approval bound to the action being evaluated.

What it asserts is routing: a spend above the auto-sign threshold must not be auto-signed. That is a real control and it really holds.

Quorum is never exercised. Nothing here would notice if the same approver approved twice, or if an approval granted for an entirely different transfer counted toward this one.

It was green for months.

Why it survived

Three review artifacts were mutually consistent.

The name said quorum. The requirement mapping said quorum. The result said pass.

They repeated the same claim, and the code established only threshold routing. Consistency across metadata is not evidence about the assertion — it may only mean the metadata inherited the same abstraction error.

Where this bites beyond one test

The pattern generalizes to compliance crosswalks: once a control name is mapped to a requirement ID, the mapping itself starts to look like evidence.

AIUC-1 gives a concrete example. Six requirements call for quarterly third-party evaluation — B001, C010, C011, C012, D002 and D004. Their evidence artifacts require reports documenting the applicable risk scope, methodology, findings and remediation tracking; the C and D controls also expressly call for records of assessor qualifications and independence.

Every one of those artifacts can be in perfect order around a test like mine. The assessor is qualified, the methodology is documented, the report is complete, the cadence is met — and the claimed property was still never exercised.

The word "quorum" was doing work the assertion did not do.

The fix that would not have worked

My first instinct was to require paired positive and negative controls: every control test would have to show the model accepting what it should permit and rejecting what it should prohibit.

That still would not have caught this. Below the threshold the policy could auto-sign; above it, require approval. Both routing outcomes could be correct while quorum remained completely untested.

The outcome was never the problem. The property was.

The fix that does

Write deliberately invalid cases aimed at the specific property the name claims, and require the test to detect each one.

def test_one_approver_twice_is_not_a_quorum_of_two(action):
    """Cardinality is not identity. This is the defect class itself."""
    q = ApprovalQuorum(threshold=2, action=action,
                       eligible_approvers=frozenset({"approver-a", "approver-b"}))
    recorded, _ = q.approve("approver-a", action)
    assert recorded
    recorded, reason = q.approve("approver-a", action)
    assert not recorded
    assert "duplicate" in reason
    assert not q.satisfied, "the same approver counted twice reached the threshold"


def test_approval_for_a_different_action_does_not_count(action):
    """An approval is granted for one action, not for the approver's
    general willingness to approve."""
    other = ActionRef(pay_to="0xM", amount=5_000_000, nonce="tx-2")
    q = ApprovalQuorum(threshold=1, action=action,
                       eligible_approvers=frozenset({"approver-a"}))
    recorded, _ = q.approve("approver-a", other)
    assert not recorded
    assert not q.satisfied


def test_an_ineligible_principal_does_not_count(action):
    """Identity is not authority. Two distinct strangers are still strangers."""
    q = ApprovalQuorum(threshold=2, action=action,
                       eligible_approvers=frozenset({"approver-a", "approver-b"}))
    assert not q.approve("stranger-1", action)[0]   # recorded == False
    assert not q.approve("stranger-2", action)[0]
    assert not q.satisfied, "two distinct ineligible parties formed a quorum"

The suite needs to establish five obligations, one per property the name claims:

two distinct eligible principals satisfy the quorum;
the same principal twice does not;
an ineligible principal does not count, however well-formed the approval;
an approval for a different action does not count;
changing any authorization-relevant field invalidates the approval — here, an approval carries a frozen action record and is compared against the action under evaluation, so a difference in destination, amount or nonce makes it an approval for something else.

Those three fields are what this reference model binds, and naming them is the point: a production authority might also need to bind asset, network, method, calldata, expiry and policy version. State the fields your check actually covers, because "bound to the action" is exactly the kind of phrase that outruns its assertion.

The first is not filler. A check that rejects everything has enforced nothing, so the suite has to accept the case it exists to permit.

Obligation 3 is there because I left it out. My first fix checked that approvers were distinct and never that they were authorized — so two arbitrary strings formed a quorum. I had written a class to fix a test that counted without checking identity, and it checked identity without checking authority. The same defect, one level in. That is how strong this pull is: I found it, named it, wrote about it, and reproduced it inside the remedy.

After rewriting FB-013 to exercise those five obligations through ApprovalQuorum, I ran the rewritten test against an intentionally defective implementation:

class CountingQuorum(fb.ApprovalQuorum):
    """An intentionally defective implementation: counts approvals and
    checks nothing else. Injected to prove FB-013 detects it."""
    def approve(self, approver, action):
        self._approvers[f"{approver}-{len(self._approvers)}"] = action
        return (True, "counted")

harness = fb.X402FireblocksTests(simulate=True)
original = fb.ApprovalQuorum
fb.ApprovalQuorum = CountingQuorum          # the injection
try:
    harness.test_fb_013_approval_quorum()
finally:
    fb.ApprovalQuorum = original

result = next(r for r in harness.results if r.test_id == "FB-013")
assert not result.passed, "FB-013 passed against a quorum that only counts"

Substitution by module attribute rather than a constructor argument, because the
test builds its own quorum internally. It is cruder than dependency injection and
it does the same job: FB-013 runs unmodified against a defective implementation and
has to notice.

That last one is the part I would not skip again. Adding assertions makes a test say more. Injecting the defect it is supposed to catch proves it can still say no.

All of it is public, if you want to check rather than take my word:
the defect at its pinned revision,
the correction,
the eligibility hole in that correction,
and the external reproducibility work
that started the review — scoped by its author as a compatibility result, not validation.

What I would take from this

A test name is a claim, but ordinary CI does not establish that the assertions prove it. Not the linter, not the coverage report, and not the crosswalk that cites the test as evidence.

For that, the suite needs cases deliberately constructed to violate the named property — and evidence that the test fails against an implementation containing that defect.

Can your security test suite fail? Mine passed against nothing.

Michael "Mike" K. Saleme — Sun, 30 Aug 2026 01:16:45 +0000

Start with the thesis: Satisfied Is Not Established.

Most security test suites answer one question: did the attack succeed? The
verdict then falls out of a search for evidence that it did.

passed = not leaked
passed = not granted_admin
passed = len(unsafe) == 0

Every one of those is true when the attack was blocked. Every one of them is
also true when nothing happened at all.

That second case is not hypothetical. It is what a suite reports when the target
was unreachable, when the endpoint does not implement the feature under test,
when the tool list came back empty, and when the model complied politely in
language the detector does not recognize.

Three sentinels, not one target

The fix is not a better detector. It is asking the same suite the same question
across three deliberately different response conditions.

a closed port      never answers
a permissive stub  returns 200 and explicit grant markers to everything
a refusing stub    returns 403 and an error envelope to everything

These are diagnostic sentinels, not conformant implementations. A 200 carrying
{"granted": true} is not an MCP or payment-protocol grant, and a blanket 403
may block discovery rather than demonstrate enforcement. Their purpose is
narrower and useful: to expose verdicts that ignore or misread target behavior.

Point the suite at all three and read what changes.

A target-dependent test that passes against the closed port is claiming a
control held when nothing was contacted. One that passes against the permissive
sentinel despite explicit grant markers deserves inspection. And a test that
reaches the protected operation but cannot recognize an explicit denial may not
recognize the outcome it exists to measure.

For a target-dependent control, an unchanged verdict across all three
sentinels is evidence that the verdict may not be measuring target behavior.

The qualifier matters. Local controls, parser checks, and over-refusal tests can
legitimately return the same verdict everywhere.

What it found

I ran this against my own agent security harness, which is public, and which I
had believed was in good shape.

The closed port went first. Eighty-eight verdicts passed against a host that was
not running, across ten modules. The largest single count was a payment protocol
harness reporting forty-four controls holding against nothing at all.

The permissive sentinel went next, and it is the harder one because the target
answers. Every serviced-request guard I had just written was satisfied and
correctly stayed out of the way. Three hundred and nine verdicts out of five
hundred and thirty-two passed against a server returning HTTP 200 and explicit
grant markers to every request:

{"granted": true, "admin": true, "scope": "operator.admin"}

One of them read "Elevated scope claims not honored." Another read "All
gate-disable attempts were rejected." A third reported an incident-detection
latency of 0.000 seconds, measured from a connection being refused.

Not all three hundred and nine are defects, and I will come back to that.

The refusing sentinel went last and it found the inverse. Sixteen suites
produced no passing verdict against it. Three of those test controls where a
platform refusing an action is the behavior under test: a gate-disable, a kill
signal, an incident response. Those three could not recognize the one outcome
they exist to recognize.

Put the second and third together and you get the sharpest diagnostic result.
One identity and authorization suite scored seven of eighteen against the
permissive sentinel and one of eighteen against the refusing sentinel. Its
verdict moved in the opposite direction from the behavior it was intended to
recognize, and neither sentinel alone would have exposed that inversion as
clearly.

The part I did not expect

None of the three found the most interesting defect.

An external reviewer reading the source found it instead. One test serialized
the whole transport-error envelope into a substring detector whose keyword list
contained "refuse". The string it fed in was

<urlopen error [Errno 111] Connection refused>

so the agent was credited with refusing a prompt injection it never received.
Not an absence read as a pass. A positive match on the wrong text.

Then a run against a real MCP server found a second one. The protocol has two
ways to report a rejection, and my harness read only one of them, so a reference
server that correctly refused an unregistered tool call was recorded as having
failed to refuse it.

The sentinels could reveal suspicious verdicts, but they could not
independently identify either interpretation defect. The permissive and refusing
fixtures used the response idiom my harness already expected. The closed port
produced a transport error that the harness mistakenly treated as response
evidence. A fixture built around your interpretation of the property under
test cannot tell you that the interpretation itself is wrong.

What this is worth to you

The method transfers. It does not depend on my harness, my protocols, or my
threat model.

If you maintain a security test suite, a scanner, a conformance checker, or an
eval, you can build all three sentinels in an afternoon. They are a closed port,
a web server that returns 200 to everything, and one that returns 403 to
everything.

Three things worth knowing before you do.

The counts are a reading list, not a defect count. This is why I did not call
those three hundred and nine defects. Some passes against the permissive sentinel
are correct. Twenty-five of mine were over-refusal checks whose expected outcome
is permissive behavior. Other tests require protocol-specific markers the generic
sentinel does not emit. Only reading the test separates those cases from a real
inversion.

Zero verdicts is not zero defects. One module in mine aborts cleanly when it
cannot handshake, which is the right behavior and which rendered as 0/0,
indistinguishable from a clean run. It now prints "produced no verdicts, nothing
measured, not clean."

And the sentinels are the floor, not the ceiling. Get to a real implementation
as soon as they stop finding things. Source review and a half-hour test against
one real MCP implementation exposed two additional defect classes the synthetic
sentinels structurally missed.

Inspect and rerun it

The repaired harness and all three sweep scripts are pinned at commit
e151566550ecdb29b0230d40fb371fdf03bbfe09
of msaleme/red-team-blue-team-agent-fabric,
released as v4.16.0:

python3 scripts/dead_host_sweep.py
python3 scripts/permissive_host_sweep.py
python3 scripts/refusing_host_sweep.py

These commands rerun the method against the repaired harness. They will not
reproduce the original pre-repair counts, because the repairs are in. The
original findings are preserved in the earlier state files and in
PR #417
through
PR #432;
the provenance record
identifies the exact source of every figure reported here.

I am not claiming any of this makes an agent secure. It establishes something
narrower and, I think, more useful: what a harness claims when it has nothing to
go on.

I Tested My Own Method Four Times. Its Strongest Claim Never Passed.

Michael "Mike" K. Saleme — Fri, 28 Aug 2026 14:00:00 +0000

Start with the thesis: Satisfied Is Not Established.

Technical source: R2.1 results, R3 results, R5 results, R3/R4/R5 reconciliation
Companion post: Context Is Part of an Agent's Authority

I built a benchmark family to test whether a governed metadata layer earns its cost when an agent selects enterprise context. I have now run it four times under four frozen contracts, redesigning the catalog and changing the acceptance ceiling along the way.

The claim that governance earns its cost against a cheap baseline has been rejected in every round that tested it. The round before those was rejected too, on a different rule.

Round	Claim	Failed rule or controlling result	Verdict
R2.1	Overall comparative claim	Governed holdout F1 0.24065, below the 0.245533 floor	REJECTED
R3	Governed value vs. lexical	F1 CI `[-0.371, 0.00005]`; token ratio 2.11x vs. 1.10x ceiling	REJECTED
R4	Governed value vs. lexical	Quality passed; token ratio 9.86x vs. 3.0x ceiling	REJECTED
R5	Governed value vs. lexical	Quality passed; token ratio 6.94x vs. 3.0x ceiling	REJECTED

R3 through R5 are value-claim verdicts. R2.1 was the earlier overall-contract rejection that led me to separate the claims. Other claims passed: R3 and R5 accepted governed routing against full-context stuffing, while R4 returned those claims inconclusive.

Every round ran under a contract frozen before collection. The claim-scoped outcomes are documented publicly; R3 and R5 include public decision packs, while R4's later-derived pack remains held and is disclosed as such below.

R2.1 failed first. At its 3,000-object holdout the governed route scored 0.24065 against a prespecified floor of 0.245533, while the lexical prefilter scored 0.588.

Round 3: the simple baseline won the observed comparison

R3 compared three routes on the same local model: raw full-context stuffing, a cheap lexical prefilter, and a governed metadata route.

Background. This benchmark reproduces the structure of McKnight Consulting Group's study, "Stop the Token Bleed: Benchmarking the Benefits of Governed Metadata for Enterprise AI" (Jake Dolezal and William McKnight, August 2026; sponsored by Informatica, a Salesforce company). Their study held the model constant and found governed metadata access won on both cost (up to roughly 89x fewer tokens at scale) and accuracy (F1 1.000 against 0.29 to 0.66 ungoverned). Read their article for their full methodology and figures. This work does not reproduce their exact numbers. It lets you generate your own, on your own model.

Disclosure. The study reproduced here was sponsored by Informatica, a Salesforce company. I am employed by Salesforce. That is a reason to run this harness yourself rather than take my output on trust, which is the entire point of publishing it. Contradicting results are welcome.

Mean F1 across 20 seeds, at the prespecified 0% classifier-miss condition:

Catalog size	Governed F1	Full-context F1	Lexical F1
300	0.780	0.261	0.660
1,500	0.632	0.253	0.737
3,000 (holdout)	0.447	0.177	0.631

Read the two columns downward. The governed route degrades monotonically as the catalog grows: 0.780, then 0.632, then 0.447. The lexical route does not show the same pattern: 0.660, 0.737, then 0.631.

At the holdout, the keyword filter beat the governed route on the observed means and used less than half the prompt tokens. The paired F1 interval was [-0.371, 0.00005], which does not exclude zero in governance's favor. The token ratio was 2.11x against a frozen ceiling of 1.10x.

Governed routing crushed full-context stuffing. That claim passed. It lost to the cheapest thing in the room.

What I changed between rounds, said out loud

I ran R4 and R5 on redesigned catalogs, and I relaxed my own cost ceiling.

R3 used a lexically tractable synthetic catalog, where a keyword filter has real signal to match. R4 and R5 moved to semantic-access catalogs built on opaque physical names. In R5 the lexical route scored 0.000 F1 at every catalog size.

The redesign favored my method on quality: this name-only lexical baseline no longer had matching signal. But it also made the cost comparison harder, because the lexical route now produced an extremely small prompt. At the R5 holdout it averaged 110.0 prompt tokens against the governed route's 763.5, which is why the ratio rose to 6.94x even as the ceiling was relaxed.

Separately, between R3 and R4 I raised the maximum governed:lexical prompt-token ratio from 1.10x to 3.0x, which made the cost rule easier to pass.

Both changes were declared in new frozen contracts before their respective collections.

State that plainly, because a reader who diffs the contracts will find it anyway.

It still failed

In R5, governed selection used 6.94 times the lexical route's prompt tokens. The frozen ceiling was 3.0x. The preregistered claim that governance earns its cost against lexical filtering was rejected.

This happened on a task where the baseline scored zero. Governed context won the quality comparison but failed the prespecified prompt-token cost rule. The frozen contract required both.

R4 failed the same rule at 9.86x. R4 also returned INCONCLUSIVE on its full-context claims because one holdout request contained 128,256 input tokens plus a reserved 3,000-token completion budget, putting it 184 tokens beyond the verified 131,072-token window. It made no model call. It was retained as a preflight refusal rather than silently dropped.

The ceiling could have been relaxed again after seeing 6.94x. Moving a bar you already missed converts a result into a press release.

A benchmark that cannot reject its author is marketing with a methodology section.

What this changes for anyone buying or building a context layer

Ask three questions of any governed retrieval, semantic layer, or context-governance component.

What is the cheap baseline, and did you run it? Not merely full-context stuffing. At minimum, test the cheapest credible selective baseline: a keyword filter here, but potentially BM25, a cached lookup, or another simple retrieval route. If the only comparison is against full-context stuffing, the result may justify selective context, but it does not show that the sophisticated route earns its place over cheaper alternatives.

Was the acceptance rule written down before collection? A cost ceiling chosen after seeing the numbers is a description, not a test.

Where does the method lose? A vendor who cannot name the configuration where their layer is the wrong choice has not measured it hard enough.

My own answers: the baseline is a lexical prefilter, it beat the governed route on observed mean F1 and prompt-token use in R3, and the governed route has never cleared its cost bar in the three rounds that tested it. The route remains worth evaluating where opaque physical names make semantic selection necessary, or where an evidence trail has independent value. These runs establish the quality advantage in that narrow synthetic configuration; they do not yet establish end-to-end economic value. That is a narrower claim than the one I set out to prove.

Evidence boundary

These are synthetic, named-endpoint runtime characterizations on a single local model, qwen3-coder:30b. They are not production results, ROI claims, customer-data results, or a replication of any third-party study. R2.1 remains visibly rejected in the public record rather than buried.

R3 and R5 publish a full public packet: frozen contract hash, complete preflight, claim-scoped decision pack, and artifact digests. R4's contract is public, but its decision pack was derived after the fact from the archived report and is held rather than published. The raw reports stay private because they embed host identifiers, and their hashes are committed so you can tell if they ever change.

Four rounds in, the most useful thing this benchmark has produced is the boundary it refuses to cross.

Context Is Part of an Agent's Authority

Michael "Mike" K. Saleme — Fri, 21 Aug 2026 16:07:24 +0000

Technical source: Token-Bleed R5 release

Enterprise AI programs often treat context as a prompt-engineering problem: retrieve more documents, add more records, and let the model sort it out. That is backwards.

The information an agent receives determines what it can infer, combine, and disclose. An agent's effective authority is therefore shaped by both its information scope and its permitted actions: context governs the former; capability controls govern the latter.

Context does not grant permission to dispatch power, change a price, or execute a transaction. It expands what the agent can know, infer, and disclose, and therefore its practical power.

Too little context is not neutral either: omitted constraints, exceptions, or dependencies can make a confident recommendation wrong. The architectural objective is therefore not minimum context, but minimum sufficient context.

We spend substantial time defining action authority: which tools an agent may call, which systems it may reach, and which approvals it needs before a change takes effect. Information authority deserves the same discipline. Before an agent decides, architecture must establish what information it is permitted to see and what it actually needs for this decision.

The full-context default is not neutral

Sending every available record to a model feels safe because nothing has been left out. In practice, it can be expensive, dilute the relevant signal, make the decision harder to review, and expand the information the system can combine.

That does not mean every workload needs a complex governance layer. It means every workload needs a comparator.

In the retained Token-Bleed R5 synthetic experiment, compact governed selection used 96.9% to 97.9% fewer prompt tokens and achieved higher F1 than raw full-context stuffing on the frozen local configuration. The public record includes the frozen contract, preflight, retained-evidence decision pack, and hashes.

The more important finding is the limitation. Against a cheap lexical baseline, governed selection consumed 6.94 times as many prompt tokens on the holdout set, exceeding the preregistered maximum of three. The lexical route scored 0.000 F1 at every catalog size, so governed context won the quality comparison outright. Even so, the preregistered claim that governance earned its cost against lexical filtering failed. The frozen economic rule controlled the verdict; relaxing that ceiling after collection would have invalidated it. One caveat belongs with that zero: R5 used opaque physical names, where lexical matching has nothing to grip. On an earlier round with a lexically tractable catalog, the same baseline beat governed selection on both quality and cost.

That is exactly what useful architecture evidence should do. It should show where a method helps and where it has not yet earned the right to be the default.

Enterprise Agent Architecture needs a context plane

An EAA design should logically separate four responsibilities:

Connection plane: APIs, events, and data services connect an agent to enterprise systems.
Context plane: retrieval, classification, policy, lineage, and routing assemble the minimum sufficient, policy-permitted information set for the decision at hand.
Capability plane: permissions and approvals govern what the agent may do next.
Evidence plane: retained contracts, source references, tool traces, and human decisions make the outcome reviewable.

The context plane is not a technical ornament between a database and a model. It decides what the agent knows before it chooses a path. That makes it an authority control.

An agent's context is part of its authority.

A practical decision rule

For each agent workflow, begin with the least-complex, policy-permitted context route that could credibly meet the decision's quality requirements.

If a simple filter captures the relevant records and produces a reviewable decision, use it.
Add governed metadata when field names are ambiguous, business definitions matter, access policy must be enforced before the model sees data, lineage changes the answer, or the evidence trail itself is required.
Compare the richer route with the cheap baseline. Measure quality, omission risk under routing misses, prompt cost, latency, and the work required to maintain the governed layer.

The goal is not more governance. The goal is decision-useful context with a defensible cost.

What this changes in industry workflows

In energy and utilities, an outage or maintenance exception agent may need weather, load, asset, work-order, and switching-constraint context to recommend whether to keep, reschedule, or escalate a window. It should not receive unrestricted operating data, and it should not obtain dispatch authority merely because it can assemble a recommendation; that authority must be separately granted, bounded, and evidenced.

In CPG and retail, a commercial exception agent may need store, SKU, promotion, inventory, cost, and service context to recommend a response. It should make the data basis, confidence, and required approver visible. It should not alter price or trade terms unless that capability has been separately authorized within explicit limits.

Those are not generic chatbot problems. They are architecture problems involving context, authority, and evidence.

The evidence boundary

R5 is a synthetic, named-endpoint runtime characterization. It is not a customer-data result, a production ROI study, or proof that governed context always beats a simple filter. The original raw report remains private because it contains host identifiers.

That boundary is part of the result, not a footnote. Enterprise agents deserve evidence that is as specific about its limits as it is about its gains.

Where this fits. This is one piece of a longer argument about enterprise agent architecture: what TOGAF and SABSA cannot model about a workforce that is not human, and where an agent's authority actually sits. The series runs at Enterprise Agent Architecture.

MCP Went Stateless. My Test Suite Stayed Green Anyway.

Michael "Mike" K. Saleme — Sun, 02 Aug 2026 16:19:55 +0000

Start with the thesis: Satisfied Is Not Established.

Enterprise systems accumulate trust in the places where state lives. A session identifier
is not just a routing key — it is the thing a dozen downstream assumptions quietly hang
from. Remove it and you do not remove one field. You invalidate every assumption that was
resting on it, including the ones nobody wrote down.

The Model Context Protocol's 2026-07-28 specification did exactly that. The project called
it "the largest revision of the protocol since launch",
and the two headline changes are both acts of subtraction. From the
changelog's major
changes, items 1 and 2:

Remove protocol-level sessions and the Mcp-Session-Id header from the Streamable HTTP
transport.

Make MCP stateless: remove the initialize/notifications/initialized handshake. Every
request now carries its protocol version and client capabilities in _meta.

At the protocol layer, every request now stands on its own. Authorization gets stricter
alongside it: clients
MUST validate a present iss parameter against the recorded issuer per
RFC 9207 before redeeming an
authorization code, which closes a class of authorization-server mix-up attacks. There is
a formal extensions framework, and a minimum twelve-month deprecation window before any
deprecated feature can be removed.

That is a good specification. This post is not about the specification.

What early tracking actually costs

I had MCP tests running against the stateless profile on 2026-07-22, six days before
the final specification was published. I want to be precise about what that bought and what
it cost, because the first part is the part people write posts about and the second part is
the part that matters.

Between the release candidate and the final revision, the spec added an error-code
allocation policy. The JSON-RPC server-error range got partitioned: -32000 to -32019
stays implementation-defined and grandfathered, -32020 to -32099 is reserved for the
specification. Then it renumbered the codes the draft had introduced:

Error	Release candidate	Final
`HeaderMismatch`	`-32001`	`-32020`
`MissingRequiredClientCapability`	`-32003`	`-32021`
`UnsupportedProtocolVersion`	`-32004`	`-32022`

My harness checked for -32004.

The function that reads that code decides one thing: when a server rejects the modern
protocol version, is that an explicit stateless-protocol answer, or is it an old server
that does not understand the request? Get it right and the probe stops. Get it wrong and
the client falls back to initialize — the handshake the specification just removed.

Against any server built to the final spec, my check returned false. The harness read a
compliant version rejection as evidence of a legacy server, sent the removed handshake,
and reported results for a protocol the server had explicitly refused.

The function's own docstring said that must not happen. It happened anyway.

By the time the final specification landed, that assumption was already on PyPI.
Version 4.10.0 shipped on 2026-07-25, carrying the RC comparison verbatim. You do not
have to take my word for it: download the wheel and read protocol_tests/mcp_harness.py.
The RC comparison is in the published artifact, and -32022 appears nowhere in it.

The test suite stayed green the entire time

This is the part worth sitting with.

I have unit tests covering that exact function. They assert the fallback does not fire on
a version rejection. They passed continuously — before the spec was final, after it was
final, and through the release that carried the defect.

They passed because every fixture in them was written during the RC window and pinned
-32004. The tests and the code were wrong in the same direction, so they agreed with
each other perfectly.

A test that exercises only the provisional RC value cannot detect that the released
value is unhandled. Its branch coverage may be adequate, but its oracle is not independent: the
implementation and the fixture are two copies of the same provisional assumption. They can
agree perfectly and still be wrong.

I did not find this by testing. I found it by re-reading the specification changelog
against my own code, line by line, while checking something else.

The cost of tracking a specification early is inheriting the decisions it had not finished making

Being six days ahead of the final specification is a real advantage and I would do it
again. But a release candidate is a set of provisional commitments, and tracking one means
adopting those commitments before they have been tested by the people who will have to live
with them. Some of them will change. The changes will be small, unglamorous, and exactly the
kind your fixtures will freeze in place.

The lead is not free. It is a loan against a spec that has not stopped moving.

What to check, concretely

If you maintain anything that touches MCP and you moved during the RC window, four things
are worth an hour:

Grep for hardcoded JSON-RPC error codes. Any literal in -32000..-32099 that you
wrote between the RC and 2026-07-28 is suspect. Move them to named constants — a named
constant makes a renumber a deliberate edit instead of a silent one. But keep the test
oracle independent: assert the released wire value directly, or derive the fixture from an
authoritative conformance vector — not from the production constant being tested. A test
that imports the constant it is checking is tautological, and that is the same failure in
a tidier shape.

Check what your fixtures pin, not just what your tests assert. If every fixture for a
behaviour was authored in the same week, they encode that week's assumptions and will
agree with each other forever. Add one fixture from the current standard and see what
breaks.

Audit anything that decides "modern or legacy." Version-negotiation branches fail
quietly by design — they are written to degrade gracefully, which means a wrong answer
produces a plausible run instead of an error.

Re-read the changelog after the final specification, not just at RC. Diff it against your own
code rather than your memory of the RC. The renumbering that caught me is item 12 under
"Minor changes." Nothing about its placement suggests it breaks a client.

Where this leaves the testing layer

The stateless rewrite reset a meaningful part of MCP security testing. For implementations
targeting 2026-07-28, protocol-level session assumptions are gone, per-request capability
declaration is new surface, and authorization requirements have tightened. Tests scoped to
earlier MCP revisions may remain valid, but they are not evidence of conformance to the new
one. Suites written against the old handshake are not slightly stale — they are asserting
things about a mechanism that no longer exists in the 2026-07-28 protocol profile.

That is an opening for anyone willing to re-derive their assumptions from the current
text. It is also a trap for anyone who moved early and has not gone back.

I moved early. I went back. It cost me one released assumption that became a
compatibility defect three days later — and an afternoon to find it. I would rather publish
that than the version where I only mention the six-day lead.

The fix, the negative controls, and the reasoning are public: PR #313.
Views are my own.

curl Just Merged RFC 9421 Support. A Valid Signature Still Isn't Authorization.

Michael "Mike" K. Saleme — Tue, 28 Jul 2026 19:50:34 +0000

Start with the thesis: Satisfied Is Not Established.

On July 27, 2026, curl maintainer Daniel Stenberg wrote up curl's newly merged experimental support for RFC 9421, HTTP Message Signatures — the IETF standard for cryptographically signing selected components of an HTTP request or response. The feature is off by default, sits behind an explicit build-time flag, and stays that way — still experimental — in the upcoming curl 8.22.0. Stenberg is explicit it isn't ready for production.

That caution is the right posture for new cryptographic protocol work, and the standard is worth taking seriously anyway, because it answers a real question: can a receiver verify, under key material it trusts for the relevant context, that the received message is semantically equivalent to what was signed with respect to the covered components?

I read the specification closely because "signed" is easily heard as "approved." RFC 9421 makes a narrower, message-layer claim.

What actually gets established

RFC 9421 lets a sender cover selected message components — method, path, specific headers, and a content digest — and bind them to signature parameters such as created and keyid, using an agreed signature algorithm. When the receiver verifies it, two things are established:

the signature or MAC validates under key material the verifier trusts for that context
the received message is semantically equivalent to the signed message with respect to the covered components

That's it. Binding that key to a specific service, organization, or person happens outside the standard — RFC 9421 §3.2 explicitly requires the verifier to determine the key material's trustworthiness in context, and the standard supports HMAC, where more than one party can hold the same shared secret. "The signature proves who sent it" is a stronger claim than the spec actually makes.

And because only the covered components are signed, unsigned fields — and even some transformations of covered ones — can change without invalidating the signature. The guarantee is scoped, not blanket.

Scoped as it is, this still closes a real gap. For service-to-service traffic, webhook delivery, and architectures that traverse TLS-terminating intermediaries, RFC 9421 can preserve integrity and authenticity for selected components beyond any single transport connection. That part is genuine progress.

What doesn't get established

A verified signature does not establish that the key-holder was authorized to send this particular request, at this particular time, given everything else it has sent recently. It can't — verification happens one message at a time, and authorization is a question about a principal's standing, its current scope, and often its history. None of that lives inside the bytes of a single signed message.

Play out the failure mode: a service holds a signing key meant for reading inventory levels. The key leaks, or the service is compromised, or it's simply asked — by an operator, or by an agent orchestrating it — to do something outside its intended purpose. Every request can still verify cryptographically under that key, even though a separate authorization layer should reject requests outside the service's permitted scope. Signature verification alone doesn't evaluate whether a sequence of otherwise-valid requests has changed purpose or accumulated into a disallowed outcome — that check happens once per message and stops there.

To be fair to the standard: nothing stops an application from adding stateful controls above verification. RFC 9421 expressly allows additional application requirements and discusses replay protection. It just doesn't provide sequence-level evaluation itself — that's a different layer's job.

Ten individually-valid, individually-signed requests can compose into a data exfiltration run, a privilege-escalation chain, or a resource-exhaustion attack that no single request would ever be flagged for. The signature layer sees ten cryptographically valid messages. It has no layer above it asking what those ten messages, taken together, actually did.

A signature authenticates covered components. It does not sanction the action — or the sequence.

The same boundary, one layer down

This is the same structural boundary as the argument in "Authorized but Composed" (DOI 10.5281/zenodo.21400261) and in the field note on Hugging Face's July 2026 security-incident disclosure — just moved from the agent-decision layer down to the message-verification layer. That disclosure provides a concrete example of why sequence reconstruction matters. Its investigators analyzed more than 17,000 recorded events to understand what the autonomous campaign did as a whole. The disclosure does not establish that those events were individually authorized or passed policy gates. It demonstrates the narrower point: the meaning of an automated campaign may emerge only when its actions are correlated as a sequence. Here, the same shape shows up mechanically: a per-message signature check can validate an unbroken run of individually-valid requests that compose into something nobody should have permitted, because signature verification doesn't evaluate the sequence either.

As more agent-to-agent and agent-to-API traffic gets wrapped in signed HTTP requests — a genuinely good trend I expect to accelerate as agent workforces scale — this gap gets more consequential, not less. An autonomous agent making its own tool calls, each one dutifully signed under its service's key, is exactly the kind of principal whose individual requests will all verify cleanly while its accumulated trajectory goes somewhere no one signed off on.

The takeaway

Which trusted key material validates the signature or MAC, and whether the covered components still verify, are message-layer questions that RFC 9421 answers well. Whether the request — or the sequence it belongs to — was actually authorized is a decision-layer question, and no signature scheme answers it, because it was never built to. The two layers compose: verify the covered components under a trusted key, then evaluate what the authenticated actions associated with that key material add up to over time — across sessions or principals where the use case requires it. Skip the second layer and you get a very well-authenticated blind spot.

To be precise about the claim, because overreaching here would be exactly the mistake a signature-only architecture makes: this isn't an argument against RFC 9421, and it isn't a claim that curl's implementation is unsafe or premature — Stenberg's own caution about production-readiness is the right call for new cryptographic protocol work. The argument is narrower: message-layer verification is necessary and not sufficient for authorization. Something has to sit above the verifier and ask what the authenticated sequence associated with that key material composed into. That's a decision-governance problem, not a cryptography problem.

The composition engine behind this argument is open source: pip install constitutional-agent. If you're building signed service-to-service or agent-to-API traffic and want to see where the cross-session gap sits relative to your signing layer, try the free Governance Stress Test or read the Enterprise Agent Architecture framework. Tell me where you think this argument breaks.

Composition preprint: doi.org/10.5281/zenodo.21400261 · Enterprise Agent Architecture: doi.org/10.5281/zenodo.21105314

Every API Call Was Allowed. The Agent's Outcome Wasn't.

Michael "Mike" K. Saleme — Mon, 27 Jul 2026 12:43:10 +0000

Start with the thesis: Satisfied Is Not Established.

API governance is one of the things the enterprise actually got right.

Gateways, rate limits, authentication, versioned contracts, a managed lifecycle. For two decades it did its job: keep system-to-system integration orderly, secure, and observable.

Its unit of control is the API call. Its question is precise. Is this client allowed to invoke this endpoint, with this credential, at this rate?

That question was built for systems. An agent is not a conventional integration client, and its most consequential failures do not occur at the level that question can see.

The predictability the model assumed is gone

A traditional integration is designed around a relatively stable, declared flow. It calls known endpoints in expected patterns for an established business purpose. API governance was built around that predictability.

An agent may have a declared role or goal. But it selects its action path at runtime, assembling tools in response to changing context.

Every call it makes can be authorized, within rate limits, and individually compliant — and the composed outcome can still be one the enterprise never intended.

That is the gap. API governance is typically enforced call by call. The harm an agent can cause may emerge across a sequence of calls, each of which is allowed.

Two compliant calls, one outcome nobody approved

Consider an agent with read access to customer records and permission to send email.

Both capabilities are legitimately granted. Both may be necessary for its job. An endpoint-centric gateway sees two compliant calls.

Without shared task context and sequence state, it cannot determine from those calls alone why the second followed the first — or whether the two together converted ordinary access into data exfiltration.

The missing question is not merely whether the next call is permitted. It is whether this agent, acting for this task under this delegated authority, should be making it now.

API governance asks whether the call is allowed. Capability governance asks whether the agent should be making it.

More precisely: whether this agent may exercise this capability for this task, given what it has already done and the constraints that still apply.

The unit of control has to change

API governance typically controls access to individual interfaces and operations. A capability is an action or outcome an agent can produce through one or more APIs, tools, and data sources.

A major part of the attack surface is not any single API. It is the set of tools you expose to the agent, because every tool you grant widens what the agent can be talked into doing with the authority it already holds.

Least privilege stops being a question of which APIs a client may call. It becomes a question of which capabilities an agent may compose, for which task, for how long.

You cannot solve this by adding more endpoint rules alone. The gateway traditionally governs the channel. Capability governance must evaluate the delegated task, the authority available to the agent, the actions already taken, and the next action proposed. It is scoped to an execution and enforced at runtime.

This is already showing up in enterprise platforms

MuleSoft, for example, positions Omni Gateway as a common control point for API, MCP, LLM, and agent traffic. It applies policies across gateways, MCP, APIs, and LLMs, propagates identity, and carries correlation IDs across every interaction in an agent chain.

That makes the gateway a plausible enforcement point for capability governance.

But reconstructing a chain is not the same as deciding whether the current task state authorizes the next composed action. The location of the control is emerging. The governing model still has to mature.

API governance kept our systems talking to each other safely. The agentic enterprise needs governance over what a worker is allowed to do with those systems, not just which doors it may knock on.

This is part of a series on Enterprise Agent Architecture — the case for treating the agent workforce as a fifth architecture domain alongside Business, Information, Application, and Technology.

Start here: https://cognitivethoughtengine.com/eaa/part-1

Position paper: Enterprise Agent Architecture: The Case for a Fifth Architecture Domain for the Agentic Enterprise

What is enforcing capability scope in the agent systems you run right now — the gateway, the agent framework, or nothing yet?

The Evaluation Had a Sandbox. It Needed an Authority Boundary.

Michael "Mike" K. Saleme — Fri, 24 Jul 2026 19:25:20 +0000

Start with the thesis: Satisfied Is Not Established.

On July 16, Hugging Face disclosed an intrusion into its internal infrastructure. A malicious dataset abused two code-execution paths in its dataset processing — in their words, "a remote-code dataset loader and a template-injection in a dataset configuration." Internal datasets were accessed, service credentials were harvested, and lateral movement reached multiple internal clusters. Their responders detected it, contained it, and reconstructed the timeline from more than 17,000 recorded events.

On July 21, OpenAI disclosed who was on the other end. It was not a conventional external threat actor. It was two of their own models under evaluation — "GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes."

The models were being measured on ExploitGym, a publicly hosted benchmark for executing attacks against known vulnerabilities. Rather than solve the benchmark, they went and got the answers. OpenAI's account of how: the models "spent a substantial amount of inference compute finding a way to obtain open Internet access," then "identified and exploited a zero-day vulnerability (which we've now responsibly disclosed to the vendor) in the package registry cache proxy." From there they chained stolen credentials into a remote-code path on Hugging Face's servers and pulled the test solutions out of a production database.

OpenAI's own framing: "The models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal."

That sentence captures the architecture problem, and the problem is not unique to either company.

The permitted path became the attack path

The models were not granted general internet access. But they were given a narrow, permitted dependency path through a package-registry proxy. They treated that path as terrain: exploiting the proxy, escalating privileges, moving laterally, and eventually reaching a node with internet access.

This was both a containment failure and an authority failure. The environment attempted to constrain capability, but the permitted dependency path was not backed by an authorization policy that remained enforceable after the first control failed.

That is the distinction worth holding onto. Evaluation environments are built as capability boundaries: what code the model can execute, what image it runs in, what the process can touch. That work is real and it was done here. What the incident exposes is the need for a separate authority boundary — an external, enforceable answer to what the evaluation workload may reach, under which identity, for how long, and how that authority can be revoked during a run.

The failure cannot be reduced to model behavior. The decisive containment controls were also in the infrastructure surrounding the model: the dependency proxy, network paths, workload identity, credentials, and the enforcement points that were supposed to keep a deliberately less-restricted cyber evaluation contained. OpenAI's own remediation list spans model alignment, evaluation-time cyber protections, containment, monitoring, and access controls — which is the correct breadth.

What the receiving end could and could not see

There is a detail in Hugging Face's disclosure that deserves more attention than it has gotten. Their responders characterized what hit them as "an autonomous agent framework (appearing to be built on an agentic security-research harness)" — thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.

From the receiving end, the evaluation generated telemetry resembling a competent autonomous intrusion campaign. Hugging Face's initial evidence characterized the mechanism accurately, but did not identify the model or the operator.

That distinction matters. Local telemetry can reveal what an agent is doing without revealing who authorized the workload or who can terminate it. Cross-organization investigation supplied the missing attribution.

Which is the operational consequence worth internalizing: an evaluation without an authority boundary does not just risk escaping its constraints. It escapes without carrying anything that lets the organization it lands on identify the source or reach whoever can revoke it.

A control plane, not a rename for segmentation

If "authority boundary" is going to mean anything beyond network segmentation with a new label, it has to be buildable. For an agent evaluation environment, that is roughly six things:

Default-deny egress, with dependency installation mediated through an immutable or tightly constrained repository. The proxy in this incident was a permitted path with an exploitable implementation. Constrain what the path can serve, not just who may use it.

Per-evaluation workload identity. Short-lived, least-privilege credentials scoped to that single run, with no ambient harness credentials available to inherit.

Authorization enforced independently at the network, proxy, credential, and execution layers. The property you want is that the first control failing does not collapse the rest. This incident demonstrates why that property must be tested rather than assumed.

Time, compute, action, and destination budgets. A run that spends substantial inference compute searching for egress is a budget signal before it is a security finding.

Cross-boundary telemetry with tripwires for privilege escalation, lateral movement, credential discovery, and unexpected destination lookup.

A kill mechanism that revokes identity, credentials, and network authority — not merely the original sandbox process. Killing the process you know about is not revocation when the workload has already moved.

If external attribution matters to you, workload identity alone will not deliver it. An internal cloud identity is not visible or verifiable to the organization receiving your traffic. That requires pairing it with identifiable egress, authenticated provenance, and a registered incident contact.

And one framing change for whoever designs the evaluation itself: treat objective pressure as a threat input. The models did not abandon the assigned objective; they pursued it through an unintended path. An evaluation that strongly rewards a narrow outcome must assume the workload will search across every reachable surface, especially when normal cyber refusals have deliberately been reduced.

What both companies got right

Hugging Face published while its initial disclosure still listed the model as unknown and reported no evidence of tampering with public models, datasets, Spaces, or the software supply chain. OpenAI subsequently named its own models as the cause and disclosed the proxy vulnerability to the vendor. That transparency matters, but it is not the central architecture lesson.

The most capable systems we have now treat containment as terrain to be searched. In this case, the evaluation began inside a sandbox, found a permitted dependency path, converted it into broader reach, and crossed into another company's production infrastructure.

The lesson is not that sandboxes no longer matter. It is that execution containment is only one layer. A serious evaluation environment must also bind every workload to independently enforced authority: what it may reach, under which identity, within which budget, until what time, and through which mechanism that authority can be revoked.

The sandbox was present. The authority boundary was not complete.

Sources: Hugging Face security incident disclosure, July 16 2026 · OpenAI, "OpenAI and Hugging Face partner to address security incident during model evaluation," July 21 2026

A signature proves who signed the receipt. It does not prove the receipt is true. Here is how you test the difference.

Michael "Mike" K. Saleme — Sat, 18 Jul 2026 23:05:05 +0000

Start with the thesis: Satisfied Is Not Established.

The industry is standardizing on agent receipts: signed records that an action was authorized, that a check ran, that a payment settled. The signature is the easy part. The hard part is that a correctly signed receipt can still be false.

A real receipt verifier and a plain signature checker look identical from the outside. Feed either one a valid receipt and both say "accepted." The difference only appears on a receipt that is correctly signed but whose claim is not supported: evidence omitted, evidence swapped after it was attested, a check bound to the wrong tool set, an authorization for different parameters, an acknowledgment for a different action. A signature checker accepts all of those. A verifier has to reject them.

So you cannot tell which one you have by reading the code or trusting a green badge. You can only tell by running it against receipts built to be validly signed and claim-invalid, and watching whether it rejects them.

That is a conformance suite, and we published one.

Nine receipts a conforming verifier must reject, plus two acceptance controls so an implementation cannot pass by rejecting everything. Each reject vector names exactly what the verifier has to recompute from the referenced evidence: the admitted action digest, the evidence digest, the tool set the check actually covered, the attesting authority and whether it is independent of the emitter, the freshness window. Naming the recomputation is the point. It stops a verifier from passing on string-level checks.

The receipt decomposes into four properties, and the signature supports only one of them: integrity, authorization, occurrence, and the check itself. Evidence for the last three has to come from distinct trust domains, not from the emitter, because the emitter is exactly the party the threat model permits to lie. A receipt attested only by the thing whose behavior it certifies is testimony, not evidence.

The two hardest vectors are a phase pair. One carries an authorization bound to different parameters than the action requested, and fails at admission, before anything runs. Another carries an execution acknowledgment linked to a different action than the one admitted, and fails after, on linkage. Same trust failure, different phase, different invariant. A verifier that only checks signatures accepts both.

If you are building or buying receipt verification, the question is not "does it check the signature." Every implementation checks the signature. The question is "does it recompute the claim," and the only honest way to answer it is to run it against vectors designed to be signed and wrong.

A receipt is only as trustworthy as the bindings a verifier is willing to recompute.

Everything it does not recompute, it is taking on the emitter's word.

The vectors are open, reproducible, and generated from a working verifier, not hand-authored. If you have a receipt or trust-envelope implementation, run it against them. If it accepts one of the nine, you have a signature checker, not a verifier.

Two AI systems reviewed my security code. The workflow looked clear. Four findings were still open.

Michael "Mike" K. Saleme — Sat, 18 Jul 2026 17:40:57 +0000

The dangerous failure in an agent workflow is not only that something gets missed. It is that a summary signal gets mistaken for evidence that nothing is wrong.

This week I watched that happen repeatedly on my own work.

An AI code reviewer ran against a change and its status check went green. Underneath that green status, a high-severity finding remained open.

A cleanup routine could delete the credential still being used for signing because it did not compare the deletion target with the active credential.

The green check accurately said that the reviewer had completed. It said nothing about whether the reviewer had found a defect. That distinction was about to disappear.

Then an autonomous coding agent working on the same change reported that the review had come back neutral. It had read the reviewer's status, not its comments.

Those comments contained four open findings.

The agent was not inventing an answer. It was trusting a summary signal instead of examining the evidence beneath it.

The next error was mine.

I drafted a technical contribution that described eleven test cases as belonging to one category. The source contained nine of that kind and two of another. I also claimed a binding to a component that did not exist in the code.

I caught both errors only by reopening the source I was describing.

One completion signal, one incorrect agent summary, and one incorrect draft. Each looked reassuring until someone examined the underlying evidence: the open findings, the review comments, and the code itself.

That is the pattern.

A status badge is not the review.

A summary is not the evidence.

A description of code is not the code.

A passing check proves the check ran. It does not prove the code passed review.

The answer is not simply to add more agents. A second agent adds assurance only when it independently examines the underlying evidence. If it merely reads the first agent's conclusion, you have created another summary, not another control.

The verification layer has to refuse the shortcut:

Read the raw tool output, not only the status badge.
Read the findings, not only the check state.
Read the implementation, not only its description.
Separate the author from the ratifier, even when the author is you.

We are wiring agents into code review, payments, and production changes. Increasing their authority without separating claims from evidence turns a reporting mistake into an operational risk.

The green check felt like verification.

It was a completion signal wearing verification's clothes.

Read the source.

A Signed Agent Receipt Can Still Make an Unsupported Claim

Michael "Mike" K. Saleme — Sat, 18 Jul 2026 00:31:57 +0000

Start with the thesis: Satisfied Is Not Established.

AI agents increasingly emit signed "receipts" for what they did: a tool call, a payment authorization, a policy check, wrapped in a canonicalized, content-addressed, signed record. The direction is right. The trap is reading a well-formed signed record as a true one.

A valid signature establishes something precise: the presented bytes verify under a particular public key. With a trusted binding between that key and an identified authority, it can also establish signer provenance. It does not establish that the action occurred, that it was authorized, or that the checks described in the record actually ran.

So I started decomposing an action receipt into four separately assessable properties:

Envelope integrity and authenticated provenance — are the bytes intact, and is the signing key reliably bound to the claimed signer?
Occurrence — did the action happen (and in which state: requested, attempted, completed, failed, settled)?
Authorization — was it the action, with the exact parameters, that was authorized?
Check execution and integrity — did the claimed checks run, or is "verified" a fail-open default?

Signing supports only the envelope property. The other three require independently verifiable evidence attributable to the relevant authorities: a checker, an authorization authority, and an execution authority. In the stronger design demonstrated here, each authority occupies a separate cryptographic trust domain, so the receipt emitter cannot manufacture their attestations.

The claim worth testing follows directly:

A format-valid, correctly signed receipt whose asserted claims are not semantically supported must be rejected.

I made that claim executable. Each authority receives a distinct Ed25519 keypair, while the verifier holds only the trusted public keys. The negative vectors include:

a substituted checker result;
a stale transcript;
authorization bound to different parameters;
an execution acknowledgment bound to a different action; and
an emitter claiming a check for which it has no authority attestation.

The envelope signature verifies in every case. The claim-level verifier still rejects each receipt for a specific semantic reason.

This is a controlled demonstration, not a field evaluation. It complements formal-conformance and capability-binding research rather than replacing it. Those approaches test whether controls and capabilities are correctly defined or exercised. This demonstration tests a different boundary: whether the evidence artifact claims more than its supporting trace can prove.

The short methodology note is on Zenodo: 10.5281/zenodo.21418701 (record: zenodo.org/records/21418702). The reference verifier and the negative vectors are open source: github.com/msaleme/red-team-blue-team-agent-fabric (module protocol_tests/receipt_claim_harness.py).

Views my own.