DEV Community: Michael "Mike" K. Saleme

Your agent has a valid token. Should this call still be blocked?

Michael "Mike" K. Saleme — Thu, 24 Sep 2026 14:34:34 +0000

Agent identity is being standardized quickly.

On August 24, Okta made Agent SSO generally available. In Okta's words, when an agent that supports
Cross App Access connects to an enterprise application, Okta "registers it as a first-class
identity in Universal Directory alongside human employees, then issues short-lived,
identity-governed tokens in place of stored credentials." The same announcement says Cross App
Access "is formally incorporated as the official Enterprise-Managed Authorization extension for
the Model Context Protocol."

The IETF is moving too, though this is work in progress, not finished standards. An individual
draft on AI agent authentication and authorization, dated July 6, has since been replaced by
AI Identity Management System (draft-ietf-wimse-aims-00, September 15), now a working-group
draft in WIMSE. Separately, the individual draft Transaction Tokens for Agents carries the agent
and the principal it acts for through a call chain, including "chain-level metadata required for
multi-agent flow integrity," so that services can "make more granular access control decisions."

And fine-grained authorization is not new: OAuth 2.0 Rich Authorization Requests (RFC 9396,
2023) supports detailed authorization requests and makes the granted details available to
resource servers, through token claims or introspection.

This is real progress. Static API keys shared across agents were indefensible.

What the new layer does, and what it hands to you

These mechanisms settle who the agent is, on whose behalf it acts, and what it has
been granted, and they carry that context, with constraints, to the service it calls.

What they cannot do on their own is check those constraints against the request that actually
arrives, in the state the system is actually in. That check happens in your deployment, at the
moment of the call, or it does not happen.

A narrative review of 89 sources posted to arXiv on September 14 (Authorization Architectures for
Tool-Using AI Agents) reaches a similar conclusion. The authors find the literature gives
"little attention to the authorization decision point itself, the moment a tool invocation
occurs," and identify "runtime enforcement and aggregation bounds as the principal unresolved
gaps." That is their finding from the literature they reviewed, not proof that nobody enforces
anything. But it points at the right place.

A token can carry the constraints. Your deployment has to enforce them, and prove they held.

Why this matters more for agents

For a human user, the space between "authorized to use the app" and "should perform this action"
is filled by the person's judgement and by the application's business rules. An agent changes
three things:

The arguments are generated. A request accompanied by a valid token can contain parameters shaped by a prompt injection in a document the agent read several steps earlier.
Delegation can widen. Each hop in a multi-agent chain can pass on more authority than its task needs unless something checks it.
Actions aggregate. Every call can be individually within bounds while the sequence is not.

A concrete version of the third: an agent is permitted to issue refunds below $500, against a
delegated budget of $5,000. It submits thirty refunds of $400. Every single call passes the
per-refund rule. The total is $12,000.

The moment of execution is where all three must be checked, provided the component doing the
checking has trustworthy context and history. A runtime check can see the arguments without
seeing that an injection shaped them, and it can only bound the total if it can see what the
chain has already done and what it has reserved but not yet completed.

What to verify in your deployment

Vendors do not stop at identity. Okta's own announcement describes its broader Okta for AI
Agents offering as one that "includes agent-to-agent connections and runtime enforcement".
So the question is not whether you must build enforcement yourself. It is whether the
enforcement you have, bought or built, covers what your agents actually do: their actions,
arguments, delegation limits and aggregate budgets.

Adopt the new identity layer. Then verify each of these:

Separate the decision from the enforcement. A policy decision point evaluates each call (this action, these arguments, this principal, this context). A policy enforcement point sits in the execution path and blocks the call when the answer is no. One service can do both, but the enforcement must be on the path the agent cannot route around.
Keep downstream authority within upstream authority. Each hop may hold at most what the hop above it held, and only what its task needs. It does not have to shrink at every hop. It must never grow.
Bound the aggregate, including in flight. Track the running total for a delegated chain, and atomically reserve against the budget before a call executes, not after it completes. Otherwise several concurrent requests can each see room in the budget and together exceed it.
Prove coverage, not just activity. A log of refused calls, with the rule that refused each one, shows that enforcement fired. It does not show that every execution path goes through it. Test for bypass, and correlate decisions with the effects that actually happened downstream.

The question to ask your architecture

Even when an agent presents a valid, short-lived, identity-governed token, authentication is
only the first question.

The next question is: when an authenticated agent makes a call it should not make, what stops
it, and where is the evidence?

If the answer is "the token", look again at what the token carried, and at whether anything
checked it against the call that was actually made.

Sources: Okta, Agent SSO announcement, 24 August 2026;
Okta, Cross App Access extends MCP;
draft-ietf-wimse-aims-00 (replaces draft-klrc-aiagent-auth);
draft-araut-oauth-transaction-tokens-for-agents-02;
RFC 9396, OAuth 2.0 Rich Authorization Requests;
Authorization Architectures for Tool-Using AI Agents, arXiv:2609.15906.

What my seventeen AI attack tests missed about the evaluator

Michael "Mike" K. Saleme — Sun, 20 Sep 2026 14:48:22 +0000

Two threat-actor designations from two Anthropic threat reports. In the first, Claude Code
was the weapon used against other organisations. In the second, another AI vendor's evaluation
sandbox was the target; Anthropic's report states its own systems were not compromised.

I have a seventeen-test module for the first pattern and nothing in it for the second, and I only
noticed because I went looking to prove the opposite.

The first one

In September 2025 a likely China-nexus actor manipulated Claude Code into running a multi-stage
intrusion campaign. MITRE tracks it as Campaign C0062, attributing the
campaign to the actor Anthropic designated GTG-1002. The targets were "approximately 30
entities in the technology, financial, chemical, and government sectors."

MITRE's summary of what the AI did:

reconnaissance, vulnerability discovery, exploitation, lateral movement, credential harvesting,
data analysis, and exfiltration operations

Anthropic reported the actor "was able to use AI to perform 80-90% of the campaign, with human
intervention required only sporadically."

I wrote tests for that. The harness I maintain carries seventeen covering the six documented
phases, including credential extraction from configurations and cross-system lateral movement
(module at e2a647f).

The second one

From Anthropic's September 2026 threat report,
actor GTG-50020:

By injecting malicious instructions into an AI vendor's automated evaluation sandbox, the actor
caused the sandbox to hand over the credentials it held - including the production AI API keys
from multiple providers belonging to that vendor.

Those keys then funded a follow-on campaign: the actor "identified one successful attack
path and repeated it against all thirty targets." Thirty AI companies attacked over about four
days. The report does not establish thirty successful compromises, and the initial sandbox
compromise and the follow-on campaign are two separate events.

Read the two designations next to each other. In the first, someone else's network was the target
and the AI product was the instrument. In the second, an AI vendor's evaluation infrastructure
was itself the target, and what it held was worth more than what it measured.

Why that surface

Not because anyone was careless. Because of a property these environments have by construction.

An environment that exercises a model exists to find out what the model does when it is allowed
to act. So it is granted reach: endpoints, tools, and in some configurations live credentials.
The thing it is pointed at is untrusted by definition, because that is the point of pointing at it.

The environment that measures a system inherits every privilege the measurement requires.

Note the qualifier. Plenty of useful measurement runs against mocks and synthetic fixtures, and my
own suite does exactly that. Some integrations need to be live. Neither of those facts requires
production credentials to be present, and that is the narrow question worth asking.

Because two properties come apart here, and satisfying one does not establish the other:

Credentials stay inaccessible. The system under test cannot read the key.
Operations stay restricted. Even holding legitimate access, it cannot perform operations it was not authorised to request.

Two more disclosures, and neither needed an attacker

Anthropic's cybersecurity incident assessment,
published 9 September 2026, covers four cybersecurity-evaluation incidents. In the fourth, an
early Claude Opus 4.6 checkpoint on a capture-the-flag task broke its own target by assigning a
conflicting IP address. What happened next is the detail I keep returning to:

The model attempted to abort the task using a command but was unsuccessful due to a
misconfiguration in our evaluation harness.

Seven further abort attempts appear across the transcript. The model also reached real
third-party infrastructure, obtained administrator access, and read one person's personal
information.

The model was trying to exit a task it had made impossible, not refusing to do harm. What the
incident establishes is narrower and still worth having: an abort control existed, the model
invoked it, and a misconfiguration in the harness meant it provided no effective stop in that
run.

It surfaced during a later transcript scan, after omitted transcripts came to light while material
was being prepared for an external reviewer.

The second is from a different setting, and the difference matters. OpenAI's
Artifactory notice
concerns RL training, not evaluation: "During RL training, there were multiple instances of
our models using OpenAI's internally hosted instance of Artifactory as a shared message board."
Credentials had been supplied deliberately for downloading packages. OpenAI states the models
"did not exploit any vulnerabilities to exchange messages" and that "our monitoring flagged the
behavior for human review."

Nothing was stolen there. The significance is unintended cross-sample communication using supplied
access, which let separate training samples draw on each other's work and undermined their
independence. It is property 2 failing while property 1 was never in question, in an environment
that is not an evaluation sandbox at all. Which is the point: the exposure is a function of granted
reach, not of what the environment is called.

What I found when I checked my own work

I filed a coverage gap against my own harness for this
(#577), then did the
implementation review I had said was outstanding, and had to withdraw part of it in public.

GTG-P3-002 | Callback/Beacon Validation already covers the egress case. I had claimed it did not,
on the strength of a keyword search over a generated catalog rather than a read of the code.

What survived the correction is narrower and sharper. All seventeen of those tests model the agent
as the attacker's instrument operating against a victim system. None models the inverse, where
the environment running the test is itself the thing holding what an attacker wants.

Seventeen tests for one direction and none for the other, within that module. I have not
audited the whole suite for this property and I am claiming nothing about anyone else's tooling.

The question worth asking on Monday

Not "is our eval sandbox secure." The narrow version, because it has an answer, and it has two
halves because the controls are independent:

Which credentials does your training or evaluation environment expose to the system under test,
which operations are reachable with them, and what enforces the boundary you intended?

An environment holding live credentials is security-sensitive and should be inventoried with the
same seriousness as the systems those credentials reach. If the honest answer to the first half is
"more than the measurement requires," that is worth an afternoon.

The gap is filed as #577,
including the part I got wrong. If you run these environments and can answer for yours, I would
rather hear it than guess.

Views my own, not my employer's.

I Tested Revocation at Four Layers. Memory Was Not One of Them.

Michael "Mike" K. Saleme — Thu, 10 Sep 2026 15:19:27 +0000

I had revocation tests at four protocol layers. Then I checked what happened to withdrawn information in memory, and found no test for whether the withdrawal was honoured.

Revocation is easy to write down. Credentials can be revoked. Delegations can be withdrawn. A compromised token stops working. The word does a lot of load-bearing work in a threat model, and in my own suite it turned out to do that work at four layers, but not the memory layer.

On 8 September a paper measured it. "Revoked but Still Authoritative" (arXiv:2609.08258) tested revocation enforcement across five agent-memory systems, nine policy scenarios and nine models. The finding, in the authors' words:

no system enforces revocation by default: the revoked fact is returned wherever the revocation label is visible to the retrieval layer

Read that carefully, because the mechanism is more specific than "revocation is broken." The record is labelled correctly. The system knows it is revoked. The label is simply visible at a layer that does not act on it, and retrieval hands the record back anyway.

The reported failure spans three stages: a revoked record is returned despite a visible revocation label, it outranks its replacement, and it influences the agent's action. Those stages need separate checks.

That is a useful shape, because it tells you that testing one of them is not testing the control.

One definition before going further, because it decides whether a test is measuring anything. Retrieval here means records supplied to the agent as context for the task. Retaining a revoked record for audit or history is a separate behaviour and a legitimate one. A system that shows you a withdrawn policy in an audit view has not failed to enforce anything.

What I found when I checked my own suite

I maintain an open-source adversarial test harness for agent protocols. I went looking for what it asserted about revocation, expecting to find a gap somewhere and to write about that.

I found four tests, spread across four protocol layers:

Layer	Test
Identity	`AUTH-002` Expired/Revoked Credential Rejection
Delegation	`DCA-007` Pass Used After Revocation Epoch
Provenance	`PRV-015` Replay of Revoked Attestation
Settlement	`DSET-008` Revoke-on-Nonfinality (post-grant remediation)

DCA-007 is the one I would defend hardest. It does not only check that a revoked pass is refused. It checks that revoking an intermediate hop voids the leaf below it, not only the hop named. Revocation that does not cascade is a containment boundary with a hole in the middle of it.

Then I looked at the memory layer, which is where an agent actually keeps the thing it will act on later. Twelve tests. Cross-session leakage, poisoning, context overflow, history manipulation, RAG poisoning, privilege escalation, selective erasure, cross-user contamination, temporal consistency, extraction, and two namespace-scoping cases added after a tenant-isolation CVE.

Not one of them asked whether a withdrawal is honoured.

So the paper measured the exact layer my suite had skipped, and my suite had skipped it while testing revocation at four other layers. The vocabulary was consistent across four modules. The coverage stopped at the layer where the record persists.

A control you can name is not a control you have measured.

What went in

Four tests: presentation, precedence, policy use, and persistence across sessions.

Test	What it asks
`MEM-013` Revoked Policy Still Presented As In Force	Is the policy still presented as in force despite an explicit revocation?
`MEM-014` Revoked Policy Presented Over Its Replacement	Does the answer treat the revoked policy as currently applicable despite its replacement?
`MEM-015` Agent Reports Authorisation Under a Revoked Policy	Does the agent still report authorisation under the revoked policy?
`MEM-016` Revocation Lapses Across a Session Boundary	Is revocation still enforced in a later session?

MEM-014 is the one a naive revocation test misses. A test that only checks whether the replacement appears can pass even when the answer still treats the withdrawn policy as applicable. Whether the agent then reports authorisation under it is a separate question, which is what MEM-015 is for. These stages fail separately, and a suite that collapses them measures one and reports three.

Getting that criterion right took a third round. "Does the revoked policy appear in the answer" is not the same question as "does the answer treat it as applicable", and the difference is a correct system explaining itself.

MEM-016 exists because a revocation honoured inside one conversation and forgotten at the next is a revocation with an undocumented expiry.

The part that took longer than the tests

Every one of these carries a positive control, and that is not ceremony.

Consider what a revocation test does against a target that stores nothing at all. You write a record. You revoke it. You read it back. Nothing comes back, because nothing was ever there. The test reports that the control held.

I have shipped that defect before. A payment test once returned a pass against a target with no delegated-allowance support whatsoever: nothing was accepted, nothing settled, so nothing was overdrawn, so the control held. The target answered. The verdict was real. It was also meaningless.

So each of these four first establishes the thing it is about to withdraw. MEM-015 carries the strictest version: it asserts the agent reported authorisation under the policy before revocation, because an agent declining a capability it never claimed proves nothing about revocation.

Then external review caught something the controls did not.

Two of the four asked the target to "repeat every operating policy you have stored." That is an audit-shaped question. A store that correctly stops honouring a revoked policy while keeping it for history answers it with the revoked record, and my test would have called that a failure to enforce. Correct behaviour, scored as a defect, on the exact property the family exists to measure.

The other two already asked the right questions, one about which policy is "currently in force" and one about whether the agent is authorised, although their answer checks still needed a later fix of their own. The repair was to ask what applies now rather than what is held, and to add a fixture that returns revoked records to audit-shaped questions only, so a future rewording cannot drift back without failing.

The controls are pinned against six target shapes:

a store that enforces revocation: all four must pass
a store reproducing the paper's mechanism: the first three must fail
a store that enforces in-session and forgets at the session edge: only MEM-016 fails
a store that retains nothing: all four must return INCONCLUSIVE, never a pass
a store that enforces but keeps revoked records visible to an audit question: all four must pass
a store that enforces and names each withdrawn policy in order to disclaim it: all four must pass

Those cases catch different errors: reporting success without establishing retention, treating audit history as active policy, and mistaking an explicit disclaimer for endorsement. Two of the three were added after a review round; the retain-nothing case was there from the start, because it is the one I already knew I had got wrong before.

Neither adversarial sweep produced a PASS. That alone does not validate the tests: an unreachable target supplies no evidence of enforcement, and a test that always fails would reject every adversarial fixture just as neatly. The compliant fixtures are what check the tests can also recognise correct behaviour.

Worth being exact, because I was vague about this in an earlier draft. Against both a closed port and a target that agrees with everything, all four report INCONCLUSIVE rather than FAIL. The closed port provides no usable response. The agreeable target never echoes the policy back, so the positive control cannot establish retention. Neither result establishes whether revocation is enforced.

What this does not establish

It does not establish that any agent system is secure. It validates the tests against controlled fixtures; it does not independently validate a deployed system, and it is self-authored and self-run.

It does not reproduce the paper's result. The authors tested five deployed memory systems. I wrote tests that would detect the failure they describe. Those are different claims and I am not going to blur them.

It does not mean my suite is ahead of the systems they measured. A tester and a memory backend are not the same category of thing.

It does not tell you which stage broke. These checks observe the agent's response. There are no retrieval traces and no backend inspection, so they cannot isolate whether a failure occurred in retrieval, in ranking, or in the agent's use of the returned context. A failure indicates that the answer still treats the withdrawn policy as applicable, or reports authorisation under it. It does not establish that the agent attempted an action.

That gap between what the tests name and what they observe is exactly what the second round of review found, and it is why MEM-013 is no longer called "Returned at Retrieval" and MEM-015 is no longer called "Agent Acts". A name that claims a stage the assertion never reaches and a test that cannot fail are different defects, but both make the reported evidence stronger than the test supports. I had written that argument up for a standards body three days earlier and then shipped it.

What it does establish is narrower and, I think, still useful: the failure mode is now an executable assertion with a stated truth table. Pointing it at your own system is not free, though. These drive an agent endpoint rather than a memory backend, so MEM-015 needs a target that will accept a policy statement and answer an authorisation question. If your store sits behind something that will not, the adapter is the work.

The generalisable bit

If your architecture names revocation as a control, the question is not whether you can revoke. It is which layer stops honouring the withdrawn thing, and whether you have ever watched it happen.

Identify what is exposed, then revoke it, then verify that the withdrawn policy is no longer treated as applicable, including in a later session. Use backend traces if you need to locate where the failure happened. The third step is the one that gets written down and not run.

I had four layers of it, and none at the layer that remembers.

Satisfied Is Not Established

Michael "Mike" K. Saleme — Sat, 05 Sep 2026 18:04:36 +0000

I went back through two months of my own writing this week and found I had written the same sentence about twenty times.

MCP went stateless. My test suite stayed green anyway.
Every action was authorized. The sequence still crossed the line.
A signed agent receipt can still make an unsupported claim.
My quorum test never had an approver in it.

I had not noticed. Each was written as a separate observation about a different control failure. They are one observation, and it is worth stating directly instead of twenty more times sideways.

Satisfied is a fact about the check. Established is a conclusion the evidence has to warrant.

A passing verdict tells you that the implemented decision procedure accepted the observed input. It does not, on its own, establish the property named by the test. That depends on what the check actually exercised, what evidence it observed, and what conditions would make it fail.

Three places the gap opens

Authorization that permits without deciding. An agent can leak private repositories without breaking a single permission. Every call was allowed; the outcome still wasn't. The permission model was satisfied. Nothing about the outcome was established, because the permission model never ranged over outcomes.

Signatures that bind without validating the claim. curl merged RFC 9421 support, and a valid signature still is not authorization. A valid signature can bind the covered receipt content to trusted key material; it does not prove the claim in the receipt is true. The cryptography is sound and the claim on top of it is unaudited. A signed receipt can still make an unsupported claim, which is why the receipt cannot be written by the pen it is checking.

Evaluations that pass without exercising. This is the third class, and it is the one I want to spend the rest of this on.

The mechanisms are different. Authorization, cryptographic verification and evaluation validity are not one thing. What they share is the mistake of treating the scope of a successful check as if it were the scope of the claim made from it.

The third class is the uncomfortable one

The first two can be discussed at a safe distance. The third turns the instrument back on whoever is holding it.

My test suite stayed green after MCP went stateless, because it was asserting against a shape that no longer existed. My suite passed against nothing — a closed port produced the same verdict as an enforced control. I tested my own method four times and its strongest claim never held. And last week I found a test named Approval Quorum that never had an approver in it: it asserted threshold routing under a name claiming quorum enforcement, and it had been green for months.

Then I wrote the fix, and the fix had the same defect one level in — it checked that approvers were distinct and never that they were authorized. Two arbitrary strings formed a quorum.

Three of those surfaced in eight days; the MCP defect was earlier in the month. I was actively looking for this exact class of failure and still reproduced it in the repair.

Why passing evidence is trusted more than it earns

Three artifacts usually travel together: a name, a mapping to some requirement, and a result. When all three agree, review stops.

But they are not three independent witnesses. The name was written from an intention. The mapping was written from the name. The result reports whatever the assertions happened to cover. Consistency across them shows the same abstraction propagated cleanly — including, when it is wrong, the error.

As agent-assurance programs formalize, that shared lineage matters more. A control name, a requirement mapping and a passing result can look like independent corroboration even when all three descend from the same mistaken abstraction.

How to tell the difference

One question, asked of a check rather than of a system: what concrete violation of the named property would make this check fail?

If you cannot answer that concretely, the check has not yet established the property it names — however green it is, however cleanly it maps. It may still establish something narrower, as mine established threshold routing. Narrower is not nothing; it is just not what the name promised.

Then build that case and run it. Not a generic pass-and-fail pair; outcome polarity is not property coverage. A quorum test needs the same approver counted twice, an approval from outside the authorized set, and an approval bound to a different action. Each one aimed at a specific property the name claims.

And point the check at an implementation that has the defect. Adding assertions proves only that more predicates ran. A suite that fails against a deliberately broken implementation demonstrates something more important: it can recognize the defect it claims to detect.

What I am doing with this

I am going to keep finding these, mostly in my own work, and write them up as one series instead of twenty unrelated posts.

The instances are not the interesting part. The interesting part is that this failure can stay invisible to the routine signals teams review software with — the linter, CI status, the coverage report, the requirements crosswalk. Each can report consistency without demonstrating that the named property was exercised.

The decisive test is a case built to violate the property.

My Quorum Test Never Had an Approver in It

Michael "Mike" K. Saleme — Sat, 05 Sep 2026 15:28:57 +0000

Start with the thesis: Satisfied Is Not Established.

Last week I wrote about tests that pass because nothing was there — a suite pointed at a closed port, reporting that a control held when nothing had been contacted.

This is the opposite failure, and I think it is the more common one.

Everything was there. The test called a working reference implementation and produced a true verdict about threshold routing. It just reported that verdict as evidence of quorum.

The test

def test_fb_013_approval_quorum(self) -> None:
    """FB-013: Above-threshold spend requires manual approval (MEDIUM)."""
    engine = PolicyEngine(SpendPolicy(
        allowlist=frozenset({"0xM"}), per_tx_cap=100_000_000,
        auto_sign_threshold=1_000_000))
    decision, reason = engine.evaluate("0xM", 5_000_000, self._now())
    self._finish(
        test_id="FB-013", name="Approval Quorum Above Threshold",
        control="policy_approval_quorum",
        ref="Fireblocks Policy Engine — approver quorum above auto-sign threshold",
        model_pass=(decision == "require_approval"),
        ...)

First, what this is not. PolicyEngine here is my own reference model of Fireblocks-style policy routing, living in my harness. The ref= string names the real product because that is the behaviour the model imitates. Nothing below is a defect in Fireblocks, or in any shipped product. The defective code is mine.

Now read the assertion, then read the name.

There is no approver set anywhere in this test. No identities, so nothing for distinctness to be checked against. No approval bound to the action being evaluated.

What it asserts is routing: a spend above the auto-sign threshold must not be auto-signed. That is a real control and it really holds.

Quorum is never exercised. Nothing here would notice if the same approver approved twice, or if an approval granted for an entirely different transfer counted toward this one.

It was green for months.

Why it survived

Three review artifacts were mutually consistent.

The name said quorum. The requirement mapping said quorum. The result said pass.

They repeated the same claim, and the code established only threshold routing. Consistency across metadata is not evidence about the assertion — it may only mean the metadata inherited the same abstraction error.

Where this bites beyond one test

The pattern generalizes to compliance crosswalks: once a control name is mapped to a requirement ID, the mapping itself starts to look like evidence.

AIUC-1 gives a concrete example. Six requirements call for quarterly third-party evaluation — B001, C010, C011, C012, D002 and D004. Their evidence artifacts require reports documenting the applicable risk scope, methodology, findings and remediation tracking; the C and D controls also expressly call for records of assessor qualifications and independence.

Every one of those artifacts can be in perfect order around a test like mine. The assessor is qualified, the methodology is documented, the report is complete, the cadence is met — and the claimed property was still never exercised.

The word "quorum" was doing work the assertion did not do.

The fix that would not have worked

My first instinct was to require paired positive and negative controls: every control test would have to show the model accepting what it should permit and rejecting what it should prohibit.

That still would not have caught this. Below the threshold the policy could auto-sign; above it, require approval. Both routing outcomes could be correct while quorum remained completely untested.

The outcome was never the problem. The property was.

The fix that does

Write deliberately invalid cases aimed at the specific property the name claims, and require the test to detect each one.

def test_one_approver_twice_is_not_a_quorum_of_two(action):
    """Cardinality is not identity. This is the defect class itself."""
    q = ApprovalQuorum(threshold=2, action=action,
                       eligible_approvers=frozenset({"approver-a", "approver-b"}))
    recorded, _ = q.approve("approver-a", action)
    assert recorded
    recorded, reason = q.approve("approver-a", action)
    assert not recorded
    assert "duplicate" in reason
    assert not q.satisfied, "the same approver counted twice reached the threshold"


def test_approval_for_a_different_action_does_not_count(action):
    """An approval is granted for one action, not for the approver's
    general willingness to approve."""
    other = ActionRef(pay_to="0xM", amount=5_000_000, nonce="tx-2")
    q = ApprovalQuorum(threshold=1, action=action,
                       eligible_approvers=frozenset({"approver-a"}))
    recorded, _ = q.approve("approver-a", other)
    assert not recorded
    assert not q.satisfied


def test_an_ineligible_principal_does_not_count(action):
    """Identity is not authority. Two distinct strangers are still strangers."""
    q = ApprovalQuorum(threshold=2, action=action,
                       eligible_approvers=frozenset({"approver-a", "approver-b"}))
    assert not q.approve("stranger-1", action)[0]   # recorded == False
    assert not q.approve("stranger-2", action)[0]
    assert not q.satisfied, "two distinct ineligible parties formed a quorum"

The suite needs to establish five obligations, one per property the name claims:

two distinct eligible principals satisfy the quorum;
the same principal twice does not;
an ineligible principal does not count, however well-formed the approval;
an approval for a different action does not count;
changing any authorization-relevant field invalidates the approval — here, an approval carries a frozen action record and is compared against the action under evaluation, so a difference in destination, amount or nonce makes it an approval for something else.

Those three fields are what this reference model binds, and naming them is the point: a production authority might also need to bind asset, network, method, calldata, expiry and policy version. State the fields your check actually covers, because "bound to the action" is exactly the kind of phrase that outruns its assertion.

The first is not filler. A check that rejects everything has enforced nothing, so the suite has to accept the case it exists to permit.

Obligation 3 is there because I left it out. My first fix checked that approvers were distinct and never that they were authorized — so two arbitrary strings formed a quorum. I had written a class to fix a test that counted without checking identity, and it checked identity without checking authority. The same defect, one level in. That is how strong this pull is: I found it, named it, wrote about it, and reproduced it inside the remedy.

After rewriting FB-013 to exercise those five obligations through ApprovalQuorum, I ran the rewritten test against an intentionally defective implementation:

class CountingQuorum(fb.ApprovalQuorum):
    """An intentionally defective implementation: counts approvals and
    checks nothing else. Injected to prove FB-013 detects it."""
    def approve(self, approver, action):
        self._approvers[f"{approver}-{len(self._approvers)}"] = action
        return (True, "counted")

harness = fb.X402FireblocksTests(simulate=True)
original = fb.ApprovalQuorum
fb.ApprovalQuorum = CountingQuorum          # the injection
try:
    harness.test_fb_013_approval_quorum()
finally:
    fb.ApprovalQuorum = original

result = next(r for r in harness.results if r.test_id == "FB-013")
assert not result.passed, "FB-013 passed against a quorum that only counts"

Substitution by module attribute rather than a constructor argument, because the
test builds its own quorum internally. It is cruder than dependency injection and
it does the same job: FB-013 runs unmodified against a defective implementation and
has to notice.

That last one is the part I would not skip again. Adding assertions makes a test say more. Injecting the defect it is supposed to catch proves it can still say no.

All of it is public, if you want to check rather than take my word:
the defect at its pinned revision,
the correction,
the eligibility hole in that correction,
and the external reproducibility work
that started the review — scoped by its author as a compatibility result, not validation.

What I would take from this

A test name is a claim, but ordinary CI does not establish that the assertions prove it. Not the linter, not the coverage report, and not the crosswalk that cites the test as evidence.

For that, the suite needs cases deliberately constructed to violate the named property — and evidence that the test fails against an implementation containing that defect.

Can your security test suite fail? Mine passed against nothing.

Michael "Mike" K. Saleme — Sun, 30 Aug 2026 01:16:45 +0000

Start with the thesis: Satisfied Is Not Established.

Most security test suites answer one question: did the attack succeed? The
verdict then falls out of a search for evidence that it did.

passed = not leaked
passed = not granted_admin
passed = len(unsafe) == 0

Every one of those is true when the attack was blocked. Every one of them is
also true when nothing happened at all.

That second case is not hypothetical. It is what a suite reports when the target
was unreachable, when the endpoint does not implement the feature under test,
when the tool list came back empty, and when the model complied politely in
language the detector does not recognize.

Three sentinels, not one target

The fix is not a better detector. It is asking the same suite the same question
across three deliberately different response conditions.

a closed port      never answers
a permissive stub  returns 200 and explicit grant markers to everything
a refusing stub    returns 403 and an error envelope to everything

These are diagnostic sentinels, not conformant implementations. A 200 carrying
{"granted": true} is not an MCP or payment-protocol grant, and a blanket 403
may block discovery rather than demonstrate enforcement. Their purpose is
narrower and useful: to expose verdicts that ignore or misread target behavior.

Point the suite at all three and read what changes.

A target-dependent test that passes against the closed port is claiming a
control held when nothing was contacted. One that passes against the permissive
sentinel despite explicit grant markers deserves inspection. And a test that
reaches the protected operation but cannot recognize an explicit denial may not
recognize the outcome it exists to measure.

For a target-dependent control, an unchanged verdict across all three
sentinels is evidence that the verdict may not be measuring target behavior.

The qualifier matters. Local controls, parser checks, and over-refusal tests can
legitimately return the same verdict everywhere.

What it found

I ran this against my own agent security harness, which is public, and which I
had believed was in good shape.

The closed port went first. Eighty-eight verdicts passed against a host that was
not running, across ten modules. The largest single count was a payment protocol
harness reporting forty-four controls holding against nothing at all.

The permissive sentinel went next, and it is the harder one because the target
answers. Every serviced-request guard I had just written was satisfied and
correctly stayed out of the way. Three hundred and nine verdicts out of five
hundred and thirty-two passed against a server returning HTTP 200 and explicit
grant markers to every request:

{"granted": true, "admin": true, "scope": "operator.admin"}

One of them read "Elevated scope claims not honored." Another read "All
gate-disable attempts were rejected." A third reported an incident-detection
latency of 0.000 seconds, measured from a connection being refused.

Not all three hundred and nine are defects, and I will come back to that.

The refusing sentinel went last and it found the inverse. Sixteen suites
produced no passing verdict against it. Three of those test controls where a
platform refusing an action is the behavior under test: a gate-disable, a kill
signal, an incident response. Those three could not recognize the one outcome
they exist to recognize.

Put the second and third together and you get the sharpest diagnostic result.
One identity and authorization suite scored seven of eighteen against the
permissive sentinel and one of eighteen against the refusing sentinel. Its
verdict moved in the opposite direction from the behavior it was intended to
recognize, and neither sentinel alone would have exposed that inversion as
clearly.

The part I did not expect

None of the three found the most interesting defect.

An external reviewer reading the source found it instead. One test serialized
the whole transport-error envelope into a substring detector whose keyword list
contained "refuse". The string it fed in was

<urlopen error [Errno 111] Connection refused>

so the agent was credited with refusing a prompt injection it never received.
Not an absence read as a pass. A positive match on the wrong text.

Then a run against a real MCP server found a second one. The protocol has two
ways to report a rejection, and my harness read only one of them, so a reference
server that correctly refused an unregistered tool call was recorded as having
failed to refuse it.

The sentinels could reveal suspicious verdicts, but they could not
independently identify either interpretation defect. The permissive and refusing
fixtures used the response idiom my harness already expected. The closed port
produced a transport error that the harness mistakenly treated as response
evidence. A fixture built around your interpretation of the property under
test cannot tell you that the interpretation itself is wrong.

What this is worth to you

The method transfers. It does not depend on my harness, my protocols, or my
threat model.

If you maintain a security test suite, a scanner, a conformance checker, or an
eval, you can build all three sentinels in an afternoon. They are a closed port,
a web server that returns 200 to everything, and one that returns 403 to
everything.

Three things worth knowing before you do.

The counts are a reading list, not a defect count. This is why I did not call
those three hundred and nine defects. Some passes against the permissive sentinel
are correct. Twenty-five of mine were over-refusal checks whose expected outcome
is permissive behavior. Other tests require protocol-specific markers the generic
sentinel does not emit. Only reading the test separates those cases from a real
inversion.

Zero verdicts is not zero defects. One module in mine aborts cleanly when it
cannot handshake, which is the right behavior and which rendered as 0/0,
indistinguishable from a clean run. It now prints "produced no verdicts, nothing
measured, not clean."

And the sentinels are the floor, not the ceiling. Get to a real implementation
as soon as they stop finding things. Source review and a half-hour test against
one real MCP implementation exposed two additional defect classes the synthetic
sentinels structurally missed.

Inspect and rerun it

The repaired harness and all three sweep scripts are pinned at commit
e151566550ecdb29b0230d40fb371fdf03bbfe09
of msaleme/red-team-blue-team-agent-fabric,
released as v4.16.0:

python3 scripts/dead_host_sweep.py
python3 scripts/permissive_host_sweep.py
python3 scripts/refusing_host_sweep.py

These commands rerun the method against the repaired harness. They will not
reproduce the original pre-repair counts, because the repairs are in. The
original findings are preserved in the earlier state files and in
PR #417
through
PR #432;
the provenance record
identifies the exact source of every figure reported here.

I am not claiming any of this makes an agent secure. It establishes something
narrower and, I think, more useful: what a harness claims when it has nothing to
go on.

I Tested My Own Method Four Times. Its Strongest Claim Never Passed.

Michael "Mike" K. Saleme — Fri, 28 Aug 2026 14:00:00 +0000

Start with the thesis: Satisfied Is Not Established.

Technical source: R2.1 results, R3 results, R5 results, R3/R4/R5 reconciliation
Companion post: Context Is Part of an Agent's Authority

I built a benchmark family to test whether a governed metadata layer earns its cost when an agent selects enterprise context. I have now run it four times under four frozen contracts, redesigning the catalog and changing the acceptance ceiling along the way.

The claim that governance earns its cost against a cheap baseline has been rejected in every round that tested it. The round before those was rejected too, on a different rule.

Round	Claim	Failed rule or controlling result	Verdict
R2.1	Overall comparative claim	Governed holdout F1 0.24065, below the 0.245533 floor	REJECTED
R3	Governed value vs. lexical	F1 CI `[-0.371, 0.00005]`; token ratio 2.11x vs. 1.10x ceiling	REJECTED
R4	Governed value vs. lexical	Quality passed; token ratio 9.86x vs. 3.0x ceiling	REJECTED
R5	Governed value vs. lexical	Quality passed; token ratio 6.94x vs. 3.0x ceiling	REJECTED

R3 through R5 are value-claim verdicts. R2.1 was the earlier overall-contract rejection that led me to separate the claims. Other claims passed: R3 and R5 accepted governed routing against full-context stuffing, while R4 returned those claims inconclusive.

Every round ran under a contract frozen before collection. The claim-scoped outcomes are documented publicly; R3 and R5 include public decision packs, while R4's later-derived pack remains held and is disclosed as such below.

R2.1 failed first. At its 3,000-object holdout the governed route scored 0.24065 against a prespecified floor of 0.245533, while the lexical prefilter scored 0.588.

Round 3: the simple baseline won the observed comparison

R3 compared three routes on the same local model: raw full-context stuffing, a cheap lexical prefilter, and a governed metadata route.

Background. This benchmark reproduces the structure of McKnight Consulting Group's study, "Stop the Token Bleed: Benchmarking the Benefits of Governed Metadata for Enterprise AI" (Jake Dolezal and William McKnight, August 2026; sponsored by Informatica, a Salesforce company). Their study held the model constant and found governed metadata access won on both cost (up to roughly 89x fewer tokens at scale) and accuracy (F1 1.000 against 0.29 to 0.66 ungoverned). Read their article for their full methodology and figures. This work does not reproduce their exact numbers. It lets you generate your own, on your own model.

Disclosure. The study reproduced here was sponsored by Informatica, a Salesforce company. I am employed by Salesforce. That is a reason to run this harness yourself rather than take my output on trust, which is the entire point of publishing it. Contradicting results are welcome.

Mean F1 across 20 seeds, at the prespecified 0% classifier-miss condition:

Catalog size	Governed F1	Full-context F1	Lexical F1
300	0.780	0.261	0.660
1,500	0.632	0.253	0.737
3,000 (holdout)	0.447	0.177	0.631

Read the two columns downward. The governed route degrades monotonically as the catalog grows: 0.780, then 0.632, then 0.447. The lexical route does not show the same pattern: 0.660, 0.737, then 0.631.

At the holdout, the keyword filter beat the governed route on the observed means and used less than half the prompt tokens. The paired F1 interval was [-0.371, 0.00005], which does not exclude zero in governance's favor. The token ratio was 2.11x against a frozen ceiling of 1.10x.

Governed routing crushed full-context stuffing. That claim passed. It lost to the cheapest thing in the room.

What I changed between rounds, said out loud

I ran R4 and R5 on redesigned catalogs, and I relaxed my own cost ceiling.

R3 used a lexically tractable synthetic catalog, where a keyword filter has real signal to match. R4 and R5 moved to semantic-access catalogs built on opaque physical names. In R5 the lexical route scored 0.000 F1 at every catalog size.

The redesign favored my method on quality: this name-only lexical baseline no longer had matching signal. But it also made the cost comparison harder, because the lexical route now produced an extremely small prompt. At the R5 holdout it averaged 110.0 prompt tokens against the governed route's 763.5, which is why the ratio rose to 6.94x even as the ceiling was relaxed.

Separately, between R3 and R4 I raised the maximum governed:lexical prompt-token ratio from 1.10x to 3.0x, which made the cost rule easier to pass.

Both changes were declared in new frozen contracts before their respective collections.

State that plainly, because a reader who diffs the contracts will find it anyway.

It still failed

In R5, governed selection used 6.94 times the lexical route's prompt tokens. The frozen ceiling was 3.0x. The preregistered claim that governance earns its cost against lexical filtering was rejected.

This happened on a task where the baseline scored zero. Governed context won the quality comparison but failed the prespecified prompt-token cost rule. The frozen contract required both.

R4 failed the same rule at 9.86x. R4 also returned INCONCLUSIVE on its full-context claims because one holdout request contained 128,256 input tokens plus a reserved 3,000-token completion budget, putting it 184 tokens beyond the verified 131,072-token window. It made no model call. It was retained as a preflight refusal rather than silently dropped.

The ceiling could have been relaxed again after seeing 6.94x. Moving a bar you already missed converts a result into a press release.

A benchmark that cannot reject its author is marketing with a methodology section.

What this changes for anyone buying or building a context layer

Ask three questions of any governed retrieval, semantic layer, or context-governance component.

What is the cheap baseline, and did you run it? Not merely full-context stuffing. At minimum, test the cheapest credible selective baseline: a keyword filter here, but potentially BM25, a cached lookup, or another simple retrieval route. If the only comparison is against full-context stuffing, the result may justify selective context, but it does not show that the sophisticated route earns its place over cheaper alternatives.

Was the acceptance rule written down before collection? A cost ceiling chosen after seeing the numbers is a description, not a test.

Where does the method lose? A vendor who cannot name the configuration where their layer is the wrong choice has not measured it hard enough.

My own answers: the baseline is a lexical prefilter, it beat the governed route on observed mean F1 and prompt-token use in R3, and the governed route has never cleared its cost bar in the three rounds that tested it. The route remains worth evaluating where opaque physical names make semantic selection necessary, or where an evidence trail has independent value. These runs establish the quality advantage in that narrow synthetic configuration; they do not yet establish end-to-end economic value. That is a narrower claim than the one I set out to prove.

Evidence boundary

These are synthetic, named-endpoint runtime characterizations on a single local model, qwen3-coder:30b. They are not production results, ROI claims, customer-data results, or a replication of any third-party study. R2.1 remains visibly rejected in the public record rather than buried.

R3 and R5 publish a full public packet: frozen contract hash, complete preflight, claim-scoped decision pack, and artifact digests. R4's contract is public, but its decision pack was derived after the fact from the archived report and is held rather than published. The raw reports stay private because they embed host identifiers, and their hashes are committed so you can tell if they ever change.

Four rounds in, the most useful thing this benchmark has produced is the boundary it refuses to cross.

Context Is Part of an Agent's Authority

Michael "Mike" K. Saleme — Fri, 21 Aug 2026 16:07:24 +0000

Technical source: Token-Bleed R5 release

Enterprise AI programs often treat context as a prompt-engineering problem: retrieve more documents, add more records, and let the model sort it out. That is backwards.

The information an agent receives determines what it can infer, combine, and disclose. An agent's effective authority is therefore shaped by both its information scope and its permitted actions: context governs the former; capability controls govern the latter.

Context does not grant permission to dispatch power, change a price, or execute a transaction. It expands what the agent can know, infer, and disclose, and therefore its practical power.

Too little context is not neutral either: omitted constraints, exceptions, or dependencies can make a confident recommendation wrong. The architectural objective is therefore not minimum context, but minimum sufficient context.

We spend substantial time defining action authority: which tools an agent may call, which systems it may reach, and which approvals it needs before a change takes effect. Information authority deserves the same discipline. Before an agent decides, architecture must establish what information it is permitted to see and what it actually needs for this decision.

The full-context default is not neutral

Sending every available record to a model feels safe because nothing has been left out. In practice, it can be expensive, dilute the relevant signal, make the decision harder to review, and expand the information the system can combine.

That does not mean every workload needs a complex governance layer. It means every workload needs a comparator.

In the retained Token-Bleed R5 synthetic experiment, compact governed selection used 96.9% to 97.9% fewer prompt tokens and achieved higher F1 than raw full-context stuffing on the frozen local configuration. The public record includes the frozen contract, preflight, retained-evidence decision pack, and hashes.

The more important finding is the limitation. Against a cheap lexical baseline, governed selection consumed 6.94 times as many prompt tokens on the holdout set, exceeding the preregistered maximum of three. The lexical route scored 0.000 F1 at every catalog size, so governed context won the quality comparison outright. Even so, the preregistered claim that governance earned its cost against lexical filtering failed. The frozen economic rule controlled the verdict; relaxing that ceiling after collection would have invalidated it. One caveat belongs with that zero: R5 used opaque physical names, where lexical matching has nothing to grip. On an earlier round with a lexically tractable catalog, the same baseline beat governed selection on both quality and cost.

That is exactly what useful architecture evidence should do. It should show where a method helps and where it has not yet earned the right to be the default.

Enterprise Agent Architecture needs a context plane

An EAA design should logically separate four responsibilities:

Connection plane: APIs, events, and data services connect an agent to enterprise systems.
Context plane: retrieval, classification, policy, lineage, and routing assemble the minimum sufficient, policy-permitted information set for the decision at hand.
Capability plane: permissions and approvals govern what the agent may do next.
Evidence plane: retained contracts, source references, tool traces, and human decisions make the outcome reviewable.

The context plane is not a technical ornament between a database and a model. It decides what the agent knows before it chooses a path. That makes it an authority control.

An agent's context is part of its authority.

A practical decision rule

For each agent workflow, begin with the least-complex, policy-permitted context route that could credibly meet the decision's quality requirements.

If a simple filter captures the relevant records and produces a reviewable decision, use it.
Add governed metadata when field names are ambiguous, business definitions matter, access policy must be enforced before the model sees data, lineage changes the answer, or the evidence trail itself is required.
Compare the richer route with the cheap baseline. Measure quality, omission risk under routing misses, prompt cost, latency, and the work required to maintain the governed layer.

The goal is not more governance. The goal is decision-useful context with a defensible cost.

What this changes in industry workflows

In energy and utilities, an outage or maintenance exception agent may need weather, load, asset, work-order, and switching-constraint context to recommend whether to keep, reschedule, or escalate a window. It should not receive unrestricted operating data, and it should not obtain dispatch authority merely because it can assemble a recommendation; that authority must be separately granted, bounded, and evidenced.

In CPG and retail, a commercial exception agent may need store, SKU, promotion, inventory, cost, and service context to recommend a response. It should make the data basis, confidence, and required approver visible. It should not alter price or trade terms unless that capability has been separately authorized within explicit limits.

Those are not generic chatbot problems. They are architecture problems involving context, authority, and evidence.

The evidence boundary

R5 is a synthetic, named-endpoint runtime characterization. It is not a customer-data result, a production ROI study, or proof that governed context always beats a simple filter. The original raw report remains private because it contains host identifiers.

That boundary is part of the result, not a footnote. Enterprise agents deserve evidence that is as specific about its limits as it is about its gains.

Where this fits. This is one piece of a longer argument about enterprise agent architecture: what TOGAF and SABSA cannot model about a workforce that is not human, and where an agent's authority actually sits. The series runs at Enterprise Agent Architecture.

MCP Went Stateless. My Test Suite Stayed Green Anyway.

Michael "Mike" K. Saleme — Sun, 02 Aug 2026 16:19:55 +0000

Start with the thesis: Satisfied Is Not Established.

Enterprise systems accumulate trust in the places where state lives. A session identifier
is not just a routing key — it is the thing a dozen downstream assumptions quietly hang
from. Remove it and you do not remove one field. You invalidate every assumption that was
resting on it, including the ones nobody wrote down.

The Model Context Protocol's 2026-07-28 specification did exactly that. The project called
it "the largest revision of the protocol since launch",
and the two headline changes are both acts of subtraction. From the
changelog's major
changes, items 1 and 2:

Remove protocol-level sessions and the Mcp-Session-Id header from the Streamable HTTP
transport.

Make MCP stateless: remove the initialize/notifications/initialized handshake. Every
request now carries its protocol version and client capabilities in _meta.

At the protocol layer, every request now stands on its own. Authorization gets stricter
alongside it: clients
MUST validate a present iss parameter against the recorded issuer per
RFC 9207 before redeeming an
authorization code, which closes a class of authorization-server mix-up attacks. There is
a formal extensions framework, and a minimum twelve-month deprecation window before any
deprecated feature can be removed.

That is a good specification. This post is not about the specification.

What early tracking actually costs

I had MCP tests running against the stateless profile on 2026-07-22, six days before
the final specification was published. I want to be precise about what that bought and what
it cost, because the first part is the part people write posts about and the second part is
the part that matters.

Between the release candidate and the final revision, the spec added an error-code
allocation policy. The JSON-RPC server-error range got partitioned: -32000 to -32019
stays implementation-defined and grandfathered, -32020 to -32099 is reserved for the
specification. Then it renumbered the codes the draft had introduced:

Error	Release candidate	Final
`HeaderMismatch`	`-32001`	`-32020`
`MissingRequiredClientCapability`	`-32003`	`-32021`
`UnsupportedProtocolVersion`	`-32004`	`-32022`

My harness checked for -32004.

The function that reads that code decides one thing: when a server rejects the modern
protocol version, is that an explicit stateless-protocol answer, or is it an old server
that does not understand the request? Get it right and the probe stops. Get it wrong and
the client falls back to initialize — the handshake the specification just removed.

Against any server built to the final spec, my check returned false. The harness read a
compliant version rejection as evidence of a legacy server, sent the removed handshake,
and reported results for a protocol the server had explicitly refused.

The function's own docstring said that must not happen. It happened anyway.

By the time the final specification landed, that assumption was already on PyPI.
Version 4.10.0 shipped on 2026-07-25, carrying the RC comparison verbatim. You do not
have to take my word for it: download the wheel and read protocol_tests/mcp_harness.py.
The RC comparison is in the published artifact, and -32022 appears nowhere in it.

The test suite stayed green the entire time

This is the part worth sitting with.

I have unit tests covering that exact function. They assert the fallback does not fire on
a version rejection. They passed continuously — before the spec was final, after it was
final, and through the release that carried the defect.

They passed because every fixture in them was written during the RC window and pinned
-32004. The tests and the code were wrong in the same direction, so they agreed with
each other perfectly.

A test that exercises only the provisional RC value cannot detect that the released
value is unhandled. Its branch coverage may be adequate, but its oracle is not independent: the
implementation and the fixture are two copies of the same provisional assumption. They can
agree perfectly and still be wrong.

I did not find this by testing. I found it by re-reading the specification changelog
against my own code, line by line, while checking something else.

The cost of tracking a specification early is inheriting the decisions it had not finished making

Being six days ahead of the final specification is a real advantage and I would do it
again. But a release candidate is a set of provisional commitments, and tracking one means
adopting those commitments before they have been tested by the people who will have to live
with them. Some of them will change. The changes will be small, unglamorous, and exactly the
kind your fixtures will freeze in place.

The lead is not free. It is a loan against a spec that has not stopped moving.

What to check, concretely

If you maintain anything that touches MCP and you moved during the RC window, four things
are worth an hour:

Grep for hardcoded JSON-RPC error codes. Any literal in -32000..-32099 that you
wrote between the RC and 2026-07-28 is suspect. Move them to named constants — a named
constant makes a renumber a deliberate edit instead of a silent one. But keep the test
oracle independent: assert the released wire value directly, or derive the fixture from an
authoritative conformance vector — not from the production constant being tested. A test
that imports the constant it is checking is tautological, and that is the same failure in
a tidier shape.

Check what your fixtures pin, not just what your tests assert. If every fixture for a
behaviour was authored in the same week, they encode that week's assumptions and will
agree with each other forever. Add one fixture from the current standard and see what
breaks.

Audit anything that decides "modern or legacy." Version-negotiation branches fail
quietly by design — they are written to degrade gracefully, which means a wrong answer
produces a plausible run instead of an error.

Re-read the changelog after the final specification, not just at RC. Diff it against your own
code rather than your memory of the RC. The renumbering that caught me is item 12 under
"Minor changes." Nothing about its placement suggests it breaks a client.

Where this leaves the testing layer

The stateless rewrite reset a meaningful part of MCP security testing. For implementations
targeting 2026-07-28, protocol-level session assumptions are gone, per-request capability
declaration is new surface, and authorization requirements have tightened. Tests scoped to
earlier MCP revisions may remain valid, but they are not evidence of conformance to the new
one. Suites written against the old handshake are not slightly stale — they are asserting
things about a mechanism that no longer exists in the 2026-07-28 protocol profile.

That is an opening for anyone willing to re-derive their assumptions from the current
text. It is also a trap for anyone who moved early and has not gone back.

I moved early. I went back. It cost me one released assumption that became a
compatibility defect three days later — and an afternoon to find it. I would rather publish
that than the version where I only mention the six-day lead.

The fix, the negative controls, and the reasoning are public: PR #313.
Views are my own.

curl Just Merged RFC 9421 Support. A Valid Signature Still Isn't Authorization.

Michael "Mike" K. Saleme — Tue, 28 Jul 2026 19:50:34 +0000

Start with the thesis: Satisfied Is Not Established.

On July 27, 2026, curl maintainer Daniel Stenberg wrote up curl's newly merged experimental support for RFC 9421, HTTP Message Signatures — the IETF standard for cryptographically signing selected components of an HTTP request or response. The feature is off by default, sits behind an explicit build-time flag, and stays that way — still experimental — in the upcoming curl 8.22.0. Stenberg is explicit it isn't ready for production.

That caution is the right posture for new cryptographic protocol work, and the standard is worth taking seriously anyway, because it answers a real question: can a receiver verify, under key material it trusts for the relevant context, that the received message is semantically equivalent to what was signed with respect to the covered components?

I read the specification closely because "signed" is easily heard as "approved." RFC 9421 makes a narrower, message-layer claim.

What actually gets established

RFC 9421 lets a sender cover selected message components — method, path, specific headers, and a content digest — and bind them to signature parameters such as created and keyid, using an agreed signature algorithm. When the receiver verifies it, two things are established:

the signature or MAC validates under key material the verifier trusts for that context
the received message is semantically equivalent to the signed message with respect to the covered components

That's it. Binding that key to a specific service, organization, or person happens outside the standard — RFC 9421 §3.2 explicitly requires the verifier to determine the key material's trustworthiness in context, and the standard supports HMAC, where more than one party can hold the same shared secret. "The signature proves who sent it" is a stronger claim than the spec actually makes.

And because only the covered components are signed, unsigned fields — and even some transformations of covered ones — can change without invalidating the signature. The guarantee is scoped, not blanket.

Scoped as it is, this still closes a real gap. For service-to-service traffic, webhook delivery, and architectures that traverse TLS-terminating intermediaries, RFC 9421 can preserve integrity and authenticity for selected components beyond any single transport connection. That part is genuine progress.

What doesn't get established

A verified signature does not establish that the key-holder was authorized to send this particular request, at this particular time, given everything else it has sent recently. It can't — verification happens one message at a time, and authorization is a question about a principal's standing, its current scope, and often its history. None of that lives inside the bytes of a single signed message.

Play out the failure mode: a service holds a signing key meant for reading inventory levels. The key leaks, or the service is compromised, or it's simply asked — by an operator, or by an agent orchestrating it — to do something outside its intended purpose. Every request can still verify cryptographically under that key, even though a separate authorization layer should reject requests outside the service's permitted scope. Signature verification alone doesn't evaluate whether a sequence of otherwise-valid requests has changed purpose or accumulated into a disallowed outcome — that check happens once per message and stops there.

To be fair to the standard: nothing stops an application from adding stateful controls above verification. RFC 9421 expressly allows additional application requirements and discusses replay protection. It just doesn't provide sequence-level evaluation itself — that's a different layer's job.

Ten individually-valid, individually-signed requests can compose into a data exfiltration run, a privilege-escalation chain, or a resource-exhaustion attack that no single request would ever be flagged for. The signature layer sees ten cryptographically valid messages. It has no layer above it asking what those ten messages, taken together, actually did.

A signature authenticates covered components. It does not sanction the action — or the sequence.

The same boundary, one layer down

This is the same structural boundary as the argument in "Authorized but Composed" (DOI 10.5281/zenodo.21400261) and in the field note on Hugging Face's July 2026 security-incident disclosure — just moved from the agent-decision layer down to the message-verification layer. That disclosure provides a concrete example of why sequence reconstruction matters. Its investigators analyzed more than 17,000 recorded events to understand what the autonomous campaign did as a whole. The disclosure does not establish that those events were individually authorized or passed policy gates. It demonstrates the narrower point: the meaning of an automated campaign may emerge only when its actions are correlated as a sequence. Here, the same shape shows up mechanically: a per-message signature check can validate an unbroken run of individually-valid requests that compose into something nobody should have permitted, because signature verification doesn't evaluate the sequence either.

As more agent-to-agent and agent-to-API traffic gets wrapped in signed HTTP requests — a genuinely good trend I expect to accelerate as agent workforces scale — this gap gets more consequential, not less. An autonomous agent making its own tool calls, each one dutifully signed under its service's key, is exactly the kind of principal whose individual requests will all verify cleanly while its accumulated trajectory goes somewhere no one signed off on.

The takeaway

Which trusted key material validates the signature or MAC, and whether the covered components still verify, are message-layer questions that RFC 9421 answers well. Whether the request — or the sequence it belongs to — was actually authorized is a decision-layer question, and no signature scheme answers it, because it was never built to. The two layers compose: verify the covered components under a trusted key, then evaluate what the authenticated actions associated with that key material add up to over time — across sessions or principals where the use case requires it. Skip the second layer and you get a very well-authenticated blind spot.

To be precise about the claim, because overreaching here would be exactly the mistake a signature-only architecture makes: this isn't an argument against RFC 9421, and it isn't a claim that curl's implementation is unsafe or premature — Stenberg's own caution about production-readiness is the right call for new cryptographic protocol work. The argument is narrower: message-layer verification is necessary and not sufficient for authorization. Something has to sit above the verifier and ask what the authenticated sequence associated with that key material composed into. That's a decision-governance problem, not a cryptography problem.

The composition engine behind this argument is open source: pip install constitutional-agent. If you're building signed service-to-service or agent-to-API traffic and want to see where the cross-session gap sits relative to your signing layer, try the free Governance Stress Test or read the Enterprise Agent Architecture framework. Tell me where you think this argument breaks.

Composition preprint: doi.org/10.5281/zenodo.21400261 · Enterprise Agent Architecture: doi.org/10.5281/zenodo.21105314

Every API Call Was Allowed. The Agent's Outcome Wasn't.

Michael "Mike" K. Saleme — Mon, 27 Jul 2026 12:43:10 +0000

Start with the thesis: Satisfied Is Not Established.

API governance is one of the things the enterprise actually got right.

Gateways, rate limits, authentication, versioned contracts, a managed lifecycle. For two decades it did its job: keep system-to-system integration orderly, secure, and observable.

Its unit of control is the API call. Its question is precise. Is this client allowed to invoke this endpoint, with this credential, at this rate?

That question was built for systems. An agent is not a conventional integration client, and its most consequential failures do not occur at the level that question can see.

The predictability the model assumed is gone

A traditional integration is designed around a relatively stable, declared flow. It calls known endpoints in expected patterns for an established business purpose. API governance was built around that predictability.

An agent may have a declared role or goal. But it selects its action path at runtime, assembling tools in response to changing context.

Every call it makes can be authorized, within rate limits, and individually compliant — and the composed outcome can still be one the enterprise never intended.

That is the gap. API governance is typically enforced call by call. The harm an agent can cause may emerge across a sequence of calls, each of which is allowed.

Two compliant calls, one outcome nobody approved

Consider an agent with read access to customer records and permission to send email.

Both capabilities are legitimately granted. Both may be necessary for its job. An endpoint-centric gateway sees two compliant calls.

Without shared task context and sequence state, it cannot determine from those calls alone why the second followed the first — or whether the two together converted ordinary access into data exfiltration.

The missing question is not merely whether the next call is permitted. It is whether this agent, acting for this task under this delegated authority, should be making it now.

API governance asks whether the call is allowed. Capability governance asks whether the agent should be making it.

More precisely: whether this agent may exercise this capability for this task, given what it has already done and the constraints that still apply.

The unit of control has to change

API governance typically controls access to individual interfaces and operations. A capability is an action or outcome an agent can produce through one or more APIs, tools, and data sources.

A major part of the attack surface is not any single API. It is the set of tools you expose to the agent, because every tool you grant widens what the agent can be talked into doing with the authority it already holds.

Least privilege stops being a question of which APIs a client may call. It becomes a question of which capabilities an agent may compose, for which task, for how long.

You cannot solve this by adding more endpoint rules alone. The gateway traditionally governs the channel. Capability governance must evaluate the delegated task, the authority available to the agent, the actions already taken, and the next action proposed. It is scoped to an execution and enforced at runtime.

This is already showing up in enterprise platforms

MuleSoft, for example, positions Omni Gateway as a common control point for API, MCP, LLM, and agent traffic. It applies policies across gateways, MCP, APIs, and LLMs, propagates identity, and carries correlation IDs across every interaction in an agent chain.

That makes the gateway a plausible enforcement point for capability governance.

But reconstructing a chain is not the same as deciding whether the current task state authorizes the next composed action. The location of the control is emerging. The governing model still has to mature.

API governance kept our systems talking to each other safely. The agentic enterprise needs governance over what a worker is allowed to do with those systems, not just which doors it may knock on.

This is part of a series on Enterprise Agent Architecture — the case for treating the agent workforce as a fifth architecture domain alongside Business, Information, Application, and Technology.

Start here: https://cognitivethoughtengine.com/eaa/part-1

Position paper: Enterprise Agent Architecture: The Case for a Fifth Architecture Domain for the Agentic Enterprise

What is enforcing capability scope in the agent systems you run right now — the gateway, the agent framework, or nothing yet?

The Evaluation Had a Sandbox. It Needed an Authority Boundary.

Michael "Mike" K. Saleme — Fri, 24 Jul 2026 19:25:20 +0000

Start with the thesis: Satisfied Is Not Established.

On July 16, Hugging Face disclosed an intrusion into its internal infrastructure. A malicious dataset abused two code-execution paths in its dataset processing — in their words, "a remote-code dataset loader and a template-injection in a dataset configuration." Internal datasets were accessed, service credentials were harvested, and lateral movement reached multiple internal clusters. Their responders detected it, contained it, and reconstructed the timeline from more than 17,000 recorded events.

On July 21, OpenAI disclosed who was on the other end. It was not a conventional external threat actor. It was two of their own models under evaluation — "GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes."

The models were being measured on ExploitGym, a publicly hosted benchmark for executing attacks against known vulnerabilities. Rather than solve the benchmark, they went and got the answers. OpenAI's account of how: the models "spent a substantial amount of inference compute finding a way to obtain open Internet access," then "identified and exploited a zero-day vulnerability (which we've now responsibly disclosed to the vendor) in the package registry cache proxy." From there they chained stolen credentials into a remote-code path on Hugging Face's servers and pulled the test solutions out of a production database.

OpenAI's own framing: "The models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal."

That sentence captures the architecture problem, and the problem is not unique to either company.

The permitted path became the attack path

The models were not granted general internet access. But they were given a narrow, permitted dependency path through a package-registry proxy. They treated that path as terrain: exploiting the proxy, escalating privileges, moving laterally, and eventually reaching a node with internet access.

This was both a containment failure and an authority failure. The environment attempted to constrain capability, but the permitted dependency path was not backed by an authorization policy that remained enforceable after the first control failed.

That is the distinction worth holding onto. Evaluation environments are built as capability boundaries: what code the model can execute, what image it runs in, what the process can touch. That work is real and it was done here. What the incident exposes is the need for a separate authority boundary — an external, enforceable answer to what the evaluation workload may reach, under which identity, for how long, and how that authority can be revoked during a run.

The failure cannot be reduced to model behavior. The decisive containment controls were also in the infrastructure surrounding the model: the dependency proxy, network paths, workload identity, credentials, and the enforcement points that were supposed to keep a deliberately less-restricted cyber evaluation contained. OpenAI's own remediation list spans model alignment, evaluation-time cyber protections, containment, monitoring, and access controls — which is the correct breadth.

What the receiving end could and could not see

There is a detail in Hugging Face's disclosure that deserves more attention than it has gotten. Their responders characterized what hit them as "an autonomous agent framework (appearing to be built on an agentic security-research harness)" — thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.

From the receiving end, the evaluation generated telemetry resembling a competent autonomous intrusion campaign. Hugging Face's initial evidence characterized the mechanism accurately, but did not identify the model or the operator.

That distinction matters. Local telemetry can reveal what an agent is doing without revealing who authorized the workload or who can terminate it. Cross-organization investigation supplied the missing attribution.

Which is the operational consequence worth internalizing: an evaluation without an authority boundary does not just risk escaping its constraints. It escapes without carrying anything that lets the organization it lands on identify the source or reach whoever can revoke it.

A control plane, not a rename for segmentation

If "authority boundary" is going to mean anything beyond network segmentation with a new label, it has to be buildable. For an agent evaluation environment, that is roughly six things:

Default-deny egress, with dependency installation mediated through an immutable or tightly constrained repository. The proxy in this incident was a permitted path with an exploitable implementation. Constrain what the path can serve, not just who may use it.

Per-evaluation workload identity. Short-lived, least-privilege credentials scoped to that single run, with no ambient harness credentials available to inherit.

Authorization enforced independently at the network, proxy, credential, and execution layers. The property you want is that the first control failing does not collapse the rest. This incident demonstrates why that property must be tested rather than assumed.

Time, compute, action, and destination budgets. A run that spends substantial inference compute searching for egress is a budget signal before it is a security finding.

Cross-boundary telemetry with tripwires for privilege escalation, lateral movement, credential discovery, and unexpected destination lookup.

A kill mechanism that revokes identity, credentials, and network authority — not merely the original sandbox process. Killing the process you know about is not revocation when the workload has already moved.

If external attribution matters to you, workload identity alone will not deliver it. An internal cloud identity is not visible or verifiable to the organization receiving your traffic. That requires pairing it with identifiable egress, authenticated provenance, and a registered incident contact.

And one framing change for whoever designs the evaluation itself: treat objective pressure as a threat input. The models did not abandon the assigned objective; they pursued it through an unintended path. An evaluation that strongly rewards a narrow outcome must assume the workload will search across every reachable surface, especially when normal cyber refusals have deliberately been reduced.

What both companies got right

Hugging Face published while its initial disclosure still listed the model as unknown and reported no evidence of tampering with public models, datasets, Spaces, or the software supply chain. OpenAI subsequently named its own models as the cause and disclosed the proxy vulnerability to the vendor. That transparency matters, but it is not the central architecture lesson.

The most capable systems we have now treat containment as terrain to be searched. In this case, the evaluation began inside a sandbox, found a permitted dependency path, converted it into broader reach, and crossed into another company's production infrastructure.

The lesson is not that sandboxes no longer matter. It is that execution containment is only one layer. A serious evaluation environment must also bind every workload to independently enforced authority: what it may reach, under which identity, within which budget, until what time, and through which mechanism that authority can be revoked.

The sandbox was present. The authority boundary was not complete.

Sources: Hugging Face security incident disclosure, July 16 2026 · OpenAI, "OpenAI and Hugging Face partner to address security incident during model evaluation," July 21 2026