Most security test suites answer one question: did the attack succeed? The
verdict then falls out of a search for evidence that it did.
passed = not leaked
passed = not granted_admin
passed = len(unsafe) == 0
Every one of those is true when the attack was blocked. Every one of them is
also true when nothing happened at all.
That second case is not hypothetical. It is what a suite reports when the target
was unreachable, when the endpoint does not implement the feature under test,
when the tool list came back empty, and when the model complied politely in
language the detector does not recognize.
Three sentinels, not one target
The fix is not a better detector. It is asking the same suite the same question
across three deliberately different response conditions.
a closed port never answers
a permissive stub returns 200 and explicit grant markers to everything
a refusing stub returns 403 and an error envelope to everything
These are diagnostic sentinels, not conformant implementations. A 200 carrying
{"granted": true} is not an MCP or payment-protocol grant, and a blanket 403
may block discovery rather than demonstrate enforcement. Their purpose is
narrower and useful: to expose verdicts that ignore or misread target behavior.
Point the suite at all three and read what changes.
A target-dependent test that passes against the closed port is claiming a
control held when nothing was contacted. One that passes against the permissive
sentinel despite explicit grant markers deserves inspection. And a test that
reaches the protected operation but cannot recognize an explicit denial may not
recognize the outcome it exists to measure.
For a target-dependent control, an unchanged verdict across all three
sentinels is evidence that the verdict may not be measuring target behavior.
The qualifier matters. Local controls, parser checks, and over-refusal tests can
legitimately return the same verdict everywhere.
What it found
I ran this against my own agent security harness, which is public, and which I
had believed was in good shape.
The closed port went first. Eighty-eight verdicts passed against a host that was
not running, across ten modules. The largest single count was a payment protocol
harness reporting forty-four controls holding against nothing at all.
The permissive sentinel went next, and it is the harder one because the target
answers. Every serviced-request guard I had just written was satisfied and
correctly stayed out of the way. Three hundred and nine verdicts out of five
hundred and thirty-two passed against a server returning HTTP 200 and explicit
grant markers to every request:
{"granted": true, "admin": true, "scope": "operator.admin"}
One of them read "Elevated scope claims not honored." Another read "All
gate-disable attempts were rejected." A third reported an incident-detection
latency of 0.000 seconds, measured from a connection being refused.
Not all three hundred and nine are defects, and I will come back to that.
The refusing sentinel went last and it found the inverse. Sixteen suites
produced no passing verdict against the refusing sentinel.
Put the second and third together and you get the sharpest diagnostic result.
One identity and authorization suite scored seven of eighteen against the
permissive sentinel and one of eighteen against the refusing sentinel. Its
verdict moved in the opposite direction from the behavior it was intended to
recognize, and neither sentinel alone would have exposed that inversion as
clearly.
The part I did not expect
None of the three found the most interesting defect.
An external reviewer reading the source found it instead. One test serialized
the whole transport-error envelope into a substring detector whose keyword list
contained "refuse". The string it fed in was
<urlopen error [Errno 111] Connection refused>
so the agent was credited with refusing a prompt injection it never received.
Not an absence read as a pass. A positive match on the wrong text.
Then a run against a real MCP server found a second one. The protocol has two
ways to report a rejection, and my harness read only one of them, so a reference
server that correctly refused an unregistered tool call was recorded as having
failed to refuse it.
These three sentinels could not isolate that defect. They all spoke the response
idiom my harness already expected, because I had written both the harness and
the sentinels. A fixture built to have the property under test cannot tell you
the property is the wrong one.
What this is worth to you
The method transfers. It does not depend on my harness, my protocols, or my
threat model.
If you maintain a security test suite, a scanner, a conformance checker, or an
eval, you can build all three sentinels in an afternoon. They are a closed port,
a web server that returns 200 to everything, and one that returns 403 to
everything.
Three things worth knowing before you do.
The counts are a reading list, not a defect count. This is why I did not call
those three hundred and nine defects. Some passes against the permissive sentinel
are correct. Twenty-five of mine were over-refusal checks whose expected outcome
is permissive behavior. Other tests require protocol-specific markers the generic
sentinel does not emit. Only reading the test separates those cases from a real
inversion.
Zero verdicts is not zero defects. One module in mine aborts cleanly when it
cannot handshake, which is the right behavior and which rendered as 0/0,
indistinguishable from a clean run. It now prints "produced no verdicts, nothing
measured, not clean."
And the sentinels are the floor, not the ceiling. Get to a real implementation
as soon as they stop finding things. Source review and a half-hour test against
one real MCP implementation exposed two additional defect classes the synthetic
sentinels structurally missed.
Inspect and rerun it
The repaired harness and all three sweep scripts are pinned at commit
e151566550ecdb29b0230d40fb371fdf03bbfe09
of msaleme/red-team-blue-team-agent-fabric,
released as v4.16.0:
python3 scripts/dead_host_sweep.py
python3 scripts/permissive_host_sweep.py
python3 scripts/refusing_host_sweep.py
These commands rerun the method against the repaired harness. They will not
reproduce the original pre-repair counts, because the repairs are in. The
original findings are preserved in the earlier state files and in
PR #417
through
PR #432;
the provenance record
identifies the exact source of every figure reported here.
I am not claiming any of this makes an agent secure. It establishes something
narrower and, I think, more useful: what a harness claims when it has nothing to
go on.
Top comments (0)