How I benchmark an agent guardrail without lying to myself

It would be easy to write a benchmark for Doberman that makes Doberman look good: pick the attacks it already stops, skip the ones it doesn't, round up. I've tried to build the opposite of that - a methodology that reports the numbers whether they're flattering or not, and a results page (docs/BENCHMARKS.md) that lists what it missed before what it caught.

The metrics, and why there are three of them

Doberman is scored as a decision function over tool-call cases, not by driving a live LLM agent. Each case - a tool call or an ordered sequence - is labeled benign or attack, replayed through the real decision engine, and the verdict (PASS/AUTH/BLOCK) is recorded. Deterministic, no model called, reproducible from a cold clone.

Three numbers matter, and none of them alone tells the truth:

• ASR (attack success rate) - the fraction of attacks that reach the tool. A hard BLOCK counts as stopped, but AUTH also counts as "not silently succeeded" in plain ASR, which flatters the score.
• asr_strict - the same measurement, but only a hard BLOCK counts as stopped. An AUTH is not a block. A human still has to say no.
• FPR (false-positive rate) - the fraction of benign cases that get friction (AUTH or BLOCK). A guardrail that blocks everything has a perfect ASR and a useless FPR.

Report ASR alone and you can make a leaky guardrail look airtight, because every AUTH counts as a win. That's why every result in the docs reports both numbers side by side, plus operator metrics (effective_asr_deny/_approve, asr_under_fatigue) that model what happens if the human on the other end of an AUTH prompt just rubber-stamps it.

The failure case, published first

On the synthetic suite (3 attacks, 3 benign - a deterministic CI smoke gate, not a coverage claim), Doberman converts every attack from a silent bypass into an AUTH: asr drops from 1.00 to 0.00. asr_strict stays at 1.00, because none of the three get a hard BLOCK - all three route to a human decision. If that human approves, effective_asr_approve is 1.00: nothing was stopped. If they deny, effective_asr_deny is 0.00: everything was stopped. A human who approves is not protected. AUTH is a leash, not a wall, and I'd rather that be the first thing you read about the synthetic suite than the last.

The labeled corpus

The synthetic suite is a smoke test. The real per-category picture comes from a 158-row labeled fixture (tests/corpus/detection_corpus.jsonl) across seven categories: injection, exfiltration, secrets, destructive, encoded, dependency, and benign. Run it with python -m tests.benchmarks.run --suite corpus --corpus, and it reports TPR (AUTH or BLOCK counts as mitigation), tpr_strict (BLOCK only), FPR, and precision, per category.

The published run (2026-09-02: 125 attack / 33 benign rows) isn't filtered to cases the engine wins. Natural-language injection with no matching command shape scores TPR 0.00, flat, documented as a structural gap: the objective deterministic layer can't see intent, only shape, and it says so instead of hiding the category. Exfiltration is mode-gated: balanced mode deliberately lets a bare unknown host through (TPR 0.375) while strict mode steps it up (1.00). Secrets score TPR 1.00 but tpr_strict only 0.43, meaning most of that mitigation is still AUTH, not BLOCK. Overall, balanced mode lands at TPR 0.77 but tpr_strict 0.08 (strict mode: 0.81 and 0.36), which is the same "AUTH is a leash" honesty at corpus scale: almost all of the real mitigation on this fixture is a human being asked, not a hard stop. Benign FPR across the whole corpus is 0.00, and precision is 1.00, so the objective layer isn't buying that TPR by hassling legitimate traffic - but I'd rather you read the low tpr_strict than stop at the high TPR.

Every attack row's floor is calibrated to what the engine actually does today, not an aspiration, and CI gates on any row dropping below its documented floor. A generator that refuses to let a shipped floor quietly get worse is doing the same "raise-only" thing the runtime does, just for the test suite.

Poisoning, across sessions, not just within one

Static ASR says nothing about a patient attacker teaching the adaptive layer to trust something dangerous one small step at a time, spread across separate sessions so no single session looks abnormal. There's a dedicated eval for exactly that: it simulates a proxy restart between sessions (so the in-memory drift detector re-warms cold, exactly like production), and measures the admitted poisoning rate - the fraction of dangerous targets an attacker can normalize using only actions the real engine actually let through with a PASS. No cheating by injecting pre-approved actions.

The admitted poisoning rate is 0. The reason is structural: teaching the baseline to trust a dangerous action requires allowed observations of that exact action, and those are precisely the observations the engine steps up and refuses to silently learn from. The lethal-trifecta floor underneath it is score-independent, so even a worst_case attacker who's already defeated the approval gate and gets everything learned can't flip the floor's verdict by grinding the score down.

Try it on your own corpus

None of this is a leaderboard. There's no competitor comparison in the docs, on purpose. If you've got a labeled corpus of your own - attack shapes specific to your stack, your MCP tools, your CI pipeline - the harness runs on any labeled tool-call fixture, not just the one shipped in the repo. I'd like to see where it holds and, more usefully, exactly where it doesn't.

Full methodology and the raw tables: docs/BENCHMARKS.md in https://github.com/DobermanCore/Doberman-Core