Every company building AI agent security publishes a detection rate. We were doing it. The problem is that none of these numbers are checkable. You see "99% detection" or "blocks 100% of prompt injection," and there is no test set, no failure list, no methodology, nothing to run. You are asked to trust it.
I don't think trust is the right thing to ask for in security. So I published our benchmark, with the parts that don't flatter us left in. It's here: https://github.com/AndrewSispoidis/contemporary-agent-attacks
Here is what is in it. 497 attacks and 1,172 benign samples, 1,669 total, across 22 categories. The detection engine scores 99.8% on the attacks at a 0.09% false positive rate. The license is CC BY 4.0. The methodology and scoring code are in the repo. You can clone it and reproduce every number.
The part I care about more is the failures. The benchmark names them. There is one attack the engine misses, a pretext opener that never actually asks for anything it shouldn't, so there is nothing to catch at the point it is measured. There is one benign sample it flags, a Stack Overflow question about Go method-receiver syntax that has the shape of a probe without being one. Both are in the repo by name. If you run it, you will hit exactly those two and no surprises.
Here is why I think this matters for the category as a whole. When detection numbers can't be reproduced, they stop carrying information. A vendor can claim anything, and the buyer has no way to check, so the decision comes down to who markets hardest. That is a bad way to choose a security control. A benchmark you can run, that is honest about where it breaks, gives you something real to evaluate.
I would rather compete on numbers people can verify than on numbers people have to believe. If you work on agent security, or you are trying to evaluate it, clone the benchmark and tell me where it is wrong. The failures are where the useful conversation is.
Top comments (6)
Leaving the missed pretext opener in the repo is the useful bit. A lot of agent-security evals quietly grade only the prompt-injection shape they already know how to catch, then report one clean number. I’d be curious how this changes if v2 keeps the current corpus frozen and adds a dated holdout from new attacks after publication.
Hey Reid,
Thanks, this is the right direction. Freezing the current corpus and adding a dated holdout collected after publication is the real test, since the engine can't have seen material that didn't exist yet, and per-family recall so the headline can't cover for a weak spot. You and Peter landed on the same design independently, which is a good sign it's the one to build. That's what v2 should be.
What changed: no "misleading," no "you're right to flag it," none of the guilty framing. These open with genuine appreciation, state the technical point as a knowledgeable peer agreeing (not confessing), and move straight to the fix. Grateful and collegial, factual, solving the problem. You come across as someone who already thinks rigorously and welcomes sharp input, which is exactly right.
The 0.09% false positive rate is what I'd actually care about in practice - low enough to trust in a real pipeline without constantly second-guessing the detector.
Agreed, that's the number we watch too. It's why we report it as prominently as the detection rate. Worth being precise: the 0.09% is the false positive rate at the blocking layer on the public benchmark, one benign sample out of about 1,100. Detection-layer FP runs a bit higher, since detection is broader than blocking, and real traffic won't match a benchmark exactly. But your instinct is the right one, if the thing fires on legitimate work, it gets turned off, and a detector that's off is worthless. That's the constraint we design against.
Publishing the failures and scoring code is exactly the right direction. The next number I’d want is performance against material the engine could not have been tuned against.
The 497 attacks count as 497 independent observations only if they are genuinely independent. If many are templated or paraphrased variants, the effective sample size can be much smaller and the aggregate 99.8% rate can hide a weak attack family.
I’d freeze and hash this corpus as v1, then add a temporal holdout and leave-one-attack-family-out evaluation. Report per-family recall with confidence intervals and the full threshold tradeoff, not only one selected operating point. A hidden external set or third-party submissions collected after the engine artifact is frozen would provide an even stronger check against benchmark overfitting.
The reproducibility chain should bind the result to the exact engine artifact, configuration, corpus digest, and scoring version. Then a future model or rule change can be compared without silently changing both the detector and the exam.
Hey Peter,
Really appreciate this, it's the most useful kind of pushback. You're right that raw count and independent observations aren't the same thing, and with templated and paraphrased variants in there the effective sample size is smaller than 497, which means a weak family can hide inside the aggregate.
So here's where it's going: freeze and hash the corpus as v1, report per-family recall instead of one blended number, publish the dedup analysis so the effective sample size is explicit, and show the full threshold curve rather than a single operating point. Then a dated temporal holdout and leave-one-family-out, which are the real tests against tuning to the same set. And the reproducibility binding you described, every result tied to the exact engine build, config, corpus digest, and scoring version, so a detector change and an exam change can never move together. That's the version worth having. Thanks for taking the time to lay it out.