Every company building AI agent security publishes a detection rate. We were doing it. The problem is that none of these numbers are checkable. You see "99% detection" or "blocks 100% of prompt injection," and there is no test set, no failure list, no methodology, nothing to run. You are asked to trust it.
I don't think trust is the right thing to ask for in security. So I published our benchmark, with the parts that don't flatter us left in. It's here: https://github.com/AndrewSispoidis/contemporary-agent-attacks
Here is what is in it. 497 attacks and 1,172 benign samples, 1,669 total, across 22 categories. The detection engine scores 99.8% on the attacks at a 0.09% false positive rate. The license is CC BY 4.0. The methodology and scoring code are in the repo. You can clone it and reproduce every number.
The part I care about more is the failures. The benchmark names them. There is one attack the engine misses, a pretext opener that never actually asks for anything it shouldn't, so there is nothing to catch at the point it is measured. There is one benign sample it flags, a Stack Overflow question about Go method-receiver syntax that has the shape of a probe without being one. Both are in the repo by name. If you run it, you will hit exactly those two and no surprises.
Here is why I think this matters for the category as a whole. When detection numbers can't be reproduced, they stop carrying information. A vendor can claim anything, and the buyer has no way to check, so the decision comes down to who markets hardest. That is a bad way to choose a security control. A benchmark you can run, that is honest about where it breaks, gives you something real to evaluate.
I would rather compete on numbers people can verify than on numbers people have to believe. If you work on agent security, or you are trying to evaluate it, clone the benchmark and tell me where it is wrong. The failures are where the useful conversation is.
Top comments (2)
Leaving the missed pretext opener in the repo is the useful bit. A lot of agent-security evals quietly grade only the prompt-injection shape they already know how to catch, then report one clean number. I’d be curious how this changes if v2 keeps the current corpus frozen and adds a dated holdout from new attacks after publication.
Publishing the failures and scoring code is exactly the right direction. The next number I’d want is performance against material the engine could not have been tuned against.
The 497 attacks count as 497 independent observations only if they are genuinely independent. If many are templated or paraphrased variants, the effective sample size can be much smaller and the aggregate 99.8% rate can hide a weak attack family.
I’d freeze and hash this corpus as v1, then add a temporal holdout and leave-one-attack-family-out evaluation. Report per-family recall with confidence intervals and the full threshold tradeoff, not only one selected operating point. A hidden external set or third-party submissions collected after the engine artifact is frozen would provide an even stronger check against benchmark overfitting.
The reproducibility chain should bind the result to the exact engine artifact, configuration, corpus digest, and scoring version. Then a future model or rule change can be compared without silently changing both the detector and the exam.