Several autonomous pentest tools now advertise scores around 96 percent on the XBOW XBEN benchmark. A percentage with no context is marketing, not evidence. Here is how to read those claims like a practitioner.
What XBEN actually is
XBOW open sourced a set of 104 web challenges built around exploit validation rather than detection. That focus is healthy: proving exploitation is a much harder and more honest bar than flagging a potential issue. But a benchmark is only as meaningful as the conditions you run it under.
The variables that inflate a score
Hint free or hinted? Single attempt or unlimited retries? What is the false positive rate behind the headline number? A tool that scores high with hints and retries is not comparable to one that scores lower hint free on the first try.
The controlled to the wild gap
Recent research evaluating agents on real targets, not just labs, keeps finding the same thing: lab benchmarks overstate real capability. CTF style completion is not the same as validated discovery on a messy production system.
A transparency checklist
Before you trust any autonomous pentest claim, ask for a reproducible run, the proof of exploit artifacts, the false positive rate, and whether it was hint free. If those are missing, the number is a slide, not a result.
Try it
Reproducibility is the whole point. Run the benchmark yourself against whatever tool you are evaluating and compare honestly.
- Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
- Docs: https://docs.dark-moon.org/
- Demo: https://youtu.be/1bFRVuMkZzY
Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.
Top comments (0)