Disclosure: Darkmoon is my project. But this is a reproducible benchmark, not a ranking I get to fudge. Run it yourself.
The question
Every AI pentester ships a headline number under different conditions. So I fixed the conditions: same public lab (OWASP Juice Shop, default docker image), black-box, method published.
The result so far
Darkmoon, black-box, on a local LLM: 57 findings (8 critical, 24 high, 21 medium, 4 low) in 28.5 minutes, proof of exploitation per finding. It compounds across runs: 14, 28, 38, 44, 49, 57.
The other tools (strix, shannon, PentAGI, PentestGPT, CAI) sit on a "run pending" row. That is the honest part: I am not going to invent their numbers. It is a community leaderboard, run your favorite tool against the same image and send a PR.
Why black-box on a public lab
Because it is the only setup anyone can verify. White-box numbers on a self-cleaned validation set are impressive but not comparable. A public image plus a documented method is falsifiable, which is the whole point of a benchmark.
Reproduce or contest
docker run -p 3000:3000 bkimminich/juice-shop, then run the tool. Full method and raw output: https://github.com/ASCIT31/Darkmoon-Benchmarks
If your tool beats 57 black-box on Juice Shop, I want that PR. If it does not, that is useful data too.
Top comments (0)