Full disclosure up front: I work on Darkmoon, one of the tools below. I have tried to keep this a fair teardown, not an ad. Every number is reproducible.
The setup
I wanted one honest answer: point an autonomous AI pentester at a target it has never seen, black-box, like a real engagement. What does it actually find? Target: OWASP Juice Shop, the standard public vulnerable app (docker run -p 3000:3000 bkimminich/juice-shop).
The twist I care about: most AI pentesters ship your traffic and code to a cloud LLM. For anything near production or a client environment, that is a non-starter. So the real question is not just "does it find bugs". It is "does it find bugs without exfiltrating your data".
What happened
Running black-box against Juice Shop on a local model (Ollama/llama.cpp), Darkmoon returned:
| Metric | Result |
|---|---|
| Vulnerabilities | 57 (8 critical, 24 high, 21 medium, 4 low) |
| Time | 28.5 min |
| Proof-of-exploitation | per finding |
| Data sent to a third party | none (local model) |
And it compounds across runs on the same target: 14, then 28, 38, 44, 49, then 57.
Raw output and methodology so you can reproduce (or contest) it: https://github.com/ASCIT31/Darkmoon-Benchmarks. PRs with other tools' runs are welcome, this is meant to be a community leaderboard.
Why "local" is the whole point
The impressive numbers you see elsewhere (for example white-box benchmarks in the 90s%) are real but come with conditions: they read your source and call a cloud model. That is a different trade-off. A local, black-box run is the one you can actually run against a sensitive target without a data-egress conversation.
Honest limitations
One lab is a starting point, not the whole story. Black-box favors breadth; white-box tools optimize a different axis. Numbers move as tools improve, which is exactly why the benchmark repo is a living, reproducible thing rather than a screenshot.
Try it
git clone https://github.com/ASCIT31/Dark-Moon && cd Dark-Moon && ./install.sh. GPL-3.0, self-hosted, runs on a local or cloud LLM. If it saves you a manual pass, a star helps other defenders find it.
What black-box results do you get on Juice Shop? Drop them in the comments or send a PR to the benchmark.
Top comments (0)