DEV Community: Mehdi BOUTAYEB

How to read AI pentest benchmark claims: XBEN, 96 percent scores, and what they hide

Mehdi BOUTAYEB — Tue, 07 Jul 2026 20:30:42 +0000

Several autonomous pentest tools now advertise scores around 96 percent on the XBOW XBEN benchmark. A percentage with no context is marketing, not evidence. Here is how to read those claims like a practitioner.

What XBEN actually is

XBOW open sourced a set of 104 web challenges built around exploit validation rather than detection. That focus is healthy: proving exploitation is a much harder and more honest bar than flagging a potential issue. But a benchmark is only as meaningful as the conditions you run it under.

The variables that inflate a score

Hint free or hinted? Single attempt or unlimited retries? What is the false positive rate behind the headline number? A tool that scores high with hints and retries is not comparable to one that scores lower hint free on the first try.

The controlled to the wild gap

Recent research evaluating agents on real targets, not just labs, keeps finding the same thing: lab benchmarks overstate real capability. CTF style completion is not the same as validated discovery on a messy production system.

A transparency checklist

Before you trust any autonomous pentest claim, ask for a reproducible run, the proof of exploit artifacts, the false positive rate, and whether it was hint free. If those are missing, the number is a slide, not a result.

Try it

Reproducibility is the whole point. Run the benchmark yourself against whatever tool you are evaluating and compare honestly.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

The real bottleneck in AI pentesting is validated exploitation, not discovery

Mehdi BOUTAYEB — Tue, 07 Jul 2026 20:30:05 +0000

In 2026 the AI slop problem got impossible to ignore: bug bounty platforms paused programs, maintainers like curl walked away from bounties, and autonomous agents started flooding open source projects with low quality reports. The lesson is that discovery got cheap and proof got scarce.

Discovery is now the easy part

An LLM can generate a plausible looking finding in seconds. That is exactly why platforms are drowning. When the cost of producing a report drops to near zero, volume stops being a signal of value and starts being noise.

Proof of exploit is the scarce thing

The finding that matters is the one you can actually exploit, with the exact commands and raw output to back it. An autonomous tool that only reports what it can prove, and attaches the evidence, produces a short list a human can trust instead of a long list a human has to triage.

Responsible behaviour toward maintainers

Autonomous offensive tooling that scrubs public repos and files unsolicited reports is part of the problem, not the solution. Scope proof, deduplication, rate limiting and a human sign off should be defaults, not afterthoughts.

The design principle

Optimise for validated exploitation and an auditable evidence trail, not for the number of findings. That is the antidote to slop, and it is the only version of autonomous pentesting that survives contact with a real remediation team.

Try it

If you care about proof over volume, the methodology and evidence trail are the part worth scrutinising.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

Shannon and Darkmoon: two Claude powered pentesters, compared

Mehdi BOUTAYEB — Tue, 07 Jul 2026 20:26:40 +0000

Disclosure: I work on Darkmoon. Shannon (KeygraphHQ) is the closest technical twin, both lean on Claude, so a fair comparison is useful.

Shannon's strength

Shannon is a white box pentester that reads your source, identifies attack vectors and executes real exploits to prove them, and it scores very well on the XBOW benchmark. For source available web and API testing, it is excellent.

Darkmoon's focus

Darkmoon is black box and multi surface: it chains across web, cloud, AD and Kubernetes without needing source, orchestrating 80 plus tools via MCP with an evidence trail per finding.

Licensing and model

Shannon is AGPL-3.0, Darkmoon is GPL-3.0. Both are model agnostic in spirit and tuned around Claude Opus for planning stability.

Which to pick

If you have source and want white box proof, Shannon. If you need broad black box chaining across an internal estate, Darkmoon. Both are open, so you can try both.

Try it

Run both against a lab and judge for your scope.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

NodeZero is excellent, but it is SaaS: the self hosted open source path

Mehdi BOUTAYEB — Tue, 07 Jul 2026 20:25:59 +0000

Disclosure: I work on Darkmoon. NodeZero is a genuinely good autonomous pentest platform. This is for the teams that cannot use it because the targets cannot leave their environment.

Why SaaS is a blocker for some

Regulated, air gapped or data sensitive environments often cannot send targets or results to a vendor cloud. That rules out most hosted platforms regardless of quality.

What self hosted changes

A self hosted engine keeps scope, data and evidence inside your environment. You run it on your own model and infrastructure.

What you get with an open source MCP host

Darkmoon is GPL-3.0, model agnostic, and orchestrates 80 plus offensive tools via MCP across web, cloud, AD and Kubernetes, keeping a proof trail per finding.

Being fair

You give up the polish and support of a funded SaaS. You gain control, privacy and the ability to read and change the method.

Try it

If data residency is the constraint, a self hosted open source engine is worth a lab run.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

Open source autonomous pentesting: Darkmoon compared to Strix, PentAGI and Shannon

Mehdi BOUTAYEB — Tue, 07 Jul 2026 20:25:17 +0000

Disclosure: I work on Darkmoon, and I tried to make this a fair map of the open source field rather than a pitch. If you are choosing an open source AI pentester, here is how the main options actually differ.

Strix

usestrix/strix runs code dynamically and validates web and app vulnerabilities with real proof of concept exploits. It is strong and fast on the application layer. If your scope is mostly web and API, look at it seriously.

PentAGI and Shannon

PentAGI (vxcontrol) is a self hosted multi agent system with 20 plus built in tools. Shannon (KeygraphHQ) is a white box, Claude based pentester that reads source and proves exploitation, and it scores very well on the XBOW benchmark. Both are excellent at what they focus on.

Where Darkmoon is different

Darkmoon focuses on multi surface coverage (web, cloud, Active Directory, Kubernetes, API and internal networks) orchestrated through MCP, with the methodology written as Markdown playbooks you can read and fork, and an evidence trail on every finding.

How to choose honestly

For source aware web testing, Shannon and Strix are compelling. For broad multi host chaining across AD and Kubernetes with an auditable method, try Darkmoon. They are not mutually exclusive.

Try it

Read the playbooks and run whichever fits your scope against a lab. Feedback and corrections to this comparison are welcome.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

The open source AI pentest tools worth knowing in 2026

Mehdi BOUTAYEB — Sat, 04 Jul 2026 22:09:54 +0000

Disclosure: I work on Darkmoon, one entry in this list. This is a neutral survey of the open source AI pentest field, not a ranking, so you can pick what fits your scope.

The assistants

PentestGPT (GreyDGL) and Nebula (berylliumsec) put an LLM next to the operator for recon, reasoning and notes. Great for learning and augmenting a human.

The autonomous agents

Strix, PentAGI, Shannon and CAI push toward autonomy on the application or source layer, each with a different focus and license.

The MCP and multi surface tools

HexStrike exposes 150 plus tools over MCP. Darkmoon runs an MCP host across web, cloud, AD, Kubernetes and internal networks with playbooks and an evidence trail.

How to read the field

There is no single winner. Match the tool to your scope, your license needs and whether you want an assistant, an agent or an orchestrator.

Try it

Star and try the ones that fit your work. Corrections to this survey are welcome.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

Autonomous pentesting against Active Directory, without the black box

Mehdi BOUTAYEB — Sat, 04 Jul 2026 22:09:28 +0000

Active Directory is where most internal compromises happen and where most AI tools give up. Darkmoon runs the AD attack path autonomously and shows every step.

Why AD is hard for AI

AD attack paths are stateful and multi step. A single prompt cannot hold the whole graph. You need an agent loop that enumerates, reasons and pivots while keeping state.

Playbooks as the state engine

In Darkmoon the AD methodology is a Markdown playbook you can read and fork. The agent follows it, the proxy keeps the state, and every tool call is explicit.

From foothold to domain

The agent enumerates with BloodHound style logic, identifies attack paths, and executes them with real tools, attaching the output of each step.

Auditable by design

Because the method is a file and the execution is logged, a reviewer can follow exactly how the domain fell.

Try it

Run it on GOAD and read the generated attack path end to end.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

MCP for offensive security: orchestrating 80+ tools through an MCP host

Mehdi BOUTAYEB — Sat, 04 Jul 2026 22:09:01 +0000

Model Context Protocol is becoming the clean way for an agent to call tools explicitly. Here is how an MCP host drives real offensive tooling, and how Darkmoon and HexStrike approach it.

Why MCP fits offense

MCP makes every tool call explicit, scoped and logged, which is exactly what offensive work needs for auditability.

HexStrike's approach

0x4m4/hexstrike-ai exposes 150 plus tools to an agent as an MCP server. If you want a broad tool server to plug into your own agent, it is a strong option.

Darkmoon's approach

Darkmoon is an MCP host: it runs the agent loop and the methodology playbooks itself, and drives 80 plus tools through MCP across web, cloud, AD and Kubernetes, with an evidence trail per finding.

Server versus host

A server exposes tools. A host reasons and orchestrates. Depending on whether you are building your own agent or want a ready one, you want one or the other, or both.

Try it

If you are wiring MCP into offensive security, both approaches are open source and worth reading.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

Pentera alternatives in 2026, including the open source options

Mehdi BOUTAYEB — Sat, 04 Jul 2026 22:08:35 +0000

Disclosure: I work on Darkmoon, one of the tools below. Pentera is a mature, enterprise grade validation platform. If you are looking at alternatives, here is an honest map including the open source path.

What Pentera does well

Agentless, safe by design validation across internal and external surface, with enterprise support and maturity. If you have the budget and want a supported product, it is a strong choice.

The new autonomous wave

NodeZero, XBOW and RunSybil are well funded SaaS platforms that autonomously find and prove exploitable paths. They are fast and polished, and they run in the vendor cloud.

The open source option

If you cannot send targets to a vendor cloud, or you want to read and fork the methodology, a self hosted GPL tool like Darkmoon covers web, cloud, AD, Kubernetes and internal networks, orchestrating 80 plus tools via MCP with an evidence trail per finding.

Honest trade off

The SaaS platforms are more mature and hands off. The open source path gives you data residency, no per test fee and full auditability, in exchange for running it yourself.

Try it

If self hosted and auditable matters to you, try the open source path against a lab first.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

Why evidence matters more than model memory in AI pentesting

Mehdi BOUTAYEB — Sat, 04 Jul 2026 22:06:04 +0000

An AI finding you cannot reproduce is a liability, not a result. Darkmoon attaches the exact commands and raw tool output to every finding so a human can peer review it.

The trust problem

Most AI security tools return a confidence score and a paragraph. In offensive security that is not enough. If you cannot show the command that proved a vulnerability, you cannot defend it in a report or a remediation meeting.

What an evidence trail actually contains

For every finding Darkmoon keeps the executed command, the raw output, and the reasoning that connected them. The finding is a reproducible artifact, not a claim you have to take on faith.

Why this beats a bigger model

A larger model reduces some errors but never removes them. The evidence trail is what lets a human catch the ones that remain, which is exactly why we made it the core of the design rather than an afterthought.

How it changes the workflow

Reviewers stop re verifying everything by hand and start spot checking the trail. Reports write themselves from real data instead of paraphrased model output.

Try it

If you want to see the evidence trail on a live target, clone the Community Edition and point it at a lab.

Repo (GPLv3): https://github.com/ASCIT31/Dark-Moon
Docs: https://docs.dark-moon.org/
Demo: https://youtu.be/1bFRVuMkZzY

Built by pentesters, open sourced for pentesters. Feedback on the methodology and the evidence trail is genuinely welcome.

We build Darkmoon: An Open-Source Autonomous Pentesting Platform

Mehdi BOUTAYEB — Thu, 18 Jun 2026 13:20:08 +0000

Darkmoon is an open-source autonomous penetration testing platform released under GPLv3.

Before talking about AI, agents or dashboards, it's important to understand that the project was built around a transparent and auditable open-source core.

Over the last few years, we've tested a lot of security products.

Some were excellent scanners.

Some generated decent reports.

Some experimented with AI-assisted workflows.

But when we tried to use them in the environments we actually encounter during professional engagements, the limitations quickly became obvious.

A real assessment rarely consists of a single web application.

More often, the target is a combination of:

Active Directory
Kubernetes
Cloud infrastructure
APIs
CMS platforms
Internal networks
Legacy systems

That's where we spend most of our time as pentesters.

And that's what eventually led us to build Darkmoon.

Why We Built Darkmoon

Darkmoon didn't begin as an AI project.

It started with a practical question:

Could we automate parts of a penetration test without reducing everything to a vulnerability scanner?

Most security tools are already very good at collecting information.

The difficult part is deciding what to do next.

An experienced pentester constantly adapts based on new information.

They change direction.

They prioritize attack paths.

They abandon dead ends.

They focus on technologies that matter.

We wanted to explore whether some of that decision-making process could be reproduced while keeping humans in control of the assessment.

Open Source First

One decision was made very early.

Darkmoon had to be open source.

Security teams already have enough black boxes to deal with.

If an autonomous system is going to participate in security assessments, practitioners should be able to understand what it is doing and why.

That's why Darkmoon is released under GPLv3.

The source code is public.

The methodologies are public.

The orchestration logic is public.

The deployment process is public.

If you disagree with how an assessment is performed, you can inspect the methodology and modify it.

For us, transparency matters more than any AI feature.

Today the project includes:

18 specialized methodology agents
80+ integrated offensive security tools
Active Directory workflows
Kubernetes workflows
Cloud infrastructure assessments
API security testing
CMS security testing
Infrastructure mapping
Evidence collection
Automated reporting

GitHub:
https://github.com/ASCIT31/Dark-Moon

Documentation:
https://docs.dark-moon.org/

Methodologies Instead of Hidden Prompts

One of the most common questions we receive is:

How do the agents work?

The answer is deliberately simple.

They're Markdown files.

Each methodology describes how Darkmoon approaches a specific technology stack.

They're reviewable.

Version-controlled.

Customizable.

And visible.

Unlike many AI-powered security products, the reasoning process isn't hidden behind proprietary prompts.

A WordPress assessment doesn't follow the same methodology as an Active Directory engagement.

A Kubernetes cluster doesn't trigger the same workflow as a GraphQL API.

The methodologies can be inspected before running an assessment.

For us, that's a critical requirement.

Understanding the Target

Darkmoon continuously builds an internal model of the environment it is assessing.

Instead of blindly launching tools, it gathers technical signals from the target:

Services
Technologies
Frameworks
CMS platforms
APIs
Infrastructure components

Those signals are then used to determine which methodologies should be activated.

Different technologies trigger different workflows.

A WordPress deployment doesn't require the same approach as:

Active Directory
Kubernetes
GraphQL
ASP.NET
Spring Boot
Cloud infrastructure

The objective is not to run every tool against every target.

The objective is to select the most relevant methodology based on what is actually discovered.

Building Around Existing Security Tools

We didn't try to reinvent twenty years of offensive security tooling.

Darkmoon relies heavily on existing projects such as:

Nuclei
SQLMap
NetExec
BloodHound
Impacket
FFUF
Hydra
Kubescape
Kubeletctl
WPScan
CMSeeK
Naabu
Masscan

and many others.

The interesting part isn't the tools themselves.

The interesting part is deciding:

Which tool should run
When it should run
Why it should run
How the results should influence the next step

That's where most of our work has gone.

Community Edition and Professional Edition

Darkmoon is built around an open-source GPLv3 core.

The Community Edition contains:

The autonomous engine
Methodologies
Orchestration logic
Docker deployment
Assessment workflows
Reporting capabilities

For organizations running larger engagements, we also maintain a Professional Edition.

The Professional Edition focuses on operational workflows rather than changing the philosophy of the project.

It extends the open-source foundation with:

Campaign management
Infrastructure visualization
Centralized dashboards
Enhanced reporting
Operational monitoring
Team-oriented workflows
Vulnerability analytics
Historical campaign tracking

The screenshots below come from the Professional Edition.

The goal is not to replace the open-source edition.

The goal is to provide operational capabilities for teams running Darkmoon at scale.

Evidence Matters

One of our biggest frustrations with automated security products is the lack of context.

A finding without evidence is difficult to trust.

Darkmoon attempts to preserve:

Commands executed
Outputs collected
Supporting evidence
Attack paths
Severity information

The objective is not simply to produce findings.

The objective is to produce findings that can be understood, validated and acted upon.

Deployment

Installation is intentionally simple.

git clone https://github.com/ASCIT31/Dark-Moon.git
cd Dark-Moon

chmod +x install.sh darkmoon.sh

./install.sh

Once installed:

./darkmoon.sh

Or directly against a target:

./darkmoon.sh "TARGET: https://target.example"

Docker handles the environment and dependencies.

Bring Your Own Model

Darkmoon does not lock users into a single provider.

Supported options include:

OpenAI
Anthropic
OpenRouter
Ollama
llama.cpp
OpenAI-compatible endpoints

Organizations can choose between:

Maximum performance
Maximum privacy
Fully local deployments

What Still Needs Work

Darkmoon is far from finished.

Today:

Web application assessments are among the most mature workflows
Active Directory coverage is highly advanced
Cloud assessments are improving rapidly
Smaller local models still struggle compared to frontier models on long reasoning loops

We're continuously improving:

Detection quality
Methodologies
Reporting
False positive reduction
Cloud coverage

The project is evolving rapidly, but we're still early.

Looking Ahead

The long-term vision is not to build another scanner.

It's not even to build another AI wrapper.

What interests us is creating systems that understand:

Infrastructure
Security methodology
Attack paths
Offensive operations
Remediation workflows

in a way that is genuinely useful to practitioners.

Today, Darkmoon orchestrates methodologies, tools and workflows.

Tomorrow, we want to push even further into cybersecurity-specific reasoning systems.

Darkmoon is our attempt at exploring that direction in public, in the open, and alongside the community.

DEV Community: Mehdi BOUTAYEB

How to read AI pentest benchmark claims: XBEN, 96 percent scores, and what they hide

What XBEN actually is

The variables that inflate a score

The controlled to the wild gap

A transparency checklist

Try it

The real bottleneck in AI pentesting is validated exploitation, not discovery

Discovery is now the easy part

Proof of exploit is the scarce thing

Responsible behaviour toward maintainers

The design principle

Try it

Shannon and Darkmoon: two Claude powered pentesters, compared

Shannon's strength

Darkmoon's focus

Licensing and model

Which to pick

Try it

NodeZero is excellent, but it is SaaS: the self hosted open source path

Why SaaS is a blocker for some

What self hosted changes

What you get with an open source MCP host

Being fair

Try it

Open source autonomous pentesting: Darkmoon compared to Strix, PentAGI and Shannon

Strix

PentAGI and Shannon

Where Darkmoon is different

How to choose honestly

Try it

The open source AI pentest tools worth knowing in 2026

The assistants

The autonomous agents

The MCP and multi surface tools

How to read the field

Try it

Autonomous pentesting against Active Directory, without the black box

Why AD is hard for AI

Playbooks as the state engine

From foothold to domain

Auditable by design

Try it

MCP for offensive security: orchestrating 80+ tools through an MCP host

Why MCP fits offense

HexStrike's approach

Darkmoon's approach

Server versus host

Try it

Pentera alternatives in 2026, including the open source options

What Pentera does well

The new autonomous wave

The open source option

Honest trade off

Try it

Why evidence matters more than model memory in AI pentesting

The trust problem

What an evidence trail actually contains

Why this beats a bigger model

How it changes the workflow

Try it

We build Darkmoon: An Open-Source Autonomous Pentesting Platform

Why We Built Darkmoon

Open Source First

Methodologies Instead of Hidden Prompts

Understanding the Target

Building Around Existing Security Tools

Community Edition and Professional Edition

Evidence Matters

Deployment

Bring Your Own Model

What Still Needs Work

Looking Ahead

Links