We built an LLM that runs real pentests. The hard part was stopping it from lying

We built an autonomous pentester on top of a large language model. It runs recon, picks and drives real offensive tools, exploits what it finds, and writes the report. The engineering that actually mattered was not teaching a model to hack. Models will happily "hack." The hard part was stopping it from telling us it found things that were not there.

This is the build story of Nemesis Red, and specifically the problem everyone building LLM security tooling smashes into: hallucinated findings.

Why most LLM pentest demos are demoware

Point a model at a target, ask it to find vulnerabilities, and it will hand you a confident, well-formatted list. Some of it is real. Some of it is invented. From the text alone you cannot tell which, and in security a false positive is not a rounding error. It burns an analyst's afternoon, it erodes trust in the tool, and at scale it makes the output worthless.

A model that says "the login form is vulnerable to SQL injection" has told you nothing you can act on. A model that says "here is the injection, here is the exact request I sent, here is the database version it returned" has told you everything. The gap between those two sentences is the entire product.

ReasonChain: prove it or it did not happen

The core of Nemesis Red is an engine we call ReasonChain. The rule it enforces is easy to state and annoying to build: the model is never trusted on its own word. Every claim it makes gets executed against the real target and checked against ground truth before it counts as a finding.

The loop:

1. The model forms a hypothesis. "This parameter looks injectable."
2. It selects and runs a real tool to test that hypothesis, against the actual target, not a simulation.
3. The result is checked against an objective signal. Did the payload return the data. Did the shell open. Did the scoreboard flip.
4. Only a hypothesis that survives step 3 becomes a finding. Everything else is discarded, including the things the model was completely sure about.

This is closer to how a human pentester works than to how a chatbot works. You do not write "SQLi confirmed" in a report because the app looked suspicious. You write it because you pulled a row out of the database. ReasonChain holds the model to that same standard.

The effect on output quality is the whole point. The model is allowed to be wrong in private as many times as it likes. It is not allowed to be wrong in the report.

Driving 290 tools instead of reimplementing them

We did not try to teach the model to exploit things from scratch. The offensive tooling already exists and is battle-tested. Nemesis Red orchestrates around 290 tools from the Kali ecosystem across 17 categories, and the model's job is to be the operator: pick the right tool for the current phase and target type, run it, read the output, and decide the next move.

That framing matters. The model is not a hacker in a vacuum, it is a decision layer on top of tools that already work. Recon feeds exploitation feeds reporting in a closed loop, with the model choosing the path and the tools doing the actual work. When the model picks wrong, the verification step catches it, because a tool that finds nothing produces no ground-truth signal to promote into a finding.

We measured it, and kept the methodology attached

Claims about "AI finds more" are cheap, so here is ours with the method attached.

In a controlled ablation, Nemesis Red produced 24.4 times more valid findings than fixed-script automation on the same targets. The setup: 30 OWASP-class targets, 120 runs, across four conditions. The full system, then three ablations that each remove one capability: no replanning, no fusion of results across steps, and randomized tool order instead of reasoned order. Wilcoxon signed-rank, W = 465, p < 0.001.

The ablations are the interesting part, because they say where the lift comes from. Removing replanning hurts. Removing cross-step fusion hurts. Randomizing the order the model would otherwise reason its way to hurts. The gain is not "we called an LLM," it is the reasoning loop wrapped around the tools. Strip the reasoning and you are back to scripted scanning.

Receipts

On known-vulnerable targets during the study, Nemesis Red autonomously identified and exploited three critical vulnerabilities rated CVSS 9.8, and confirmed each with a working exploit rather than a scanner flag:

• CVE-2024-38476 (Apache mod_rewrite SSRF)
• CVE-2024-38474 (Apache mod_rewrite substitution)
• CVE-2023-25690 (HTTP request smuggling)

To be precise about what that is and is not: these are known, published N-day CVEs, not novel zero-days. The result is that the system chained a full exploit against each on its own and proved it, with no human steering the steps. Novel zero-day discovery is a different and harder problem, and we treat it as a separate thing rather than blurring the two.

Local by default, because pentest data is sensitive

A detail that matters for anyone who would actually run this: you should not have to ship a client's live vulnerabilities to a third-party API to use the tool. Nemesis Red is bring-your-own-key for the frontier models (Anthropic, OpenAI, Gemini) if you want their reasoning, but it also runs fully local against an Ollama model (Qwen, Llama, or your own fine-tune). Local is slower and a little less sharp, but for engagements where the target data cannot leave the environment, "a little less sharp and it never leaves the box" is the correct trade.

Three ways to run it

The same engine runs at three levels of autonomy, depending on how much you want to hand over:

• Copilot. A human drives, the model reads tool output and pre-fills the next moves. For testers who want speed without giving up the wheel.
• Autonomous vulnerability assessment. The model runs the discovery and verification loop and hands you a verified findings list.
• Autonomous penetration test. Recon to exploit to report, hands off.

What it gets wrong, honestly

Before verification, the model is exactly as overconfident as every other LLM. The entire architecture exists because raw model output is not trustworthy, and if you deleted the verification layer you would have the same demoware problem as everyone else. Verification also costs time and tool executions, so a run is not free. And autonomy needs guardrails: a system driving offensive tools on its own has to be scoped hard to the authorized target set, which is a first-class design concern, not an afterthought.

None of that is a reason not to build it. It is the reason to build it with proof in the loop instead of vibes.

Where this fits

Nemesis Red is the offensive half of what we are building. The defensive half, Nemesis Blue, is a Rust EDR with a completely different build story that I wrote up separately.

If you want to see it run or get your hands on it, we are at nemesislabs.xyz.