An AI pentest agent that structurally can't hallucinate a vulnerability — and runs offline

Александр Тихонов — Tue, 25 Aug 2026 07:14:59 +0000

If you point any LLM at a target and ask it to "write a security report," it will confidently invent findings that aren't there: an imagined TLS weakness, a "likely" SQL injection, a secret leak that never happened. For a security tool, a hallucinated finding is the worst possible output — you can't hand a client a report built on fiction.

There's a second problem, and it's a hard blocker for a lot of real work: client data can't go to someone else's cloud. NDAs, air-gapped environments, regulated data, or plain lack of trust. "Just use ChatGPT" isn't an option.

I built Nexus, an autonomous red+blue agent that addresses both. Here's how.

The idea: an evidence gate

An LLM is great at some things and terrible at others, so the two jobs are split:

The model TRIES — it decides what to test and sends the payloads. This is where LLMs are strong.
Deterministic code writes the report (findings.py), and a finding only counts if the live target's response proves it: an XSS marker came back unescaped; ../../etc/passwd returned content containing root:x:; a quote triggered a SQL error; the server handed over a token on ' OR 1=1--.

Nothing in the final report comes from the model's prose. Every finding is read from the transcript — the actual response bytes — not from what the model "wrote." Hallucinating a finding isn't forbidden; it's structurally impossible.

$ nexus --target demo.testfire.net --authorized --rules-accepted
  -> /search.jsp?query=<script>alert(1)</script>
     [+] reflected UNESCAPED — XSS confirmed
  -> /index.jsp?content=../../../../etc/passwd
     [-] no root:x: in response — not confirmed, dropped
[gate]  writing report from EVIDENCE, not from prose
  [x] "weak TLS 1.0 cipher"    no evidence · DROPPED
  [+] HIGH    Reflected XSS
[done]  findings: 2 · hallucinated: 0

The model suspected path traversal and weak TLS — the gate dropped everything the target didn't confirm.

Offline: an stdlib core, data stays with you

The core is pure Python stdlib, zero dependencies, and runs in an air-gapped box. The "brain" is pluggable: a cloud model (if allowed), a local Ollama model, or a small fine-tuned model that runs fully offline. The point: for confidential work where the cloud is off the table, you still get a working AI pentester, and the client's data never leaves the machine.

Isn't a small offline model just a toy?

A generic 1.5B is nearly useless for a real assessment: on a held-out benchmark it latches onto two passive findings and exploits nothing — 35%. I fine-tuned the same 1.5B (on teacher-distilled trajectories, scored by the same evidence gate where you can't fake a finding): 95% held-out, 71% on an OOD target it had never seen, fp=0. An offline brain that does the work instead of imitating it. Numbers are reproducible (lab/ood_eval.py).

What you get

Not a scanner dump — a client-ready report: CVSS 3.1 (computed from the formula, not guessed), attack chains, CWE/OWASP/PCI mapping, a remediation plan with concrete fixes, retest deltas between runs, and a co-pilot you can ask about findings — it answers only from what was proven (ask about an XSS that doesn't exist and it says "no evidence" instead of inventing one). Red (audit) and blue (defending your own host) are one discipline.

Ethics

Authorized scope only, declared up front. Scope is an allowlist in code, checked on every request and every redirect (SSRF guard). External targets require a second confirmation. No destructive or DoS actions. This is a tool for your own infrastructure and for engagements with written permission.

Try it

Early release, source-available (BUSL-1.1).

pip install nexus-sec

Code: https://github.com/alerta200/alerta

I'm most curious where the evidence-gated approach itself breaks. The obvious trade-off is false negatives — a detector this strict will miss real bugs. If you find a case, send it.