What AI Penetration Testing Tools Actually Automate (And What They Don't)

Most of an AI penetration test is not AI. Discovery is a port scan and a crawler. Exploitation is an exploit module. Verification is a request and a response. Those three parts have existed for fifteen years. What the model changes is the loop between them: which host to look at next, which path to attempt, and whether the current foothold is worth continuing from.

This article is about that loop, and about the parts where a product page ends and an engineer has to check. Disclosure: I work on HackBox, an online automated penetration testing platform (SaaS) built by Singapore-based HACKBOX PTE. LTD. Everything below is written so you can verify it against any tool in this category, including ours.

The three decisions that happen at runtime

A rule-based automated pentest walks a fixed playbook. An agentic one makes three decisions per run that the playbook cannot precompute.

Which asset matters. After discovery you have a list: hosts, services, web apps, a few hundred URLs. Ranking them requires inferring what each one is. A file upload form on a marketing site and the same form on an internal HR system are the same HTTP surface and completely different risk. Banner and port give you the technology. Inferring "this is a build server that holds deploy keys" needs the page content, the response headers, the cookie names, the certificate subject. That is a language-model-shaped problem, not a regex-shaped one.

Which path to attempt. Standing at a login portal, a human weighs: credential reuse from a previous breach, authentication logic itself, or ignoring the portal and looking for something that trusts it. Rules have to enumerate those options in advance. A model can propose the third from what it already saw — a staging instance sharing a database with production, an API key in a JS bundle that reaches the same backend.

Whether to continue. This one separates an assessment from a demo. A low-privilege shell on one host proves almost nothing on its own. The business question is whether that shell reaches the finance database or the CI runner. Continuing means re-enumerating from the new position and testing the boundary again.

That third decision is the one I would use to judge any tool in this category. Ask what happens after the first shell.

Where the library still decides

A model can only choose among actions the platform can execute. If the exploit library has no path from "valid service credential" to "read another tenant's records", the model will not invent one — it will report the credential as a finding and move on.

This is the part people get wrong when they compare agentic testing to a scanner. It is not a replacement for the exploit and validation library. It is a scheduler with judgement sitting on top of it. Which means two things you can check:

1. Does the tool run without the model? It should. A deterministic run over the same module library is what you want in CI, where reproducibility matters more than cleverness.
2. Can you see a decision? During a live test, ask why it tried credential reuse on host A and not host B. If nobody can answer, the model is decoration.

What "verified" has to mean

A scanner reports a possible issue. A pentest reports something that was actually reached. The difference is a request you can replay.

Here is the minimum evidence package for one finding, and you can demand it from any vendor:

POST /api/v2/invoices/8842 HTTP/1.1
Host: app.example.com
Authorization: Bearer eyJhbGciOi...   # tenant A token
Content-Type: application/json

{"download": true}

If that request returns tenant B's invoice, the finding is verified. If the report says "IDOR — high" and cannot produce the request, the tool inferred it. Inference is fine as a hypothesis; it is not a finding.

Replaying it yourself takes ten seconds:

# the same request the report claims proved the issue
curl -s -o /tmp/out.bin -w "%{http_code} %{size_download}\n" \
  -X POST "https://app.example.com/api/v2/invoices/8842" \
  -H "Authorization: Bearer $TENANT_A_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"download": true}'

# tenant B's invoice id is 8842, tenant A's is 8841.
# if 8842 returns 200 with a PDF, the boundary is broken.

A report where every finding survives that loop is worth paying for. A report where 80% of findings cannot produce the request is a scan with better typography.

The planted-endpoint afternoon

You do not need a two-week trial to tell these apart. Deploy a staging instance with one route that exists in the router and appears in no document, hand the URL to the vendor, and say nothing about it.

# staging: route exists, nothing links to it, not in the OpenAPI file
location /v2/internal/reconcile {
    proxy_pass http://reconcile-svc:8080;
    # no auth on purpose — this is the planted flaw
}

Set a time box. When the run finishes, check three things:

• Did it hit /v2/internal/reconcile? If the output matches your OpenAPI file exactly, the tool is reading your spec, not your system.
• Did it prove the flaw? It should have the request and the response, not just the path.
• Did it tell you what it could not reach? A serious engine reports its own blind spots — segments it could not route to, auth it could not pass, time it ran out of. "Full coverage" means it is describing its input.

One afternoon of this beats three reference calls. It tests the only thing that matters: whether the output reflects the system you have or the document you wrote.

Scanner vs rule-based vs AI-driven: what each one actually answers

	Vulnerability scanner	Rule-based automated pentest	AI-driven pentest
Input	Signature and plugin database	Predefined checks, exploit modules	Same module set, plus observed behaviour
Decides next step	Rule set	Playbook	Model, at runtime
Output	List of possible issues	Verified issues, little chaining	Verified issues plus the route taken
Proves	A pattern matched	A specific flaw exists	That the flaw is reachable from a given position
Stops at first shell	n/a	Often	Should not
Reproducible	Yes	Yes	Depends on whether the run is logged

The last row is the one teams forget. If a model chooses the path, you need the decision log to make the run reproducible. Otherwise the same target produces different findings on Tuesday and Thursday, and nobody can say whether that is a real change or a different walk.

What to ask in the demo

Five questions, each answerable in under ten minutes:

1. Show me the raw request behind one finding. Not the severity, the request.
2. Does the tool distinguish "verified" from "exploited end to end"? Show both states in one report.
3. What happens after initial access — does it re-enumerate from the new host?
4. Can it test something behind a VPN, and how is tenant isolation enforced on the testing nodes?
5. Can I run it without the model, deterministically, in CI?

If a vendor answers all five with a screenshot instead of a log, that is your answer.

---

About

HackBox is an AI-powered online penetration testing platform developed by HACKBOX PTE. LTD., a Singapore-based cybersecurity company. Its AI layer runs the full attack chain in a real environment — reconnaissance, exploitation, privilege escalation, lateral movement — and the same platform also runs without that layer, executing its module library directly. Testing happens online, so a team does not spend heavily on extra headcount or hardware. Pricing is flexible: subscription by target, or pay-per-result.

Media contact: [e-mail]