AI-Assisted Penetration Test Report Generation

Penetration test reports consume 30–40% of total engagement time. The vulnerability discovery is done — you have notes, tool outputs, screenshots — but converting that raw evidence into a structured, client-readable document is where hours disappear. Integrating a language model into this pipeline cuts report writing time significantly while keeping the critical validation work firmly in human hands.

This is not about automating your pentesting. It is about automating the prose that wraps findings you have already confirmed.

The Architecture

The pipeline runs in three stages:

1. Ingest — parse raw finding data from your tooling (Burp Suite exports, Nmap XML, manual notes in JSON or CSV)
2. Enrich — feed each finding to a language model to generate description, impact, and remediation prose
3. Render — merge enriched findings into a Jinja2 template to produce Markdown or DOCX output

The core constraint: the LLM generates prose only. It never changes a CVSS score, never modifies the affected host, never invents a finding. Your structured data is the source of truth; the model fills the narrative gaps around it.

Structuring Raw Findings Before Touching the LLM

Define a strict schema before you write a single prompt. This eliminates hallucination drift — if the model cannot change numeric fields, it cannot fabricate severity.

from dataclasses import dataclass, asdict
from typing import Optional

@dataclass
class RawFinding:
    id: str
    title: str
    cvss_score: float
    cvss_vector: str
    affected_host: str
    port: Optional[int]
    evidence: str          # raw tool output or tester note
    cwe_id: Optional[str]
    remediation_hint: str  # one-liner from the tester

def load_findings(path: str) -> list[RawFinding]:
    import json
    with open(path) as f:
        data = json.load(f)
    return [RawFinding(**item) for item in data]

A sample input file (findings.json):

[
  {
    "id": "FIND-001",
    "title": "Reflected XSS in search parameter",
    "cvss_score": 6.1,
    "cvss_vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:C/C:L/I:L/A:N",
    "affected_host": "https://app.example.com/search",
    "port": 443,
    "evidence": "GET /search?q=<script>alert(1)</script> returned unsanitized input in response body",
    "cwe_id": "CWE-79",
    "remediation_hint": "encode output, add CSP header"
  }
]

Building the Enrichment Layer

The enrichment step sends each finding to a language model with a strict system prompt that forbids modifying structured fields. Using response_format: json_object forces the model to return parseable JSON consistently.

import json
import textwrap
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from environment

SYSTEM_PROMPT = textwrap.dedent("""\
    You are a senior penetration tester writing a client-facing security report.
    Given a JSON finding, produce exactly three fields:
    - "description": 2-3 sentences explaining the vulnerability technically
    - "impact": 2-3 sentences on concrete business and security impact
    - "remediation": 3-5 bullet points with actionable remediation steps

    Rules:
    - Return valid JSON only. No text outside the JSON object.
    - Do NOT change any field from the input (cvss_score, affected_host, etc.)
    - Do NOT invent additional findings or sub-findings
    - Write at a level appropriate for a technical security team
""")

def enrich_finding(finding: RawFinding) -> dict:
    payload = json.dumps(asdict(finding), indent=2)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Finding:\n{payload}"},
        ],
        response_format={"type": "json_object"},
        temperature=0.2,
    )

    enriched = json.loads(response.choices[0].message.content)

    # Enforce immutable fields after the merge
    result = {**asdict(finding), **enriched}
    for key in ("cvss_score", "cvss_vector", "affected_host", "port", "cwe_id"):
        result[key] = getattr(finding, key)
    return result


def enrich_all(findings: list[RawFinding]) -> list[dict]:
    results = []
    for f in findings:
        try:
            results.append(enrich_finding(f))
        except Exception as e:
            print(f"[WARN] Skipping {f.id}: {e}")
            results.append(asdict(f))  # fall back to raw data
    return results

The {**asdict(finding), **enriched} merge pattern lets the LLM add its prose keys, and then the explicit re-assignment locks every numeric and structured field to its original value. Even if the model attempts to return a modified cvss_score, it gets overwritten. This is not optional; it is what makes the tool safe to use in a client-deliverable workflow.

Rendering the Final Report

With enriched findings in hand, a Jinja2 template decouples content from formatting. Swap the template to target DOCX (via python-docx), PDF (via Pandoc), or any internal format without touching the enrichment logic.

from jinja2 import Environment, FileSystemLoader

SEVERITY = [
    (range(9, 11), "Critical"),
    (range(7, 9),  "High"),
    (range(4, 7),  "Medium"),
    (range(0, 4),  "Low"),
]

def get_severity(score: float) -> str:
    for r, label in SEVERITY:
        if int(score) in r:
            return label
    return "Informational"

def render_report(findings: list[dict], template_path: str, output_path: str):
    env = Environment(loader=FileSystemLoader("."))
    env.filters["severity"] = get_severity

    template = env.get_template(template_path)
    output = template.render(
        findings=sorted(findings, key=lambda f: f["cvss_score"], reverse=True)
    )
    with open(output_path, "w") as fh:
        fh.write(output)
    print(f"Report written to {output_path}")

A minimal Jinja2 template (report.md.j2) that sorts findings by severity:

# Penetration Test Report

{% for f in findings %}
## {{ f.id }} — {{ f.title }}

**Severity**: {{ f.cvss_score | severity }} (CVSS {{ f.cvss_score }})
**Host**: {{ f.affected_host }}
**CWE**: {{ f.cwe_id or 'N/A' }}

### Description
{{ f.description }}

### Impact
{{ f.impact }}

### Remediation
{{ f.remediation }}

---
{% endfor %}

Quality Checks You Cannot Skip

The LLM will occasionally produce descriptions that are generic or misaligned with the actual evidence. Two automated checks must run before every delivery.

1. Keyword overlap between title and description. If the finding title says "XSS" and the generated description discusses SQL injection, catch it before the client does.

def sanity_check(finding: dict) -> list[str]:
    warnings = []
    title_words = set(finding["title"].lower().split())
    desc_words  = set(finding["description"].lower().split())
    if not title_words & desc_words:
        warnings.append(
            f"{finding['id']}: title and description share no keywords — review manually"
        )
    if finding["cvss_score"] >= 7.0:
        impact_lower = finding["impact"].lower()
        if "high" not in impact_lower and "critical" not in impact_lower:
            warnings.append(
                f"{finding['id']}: CVSS {finding['cvss_score']} but impact text lacks severity language"
            )
    return warnings

2. Remediation bullet count. The prompt asks for 3–5 bullets. A one-liner remediation on a High-severity finding is a red flag, not a deliberate choice.

Run these checks after enrich_all() and print every warning to stdout. Do not suppress them with a try/except. These are the cases where a reviewer needs to read carefully before the report leaves the building.

Cross-referencing your recommended mitigations against your firm's internal standards is also worth the time — our security hardening checklists cover the most common remediation patterns and serve as a quick reference during that review pass.

The Takeaway

The full pipeline — ingest, enrich, render, validate — stays under 200 lines of Python and compresses report prose time from hours to under ten minutes. The human work shifts from drafting boilerplate to reviewing LLM output against raw evidence. That is the correct trade-off: your expertise goes into the validation layer, not the repetitive prose.

Three things make this safe for production use: immutable structured fields, strict JSON output mode from the model, and a sanity-check pass before every delivery. Skip any of them and you are shipping a liability, not a tool.

The Jinja2 template approach means your firm's formatting standards stay in a single file. You can maintain a DOCX-based template for enterprise clients and a Markdown template for technical ones, with no changes to the enrichment code.

---

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.