Beyond free-roaming Agents: Architecting a Deterministic 4-node Graph pipeline for near zero false-positive Autonomous VAPT

Beyond free-roaming Agents: Architecting a Deterministic 4-node graph pipeline for near zero false-positive Autonomous VAPT

Every AppSec engineer eventually hits a mathematical scaling wall.

In a high-growth environment, a single application security engineer is often personally responsible for securing dozens of microservices, multiple public-facing API gateways, complex native mobile apps, and continuous CI/CD pipelines. When you are outnumbered 100-to-1 by software engineers shipping code daily, 70% of your offensive testing time is eaten by repetitive friction: subdomain sweeps, manual API parameter fuzzing, writing headers, and inspecting boilerplate responses.

Naturally, when the AI wave hit, many of us tried to offload this friction by feeding targets to LLMs.

But if you’ve ever tried to run a penetration test using a standard conversational LLM chatbot, you know it fails catastrophically in production. The reasons are always the same:

• Ephemeral Context: The context window eventually resets or decays, completely wiping out hours of complex reconnaissance and target memory.
• Phantom Triage: Hallucinated vulnerabilities create massive time-sinks where you end up hunting for vulnerabilities that do not exist.
• Friction & Latency: They require constant, manual "human-in-the-loop" prompting between every single execution step.

Frustrated by these limitations, I set out to build something different.

This is the story of how I evolved my automated security testing from a brittle, free-roaming prompt loop into Okwute: a deterministic, 4-node directed graph pipeline (Mapper → Generator → Executor → Validator) that runs on a headless, self-hosted harness to achieve near zero false-positive autonomous penetration testing.

---

Phase 1: The .claude Harness (Context on Disk)

My initial attempt at solving LLM amnesia was building an engine, a .claude Harness.

Instead of letting the agent hold execution state in its active context, I decided to decouple the intelligence layer from the ephemeral chat session. The core breakthrough was "loop engineering", creating an execution engine that lives between prompts by anchoring all memory, phase progression, and vulnerability proofs in deterministic filesystem artifacts on a self-hosted workspace.

Instead of isolating learning inside independent target silos, I designed an enterprise-ready workspace topology with a robust Three-Tier Memory System under a central .claude/ root runtime:

.claude/
├── shared/
│   └── org-knowledge.md         # Tier 1: Organization-wide security baselines & false-positive filters
├── products/
│   └── <product-slug>.md        # Tier 2: Shared product-family dynamic memory (e.g., login_api)
├── working/
│   └── <session-id>/
│       └── memory.md            # Tier 3: Isolated session scratchpad & exploration sandbox
└── targets/
    └── payment-gateway/         # Isolated target workspace
        ├── state.json           # Authoritative state-machine ticker & configs
        ├── findings.md          # Cumulative, append-only verified vulnerability proofs
        ├── triage.json          # Human-in-the-loop triage overrides
        └── reports/             # Generated executive & technical vulnerability reports

The Three-Tier Memory Model

This directory layout decouples ephemeral execution logs from enterprise-grade intelligence:

1. Tier 1 (Org Knowledge): Read-only for active agents. It enforces company-wide guardrails and cataloged false-positive exemptions (e.g., "Spring Actuator 401s are intentional").
2. Tier 2 (Product Memory): Keys dynamic vectors across targets of the same product_type. If an agent discovers sequential ID exploitation patterns on a target within a product family, every other target in that family inherits that heuristic, speeding up warm-starts.
3. Tier 3 (Agent Working Memory): Restricts active session scratchpad noise, keeping unverified exploit attempts and exploratory hashes fully isolated until they are ready to be verified.

The system ran as an autonomous state machine. Instead of running an infinite script that would drain API credits, it executed one bounded, deterministic "tick" via a /scan-target command.

During each tick, the harness would:

1. Parse state.json to determine the current testing phase (recon, enumeration, vuln-discovery, chain-development, reporting).
2. Read the local filesystem on disk to reconstruct what it had already scanned.
3. Call the LLM to decide the next logical action.
4. Save the updated results back to disk.
5. Exit.

By writing every single observation, endpoint, and finding immediately to a structured state.json and a Markdown audit trail (findings.md), the context could survive host restarts, process crashes, and API disconnects.

The Concurrency Conflict

To allow multiple instances of this harness to run without stepping on each other's toes, I implemented a lightweight three-tier memory system using SHA-256 context hashing in the file frontmatter:

 `context_hash = SHA-256 (file_content ∖ {context_hash line})`

If a background worker attempted to write an update to a product's shared memory, it would first re-evaluate the hash on disk. If the current hash matched the initial state, the write was committed; if not, a conflict was declared, and the worker safely aborted.

It was an incredible step forward. But as we scaled it, we hit a massive conceptual roadblock: LLMs love to roam free. If you give an LLM agent access to a generic "run terminal command" tool, it will eventually try to skip steps, guess endpoints, or spiral into endless recursive loops trying to fix a single broken curl payload.

We needed a system that enforced absolute discipline.

---

The Epiphany: James Kettle's "HTTP Terminator"

In August 2026, at DevCon (presented also at Black Hat USA 2026 and DEF CON 34), PortSwigger's Director of Research, James Kettle, published a paradigm-shifting whitepaper titled "Can AI do novel security research? Meet the HTTP Terminator".

In his paper, Kettle tackled a profound question: can an autonomous AI system actually invent new attack techniques, bypass complex security layers, and discover zero-days on live, production systems?

To prove it, he built and open-sourced the HTTP Terminator, an autonomous research engine designed to hunt for HTTP desync vulnerabilities. But his ultimate success was built on a series of harsh, real-world lessons about the cognitive limits of SOTA LLMs. He realized that while LLMs are brilliant at high-level reasoning and pattern recognition, they are notoriously unreliable at state-tracking, structured execution, and exact payload formatting.

Left to their own devices, agents in a freeform loop behave like over-caffeinated interns. They trigger WAF rate-limits, get trapped in recursive debugging loops, hallucinate vulnerabilities, and frequently give up entirely upon seeing defensive headers like Connection: close.

To tame this chaos, Kettle structured the HTTP Terminator around a Four-Phase Discovery Loop:

1. Ideation (with Micro-Inspiration): To prevent models from over-anchoring on existing concepts (context-contamination), Kettle stripped down the input. He parsed 138 technical RFCs into 15,000 tiny, 1-to-3 sentence fragments (micro-inspirations) and prompted the LLM to generate 1-to-5 vectors per fragment, resulting in 30,000 highly unique candidate desync payloads.
2. Evaluation: A simple, non-judgmental execution harness. Instead of pre-configuring what a "successful" vulnerability should look like, the system paired a regular HTTP request with a candidate desync trigger and flagged anomalous deltas in the response status or body length.
3. Weaponization: Moving from a raw desync trigger to verified impact (e.g. Response Queue Poisoning). To bypass LLM safety guardrails ("offensive capability increases"), Kettle utilized Reality Re-framing (renaming local MCP tools to 'Turbo Simulator' so the agent believed it was in a sandbox) and Placebo Capabilities (giving the agent dummy connection-reuse tools to stop it from giving up when connection reuse was blocked). He also designed the "dangling-byte technique", using partial HTTP requests missing a single byte, to systematically defeat the front-end stacked-response connection resets.
4. The Cascade: The heart of true research. The engine took every verified finding and fed it back in to ask: "How can I detect similar behavior elsewhere? Does the origin of that behavior enable other attacks?" This systematic feedback loop led directly to his most significant conceptual breakthrough: Shared-Parser Confusion (the discovery that back-end servers use shared parser code to process both requests and responses, meaning response-only processing directives can be triggered via malicious requests) and a zero-day in Apache Traffic Server (CVE-2026-63078).

Turning the Blueprint into Code: The Genesis of Okwute

Kettle's ultimate takeaway was that "AI vs. Human" is the wrong framing. Instead, modern security engineering is a three-way collaboration: AI vs. Code vs. Human.

He found that starting with an AI-heavy loop is useful for speed, but to achieve consistent, production-grade accuracy, you must gradually move cognitive responsibility to deterministic code. In his exploitation engine, he split templates in half, ensuring deterministic code validated the evidence while the LLM handled planning.

This core design principle was the absolute catalyst for Okwute.

Okwute takes Kettle's four-phase research methodology and compiles it into a rigid, relational, and mathematically-enforced Directed Acyclic Graph (DAG). I eliminated the loose phase drift of legacy tools by locking down each step inside its own database boundary. The Mapper, Generator, Executor, and Validator are the physical implementation of Kettle's Ideation, Evaluation, Weaponization, and Cascade loops, bound forever by strict read/write contracts on SQLite.

---

Phase 2: Okwute: The Deterministic 4-Node Pipeline

I retired the free-flowing state machine of the .claude harness and completely re-engineered the platform around a rigid 4-node pipeline managed by SQLite.

   ┌──────────────┐      ┌──────────────┐      ┌──────────────┐      ┌──────────────┐
   │    MAPPER    │─────▶│  GENERATOR   │─────▶│   EXECUTOR   │─────▶│  VALIDATOR   │
   │ (Pure Recon) │      │ (Test Cases) │      │ (Burp Suite) │      │ (PoC Proof)  │
   └──────────────┘      └──────────────┘      └──────────────┘      └──────────────┘

The pipeline operates on a strict rule: Each node reads only from its upstream table and writes only to its designated downstream table. No node may skip, jump ahead, or bypass the graph.

Here is how the 4 nodes execute natively in a self-hosted environment:

Node 1: The Mapper (Pure Reconnaissance)

• Command: /run-mapper <target> [<domain-or-url>]
• The Rules: The Mapper has zero judgment responsibilities. It is not allowed to generate payloads or think about exploitation.
• The Job: It runs high-speed subdomain enumeration (subfinder), port scanning (nmap), and passive traffic analysis (httptoolkit MCP).
• Output: It inserts discovered raw endpoints directly into the endpoints table of the target's SQLite database (appsec.db).

Node 2: The Generator (The Threat Modeler)

• Command: /run-generator <target>
• The Rules: The Generator is highly intelligent, but completely sandboxed. It is strictly forbidden from sending a single network packet.
• The Job: It reads unprocessed rows from the endpoints table. Utilizing the LLM's deep contextual reasoning, it evaluates the endpoint parameters, headers, and underlying technology stack. It then plans highly specific test cases (e.g., SQLi, IDOR, CL.0 HTTP Request Smuggling desync paths, unvalidated postMessage handlers).
• Output: It writes candidate payloads and expected behaviors to the test_cases table.

Node 3: The Executor (The Mechanical Gun)

• Command: /run-executor <target>
• The Rules: The Executor is completely mechanical. It has zero intelligence and does not evaluate whether a vulnerability is real or fake.
• The Job: It takes fully-specified HTTP requests directly from the test_cases table and fires them through a headless Burp Suite Professional instance using a custom-built Java proxy (mcp-proxy) that bridges Burp's REST API into the Model Context Protocol (MCP).
• Output: It records the raw response, status codes, and response length directly into execution_results. If a response shows an anomaly (status or length delta compared to the baseline), it flags it and creates an active Repeater tab in Burp Suite for manual traceability.

Node 4: The Validator (The Jury)

• Command: /run-validator <target>
• The Rules: The Validator has the highest evidentiary bar: Zero False Positives. Nothing lands in the findings file without an automated, reproducible Proof of Concept (PoC).
• The Job: It reads anomalous rows from execution_results. It parses the raw HTTP response and compares it logically against the Generator's expected_behavior. It actively attempts to reconstruct a working, step-by-step PoC request-response pair.
• Output: Verified, iron-clad vulnerabilities are written to the verified_findings table on disk and synced to a shared cross-target SQLite memory (appsec_memory.db) to prioritize future test cases on similar tech stacks.

---

The SQLite Core: How Okwute Keeps State

Because all memory lives in SQLite, there is an immutable audit trail. Data is never deleted; rows are progressed through a state lifecycle (unprocessed $\to$ processed).

Here is a look at how clean and simple the database model is:

Table Name	Owner (Writes)	Primary Purpose
endpoints	Mapper	Mapped attack surface (URLs, methods, parameters)
test_cases	Generator	Planned payloads and expectation criteria
execution_results	Executor	Raw HTTP responses and anomaly flags
verified_findings	Validator	Verified vulnerabilities with reproducible PoCs

---

The Centralized Memory Evolution: Shifting to Obsidian

The transition from the old .claude harness to Okwute forced us to completely rethink how to manage cross-target security memory.

In the early iterations, I attempted to maintain cross-target knowledge inside flat, prose Markdown files (products/<product-slug>.md). But as multiple agents executed in parallel, this filesystem-centric model created a severe bottleneck: concurrent file writes caused clobbered updates, and the SHA-256 context hashing system would trigger frequent merge conflict halts, entirely defeating fully autonomous runs.

We solved this concurrency crisis by building a Statistical Cross-Target Memory Engine (appsec_memory.db) managed by SQLite. Instead of prose notes, the engine computes real-time mathematical hit-rates for successful exploit vectors:

    `Exploit Hit Rate = confirmed_count / tested_count`

If a specific JWT header spoofing vector yields an 85% success rate on a fintech target, that statistical weight is committed atomically to the centralized database, allowing day-one prioritization when the pipeline spins up on a new target sharing the same product_type.

Bridging Concurrency and Human Readability: The Obsidian Sync

While relational tables solve the multi-agent race conditions, they are notoriously hostile to human operators who need to quickly review, organize, and enrich security findings.

To bridge this gap, I designed a hybrid interface that ports the relational memory directly into a hyper-linked Obsidian Vault using a dedicated Python command-line utility (memory_db.py).

# Synced dynamically via a production orchestrator
python3 .opencode/scripts/memory_db.py sync-vault /workspace/obsidian-security-vault/

This synchronization engine dynamically translates binary SQL tables into a beautifully structured, human-readable Obsidian workspace:

Obsidian_Vault/
├── Index.md                     # Auto-generated, linked directory map of all product types
└── Products/
    ├── fintech_api.md       # Auto-generated vector hit-rates and metrics (Overwritten on sync)
    └── fintech_api - Notes.md # Hand-authored qualitative research and overrides (Preserved forever)

The system automatically splits product-family memory into a dual-file architecture:

1. The Machine Ledger (<product_type>.md): This is a read-only, auto-generated summary updated on every sync. It lists live empirical hit rates, historical vulnerability vectors, and automated false-positive rules. Humans never edit this file.
2. The Human Ledger (<product_type> - Notes.md): This file is created as a blank stub once. It is never overwritten during sync cycles. It is a playground for the human pentester to hand-author qualitative notes, record technology quirks, and outline manual exploitation hypotheses that the AI can read and import as context rules.

This design gives us the best of both worlds: strict relational concurrency for parallel agents, and a beautifully visualized Markdown canvas for the human product owner.

The Orchestration Layer: Multi-Modal Control Planes

While the core SQLite database acts as a decentralized state engine, how the pipeline is actively triggered and managed can be adapted across three distinct operational control planes, depending on the environment:

1. Continuous DevSecOps (Headless CI/CD)

For hands-off, continuous scanning, the entire pipeline is packaged inside a self-hosted Forgejo Actions (or GitHub Actions) CI/CD runner. This headless pipeline runs overnight on a dedicated label-node, managing process state for Burp and the AI harness dynamically.

• The Mechanism: An automated cron or repository dispatch triggers Forgejo Actions to initialize the target's workspace, spin up the headless Burp proxy instance, drive the execution pipeline sequentially, and drop a comprehensive markdown audit summary directly into the Forgejo job dashboard at completion.

2. Terminal-Native Harnesses (The Developer CLI)

For rapid, interactive local testing, security engineers can drive Okwute directly from terminal-native agent environments like Claude Code or OpenCode.

• The Mechanism: By hooking local Model Context Protocol (MCP) servers, engineers can run conversational slash commands (e.g., /run-generator or /run-validator) straight from their local terminal to triage new API endpoints or debug specific payload modifications in real time, combining autonomous execution with on-demand interactive control.

3. Custom-Built Command Dashboard (The Web Control Plane)

To abstract CLI friction entirely, Okwute's pipeline CLI outputs can be fed directly into a custom-engineered Web UI Control Plane.

• The Mechanism: This UI visualizes live telemetry of active target environments, lets security analysts draw trust boundaries on interactive SVG graphs, renders active Burp Repeater tabs side-by-side with LLM-generated payloads, and lets human-in-the-loop operators click-to-approve candidate findings before they sync to the global enterprise database.

---

Conclusion: Taming the Wave

AI is not going to replace the human-in-the-loop security engineer, but the security engineers who automate their workflows using deterministic graph engineering will completely outscale those who do not.

By moving away from open-ended chat inputs and building structured, database-backed state machines like Okwute, I can let AI do what it does best, reason, classify, and generate patterns, while forcing the underlying infrastructure to remain secure, disciplined, and deterministic.

Stop chatting with your models. Start building compilers for them.

---

What are your thoughts on agentic security pipelines? Have you explored wrapping Burp Suite or headless proxies into autonomous decision loops? Let’s discuss in the comments!

Comments

[Peter]

The Mapper/Executor split solving "LLM can't hold state reliably" is the right instinct, and pairing it with James Kettle's four-phase structure is a solid foundation. One thing I'd want measured explicitly: the Validator's zero-false-positive bar is enforced against the Generator's expected_behavior — but the Generator is also an LLM, reasoning about what a vulnerability should look like from the endpoint shape. If the Generator's expectation is wrong in a self-consistent way, the Validator can build a PoC that satisfies that wrong expectation and still call it verified. PoC-gating is a strong control against noise (false positives), but it doesn't say anything about what the pipeline never generated a test case for in the first place (false negatives) — those two failure modes need separate numbers, not one "near zero false-positive" claim. Have you run Okwute against a labeled set of intentionally vulnerable targets to get an actual recall number, separate from the precision story?

[Ekene Ejike]

You raise a fair concern about self-consistent hallucination, but the Validator's verification is more independent than the Generator's expected_behavior might suggest. The Validator doesn't just check if the response matches the Generator's prediction, it independently re-fires probes for reproducibility, polls Burp Collaborator directly for OOB vectors (SSRF, blind XXE) through channels the Generator never touches, cross-references other findings for chain opportunities, and constructs its own PoC from raw HTTP evidence. The expected_behavior is contextual guidance, not a verification oracle.

For example, the Validator has already rejected findings where the Executor flagged an anomaly but the deeper analysis revealed it was a transient 5xx, and confirmed findings where the initial HTTP response looked benign but OOB Collaborator interactions told a different story. The Validator also independently adjusts severity ratings based on evidence, not just accepting the Generator's initial classification.

That said, your core point about false negatives is absolutely valid. The pipeline mitigates it somewhat through the Mapper feeding newly discovered endpoints back into subsequent cycles, there is a Browser Investigation lane that finds hidden APIs that the mechanical pipeline picks up, and the cross-target memory engine prioritizing historically high-yield vectors. But you're right, a recall number against a labeled benchmark set is the proper metric, and I haven't published one yet.

The article only covers the 4-node graph and SQLite core, but Okwute has significantly more: a browser investigation side lane for DOM-XSS/OAuth/CSRF flows, 19+ specialist security skills (mobile, cloud, LLM red teaming), MCP abstraction for Burp/MobSF/Blutter/DevTools, and an Obsidian vault sync for human-curated memory. The full technical spec is ~880 lines, the Dev.to format just couldn't fit it all.

On the recall question specifically: Okwute has been run against several production API and mobile targets, surfacing multiple confirmed findings across critical, high, and medium severities — including RCE, SSRF, IDOR, and authentication bypass vectors, verified via Burp Collaborator OOB evidence and retested post-fix. The real question is what it missed.

I would plan a follow-up article where I run Okwute against a known vulnerable target (WebGoat or Juice Shop) to publish actual recall/precision numbers. That would directly answer your false-negative question with data rather than architecture. I'd appreciate your thoughts on the approach when it's ready.

[Peter]

Good to hear the Validator independence story is more solid than the article had room for. On the WebGoat/Juice Shop benchmark: I'd push for stratifying recall by vuln class rather than reporting one blended number. A pipeline can hit strong recall on straightforward single-shot payloads (reflected XSS, basic SQLi) while systematically missing anything that requires chaining — SSRF into internal service enumeration, IDOR that only manifests after a state transition — and a blended recall number hides that completely. If the known-vulnerable set includes multi-step chains, I'd want the report broken out by whether the finding required chaining at all, since that's usually where autonomous pipelines fall over first. Happy to look at the approach when you're ready to publish it.

[Ekene Ejike]

Fair point, a blended recall number is meaningless. I'm planning to stratify by:

1. Vuln class (XSS, SQLi, IDOR, SSRF, etc.)
2. Discovery depth: single-shot vs. multi-step chains
3. OOB vs. in-band, since blind vectors need different verification channels

Okwute's architecture does bias toward chaining, the Generator reads endpoint relationships, not isolated URLs, and the Validator cross-references findings. Earlier cycles feed into later ones, so multi-step chains emerge from the DAG itself, not manual prompting. But you're right, that needs a benchmark number, not just an architecture claim.

I'll share the stratified results when it's ready. Would appreciate your eyes on the breakdown before I publish.

[Peter]

On the stratification: worth separating by confirmation channel too, not just vuln class and depth. An OOB-confirmed finding (Collaborator interaction) and an in-band finding confirmed by response diffing have very different residual false-positive rates — OOB is close to unforgeable, response-diffing can still be fooled by things like differential error pages that aren't actually the vulnerability. If you blend them into one recall number, a class with good OOB coverage (SSRF, blind XXE) can make the aggregate look stronger than the in-band-only classes (most XSS, some IDOR) actually are. I'd also add a bucket for "true positive, unconfirmable" — cases with no OOB channel available at all (no egress, air-gapped target) — and report those separately rather than silently counting them as recall misses, since that's a measurement gap, not a detection failure.

[Ekene Ejike]

Both points are solid. OOB vs. in-band confirmation channels have fundamentally different residual FP rates, and 'true positive, unconfirmable' shouldn't be a recall miss. I'll add both to the stratification framework. Would you be open to collaborating on this benchmark together? I can set up a workspace — Slack, Discord, whatever works for you.

[Peter]

Glad the stratification split is useful. One more cut worth adding: an "unconfirmable" disposition needs its own trend line, not just its own label. OOB confirmation channels (DNS/HTTP callback) get blocked by the same egress controls that would block real exfiltration, so a rising unconfirmable rate can mean your target's network got stricter, not that your findings got less confirmable — worth tracking that ratio over time separately from the raw count, or a tightened environment silently looks like regressing recall. On the workspace — appreciate the offer, but I'll keep my end of this in the open here rather than a side channel; happy to keep trading ideas in the comments as the benchmark comes together.