DEV Community: Yoandy Ramirez Delgado

Building an AI-Assisted Pentesting Platform Without Creating an Auto-Pwn

Yoandy Ramirez Delgado — Wed, 02 Sep 2026 20:27:32 +0000

AI agents are becoming increasingly capable of interacting with tools.

In cybersecurity, that creates an obvious opportunity: let an agent inspect a target, run reconnaissance, interpret the output and decide what to test next.

It also creates an obvious problem.

What happens when the agent decides to do something that should not be done?

That question became one of the design constraints behind Dark Spear, an AI-assisted platform for authorised security assessments.

Dark Spear was not designed as an autonomous exploitation engine.

The goal was different:

Build an automated pentesting workflow that can move through an engagement while keeping scope enforcement, dangerous tool approval and evidence integrity under human control.

This article explains the architecture behind that decision.

The problem with “autonomous pentesting”

Automation is not new in offensive security.

We already automate:

Port scanning.

Directory discovery.

Technology fingerprinting.

Credential testing.

Vulnerability detection.

OSINT collection.

Report generation.

The difference with AI agents is that they can dynamically decide what to do next.

That flexibility is useful.

A deterministic scanner can only execute what its author anticipated.

An agent can potentially react to unexpected output.

But flexibility introduces a new trust boundary.

If an AI agent has access to security tools, the system needs to answer several questions:

What targets is the agent allowed to interact with?

What tools can it execute?

Which actions require explicit human approval?

Can the client-side agent bypass restrictions?

How are findings verified?

What evidence proves a finding?

Can the final report be defended and reproduced?

Dark Spear treats those questions as architecture problems rather than prompt engineering problems.

Architecture overview

The platform has three major components.

Browser-side assessment engine

The assessment engine contains two execution modes.

The first is deterministic.

The second is agentic.

The browser orchestrates the workflow and communicates with the backend only when an approved binary needs to be executed.

The main engine includes:

PTES phase orchestration.

Probe definitions.

Vulnerability knowledge bases.

Finding heuristics.

LLM agent orchestration.

Ollama Cloud integration.

Bridge client communication.

Trusted backend bridge

The backend is the enforcement point.

This is deliberate.

The browser is not trusted to enforce security boundaries.

The backend applies:

Tool allowlisting.

Target scope locking.

PTES phase gates.

Dangerous action approval.

Finding persistence.

Evidence hashing.

Encrypted API key storage.

The client can request an action.

The server decides whether that action is allowed.

That separation is fundamental when an AI agent can generate commands dynamically.

SecOps reporting console

The third component is the operational layer.

Security testing is only part of an engagement.

The results need to be reviewed, correlated and communicated.

The console provides views for:

Active engagements.

Tool approval.

Findings.

Critical vulnerabilities.

Attack graphs.

MITRE ATT&CK mapping.

OSINT.

Governance.

Risk maturity.

Remediation planning.

Reporting.

The goal was to avoid treating the scanning engine and the reporting system as separate projects.

They are parts of the same workflow.

Deterministic mode versus LLM agents

One of the design decisions was to avoid making the LLM mandatory.

The platform supports two modes.

Deterministic playbooks

The deterministic mode follows a curated catalogue of security probes.

It does not require an LLM.

It does not consume inference tokens.

And its execution path is reproducible.

The probes cover multiple categories:

Injection

SQL injection.

NoSQL injection.

Reflected XSS.

Open redirects.

Generic IDOR checks.

Authentication and sessions

Default credentials.

Cookie security attributes.

Missing security headers.

JWT testing

Weak HS256 secret cracking performed locally.

alg=none validation where applicable.

Active reconnaissance

Technology fingerprinting.

Content discovery.

robots.txt analysis.

CORS configuration.

HTTP TRACE exposure.

Code and configuration exposure

Accessible .git directories.

Environment files.

Backup files.

Source maps.

Public API documentation.

Metrics endpoints.

Secrets exposed in JavaScript bundles.

Cloud attack surface

Passive WAF and CDN fingerprinting.

Cloud infrastructure hints.

Public storage references.

SSRF checks against cloud metadata services within authorised scope.

OSINT

DNS records.

RDAP and WHOIS data.

Certificate transparency.

Historical URLs.

The important part is not only what the probes execute.

It is how findings are evaluated.

Why finding heuristics should be probe-local

A common source of false positives in automated security workflows is context contamination.

Imagine a scanner that accumulates all output generated during an assessment phase.

A finding heuristic later searches the entire accumulated output for a pattern.

A signal produced by probe A might accidentally trigger a finding that belongs to probe B.

Dark Spear avoids that by evaluating each probe against its own evidence.

The model is:

Probe → Evidence → Probe-specific heuristic → Proposed finding

Rather than:

Phase → Large output blob → Generic heuristic → Finding

This makes the relationship between a finding and its evidence easier to understand and debug.

It also improves reproducibility.

If a finding is incorrect, the auditor can inspect the exact probe that generated it.

Adding an LLM without making it the authority

The second execution mode uses a ReAct-style agent.

The agent can analyse the current state and decide which command or action should be attempted next.

This is useful when the deterministic catalogue does not cover a particular scenario.

However, the agent does not become the enforcement authority.

The execution flow remains conceptually similar to this:

The agent proposes an action.

The client sends an execution request.

The backend validates the target.

The backend validates the PTES phase.

The backend validates the tool.

Dangerous actions can require explicit operator approval.

Only then can the command execute.

This creates an important separation.

The LLM is responsible for reasoning.

The backend is responsible for enforcement.

The human operator remains responsible for approval.

Human-in-the-loop is not a marketing checkbox

“Human-in-the-loop” is becoming a popular phrase.

Sometimes it simply means that a person can watch an AI system running.

That is not enough.

In a security assessment, human control should exist at meaningful decision points.

Dark Spear introduces operator review at two levels.

Tool execution

Potentially dangerous actions can require explicit approval.

Findings

A proposed finding does not automatically become part of the final report.

The operator reviews it before accepting it.

This matters because AI-generated findings can sound convincing even when the underlying evidence is weak.

The system should not confuse a plausible explanation with a validated vulnerability.

Evidence integrity and chain of custody

A finding should not only contain a description.

It should contain the evidence that supports it.

When a finding is accepted, Dark Spear persists its evidence and generates an integrity hash.

This creates a more defensible workflow.

Instead of asking:

Why did the AI say this vulnerability exists?

The reviewer can ask:

What evidence produced this finding?

That distinction is particularly important in professional engagements.

The report should be derived from validated technical evidence.

Not from the memory of a conversation between a model and an operator.

Mapping technical findings to security frameworks

Technical findings need different forms of context depending on the audience.

A developer may need remediation steps.

A security manager may need risk prioritisation.

A compliance team may need regulatory context.

Dark Spear enriches accepted findings with mappings such as:

CVSS v3.1.

CWE.

OWASP.

MITRE ATT&CK.

ISO 27001.

ENS.

NIS2.

GDPR.

The mapping is finding-specific.

The goal is to avoid assigning frameworks purely based on severity.

A critical vulnerability is not automatically associated with every critical MITRE technique.

The relationship should come from the actual type of finding.

MITRE ATT&CK should describe what happened

MITRE mapping is another area where automation can become misleading.

A static matrix that always shows the same techniques looks impressive.

But it does not necessarily represent the actual assessment.

Dark Spear maps findings to ATT&CK techniques based on the characteristics of the finding.

The objective is to represent detected behaviour rather than display a decorative coverage matrix.

The project currently focuses on techniques relevant to non-destructive web security assessments.

Destructive post-exploitation and impact-oriented behaviour are deliberately excluded from the intended scope.

That is a product decision as much as a security decision.

A professional assessment platform should understand what it is not supposed to do.

The backend as the trust boundary

One of the strongest lessons from building this architecture is that AI agents should be treated as untrusted clients.

That statement may sound harsh.

But it simplifies the security model.

The agent can generate unexpected requests.

It can misunderstand context.

It can receive manipulated data from the target environment.

It can attempt actions outside the intended workflow.

Therefore:

Never rely exclusively on the agent to enforce the rules governing the agent.

The backend must independently enforce:

Scope.

Tools.

Execution phases.

Approval requirements.

Prompt instructions are useful.

Server-side enforcement is necessary.

What I would improve next

Dark Spear is an ongoing project.

The architecture is designed to evolve.

The areas I am most interested in exploring next include:

Better evidence correlation.

More structured attack path visualisation.

Improved agent decision auditing.

Stronger reproducibility controls.

More lab validation scenarios.

Better false-positive analysis.

Additional enterprise reporting workflows.

The interesting part of building an AI-assisted security platform is that the architecture keeps forcing new questions.

How much autonomy is useful?

Where should human approval happen?

What should the agent be allowed to infer?

What must be enforced independently?

Those questions are more important than simply connecting an LLM to a terminal.

Final thoughts

The future of automated pentesting will probably include AI agents.

But autonomous execution is not automatically better automation.

In professional security assessments, control and accountability matter.

A useful architecture should combine:

Deterministic automation where reproducibility matters.

AI reasoning where exploration adds value.

Server-side enforcement for scope and execution.

Human approval for meaningful decisions.

Evidence-backed findings.

Reporting that connects technical results to business and governance requirements.

That is the design direction behind Dark Spear.

Not an auto-pwn.

A controlled, evidence-driven platform for authorised security assessments.

The code is available under the MIT license:

Dark Spear — Yoandy Ramírez Delgado

The project is intended exclusively for authorised security testing, controlled laboratories and engagements with explicit permission.

Monté un mini-SOC en mi portátil con honeypot, Wazuh e IA local — proyecto del Máster en Evolve

Yoandy Ramirez Delgado — Sat, 16 May 2026 15:34:13 +0000

Monté un mini-SOC en mi portátil con honeypot, Wazuh e IA local — proyecto del Máster en Evolve

Cuando empecé el proyecto final del Máster en Ciberseguridad en Evolve, tenía claro qué no quería: otro informe PDF que nadie vuelva a abrir. Quería algo que se viera en una entrevista en treinta segundos — logs reales, alertas que suben solas y una interfaz donde un analista pueda trabajar sin pelearse con veinte pestañas.

Eso acabó siendo Valhalla SOC: un centro de operaciones pequeño pero completo, montado con Docker en local, sin mandar datos a la nube.

El problema que quería resolver

En muchos laboratorios de seguridad practicas con PCAPs viejos o ejercicios ya masticados. Yo quería el ruido de fondo de internet: bots que prueban root/admin, descargas con wget, intentos de persistencia… pero en un entorno controlado, en mi máquina.

La idea era encadenar tres piezas:

Una trampa que parezca un servidor SSH de verdad.
Un SIEM que clasifique lo que pasa ahí.
Una capa de análisis que no dependa de APIs de pago ni de subir alertas a terceros.

El entorno: de dónde salen los datos

No hay un CSV de Kaggle detrás. El “dataset” son los propios atacantes (y el contenedor de laboratorio que lanza ataques automáticos).

Cowrie escucha en el puerto 2222 y finge ser OpenSSH. Todo queda en JSON: logins, comandos, descargas.
Wazuh lee esos logs, aplica decoders y reglas propias (fuerza bruta, malware, reverse shell, etc.) y los mapea a MITRE ATT&CK.
OpenSearch (indexer de Wazuh) guarda el histórico para consultas y dashboards.
Ollama en el host, con qwen2.5-coder:7b, recibe las alertas más graves y devuelve un resumen corto — dos frases, sin novela.

Encima monté un frontend + API (React y FastAPI) para tickets, workspace tipo kanban, mapa de amenazas, integración con VirusTotal y chat entre operadores. No sustituye a Wazuh Dashboard; lo complementa para el día a día del analista.

El reto técnico no fue solo “que funcione”, sino que aguante en Windows (sí, con Docker Desktop, Ollama fuera del compose y el clásico lío del event loop de Python con PostgreSQL async). La primera vez que levanté todo, el backend moría al arrancar y el frontend devolvía 500. Horas perdidas hasta entender que uvicorn en Windows necesita un bucle distinto. Cosas que no salen en los diagramas de arquitectura.

Cómo lo monté (sin saltarme pasos)

Infra con Docker Compose: Wazuh manager + indexer + dashboard, Cowrie, Postgres para la app, nginx como puerta de entrada, y un contenedor “attacker” para generar ruido en el laboratorio.
Reglas a medida en wazuh_config/rules/ — por ejemplo detección de fuerza bruta (varios fallos en ventana corta), login exitoso en honeypot, descargas con wget/curl, o intentos de desactivar el firewall.
Integración Ollama vía script en wazuh_config/integrations/custom-ollama.py: solo alertas de nivel ≥ 5, prompt acotado, respuesta indexada como insight adicional.
Backend con autenticación por cookie httpOnly, CSRF en mutaciones, auditoría de acciones y sincronización de alertas Wazuh → tickets.
UI con tema oscuro/claro, workspace de incidentes, SIEM propio consultando OpenSearch, y runbooks enlazados a tickets.

Para levantarlo en limpio: clonar repo, copiar .env.example a .env, ollama pull qwen2.5-coder:7b, docker compose up -d --build, y los scripts de dashboards/monitores. En el README dejo el detalle; no voy a repetir aquí quince comandos.

Resultados que sí se pueden enseñar

En una sesión de laboratorio típica (con el honeypot expuesto y el atacante automático corriendo) el sistema se comporta como un SOC en miniatura:

Miles de eventos en el dashboard de Cowrie — en mis pruebas llegué a ver más de 8000 alertas críticas en la vista dedicada, con timeline y top de IPs.
Reglas disparando técnicas MITRE concretas (T1110 fuerza bruta, T1105 transferencia de herramientas, T1059 ejecución, etc.).
Monitores en Wazuh cada pocos minutos para brute force, malware, reverse shell, persistencia en crontab…
Tickets creados desde alertas Wazuh, asignables y movibles por estados en el workspace.
Respuestas de Ollama del estilo: “están bajando un script desde una IP externa; tratar como intento de implant” — útil como primera lectura, no como veredicto legal.

Capturas que recomiendo subir en el artículo (están en el repo bajo docs/img/):

Dashboard Cowrie en tiempo real (04-cowrie-honeypot.png)
Cobertura MITRE en agente (02-agente-mitre.png)
Pantalla de login de Valhalla (11-login-valhalla.png)

Lo que me llevé (y lo que cambiaría)

Aprendizajes:

Un honeypot enseña más en una tarde que una semana leyendo solo teoría de SIEM.
La IA local tiene sentido en SOC si acotas el prompt y el modelo; no hace falta un LLM gigante para resumir una alerta.
La parte dura es la integración (certificados Wazuh, rutas de logs, webhooks, CORS, CSRF), no el CSS del login.

Si lo retomara mañana:

Endurecería producción (webhooks firmados, websockets autenticados, cero contraseñas por defecto).
Documentaría un “modo demo” de un solo comando para corrección en clase.
Separaría más el módulo honeypot del repo principal — ahora todo vive junto y pesa, pero para el máster tenía sentido mostrar el stack entero.

Hice además una auditoría de seguridad sobre el propio código (auth, API, Docker) y salieron hallazgos serios — buena lección: montar un SOC no te convierte automáticamente en secure by default.

Código y cómo reproducirlo

Todo el proyecto está en GitHub, público, con README paso a paso, manual de usuario y capturas:

Repositorio: github.com/heindall92/Proyecto-Master-Ciberseguridad-Evolve-Yoandy

git clone https://github.com/heindall92/Proyecto-Master-Ciberseguridad-Evolve-Yoandy.git

Stack principal: Python · FastAPI · React · Docker · Wazuh · Cowrie · Ollama · PostgreSQL · OpenSearch (TypeScript ~60 % del código según GitHub).

Proyecto académico desarrollado durante el Máster en Ciberseguridad de Evolve. Si estás montando tu primer laboratorio SOC en casa, espero que te ahorre algún tropezón con Docker — o al menos que sepas que no eres el único al que el backend le devuelve 500 un domingo a las once de la noche.

¿Has montado honeypots en clase o en casa? Me interesa saber qué regla fue la primera que te hizo decir “vale, esto ya parece un SOC de verdad”.