A single hands-on reviewer pointed Strix at every challenge in a standard web-security benchmark and watched it break almost all of them — then attach a working exploit to each one. The headline number, and its price tag, are worth unpacking before you run the tool yourself.
The 96% PoC yield: what the XBEN dataset revealed about Strix
Strix is an open-source agentic penetration-testing tool (repo usestrix/strix, Apache-2.0) that runs a target application dynamically and attaches a working proof-of-concept to every finding. In one practitioner benchmark against XBEN — a set of 104 real-world web-security challenges in CTF format — Strix solved 100 of 104 (~96%), averaging about 19 minutes per challenge at roughly $337 total, or ~$3.37 per challenge in LLM API spend.
Quick Answer: In a single-reviewer XBEN benchmark, Strix — an open-source agentic pentest tool — solved 100 of 104 web-security challenges (~96%) at about $3.37 each in API cost. Unlike a scanner, each finding ships with the exact request that reproduces the break, plus a suggested fix.
Treat that result as indicative, not authoritative: it comes from one hands-on reviewer, not a vendor-published or peer-reviewed study. What makes the yield notable is the mechanism behind it. Strix orchestrates a "Graph of Agents" — specialized reconnaissance/OSINT, exploitation, and post-exploitation subagents that run in parallel and share context, so one agent's discovery reshapes another's attack path . The payoff is validation over detection: each confirmed vulnerability arrives with the exact HTTP request or payload that reproduces it, plus remediation guidance . A Strix "finding" is a proven exploit, not a candidate awaiting manual triage.
The project first appeared in November 2025, hit #1 on GitHub Trending on July 3, 2026 (+2,137 stars in a single day), and has accumulated roughly 47k stars as of mid-2026 — though exact counts vary by source, so verify the repo directly.
Configuring Strix for an initial PoC on a disposable VM
Because Strix launches real exploits rather than passive scans, treat setup as a controlled lab exercise, not a casual install. The steps below take you from authorization to triaging a proven proof-of-concept against a deliberately vulnerable practice app.
Step 0 — Get written authorization and isolate the target. Strix executes live attacks against whatever you point it at, so running it against any system you do not own or lack explicit written permission to test is illegal, not a scan . Confine it to a VM or container you fully control before you run anything.
Step 1 — Install. Two methods circulate, and the divergence itself is worth flagging. The official site advertises a one-line bootstrap, while community write-ups cite a pipx install . Verify against the current usestrix/strix README before running either, since installer instructions have diverged across sources.
# official site
curl -sSL https://strix.ai/install | bash
# community write-ups
pipx install strix-agent
Step 2 — Set the model environment variables. Strix routes to providers LiteLLM-style, so you export the model and a key:
export STRIX_LLM=<model>
export LLM_API_KEY=<key>
Official docs recommend OpenAI GPT-5.4, with Anthropic Claude Sonnet 4.6 and Google Gemini 3 Pro also supported, plus Ollama or LMStudio for fully local inference .
Step 3 — Point it at a known-vulnerable practice app. Run against a local codebase or a running URL with the --target flag:
strix --target ./app-directory
Baseline its behavior against a deliberately vulnerable app such as DVWA or WebGoat before you aim it at anything you care about .
Step 4 — Triage the PoC output. Each confirmed finding ships with the exact request or payload that reproduces it, plus a suggested fix . Replay the reproducing request to confirm it, then validate the suggested patch in an isolated branch before merging.
Where Strix underperforms: non-determinism and per-exploit expenditure
Strix trades reproducibility for autonomy, and that trade-off is its main weakness. Because each run is driven by an LLM, the same target can surface different findings on different passes — there is no guarantee that run two reproduces run one. Nuclei sits at the opposite end: an identical YAML template against an identical endpoint returns identical output every time . For CI gates and regression checks, that determinism matters more than raw exploit depth.
Cost is the second operational variable. In the XBEN evaluation, Strix averaged roughly $3.37 per challenge across 104 challenges, totaling about $337 . Extrapolate that to a large production surface and API spend accumulates quickly, whereas Nuclei and OWASP ZAP are effectively free to run at any cadence . The math is easy to model before you commit budget:
solved = 100
total = 104
cost_each = 3.37
print(f"Strix solved {solved} of {total} real-world exploits - at ${cost_each:.2f} each")
print(f"success_rate={solved / total:.1%}")
print(f"total_cost=${solved * cost_each:.2f}")
That snippet is a verified illustration (it ran, exit 0) of the reported figures: a 96.2% solve rate at a $337 total . Wall-clock is the third constraint: Strix averaged about 19 minutes per challenge, while Nuclei sweeps thousands of endpoints with community templates in seconds .
None of this makes Strix a replacement for those primitives. Its own documentation credits Nuclei and Caido as foundational dependencies — Strix wraps template-based detection inside an agentic loop rather than rebuilding it . As AppSecEngineer frames it, the tool is "popular not because it finds more than ZAP or Burp, but because it proves them" (source: AppSecEngineer).
Strix, Nuclei, and ZAP as a complementary appsec trio
Treat the three as a pipeline, not a contest. Nuclei, ZAP, and Strix each own a different stage of the loop — fast detection, broad coverage, and proof — and a small team gets the most value by chaining them in that order rather than picking one. Nuclei (ProjectDiscovery) is deterministic and template-driven, with near-zero marginal cost per run; its 2025–2026 releases added AI-assisted template generation that drafts a detection template from a plain-language vulnerability description . That makes it the ideal CI gate for known-CVE and misconfiguration sweeps. OWASP ZAP — now stewarded by Checkmarx — offers the broadest open-source DAST surface with strong Jenkins and GitHub Actions hooks, but like every scanner it emits candidate findings that a human must validate . That manual validation step is exactly what Strix automates, attaching a reproducible exploit to each confirmed finding .
| Tool | Model | Cost per run | Best for |
|---|---|---|---|
| Nuclei | Deterministic templates | Near-zero | Fast known-CVE / misconfig CI gating |
| OWASP ZAP | Deterministic active/passive DAST | Near-zero | Broad coverage of your specific app |
| Strix | Non-deterministic, LLM-driven agents | ~$3.37/exploit on XBEN | Auto-validating high-severity candidates with a PoC |
A practical triage order: run Nuclei first for fast CVE gating, follow with a ZAP sweep for breadth, then point Strix at the highest-severity candidates to auto-validate each one with a working exploit before you file it. To evaluate all three honestly, run them against the same deliberately vulnerable app and compare four things: findings count, false-positive rate, whether each finding is actually proven, and wall-clock plus dollar cost . The takeaway: keep the cheap deterministic scanners as your always-on gate, and reserve Strix's per-run spend for the findings where a reproducible proof-of-concept saves the most analyst hours .
Frequently asked questions
Is the 100/104 XBEN result from an official Strix benchmark?
No. The 100 of 104 figure on the XBEN CTF dataset comes from a single practitioner's hands-on evaluation, not a peer-reviewed study or a vendor-published benchmark . It is useful signal, but treat it as indicative. Before relying on it, reproduce the run locally against a deliberately vulnerable target such as DVWA or WebGoat to establish your own baseline.
How much does running Strix cost compared to Nuclei or OWASP ZAP?
Nuclei and OWASP ZAP are free to run at any frequency because they are deterministic and carry near-zero marginal cost. Strix spends LLM API tokens on every run: the XBEN evaluation averaged $3.37 per challenge for a total of about $337 across 104 challenges. For an app with many endpoints, budget the spend explicitly and weigh it against the hourly rate of a manual penetration tester.
Does Strix replace Nuclei or OWASP ZAP?
No. Strix's own documentation credits Nuclei as a foundational open-source dependency, so it wraps template-based scanning inside a broader agentic loop rather than replacing it . The three tools are complementary: Nuclei for fast, template-driven detection of known CVEs, ZAP for broad DAST coverage of your specific app, and Strix for autonomous proof-of-concept generation on the findings that matter most .
Which LLM gives the best results with Strix?
The official documentation recommends OpenAI GPT-5.4 as the default model . Anthropic Claude Sonnet 4.6, Google Gemini 3 Pro, and providers such as Vertex AI, Bedrock, and Azure are also supported through LiteLLM-style routing, along with local models via Ollama or LMStudio . Local inference avoids API cost but may reduce accuracy on complex, multi-step exploit chains.
Can I run Strix in CI/CD the way I run Nuclei or ZAP?
It is technically possible but slow and expensive. At roughly 19 minutes and $3.37 per challenge, Strix is not suited to every-commit gates. Use Nuclei for CI gating and broad ZAP passes for scheduled coverage, and reserve Strix for targeted pre-release validation of high-severity candidates where a reproducible exploit saves the most analyst time .
Top comments (0)