Most LLM eval frameworks score a single prompt and a single response. That's not really what ships to production anymore. What ships is an agent: a system that plans, calls tools, reads documents, opens browsers, and holds state across many turns. The place that system actually breaks is adversarial robustness, not benchmark accuracy — prompt injection through a retrieved document, tool-call confusion, a jailbreak that only lands after three turns of setup.
I built Argus to test agents the way an attacker actually would: as a black box, over the wire, against a live endpoint, with no source access. Apache 2.0, open for anyone to use or extend.
What it does
Argus ships with a library of 167 adversarial probes: hand-authored coverage of the OWASP LLM Top 10, scenarios pulled from public model system cards (crescendo, many-shot jailbreak, sleeper-agent triggers), browser-agent-specific attacks, and wrappers around NVIDIA's garak catalogue. On top of the static probes it runs algorithmic iterative attacks — TAP, PAIR, and black-box GCG — that adapt the prompt based on the target's responses.
It can talk to five kinds of targets out of the box: OpenAI-compatible APIs, Anthropic's native API, arbitrary HTTP endpoints via a templated adapter, gRPC services (with reflection-based auto-discovery), and full browser-driven agents through Playwright.
Findings are judged by an LLM (Haiku by default, escalating to Sonnet on high-severity hits) against per-probe rubrics, then written out as SARIF 2.1.0, JUnit XML, or HTML — so a scan can gate a CI pipeline through GitHub Code Scanning instead of just producing a report nobody reads.
Try it in five minutes, no API keys
The repo ships a deterministic local demo: a deliberately vulnerable FastAPI chatbot whose system prompt hides a few fake secrets, plus a runner that fires a curated probe set at it.
git clone https://github.com/gy15901580825/Argus
cd Argus
pip install fastapi httpx pydantic uvicorn slowapi pyyaml
PYTHONPATH=. python3 demo/run_local_demo.py
You'll see individual probes pass or leak in real time, then a summary attack-success rate. There's also a guardrail mode that puts a prompt-injection classifier in front of the same target, so you can compare the before-and-after numbers yourself.
Real numbers, not just a demo
RESULTS.md in the repo walks through a measured before/after run: adding an input-side guardrail (ProtectAI's deberta-v3-base-prompt-injection-v2, wrapped as "PromptGuard") drops the attack-success rate on the demo target from 22.9% to 2.9%, at roughly 275ms p50 latency overhead — and it documents the one bypass and twelve false positives we found, instead of hiding them.
Pointing it at your own agent
Targets are just a YAML file:
kind: openai_compat
base_url: "https://api.your-agent.example.com/v1"
api_key_env: AGENT_API_KEY
model: "your-agent-prod-v3"
Then run it with the CLI:
pip install argus-probe
argus-probe run --target my_target.yaml --probes owasp_,syscard_ --judge anthropic --report sarif > argus.sarif
For CI, there's a bundled argus-probe-action@v1 GitHub Action with a --block-on-critical flag to fail the build on high-severity findings.
Where it's at
Argus is pre-1.0. The CLI, REST API, probe library, judge harness, and report formats are stable; the orchestrator's internal dispatch contract may still shift. The web UI and SaaS surface are actively being built.
If you're doing any kind of AI security or red-teaming work, I'd genuinely like your feedback — probes that are missing, target adapters you need, report formats that don't fit your pipeline. Issues and PRs are welcome; the contributing guide has the repo layout and probe schema.
Top comments (0)