DEV Community

Manh Liem
Manh Liem

Posted on

I built a zero-dependency LLM red-team kit: 15 probes, one command, a 5-minute report

#programming #aisecurityllm #ai #security

You shipped a chatbot. Has anyone actually tried to break it?

Most teams run a happy-path demo and call it secure. Adversarial behavior is the opposite: the model does exactly what it was told, just for the attacker. I keep getting asked how to check this without spinning up a framework, so I built something that takes one command.

What it does

The LLM Red Team Starter Kit runs 15 crafted attack prompts across 6 categories against any OpenAI-compatible endpoint and writes a Markdown report you can paste straight into a launch review:

  • Jailbreak — DAN, roleplay, persuasion-style bypass
  • Prompt injection — direct and indirect (delimiter tricks, instruction override)
  • System prompt extraction — direct asks, encoded lures, translation laundering
  • Safety bypass — multi-turn escalation, context pollution
  • Tool abuse — out-of-scope calls, argument injection (the rm -rf and exfiltration probes)
  • Encoding — base64, rot13 obfuscation of malicious payloads

Zero dependencies, by design

Pure Python 3.8+ stdlib. No pip installs, no config files, no agent framework. If you can run python3, you can run the scan:

unzip llm-red-team-starter-kit.zip
cd llm-red-team-starter-kit
python3 -m redteam --demo   # offline, no API key needed
python3 -m redteam --base-url https://api.openai.com/v1   --api-key $KEY --model gpt-4o-mini --report report.md
Enter fullscreen mode Exit fullscreen mode

Works with any /chat/completions endpoint — OpenAI, OpenRouter, local Ollama or vLLM. The --demo flag runs against a canned target so you can see the report shape before spending a single token.

Honest limitations

The verdicts are heuristic signal matches with a manual-review flag — the kit tells you what pattern fired and lets you decide, which is the honest way to do LLM red-teaming. A YES means "a keyword/signature matched; look at this". It does not mean "vulnerable, ship blocked." Every report carries a repro manifest (engine version, probe-corpus sha256, exact probe set, target, repro-hash) so a second person can re-run the same corpus + probes + target and compare verdicts.

That's the whole product: no black box, reproducible, reviewable.

Try it free first

If you'd rather not install anything, paste your agent's system prompt here and get a real 8-probe report in ~35 seconds, no account, no key:

https://llmrt-companion.manhliemcn4euwlu.workers.dev/review

Full kit (15 probes + repro manifest + 25 offline tests): https://manhliem.gumroad.com/l/czytp

For authorized security testing of systems you own or have written permission to test.

Top comments (0)