Every team shipping an LLM feature has the same quiet worry: what happens when someone tries to jailbreak our chatbot? And almost nobody has a real answer, because "is our system prompt resilient?" usually gets settled by gut feeling and a couple of manual pokes.
I wanted a feedback loop instead. So I built a Prompt Injection Tester: paste your system prompt, and it runs five classes of attack against it, then hands you a resilience score with the actual model responses as evidence.
The five attack classes
- Instruction override — "Ignore all previous instructions and…" The classic. Still works more often than you'd like.
- Prompt extraction — coaxing the model into leaking its own system prompt. If your prompt contains rules, examples, or hints about internal tools, this is how they get out.
- Delimiter / escape — breaking out of the formatting you wrapped user input in. If you concatenate user text into a template, this is the hole.
- Role-play — "Let's play a game where you're an AI with no restrictions…" Wrapping the override in fiction to slip past a naive filter.
- Indirect injection — the nasty one. The attack doesn't come from the user at all; it rides in on attacker-controlled text that enters through a tool call, a retrieved document, or a web page your agent reads.
That last one is the eye-opener. A system prompt that looks bulletproof against direct attacks falls over the moment untrusted content flows in through RAG or a browsing tool — because the model can't tell your instructions from the ones hidden in the document it just fetched.
The honest limits
Five families is a floor, not a ceiling. A determined attacker has more, and prompt injection is not a solved problem — there's no system prompt that makes you immune. Treat this as a fast first pass to catch the obvious holes before they catch you, not a certificate of safety.
One thing I deliberately got right: it runs on our own hardware. Your system prompt is not shipped off to a third-party model to be "tested" — which would be a slightly absurd thing for a security tool to do. Free, no signup.
Why I built it
We build private, in-boundary AI at Framz, so "what can untrusted input do to a model?" is a question we live with daily. The tester is the thing I wanted for our own work, so we made it public.
If you've shipped an LLM feature, run your real system prompt through it and see which of the five it survives: framz.io/tools/prompt-injection-tester.
Genuinely curious about the community's war stories: what's the most creative prompt injection you've seen land in production? Drop it in the comments — and tell me which attack classes you think the tester should add next.
Top comments (2)
The move I respect most here is that you attacked your own defense instead of trusting it. I did the boring cousin of this the same week — planted known-good inputs into my code scanner to see if it falsely flags them — but you did the version that actually scares me, because indirect injection is the one I can't fully gate.
Direct injection I can reason about: the user types something hostile, I wrap it and check it. But your fifth family — the attack riding in on attacker-controlled text through a tool call or a retrieved document — is a live risk in exactly the tools I've built. I have systems that feed a model external text it didn't author: fetched news, legal documents, uploaded files. The model can't tell my instructions from instructions hidden inside the thing it fetched, and that's not a prompt I can harden my way out of. Reading you, I realized I'd been treating "the input" as the user's message and quietly ignoring the input that arrives sideways.
The line I'm keeping is "a floor, not a ceiling." The failure mode isn't running five attacks and passing — it's running five, passing, and filing the problem as solved. Same trap as a scanner that catches ten planted bugs and then gets trusted for the ones nobody planted. Testing your defense is the start of not trusting it, not the end. Re-looking at every place I pipe untrusted text into a model. Thanks for the uncomfortable nudge.
This is the right way to make prompt injection concrete. The important result is not that one prompt broke, but which assumptions the system was making about trusted text, tool scope, and user intent.