DEV Community: Andrei

I pulled 50 system prompts from public GitHub repos and tested them for prompt injection vulnerabilities. Average score: 3.7/100. 70% had zero defenses. The best score was 28/100. Here's the full breakdown by attack category.

Andrei — Mon, 16 Mar 2026 19:48:59 +0000

I Tested 50 AI App Prompts for Injection Attacks. 90% Scored CRITICAL.

Andrei ・ Mar 16

I Tested 50 AI App Prompts for Injection Attacks. 90% Scored CRITICAL.

Andrei — Mon, 16 Mar 2026 09:07:59 +0000

So I spent last week doing something slightly unhinged. I pulled 50 system prompts out of public AI app repos on GitHub — just sitting there in the code, plain text — and ran every single one through a prompt injection scanner.

The average score was 3.7 out of 100.

Median? Zero.

35 out of 50 had no defenses at all. Not weak defenses. Not "could be better" defenses. Literally nothing.

How I got here

Last week I published results from scanning 100 vibe-coded apps for the usual security stuff — XSS, exposed secrets, missing auth. That was bad enough. But while I was going through those repos, I kept tripping over the same thing: system prompts just... sitting there. Zero guardrails. Not even a basic "don't reveal your instructions" line. Raw instructions to an LLM with zero thought given to what happens when a user decides to be creative with their input.

I couldn't stop thinking about it. So I made it a project.

Grabbed 50 AI-powered apps from public GitHub repos — chatbots, coding assistants, productivity tools, API agents — extracted their system prompts, ran each one through a scanner that tests 10 attack categories based on OWASP LLM Top 10 (specifically LLM01: Prompt Injection).

The 10 categories:

System Prompt Extraction
Role Override
Delimiter Escape
Indirect Injection
Output Manipulation
Tool/Function Abuse
Context Window Overflow
Encoding Bypass
Social Engineering
Multi-turn Escalation

Score goes from 0 to 100. Higher is better. 100 means you're defended on every vector. Zero means the prompt might as well not exist.

The numbers

Metric	Value
Apps tested	50
Average score	3.7/100
Median score	0/100
Highest score	28/100
Apps scoring 0/100	35 (70%)
Apps scoring ≤10/100	43 (86%)
Apps scoring ≤20/100	47 (94%)
CRITICAL severity	45 (90%)
HIGH severity	5 (10%)

Nobody passed. The best score across all 50 apps was 28/100, which is still HIGH severity. Still a fail.

90% got rated CRITICAL.

What CRITICAL actually looks like in the wild

A code interpreter — Score: 0/100

The entire system prompt: "write Python code to answer the question"

162 characters. That's the whole thing. No role boundaries, no output restrictions, nothing. You could tell it to ignore its instructions and recite limericks and it would just... do that. You could ask it to dump its own prompt and it'd hand it right over. I keep calling these "vulnerabilities" but there's nothing there to be vulnerable. It's a void.

A Google Sheets integration — Score: 0/100

It connects an LLM to Google Sheets. Zero prompt injection defenses. So any cell value in your spreadsheet could contain an injection payload. Someone shares a spreadsheet with you, you open it with this tool, and now a cell in row 47 is telling the LLM what to do instead of your system prompt. Your spreadsheet is the attack surface. Wild.

Then there was a subscription tracker. Also zero. Its entire security posture was format instructions — telling the model what shape the response should be. That's it. The whole defense. Against an attacker who literally just has to type "ignore previous formatting."

A Cloudflare API agent — Score: 5/100

An AI agent that talks to the Cloudflare API. Five points out of a hundred. It had some structure — enough to not score zero, I guess — but nothing that would slow down even a lazy attacker. This thing has API access. To your infrastructure. Five points.

The "best" prompt was still bad

A learning companion app — Score: 28/100

Highest score in the dataset. Had some role definition, some behavioral constraints. Enough to block the most obvious "ignore all previous instructions" stuff. But 28/100 means most attack categories still got through — role override, encoding bypass, multi-turn escalation, all still worked fine.

28 was the ceiling. The best anyone managed. Not close to good enough.

A terminal assistant — Score: 16/100

This one's kind of funny (in a grim way). It got 16 points not because anyone was thinking about injection defense, but because its output format restrictions happened to accidentally block one attack vector. Couple other apps in the dataset had this too — a few accidental points from constraints that were never meant to be security measures. Accidental security is not a strategy.

Why this keeps happening

Most devs building AI apps don't think about prompt injection because the prompt doesn't feel like a security boundary. It feels like config. You write "You are a helpful assistant that..." and move on to the actual code. The interesting code. The UI, the API integration, the database schema. The prompt is an afterthought — last thing you write before you ship.

I get it.

But that prompt is the only thing separating user input from model behavior. It IS the security boundary, whether it looks like one or not. Prompt injection is OWASP LLM01 for a reason — it's the most common vulnerability class in LLM apps and the easiest to pull off.

70% of the apps I tested had zero defense against it. Not "weak." Zero.

And these apps aren't toys. People are shipping AI tools that connect to APIs, read files, access databases, send emails. The prompt is the one barrier between a malicious input and all those capabilities. In 35 out of 50 cases, there was no barrier.

What you can actually do about it

Not gonna pretend this is simple — prompt injection defense is hard and the attacks keep changing. But there are basics, and almost every app in this dataset skipped all of them.

Start with role anchoring. Define what the model can and can't do. Repeat it. Not just a line at the top — reinforce it throughout the prompt. Models have short attention spans (sort of) and a single instruction at the beginning gets drowned out by a long conversation. Pair that with input/output boundaries — use delimiters, tell the model explicitly that user input is data, not instructions. Will a determined attacker try to escape those delimiters? Sure. But you've moved from "zero effort to exploit" to "has to actually think about it," which filters out a lot.

Instruction hierarchy — almost nobody does this and I don't get why. Tell the model explicitly: system instructions beat user input. Always. If there's a conflict, system wins. Put it in those exact words. I've seen maybe two prompts out of 50 that even attempted this.

Then there's the boring-but-necessary layer: refusal patterns and output validation. On the prompt side, tell the model to refuse if someone tries to change its behavior or extract its instructions. On the code side — and this part isn't even LLM-specific — don't blindly trust model output before you hand it to a tool or API. You already sanitize user input (right?). Same thing here.

You won't be bulletproof after this. But you'll go from 0/100 to somewhere defensible. The scanner I built also spits out a hardened version of your prompt after each scan — takes your original instructions and wraps them with these patterns so you don't have to figure out the wording yourself.

Why I built this

I'm a solo indie dev. I built VibeWrench because I kept running into the same security gaps in AI-generated and AI-powered apps and nobody was making it easy to catch them. The prompt injection scanner is one piece of it — paste your system prompt, get scored on all 10 OWASP LLM01 categories, see exactly which attack vectors work against you, get a hardened prompt back.

Free to scan. No signup for a basic scan.

And look — I'm not trying to dunk on anyone whose repo ended up in this dataset. Most of these are side projects, experiments, people learning. I've shipped dumb stuff too. But the patterns I see in hobby repos are the exact same patterns showing up in production apps that handle real user data. Same "just tell the AI what to do" approach. Same empty defenses.

If you're building anything with an LLM — especially if it touches real data or calls real APIs — test your prompt. Takes five minutes. Beats being the example in someone's next blog post about AI security.

vibewrench.dev

Questions about methodology? Think my scoring is wrong? Drop a comment, I'll respond to everything.

— Andrei K.

I Scanned 100 Vibe-Coded Apps for Security. I Found 318 Vulnerabilities.

Andrei — Mon, 09 Mar 2026 19:46:28 +0000

In early March I scanned 100 apps built with Lovable, Bolt.new, Cursor, and v0.dev.

I wasn't looking for obscure zero-days. I was looking for the basics — missing CSRF protection, exposed API keys, no authentication. The stuff that gets you hacked on day one.

65% had security issues. 58% had at least one CRITICAL vulnerability.

The Numbers

I ran automated security scans on 100 public GitHub repos built with AI coding tools. Here's what I found:

Finding	% of Apps	Severity
Missing CSRF protection	70%	🔴 CRITICAL
Exposed secrets or API keys	41%	🔴 CRITICAL
Poor error handling	36%	🟡 WARNING
Missing input validation	28%	🟡 WARNING
No authentication on endpoints	21%	🔴 CRITICAL
Missing security headers	20%	🟡 WARNING
XSS vulnerabilities	18%	🔴 CRITICAL
Exposed Supabase credentials	12%	🔴 CRITICAL

318 total vulnerabilities. 89 of them CRITICAL.

Average Security Score: 65/100 — a D grade.

That might sound "okay" until you realize 65% of apps scored below 70 (passing), and nearly half (47%) got a D.

Platform Breakdown

Platform	Avg Score	% With Issues	% With CRITICAL
Lovable	58/100	79%	72%
Bolt.new	66/100	60%	57%
v0.dev	71/100	60%	20%
Cursor	75/100	50%	42%

These scores reflect individual apps, not the platforms themselves. The tools generate what you ask for — security is on you.

Lovable apps were the most vulnerable — 10 out of 38 had Supabase credentials exposed directly in their code.

The Scariest Find

One Lovable app had its Supabase keys — including the service role key — committed to the repo in a .env file. Not the anon key. The service role key. With that key, anyone can bypass Row Level Security and read every row in every table.

The developer had no idea. They'd built the app in Lovable, it worked, they deployed it. Lovable didn't warn them. Why would it? It's a code generator, not a security auditor.

Why This Happens

AI coding tools are incredible at generating working code. But "working" ≠ "secure."

When you tell Lovable "connect to Supabase," it generates code that queries the database. It works. But it might commit the service key to source control, because the AI optimized for "make it work," not "make it safe."

This isn't Lovable's fault. Or Bolt's. Or Cursor's. They're doing exactly what you asked — writing code that works. But nobody asked "also make it secure."

That's where the gap is.

And it's not just Lovable or Bolt. Claude, ChatGPT, Cursor, Copilot — every AI code generator optimizes for "working," not "secure." I've built apps with Claude Code myself and found the same issues. This is an industry-wide problem, not a platform-specific one.

What I Built

After seeing these results, I built VibeWrench — a tool that scans vibe-coded apps for security holes, speed issues, SEO problems, and more.

It's designed for non-programmers. Instead of "Missing CSP header on response object," it says "Your website doesn't tell browsers to block suspicious scripts — like leaving your front door unlocked."

For every problem it finds, it gives you a Fix Prompt — a copy-paste prompt for Cursor or Claude that fixes the issue automatically.

Your first scan is free. No signup required.

What it checks (18 tools):

Security: exposed keys, XSS, CSRF, missing auth, input validation, security headers
Prompt Injection Scanner: test your AI app's system prompt against 10 attack categories (OWASP LLM01)
Speed: Lighthouse analysis in plain English — why your site takes 8 seconds
SEO: missing meta tags, no sitemap, "Vite App" as page title (63% of vibe-coded apps fail basic SEO)
Accessibility: WCAG 2.1 compliance, missing alt tags, form labels
Legal: GDPR-ready privacy policy and terms (5 questions → done)
And more: error translation, deploy guides, code explainer, cost forecasting...

The Uncomfortable Truth

If you built an app with AI and deployed it without a security check — you probably have at least 3 of the issues above. The average app had 3.2 findings. I'm not saying this to scare you (ok, maybe a little). I'm saying this because it's fixable. In most cases, 10 minutes with the right prompts and you're good.

Try it: vibewrench.dev — paste your GitHub URL or site URL, get results in 30 seconds.

I'm a solo developer building tools for the vibe coding community. If you have questions or found something weird in your scan, drop a comment — I read everything.