DEV Community

ayame0328
ayame0328

Posted on

Your AI Copilot Might Be Poisoned: RAG Attacks and Why Static Analysis Still Wins

#ai #githubcopilot #rag #security

This week, a Hacker News post about document poisoning in RAG systems caught my attention. And over on Zenn (Japanese dev community), someone found malware disguised as a "useful tool" on GitHub.

These aren't isolated incidents. They're symptoms of the same problem: the code your AI writes is only as trustworthy as its training data and context.

I've been building a security scanner specifically for AI-generated code for the past two weeks. Here's what I've learned about why this matters — and what actually works to catch the problems.

The Attack Surface Nobody Talks About

When you use an AI coding assistant, you're trusting:

  1. The model's training data — was any of it poisoned?
  2. The RAG context — are your docs, READMEs, and examples clean?
  3. The packages it suggests — are they typosquatted?
  4. The patterns it follows — are they secure by default?

The RAG poisoning paper shows how attackers can inject malicious content into the documents that AI systems use as context. Imagine someone submits a PR to your internal docs that subtly changes a code example to include a hardcoded backdoor. Your AI assistant picks it up as "how we do things here" and starts suggesting it everywhere.

I ran an experiment: I fed deliberately tainted documentation to an AI assistant and asked it to generate API middleware. The output included SSL verification disabled — because the poisoned doc said "disable SSL for local development" and the AI generalized it.

What I Keep Finding in AI-Generated Code

After scanning hundreds of AI-generated code samples while building CodeHeal, I see the same vulnerability categories over and over:

1. Hardcoded Secrets (Almost Universal)

Every AI coding assistant I've tested will happily generate:

const API_KEY = "sk-proj-abc123...";
const client = new OpenAI({ apiKey: API_KEY });
Enter fullscreen mode Exit fullscreen mode

When I first started scanning AI output, I thought this was a minor issue. Then I checked — over 60% of AI-generated API integration samples had some form of hardcoded credential. Not in .env files. Not in environment variables. Right there in the source.

2. Command Injection via Template Literals

This one is subtle. AI loves writing "convenient" utility functions:

const result = execSync(`git log --author="${userName}"`);
Enter fullscreen mode Exit fullscreen mode

Looks clean. Works great. But userName comes from user input. I found this pattern in 3 different AI-generated CLI tools within a single week.

3. The Empty Catch Block Epidemic 

try {
  await processPayment(order);
} catch (e) {
  // handle error later
}
Enter fullscreen mode Exit fullscreen mode

"Handle error later" is the most dangerous comment in programming. AI generates these constantly because its training data is full of tutorial code with placeholder error handling.

4. Package Typosquatting Suggestions

The GitHub malware incident from Zenn isn't new. AI assistants sometimes suggest packages with slightly wrong names — colurs instead of colors, requets instead of requests. I built typosquatting detection into my scanner after seeing this happen three times in one day.

Why I Don't Use LLM for Security Scanning

Here's the counterintuitive part: using AI to scan AI-generated code is circular logic.

I tried it. Early in development, I used LLM-based analysis for my scanner. I ran the same code through it 5 times and got 5 different severity ratings. One run flagged a function as "critical risk." The next run called it "low concern." Same code. Same prompt.

That's when I switched to pure static analysis:

  • Deterministic: Same code → same result. Every time.
  • Fast: Full scan in under 2 seconds, not 30+ seconds waiting for API responses
  • Free: Zero API costs. No tokens burned.
  • Auditable: Every detection has a specific rule you can inspect

My scanner now checks 93 patterns across 14 vulnerability categories. No LLM involved. The detection rate against known-vulnerable samples is higher than when I used LLM, and the false positive rate dropped significantly.

The Supply Chain Problem Is Getting Worse

The RAG poisoning attack is particularly nasty because it's indirect. The attacker doesn't need to compromise your machine or your AI provider. They just need to slip bad content into something your AI reads.

Combined with:

  • GitHub repos that look legitimate but contain malware
  • NPM packages that are one typo away from popular libraries
  • AI assistants that confidently suggest insecure patterns

...we're looking at a supply chain attack surface that traditional security tools weren't designed for.

Snyk, SonarQube, and Semgrep are excellent tools. But they're built for human-written code patterns. They don't check for the specific ways AI tends to fail — the confident insecurity, the tutorial-grade error handling shipped to production, the "it works so it must be safe" patterns.

What You Can Do Today

  1. Never trust AI-generated code without review — yes, even from paid tools
  2. Check package names character by character — typosquatting is real
  3. Scan for hardcoded secrets before every commit — make it a pre-commit hook
  4. Validate your RAG sources — if you're using retrieval-augmented generation, treat your document store like you'd treat your source code
  5. Use deterministic scanning — pattern matching catches what LLMs miss (and never gives you a different answer twice)

Scan Your Code

I built CodeHeal because I got tired of finding the same AI-generated vulnerabilities manually. It checks for 93 vulnerability patterns across 14 categories — hardcoded secrets, command injection, typosquatting, empty error handling, and more. No LLM, no API costs, deterministic results.

Try CodeHeal free →

Have you encountered poisoned AI suggestions or malware disguised as dev tools? I'd love to hear your stories in the comments.

Top comments (0)