DEV Community

Cover image for IPI-Scanner: Detecting Indirect Prompt Injection Attacks Before Your LLM Reads Them
Amit Gupta
Amit Gupta

Posted on

IPI-Scanner: Detecting Indirect Prompt Injection Attacks Before Your LLM Reads Them

#security #ai #opensource #devops

IPI-ScannerAn open-source security tool for RAG pipelines and agentic AI systems

The Problem: The Silent Attack Vector

You've probably heard about prompt injection attacks. But here's what most people don't realize: 80% of prompt injection attacks are indirect.

They don't target your prompt. They target your data.

An attacker poisons a document that your RAG system later retrieves. When your LLM reads it, hidden instructions execute silently. No alerts. No warnings. Just compromised output.

Real Examples

  • EchoLeak: Malicious email to a Copilot user leaked passwords via invisible instructions
  • HashJack: URL fragments with hidden instructions steered AI summaries
  • Perplexity Comet: Reddit posts with invisible text exfiltrated user data
  • CVE-2025-53773: GitHub Copilot RCE via PR description injection

The cost? $2.3 billion in global losses (2025). OWASP lists prompt injection as the #1 vulnerability in LLM systems.

The Solution: IPI-Scanner

I built IPI-Scanner – an open-source tool that detects indirect prompt injection attacks in documents before they reach your LLM.

Think of it as a security checkpoint for your data pipeline.

pip install ipi-scanner
ipi-scan document.pdf
Enter fullscreen mode Exit fullscreen mode

How It Works

IPI-Scanner uses a 3-tier detection system:

Tier 1: Pattern Matching (Fast)

  • 50+ regex patterns across 15 attack categories
  • Detection in <100ms per document
  • 60-80% accuracy
  • No API calls needed

Tier 2: Semantic Analysis (Accurate)

Coming in v0.2

  • Claude-powered analysis for borderline cases
  • 85%+ accuracy

Tier 3: Simulation (Proof)

Coming in v0.3

  • Proves attacks actually execute
  • 90%+ accuracy

What It Detects

Critical (40 points each)

  • Data exfiltration attempts
  • Credential/API key extraction
  • Sensitive file access requests

High (25 points each)

  • System prompt override
  • Context manipulation
  • Authentication bypass

Medium (10 points each)

  • URL fragment injection
  • Hidden/steganographic instructions
  • Policy override attempts
  • Social engineering

Low (5 points each)

  • Tool execution manipulation
  • Memory poisoning
  • Citation injection

Real Attack Validation

IPI-Scanner has been tested against real CVEs from 2025-2026:

Attack Status
EchoLeak ✅ Email-based data exfiltration
HashJack ✅ URL fragment injection
Perplexity Comet ✅ Invisible text injection
CVE-2025-53773 ✅ System prompt override
Google Gemini Calendar ✅ Invite-based manipulation
ChatGPT Google Drive ✅ File extraction
Zero-Click MCP RCE ✅ Metadata poisoning

Detection rate: 85%+ on known attacks
False positive rate: <5% on benign documents

Usage Examples

Scan a single file: 

ipi-scan document.pdf
Enter fullscreen mode Exit fullscreen mode

Scan a directory: 

ipi-scan ./documents/ --recursive
Enter fullscreen mode Exit fullscreen mode

Get JSON output (for automation): 

ipi-scan file.pdf --output json
Enter fullscreen mode Exit fullscreen mode

Generate HTML report: 

ipi-scan ./docs --output html --output-file report.html
Enter fullscreen mode Exit fullscreen mode

With context (RAG pipeline): 

ipi-scan file.pdf --context rag
Enter fullscreen mode Exit fullscreen mode

In Python: 

from ipi_scanner import Scanner

scanner = Scanner()
result = scanner.scan_file('document.pdf')
print(result['risk_assessment']['score'])  # 0-100 risk score
Enter fullscreen mode Exit fullscreen mode

Risk Scoring

Results are scored 0-100 with context awareness:

  • 75-100 (Red): BLOCK - Do not feed to your LLM
  • 50-74 (Orange): REVIEW - Check before RAG ingestion
  • 25-49 (Yellow): CAUTION - Monitor for suspicious behavior
  • 0-24 (Green): SAFE - Proceed normally

Context Multipliers

Risk increases based on deployment context:

  • Untrusted source: 1.3x
  • RAG pipeline: 1.5x
  • Agent with tool access: 2.0x
  • Agent with API access: 2.5x

Why Open-Source?

IPI detection is too important to be proprietary. Organizations need to:

  • Understand how detection works
  • Audit the patterns
  • Contribute improvements
  • Build on it

IPI-Scanner is MIT-licensed and welcomes contributions.

What's Next?

v0.2.0 (coming soon)

  • Claude semantic analysis (Tier 2)
  • Multi-language support
  • Custom pattern loader

v0.3.0 (roadmap)

  • Simulation-based validation (Tier 3)
  • MCP server integration
  • Real-time monitoring

Get Started 

pip install ipi-scanner
ipi-scan --help
Enter fullscreen mode Exit fullscreen mode

Links:

Questions?

Drop them in the comments. Happy to discuss:

  • Detection accuracy
  • Attack patterns
  • Integration strategies
  • Contributing guidelines

This is v0.1.0, so feedback is invaluable as we grow.

Happy scanning! 🔒

Top comments (1)

Collapse
 
ali_muwwakkil_a776a21aa9c profile image
Ali Muwwakkil •

Prompt injection attacks often go unnoticed until they cause significant damage. In our experience with enterprise teams, a surprising insight is that building robust RAG architectures requires focusing on data hygiene as much as on the LLM's capabilities. Regularly auditing your data sources and implementing dynamic filtering in RAG pipelines can prevent many indirect attacks before they even reach your LLM. Think of it as a proactive defense layer that complements tools like IPI-Scanner. - Ali Muwwakkil (ali-muwwakkil on LinkedIn)