π° Originally published on Securityelites β AI Red Team Education β the canonical, fully-updated version of this article.
π€ AI/LLM HACKING COURSE
FREE
Part of the AI/LLM Hacking Course β 90 Days
Day 6 of 90 Β· 6.6% complete
β οΈ Authorised Targets Only: Testing for sensitive information disclosure in LLM applications must only be performed against systems you have explicit written authorisation to test. If you discover real credentials, PII, or sensitive data during authorised testing, document it without accessing or using the disclosed information beyond what is necessary to confirm the finding. SecurityElites.com accepts no liability for misuse.
The most impactful finding from the first AI red team engagement I ran was not the prompt injection. It was what the prompt injection exposed. The system prompt of the clientβs customer service AI contained four things that had no business being there: an AWS access key, the internal hostname of their primary database, the email addresses of three senior engineers, and a sentence confirming that the AI had read-write access to customer records. The developer had put all of that in the system prompt to give the AI βcontextβ about its environment. They had not considered that the system prompt could be extracted.
LLM02 Sensitive Information Disclosure is what you find after the injection from Days 4 and 5 succeeds. It is the category that determines whether a High severity finding becomes Critical β not because of what the injection does technically, but because of what it reveals. Credentials that unlock real systems. PII that triggers GDPR breach notifications. Architecture details that map the entire backend for the next phase of the attack. Day 6 covers every LLM02 disclosure variant β where the data comes from, how to find it, and how to calculate the real business impact when you do.
π― What Youβll Master in Day 6
Identify all four LLM02 disclosure mechanisms and their distinct attack paths
Extract and validate credentials embedded in system prompts
Test for PII and training data memorisation with structured prompts
Identify cross-user data leakage via shared RAG and conversation context
Calculate regulatory impact and data classification for each disclosure type
Build the complete LLM02 finding with correct CVSS severity and breach notification implications
β±οΈ Day 6 Β· 3 exercises Β· Think Like Hacker + Browser + Kali Terminal ### β Prerequisites - Day 4 β LLM01 Prompt Injection β LLM02 findings typically follow a successful LLM01 injection; the payload library from Day 4 is the entry point - Day 5 β Indirect Prompt Injection β RAG-based LLM02 disclosure often arrives through the indirect injection surfaces from Day 5 - An understanding of basic credential formats β AWS access keys, connection strings, JWT tokens β to recognise what you find ### π LLM02 Sensitive Information Disclosure β Day 6 Contents 1. Four LLM02 Disclosure Mechanisms 2. System Prompt Credential Extraction 3. Training Data Memorisation β Surfacing PII 4. RAG and Context Window Data Exposure 5. Cross-Session and Cross-User Leakage 6. Calculating Impact β Regulatory and Business Severity In Day 5 you demonstrated how injection payloads reach the AI through documents, web pages, and emails. Today the focus shifts to what those injections reveal. LLM02 overlaps with LLM07 System Prompt Leakage at the system prompt layer, but extends significantly further β into training data, RAG knowledge bases, and cross-user session data. The distinction matters for your report: LLM07 is the mechanism, LLM02 is the impact.
Four LLM02 Disclosure Mechanisms
LLM02 manifests through four distinct mechanisms, each with a different attack path and a different mitigation. Understanding which mechanism produced a finding determines how you write the remediation section and what evidence you need for the report.
System prompt leakage. The developerβs instruction set is extracted via LLM01 injection or LLM07 extraction techniques. The system prompt often contains far more than intended β credentials, architecture details, employee names, internal URLs. This is the most consistently high-severity variant because developers routinely embed operational context in system prompts without considering extractability.
Training data memorisation. The model has encoded specific text from its training corpus in its weights. With the right prompt context β partial sequences that match memorised text β the model reproduces real data: email addresses, phone numbers, code snippets, even cryptographic keys that appeared in training data. This vector targets the model itself, not the application deployment.
RAG retrieval exposure. The RAG knowledge base contains sensitive documents that should not be accessible to the querying user. A carefully constructed query retrieves those documents via the vector search, and the model incorporates their content into its response. This is often an access control problem β the knowledge base was not partitioned by user permission level.
Cross-session leakage. Application-level conversation history storage creates cross-session or cross-user exposure. If conversation history is not properly isolated β if one userβs session data can influence another userβs context β information disclosed in one session can be retrieved in another. This is not a model-level vulnerability but an application architecture vulnerability that the modelβs context window makes exploitable.
π§ EXERCISE 1 β THINK LIKE A HACKER (20 MIN Β· NO TOOLS)
Classify LLM02 Findings and Calculate Regulatory Impact
π Read the complete guide on Securityelites β AI Red Team Education
This article continues with deeper technical detail, screenshots, code samples, and an interactive lab walk-through. Read the full article on Securityelites β AI Red Team Education β
This article was originally written and published by the Securityelites β AI Red Team Education team. For more cybersecurity tutorials, ethical hacking guides, and CTF walk-throughs, visit Securityelites β AI Red Team Education.
Top comments (0)