This is the second post in a new series on AI application security testing, following the structure of the OWASP AI Testing Guide's AI Application Testing category (AITG-APP).
The previous post covered prompt injection: testing whether attacker-controlled input can change an application's behavior in ways that violate its intended instructions or security constraints. This post focuses on a related but distinct question: whether an AI application exposes information that should remain protected.
We're covering three tests from the guide. Although all three involve information being exposed in an unintended way, they target different sources of information and different security boundaries.
| Test | The question it asks | Where the information comes from |
|---|---|---|
| AITG-APP-03 — Sensitive data leak | Can the application expose confidential information through its model, memory, training data, or connected tools? | Model, memory, training data, retrieved documents, connected tools |
| AITG-APP-04 — Input leakage | Can information supplied by a user be unnecessarily reproduced or exposed through the application? | The user's own input |
| AITG-APP-07 — Prompt disclosure | Can the application reveal its system or application instructions? | The application's instructions |
The distinction matters because the same apparent symptom can have very different causes. Effective testing therefore needs to establish where the information came from, whether the disclosure was unauthorized, and which security control should have prevented it.
Sensitive Data Leak (AITG-APP-03)
Sensitive data leakage occurs when an AI application exposes confidential information that should not be available to the requesting user or should not leave a protected system boundary.
The source of the information can vary. It may come from training data, conversation memory, retrieved documents, application context, databases, or connected tools. The testing approach therefore depends on what information the application can access and which users are authorized to see it.
A useful starting point is to identify what the application should protect and where that information enters the system. Testing can then examine whether the model or surrounding application exposes it through direct requests, contextual tasks, or unintended information disclosure.
Attack Patterns Worth Knowing
Several approaches can help testers identify whether sensitive information is exposed through an AI application.
Direct and contextual requests
Simply asking the model for credentials or confidential information is an obvious starting point. A more contextual test might ask the model to summarize internal communications or retrieve information from a document that happens to contain sensitive values.
The distinction matters because an application may refuse a direct request for a password while still exposing the same information through an apparently legitimate task, such as summarization or document analysis.
Persona and pretext framing
A request might frame the interaction as a developer debugging the application, an administrator investigating an error, or a researcher examining a system's configuration.
These scenarios can help determine whether the application relies too heavily on the model's interpretation of the user's claimed role or purpose. Authorization should be enforced by the application rather than inferred from a persuasive explanation in the prompt.
Tool and application reconnaissance
For agentic systems, testers can examine whether the model reveals available tools, sub-agents, tool parameters, or other operational details.
This information is not necessarily confidential, and revealing a tool description is not automatically a vulnerability. However, unnecessary disclosure can help an attacker understand the application's capabilities and construct more targeted follow-up attacks.
Testing should establish whether the disclosed information violates a confidentiality requirement or meaningfully weakens a security boundary.
Implementation details and error disclosure
Requests for stack traces, API endpoints, framework details, internal configuration, or recent errors can reveal information about the application's implementation.
Individually, some of these details may be harmless. Others can expose sensitive values, internal infrastructure, or information that should be restricted to administrators. The security impact depends on what is disclosed and who can access it.
The Unifying Theme
A sensitive data leak is not always a model revealing a password. Information about internal systems, retrieved records, or connected tools can also be relevant when its disclosure violates the application's intended security constraints.
Distinguish actual disclosure from speculation. A model describing a hypothetical environment does not prove that it has access to the corresponding credentials, files, or infrastructure.
Input Leakage (AITG-APP-04)
Input leakage examines whether information supplied to an AI application is exposed or reused in ways that violate the application's intended privacy or security boundaries.
Unlike a sensitive data leak, where the information may originate from a protected data source or connected system, this test starts with information provided through the input channel itself.
A practical test might use a synthetic API key, a fake password, or a clearly marked confidential string. The tester then checks whether the application:
- reproduces that value in a later response,
- exposes it to another user or session, or
- passes it to a component that should not receive it.
The important question is not simply whether the model can repeat the value. It is whether the application handles that value in accordance with its security requirements.
Common Test Patterns
Repeated-input testing
Provide a synthetic sensitive value and check whether the application reproduces it when asked directly or in a later turn.
Where relevant, test session boundaries and whether information can appear in another user's context. This helps distinguish ordinary repetition within a conversation from unintended exposure across users or sessions.
Normalization and obfuscation
Represent test values using alternative character encodings or Unicode variants, then evaluate whether the application's detection and redaction controls recognize the value.
For example, a test might use Unicode full-width digits instead of standard ASCII digits. A filter that relies on a narrowly defined character pattern may handle the two representations differently, while a downstream component may normalize them to the same value.
This is not a guaranteed bypass. The result depends on how inputs are normalized, inspected, and processed throughout the application.
Contextual variation
Change the surrounding request while keeping the underlying sensitive value the same.
For example, compare a direct request to reproduce a value with a request to summarize, transform, or explain the message containing it. The objective is to determine whether the application's protections depend too heavily on specific keywords or request formats.
Affix and formatting variations
Add unrelated text immediately before or after a synthetic sensitive value, or place the value within a larger structured input.
These variations can help identify filters that rely on particular positions, delimiters, or surrounding context. Their effectiveness depends on the implementation being tested.
Combined test cases
Combine several input variations to evaluate whether the application's controls continue to work when multiple techniques are used together.
For example, a test might combine Unicode variations with additional surrounding text and a contextual request. This can help establish whether different defensive layers work together or whether one layer depends on another recognizing a particular representation.
Testing Ground Rules
These tests should use synthetic values rather than real credentials or personal information. They should also distinguish between the model's response, application logs, downstream tool calls, and any cross-session exposure being tested.
A clean result has limits. A successful output test only establishes what happened in the observed interaction. It does not establish whether the provider retains inputs, whether those inputs are used for training, or how they are handled in application logs. Those questions require separate examination of the system's data-handling configuration and policies.
Prompt Disclosure (AITG-APP-07)
Prompt disclosure tests whether an AI application reveals system or application instructions that are intended to remain confidential.
These instructions may contain proprietary workflows, internal policy details, tool descriptions, or other implementation information. Their disclosure can help an attacker understand how the application is designed and identify additional attack paths.
However, a system prompt should not be treated as a security boundary in its own right. Secrets should not be embedded in instructions, and authorization should be enforced by the application rather than relying on the model to follow a prompt.
Testing should therefore establish both whether protected instructions can be reconstructed and whether their disclosure creates a meaningful security or operational impact.
Techniques Worth Testing
Direct and oblique requests
Simply asking the model to reveal its system prompt is an obvious starting point. More indirect requests might ask it to explain its guidelines, describe its operating instructions, or summarize the rules governing its behavior.
Oblique phrasing can sometimes produce different results from a direct request, depending on the model and the application's safeguards. Testers should evaluate both approaches rather than assuming that blocking one phrase prevents disclosure.
Context and clarification framing
A request might claim that the model's behavior is confusing and ask it to quote the instructions responsible for that behavior.
This tests whether the model treats a request framed as clarification differently from an explicit request for confidential instructions. The important question is whether the application exposes information that its security requirements say should remain protected.
Partial completion and reconstruction
A tester can examine whether supplying a suspected prefix of an instruction causes the model to continue with confidential text.
Language models generate text based on their learned behavior and the context they receive, including instructions and learned refusal behavior. Prefix completion may therefore reveal information in some configurations, but it does not guarantee disclosure.
Testing should also distinguish between the model producing a plausible continuation and actually reproducing the application's real system prompt.
Incremental extraction
Instead of requesting an entire prompt at once, a tester can examine whether the application reveals protected instructions in smaller pieces.
For example, successive requests might ask for individual instructions, a summary of particular rules, or additional details about previously disclosed text. The objective is to determine whether restrictions that apply to a single large request remain effective across multiple interactions.
Adversarial suffixes (a related research area)
Adversarial suffixes are specially constructed text appended to a prompt to influence a model's behavior. Some research has demonstrated that such techniques can induce undesirable outputs or weaken refusal behavior in particular configurations.
However, causing a model to produce an unwanted response is not the same as demonstrating system-prompt extraction. A prompt-disclosure finding requires evidence that protected instructions were actually revealed or reconstructed.
The broader testing principle is to vary the request format and interaction pattern rather than assuming that one refusal demonstrates that the system prompt is protected.
What Makes These Tests Difficult
The three tests share an important challenge: the model's visible response does not necessarily tell you where the information originated or which control failed.
Model access versus application access
A model may generate a plausible-looking secret without actually having access to it. Conversely, an application may retrieve confidential information and pass it into the model's context even when the model would not otherwise know that information.
Testers should establish whether a disclosed value is real, whether the application had access to it, and how it reached the response.
Conversation context and session boundaries
A model may repeat information because it remains available in the current conversation context. That behavior is different from exposing information across isolated sessions or retaining it beyond the intended lifetime.
Testing should account for conversation history, application memory, retrieval systems, and any explicit persistence mechanisms.
Multiple defensive layers
An application may use input filters, output classifiers, retrieval permissions, redaction, and model-level safeguards.
A successful or unsuccessful request may reflect the combined behavior of these components rather than the underlying model alone. Where practical, testers should inspect application logs, retrieval traces, and tool calls to identify which control accepted or rejected the request.
Configuration and repeatability
Results can depend on the model, application instructions, available tools, conversation history, and sampling configuration.
A single successful or unsuccessful attempt is not necessarily conclusive. Relevant tests should be repeated under representative configurations, with enough information recorded to make the results reproducible.
These challenges make observability important. Without knowing what information entered the model's context and what happened after generation, it can be difficult to distinguish an actual disclosure from a plausible but unsupported response.
What Counts as a Finding
Across these three tests, a vulnerability is confirmed when the application exposes information in a way that violates its intended security or privacy constraints. Examples include:
- The application reveals credentials, API keys, personal data, or confidential records to a user who should not have access to them.
- Information supplied in one context becomes accessible in another context where it should remain isolated.
- Sensitive input is passed to a downstream tool, service, or logging component that is not authorized to receive it.
- The application discloses protected system instructions, and that disclosure violates an established confidentiality requirement or creates a demonstrable security risk.
The distinction between disclosure and impact matters. A model repeating a synthetic test string to the same user is not necessarily a vulnerability. Likewise, revealing a system prompt is not automatically a security incident if the prompt contains no protected information and its disclosure does not violate a meaningful requirement.
For each finding, establish:
- What information was exposed
- Where it originated
- Who could access it
- Which security boundary failed
For input leakage specifically, an unsuccessful attempt to reproduce a sensitive value does not prove that the application has no retention or logging risks. Those require separate tests of the relevant data-handling mechanisms.
Remediation, in Short
These tests point toward several overlapping defensive principles.
Protect sensitive data at the application layer
Apply access controls before retrieving or exposing confidential information. Enforce authorization outside the model, minimize the data included in model context, and redact sensitive values from outputs where appropriate.
Review training and retrieval pipelines for inadvertently exposed secrets, and ensure that the model receives only the information required for its task.
Isolate sensitive user input
Define how sensitive input should be handled throughout its lifecycle, including model context, conversation history, application logs, and downstream tool calls.
Use synthetic test values to verify that redaction, normalization, and access controls behave as intended. Where sensitive information should not persist, enforce that requirement through application and data-retention controls rather than relying on the model to forget it.
Treat system prompts as instructions, not secrets storage
Avoid embedding credentials, API keys, or other sensitive values in system prompts. Limit unnecessary internal implementation details, and test whether confidential instructions can be reconstructed through direct requests, partial extraction, or contextual framing.
Prompt design and model-level safeguards can reduce unwanted disclosure, but they should not be the only controls protecting confidential information.
Enforce least privilege and session isolation
Restrict access to tools, documents, databases, and other resources according to the requesting user's permissions. Ensure that conversation state and retrieved information cannot cross user or session boundaries unintentionally.
Monitor information flows
Where practical, log and monitor access to sensitive data, model inputs and outputs, retrieval operations, and tool calls. Protect those logs appropriately and use them to investigate suspected disclosure without creating additional copies of sensitive information.
The central principle: enforce confidentiality and authorization in the surrounding application. A model's willingness to refuse a request is useful, but it is not a replacement for those controls.
Tools for Testing This
Several tools can help automate or structure information-disclosure testing:
- Garak: includes probes for sensitive information disclosure and prompt-leakage-related behaviors.
- Promptfoo: supports adversarial testing and evaluations for LLM applications, including custom tests for sensitive-data exposure and prompt disclosure.
Tools are useful for generating test cases and repeating requests, but they should complement application-specific testing. A model that reproduces a test value or reveals an instruction is not necessarily vulnerable; the tester still needs to establish what information was exposed, whether the disclosure was unauthorized, and what security boundary should have prevented it.
What's Next in This Series
This post covered three information-disclosure tests. Coming up:
- AITG-APP-05 / 10 / 11 / 12 — unsafe outputs, harmful-content bias, hallucinations, and toxic output
- AITG-APP-06 / 13 — agentic behavior limits and over-reliance on AI
- AITG-APP-08 / 09 — embedding manipulation and model extraction
- AITG-APP-14 — explainability and interpretability
Official Resources
- AITG-APP-03: Testing for Sensitive Data Leak
- AITG-APP-04: Testing for Input Leakage
- AITG-APP-07: Testing for Prompt Disclosure
- OWASP Top 10 for LLM Applications — LLM02: Sensitive Information Disclosure
- OWASP AI Testing Guide — the full AITG-APP category
Final Thoughts
The previous post examined whether attacker-controlled input can change an AI application's behavior. This post examines whether the application exposes information that should remain protected.
Sensitive data leakage, input leakage, and prompt disclosure can produce similar-looking outputs, but they involve different information sources and different security boundaries.
The important testing question is therefore broader than:
"Can I make the model reveal something?"
A useful test asks:
"Can the application expose information to someone or somewhere that should not receive it?"
Answering that question requires more than testing the model's response. It means tracing where information originates, how it enters the model's context, which tools and users can access it, and what happens to it after the model responds.
Prompt design, output filtering, and model-level safeguards can all contribute to reducing disclosure. But effective protection ultimately depends on application-level access controls, appropriate data handling, session isolation, and a clear understanding of which information the model actually needs.
Next in this series: unsafe outputs, hallucinations, and toxic content — testing what happens when the problem isn't information disclosure, but the content the application generates.
Top comments (0)