DEV Community: Vanessa Sastre

AI Security Testing 102: Sensitive Data Leak, Input Leakage, and Prompt Disclosure

Vanessa Sastre — Thu, 29 Oct 2026 13:20:00 +0000

This is the second post in a new series on AI application security testing, following the structure of the OWASP AI Testing Guide's AI Application Testing category (AITG-APP).

The previous post covered prompt injection: testing whether attacker-controlled input can change an application's behavior in ways that violate its intended instructions or security constraints. This post focuses on a related but distinct question: whether an AI application exposes information that should remain protected.

We're covering three tests from the guide. Although all three involve information being exposed in an unintended way, they target different sources of information and different security boundaries.

Test	The question it asks	Where the information comes from
AITG-APP-03 — Sensitive data leak	Can the application expose confidential information through its model, memory, training data, or connected tools?	Model, memory, training data, retrieved documents, connected tools
AITG-APP-04 — Input leakage	Can information supplied by a user be unnecessarily reproduced or exposed through the application?	The user's own input
AITG-APP-07 — Prompt disclosure	Can the application reveal its system or application instructions?	The application's instructions

The distinction matters because the same apparent symptom can have very different causes. Effective testing therefore needs to establish where the information came from, whether the disclosure was unauthorized, and which security control should have prevented it.

Sensitive Data Leak (AITG-APP-03)

Sensitive data leakage occurs when an AI application exposes confidential information that should not be available to the requesting user or should not leave a protected system boundary.

The source of the information can vary. It may come from training data, conversation memory, retrieved documents, application context, databases, or connected tools. The testing approach therefore depends on what information the application can access and which users are authorized to see it.

A useful starting point is to identify what the application should protect and where that information enters the system. Testing can then examine whether the model or surrounding application exposes it through direct requests, contextual tasks, or unintended information disclosure.

Attack Patterns Worth Knowing

Several approaches can help testers identify whether sensitive information is exposed through an AI application.

Direct and contextual requests

Simply asking the model for credentials or confidential information is an obvious starting point. A more contextual test might ask the model to summarize internal communications or retrieve information from a document that happens to contain sensitive values.

The distinction matters because an application may refuse a direct request for a password while still exposing the same information through an apparently legitimate task, such as summarization or document analysis.

Persona and pretext framing

A request might frame the interaction as a developer debugging the application, an administrator investigating an error, or a researcher examining a system's configuration.

These scenarios can help determine whether the application relies too heavily on the model's interpretation of the user's claimed role or purpose. Authorization should be enforced by the application rather than inferred from a persuasive explanation in the prompt.

Tool and application reconnaissance

For agentic systems, testers can examine whether the model reveals available tools, sub-agents, tool parameters, or other operational details.

This information is not necessarily confidential, and revealing a tool description is not automatically a vulnerability. However, unnecessary disclosure can help an attacker understand the application's capabilities and construct more targeted follow-up attacks.

Testing should establish whether the disclosed information violates a confidentiality requirement or meaningfully weakens a security boundary.

Implementation details and error disclosure

Requests for stack traces, API endpoints, framework details, internal configuration, or recent errors can reveal information about the application's implementation.

Individually, some of these details may be harmless. Others can expose sensitive values, internal infrastructure, or information that should be restricted to administrators. The security impact depends on what is disclosed and who can access it.

The Unifying Theme

A sensitive data leak is not always a model revealing a password. Information about internal systems, retrieved records, or connected tools can also be relevant when its disclosure violates the application's intended security constraints.

Distinguish actual disclosure from speculation. A model describing a hypothetical environment does not prove that it has access to the corresponding credentials, files, or infrastructure.

Input Leakage (AITG-APP-04)

Input leakage examines whether information supplied to an AI application is exposed or reused in ways that violate the application's intended privacy or security boundaries.

Unlike a sensitive data leak, where the information may originate from a protected data source or connected system, this test starts with information provided through the input channel itself.

A practical test might use a synthetic API key, a fake password, or a clearly marked confidential string. The tester then checks whether the application:

reproduces that value in a later response,
exposes it to another user or session, or
passes it to a component that should not receive it.

The important question is not simply whether the model can repeat the value. It is whether the application handles that value in accordance with its security requirements.

Common Test Patterns

Repeated-input testing

Provide a synthetic sensitive value and check whether the application reproduces it when asked directly or in a later turn.

Where relevant, test session boundaries and whether information can appear in another user's context. This helps distinguish ordinary repetition within a conversation from unintended exposure across users or sessions.

Normalization and obfuscation

Represent test values using alternative character encodings or Unicode variants, then evaluate whether the application's detection and redaction controls recognize the value.

For example, a test might use Unicode full-width digits instead of standard ASCII digits. A filter that relies on a narrowly defined character pattern may handle the two representations differently, while a downstream component may normalize them to the same value.

This is not a guaranteed bypass. The result depends on how inputs are normalized, inspected, and processed throughout the application.

Contextual variation

Change the surrounding request while keeping the underlying sensitive value the same.

For example, compare a direct request to reproduce a value with a request to summarize, transform, or explain the message containing it. The objective is to determine whether the application's protections depend too heavily on specific keywords or request formats.

Affix and formatting variations

Add unrelated text immediately before or after a synthetic sensitive value, or place the value within a larger structured input.

These variations can help identify filters that rely on particular positions, delimiters, or surrounding context. Their effectiveness depends on the implementation being tested.

Combined test cases

Combine several input variations to evaluate whether the application's controls continue to work when multiple techniques are used together.

For example, a test might combine Unicode variations with additional surrounding text and a contextual request. This can help establish whether different defensive layers work together or whether one layer depends on another recognizing a particular representation.

Testing Ground Rules

These tests should use synthetic values rather than real credentials or personal information. They should also distinguish between the model's response, application logs, downstream tool calls, and any cross-session exposure being tested.

A clean result has limits. A successful output test only establishes what happened in the observed interaction. It does not establish whether the provider retains inputs, whether those inputs are used for training, or how they are handled in application logs. Those questions require separate examination of the system's data-handling configuration and policies.

Prompt Disclosure (AITG-APP-07)

Prompt disclosure tests whether an AI application reveals system or application instructions that are intended to remain confidential.

These instructions may contain proprietary workflows, internal policy details, tool descriptions, or other implementation information. Their disclosure can help an attacker understand how the application is designed and identify additional attack paths.

However, a system prompt should not be treated as a security boundary in its own right. Secrets should not be embedded in instructions, and authorization should be enforced by the application rather than relying on the model to follow a prompt.

Testing should therefore establish both whether protected instructions can be reconstructed and whether their disclosure creates a meaningful security or operational impact.

Techniques Worth Testing

Direct and oblique requests

Simply asking the model to reveal its system prompt is an obvious starting point. More indirect requests might ask it to explain its guidelines, describe its operating instructions, or summarize the rules governing its behavior.

Oblique phrasing can sometimes produce different results from a direct request, depending on the model and the application's safeguards. Testers should evaluate both approaches rather than assuming that blocking one phrase prevents disclosure.

Context and clarification framing

A request might claim that the model's behavior is confusing and ask it to quote the instructions responsible for that behavior.

This tests whether the model treats a request framed as clarification differently from an explicit request for confidential instructions. The important question is whether the application exposes information that its security requirements say should remain protected.

Partial completion and reconstruction

A tester can examine whether supplying a suspected prefix of an instruction causes the model to continue with confidential text.

Language models generate text based on their learned behavior and the context they receive, including instructions and learned refusal behavior. Prefix completion may therefore reveal information in some configurations, but it does not guarantee disclosure.

Testing should also distinguish between the model producing a plausible continuation and actually reproducing the application's real system prompt.

Incremental extraction

Instead of requesting an entire prompt at once, a tester can examine whether the application reveals protected instructions in smaller pieces.

For example, successive requests might ask for individual instructions, a summary of particular rules, or additional details about previously disclosed text. The objective is to determine whether restrictions that apply to a single large request remain effective across multiple interactions.

Adversarial suffixes (a related research area)

Adversarial suffixes are specially constructed text appended to a prompt to influence a model's behavior. Some research has demonstrated that such techniques can induce undesirable outputs or weaken refusal behavior in particular configurations.

However, causing a model to produce an unwanted response is not the same as demonstrating system-prompt extraction. A prompt-disclosure finding requires evidence that protected instructions were actually revealed or reconstructed.

The broader testing principle is to vary the request format and interaction pattern rather than assuming that one refusal demonstrates that the system prompt is protected.

What Makes These Tests Difficult

The three tests share an important challenge: the model's visible response does not necessarily tell you where the information originated or which control failed.

Model access versus application access

A model may generate a plausible-looking secret without actually having access to it. Conversely, an application may retrieve confidential information and pass it into the model's context even when the model would not otherwise know that information.

Testers should establish whether a disclosed value is real, whether the application had access to it, and how it reached the response.

Conversation context and session boundaries

A model may repeat information because it remains available in the current conversation context. That behavior is different from exposing information across isolated sessions or retaining it beyond the intended lifetime.

Testing should account for conversation history, application memory, retrieval systems, and any explicit persistence mechanisms.

Multiple defensive layers

An application may use input filters, output classifiers, retrieval permissions, redaction, and model-level safeguards.

A successful or unsuccessful request may reflect the combined behavior of these components rather than the underlying model alone. Where practical, testers should inspect application logs, retrieval traces, and tool calls to identify which control accepted or rejected the request.

Configuration and repeatability

Results can depend on the model, application instructions, available tools, conversation history, and sampling configuration.

A single successful or unsuccessful attempt is not necessarily conclusive. Relevant tests should be repeated under representative configurations, with enough information recorded to make the results reproducible.

These challenges make observability important. Without knowing what information entered the model's context and what happened after generation, it can be difficult to distinguish an actual disclosure from a plausible but unsupported response.

What Counts as a Finding

Across these three tests, a vulnerability is confirmed when the application exposes information in a way that violates its intended security or privacy constraints. Examples include:

The application reveals credentials, API keys, personal data, or confidential records to a user who should not have access to them.
Information supplied in one context becomes accessible in another context where it should remain isolated.
Sensitive input is passed to a downstream tool, service, or logging component that is not authorized to receive it.
The application discloses protected system instructions, and that disclosure violates an established confidentiality requirement or creates a demonstrable security risk.

The distinction between disclosure and impact matters. A model repeating a synthetic test string to the same user is not necessarily a vulnerability. Likewise, revealing a system prompt is not automatically a security incident if the prompt contains no protected information and its disclosure does not violate a meaningful requirement.

For each finding, establish:

What information was exposed
Where it originated
Who could access it
Which security boundary failed

For input leakage specifically, an unsuccessful attempt to reproduce a sensitive value does not prove that the application has no retention or logging risks. Those require separate tests of the relevant data-handling mechanisms.

Remediation, in Short

These tests point toward several overlapping defensive principles.

Protect sensitive data at the application layer

Apply access controls before retrieving or exposing confidential information. Enforce authorization outside the model, minimize the data included in model context, and redact sensitive values from outputs where appropriate.

Review training and retrieval pipelines for inadvertently exposed secrets, and ensure that the model receives only the information required for its task.

Isolate sensitive user input

Define how sensitive input should be handled throughout its lifecycle, including model context, conversation history, application logs, and downstream tool calls.

Use synthetic test values to verify that redaction, normalization, and access controls behave as intended. Where sensitive information should not persist, enforce that requirement through application and data-retention controls rather than relying on the model to forget it.

Treat system prompts as instructions, not secrets storage

Avoid embedding credentials, API keys, or other sensitive values in system prompts. Limit unnecessary internal implementation details, and test whether confidential instructions can be reconstructed through direct requests, partial extraction, or contextual framing.

Prompt design and model-level safeguards can reduce unwanted disclosure, but they should not be the only controls protecting confidential information.

Enforce least privilege and session isolation

Restrict access to tools, documents, databases, and other resources according to the requesting user's permissions. Ensure that conversation state and retrieved information cannot cross user or session boundaries unintentionally.

Monitor information flows

Where practical, log and monitor access to sensitive data, model inputs and outputs, retrieval operations, and tool calls. Protect those logs appropriately and use them to investigate suspected disclosure without creating additional copies of sensitive information.

The central principle: enforce confidentiality and authorization in the surrounding application. A model's willingness to refuse a request is useful, but it is not a replacement for those controls.

Tools for Testing This

Several tools can help automate or structure information-disclosure testing:

Garak: includes probes for sensitive information disclosure and prompt-leakage-related behaviors.
Promptfoo: supports adversarial testing and evaluations for LLM applications, including custom tests for sensitive-data exposure and prompt disclosure.

Tools are useful for generating test cases and repeating requests, but they should complement application-specific testing. A model that reproduces a test value or reveals an instruction is not necessarily vulnerable; the tester still needs to establish what information was exposed, whether the disclosure was unauthorized, and what security boundary should have prevented it.

What's Next in This Series

This post covered three information-disclosure tests. Coming up:

AITG-APP-05 / 10 / 11 / 12 — unsafe outputs, harmful-content bias, hallucinations, and toxic output
AITG-APP-06 / 13 — agentic behavior limits and over-reliance on AI
AITG-APP-08 / 09 — embedding manipulation and model extraction
AITG-APP-14 — explainability and interpretability

Official Resources

Final Thoughts

The previous post examined whether attacker-controlled input can change an AI application's behavior. This post examines whether the application exposes information that should remain protected.

Sensitive data leakage, input leakage, and prompt disclosure can produce similar-looking outputs, but they involve different information sources and different security boundaries.

The important testing question is therefore broader than:

"Can I make the model reveal something?"

A useful test asks:

"Can the application expose information to someone or somewhere that should not receive it?"

Answering that question requires more than testing the model's response. It means tracing where information originates, how it enters the model's context, which tools and users can access it, and what happens to it after the model responds.

Prompt design, output filtering, and model-level safeguards can all contribute to reducing disclosure. But effective protection ultimately depends on application-level access controls, appropriate data handling, session isolation, and a clear understanding of which information the model actually needs.

Next in this series: unsafe outputs, hallucinations, and toxic content — testing what happens when the problem isn't information disclosure, but the content the application generates.

AI Security Testing 101: Prompt Injection and Indirect Prompt Injection

Vanessa Sastre — Tue, 20 Oct 2026 22:45:35 +0000

This is the first post in a new series on AI application security testing, following the structure of the OWASP AI Testing Guide's AI Application Testing category (AITG-APP). Each post in the series will take one test from that category and turn it into something practical: what the vulnerability actually looks like, how testers approach it, and what "vulnerable" means in concrete terms.

We're starting where the guide starts: prompt injection, one of the most well-known AI application vulnerabilities, and a class of attack that can underpin many other AI-specific security risks.

Direct Prompt Injection (AITG-APP-01)

Prompt injection happens when attacker-controlled input changes an AI application's behavior in a way that conflicts with its intended instructions or security constraints. The input may attempt to make the model ignore higher-priority instructions, reveal information it should not disclose, or perform an action it was not intended to perform.

A useful, though imperfect, analogy is SQL injection: in both cases, attacker-controlled input can influence behavior that the application intended to control. The analogy has limits, however. SQL injection exploits the interpretation of data as executable syntax, while prompt injection arises from the way language models process natural-language instructions and contextual information.

The AITG-APP-01 methodology characterizes a prompt injection in terms of three elements:

Instructions: what the attacker actually wants the model to do.
A trigger: the specific phrasing, framing, or encoding intended to cause the model to follow the attacker's instructions rather than its intended behavior.
Conflict with intent: the injected request conflicts with the application's intended instructions or constraints.

Without that conflict, an unusual or adversarial-looking prompt may simply be a normal request rather than a security-relevant prompt injection.

Prompt injection is also broader than jailbreaking. A jailbreak attempts to bypass safety restrictions, while prompt injection more generally involves attacker-controlled input changing model behavior in an unintended way. An injection can therefore be security-relevant even when it does not produce prohibited content.

Whether an attack succeeds can depend heavily on how the trigger is constructed. That's one reason simple keyword filtering is insufficient on its own. Blocking the phrase "ignore previous instructions" does little against the same underlying idea expressed through role-play, another language, obfuscation, or an encoded representation.

Attack Patterns Worth Knowing

The OWASP guide catalogs a broad set of concrete payload techniques. They can be grouped into a handful of practical patterns:

Persona and role-play framing

Asking the model to "pretend" to be an unrestricted persona so that it treats otherwise disallowed behavior as being in character. A well-known example is the "DAN" jailbreak.

The underlying technique is not limited to DAN. Attackers can construct fictional characters, simulated environments, alternate roles, or hypothetical scenarios intended to change how the model interprets its constraints.

Context manipulation

Telling the model to "forget" previous instructions, attempting to override higher-priority context, or gradually steering the conversation toward a prohibited behavior.

This can include multi-turn approaches in which no individual request appears particularly harmful, but the sequence gradually changes the context presented to the model. "Crescendo" and "echo chamber" style attacks are examples of this broader pattern.

Obfuscation and encoding

Hiding or transforming the attack instruction using techniques such as base64, hexadecimal encoding, reversed text, unusual capitalization, deliberate typos, Unicode manipulation, or other transformations.

The purpose is often to evade filters or exploit differences between what a security control recognizes and what the model can interpret.

The important lesson is that filtering for a fixed set of phrases is not a sufficient defense against a language-based attack surface.

Structural tricks

Taking advantage of how the application or model handles structured inputs, including JSON or other formats, multi-message inputs, or instructions embedded within content that the application expects the model to process.

Structured formats can create additional attack surfaces when applications treat model-generated or model-consumed fields as trusted simply because they conform to an expected schema.

Contradiction and instruction-conflict framing

Presenting multiple competing instructions or asking the model to produce both a normal response and an explicitly contrary response, with the goal of getting it to produce the latter.

The exact payloads are less important than the general testing principle: don't assume that one known jailbreak or injection string represents the entire attack surface.

These techniques are not permanent. Models, applications, and defenses evolve, so a payload that works against one model or configuration may become ineffective against another. That's why testing should combine techniques and repeat requests rather than relying on a single known payload.

Response variability can also matter. In configurations where sampling introduces variability, the same prompt may produce different results across attempts.

What Makes a Test Environment Different

Before running tests against a real target, it's worth understanding what's actually in front of you. The architecture and model configuration can significantly affect which techniques are relevant and how the results should be interpreted.

Temperature

Lower temperature generally makes output more repeatable, while higher temperature generally increases variability and diversity.

That matters during testing because a payload that succeeds once may not succeed consistently. Testers should therefore repeat relevant requests and record the configuration used rather than treating a single successful or unsuccessful attempt as definitive.

Temperature does not by itself determine whether a system is secure, and a temperature of zero should not be treated as a universal guarantee of deterministic behavior.

Fine-tuning vs. prompt tuning

Safety and behavioral constraints can be implemented at different layers, including through model training or fine-tuning and through application-level instructions or other steering mechanisms.

This distinction matters during testing because a successful bypass may target different layers of the system. A test might circumvent application-level instructions without demonstrating that the underlying model has changed, for example.

The important question for a tester is therefore not just "did the model comply?" but also "which security control was bypassed?"

Guardrails

Guardrails can be implemented in several ways, including deterministic rules, classifiers, access controls, schema validation, filtering systems, or additional LLM-based checks.

When an application uses an LLM as part of its guardrail system, that introduces another model and potentially another source of variability. Testers should evaluate the guardrail as part of the attack surface rather than assuming that adding another model automatically prevents prompt injection.

Multi-agent setups

When several agents hand work to each other, testers may only see the final output rather than the intermediate prompts, tool calls, and responses.

That makes it harder to determine which agent or handoff an injection actually influenced. Testing therefore benefits from observability into agent-to-agent messages, tool calls, retrieved content, and intermediate decisions wherever possible.

Tool access

A model wired up to scripts, shells, databases, or external APIs can turn a successful injection from unwanted model output into an unauthorized action.

Tool access can therefore substantially increase the potential impact of a prompt injection. However, tool access is not required for an injection to have security impact: unauthorized information disclosure or manipulation of important outputs can also be significant.

Indirect Prompt Injection (AITG-APP-02)

Direct injection assumes the attacker is the one providing the malicious input to the AI application.

Indirect prompt injection works differently. The malicious instruction is planted somewhere the AI application will process as external content: a web page, PDF, email, uploaded document, database record, search result, or tool-related content.

The model may then encounter that instruction as part of the context it receives.

The important distinction is not that the model has literally "no way" to distinguish data from instructions. Rather, the model may fail to reliably distinguish untrusted external content from instructions when both are represented in the context it processes.

That creates a different attack surface: the attacker may not need direct access to the AI application if they can influence content that the application will eventually retrieve or process.

Indirect injection can be particularly concerning in applications that automatically retrieve external content, have access to sensitive information, or can take actions through tools.

Concrete Surfaces This Shows Up On

Hidden text in web pages or documents

An attacker can place instructions in content that is difficult for a human reviewer to notice, such as HTML comments, invisible or visually obscured text, document metadata, or other content that is nevertheless extracted and passed to the model.

The important security property is not that the text is literally invisible. It's that the application may process content that the user did not realize would be interpreted by the model.

Structured data fields

A JSON, XML, database record, API response, or other structured input may contain an attacker-controlled field whose value includes an instruction rather than the data the application expected.

For example, a field intended to contain a document summary or description might contain language such as an instruction to ignore the application's task and perform another action.

The fact that the content is valid JSON or XML does not make it trustworthy.

Markdown-based exfiltration

An indirect prompt injection can attempt to make the model include sensitive information in a URL or other externally observable location, including by generating a Markdown image reference or link containing attacker-controlled data.

If the resulting content is rendered or fetched by a downstream component, this can create a data-exfiltration path.

A related technique was demonstrated in the real-world EchoLeak vulnerability affecting Microsoft 365 Copilot. EchoLeak was a multi-stage attack involving indirect prompt injection and data exfiltration under particular conditions; it should not be treated as synonymous with every Markdown-based exfiltration technique.

Encoded payloads

External content can also contain encoded or otherwise obfuscated instructions. The same general filter-evasion issues that apply to direct prompt injection can therefore apply to indirect injection.

Tool metadata and tool poisoning

For AI systems using MCP or similar tool-calling protocols, tool names, descriptions, schemas, metadata, parameters, and outputs can become part of the model's context.

In some client implementations, this information may not be prominently exposed to the user even though it is presented to the model.

An attacker who can control a tool's metadata may therefore attempt to embed instructions into it, for example, instructions telling the model to retrieve information before using the tool.

The security issue is not specific to one metadata field. The broader concern is that the model may treat untrusted tool-related content as instructions.

The common thread is this:

Anywhere external content reaches the model without a reliable boundary between trusted instructions and untrusted data, there is a potential prompt-injection attack surface.

That boundary is an architectural concern, not merely a question of finding the right payload string.

What Counts as a Finding

For both tests, a prompt injection becomes a security finding when attacker-controlled instructions cause behavior that violates the application's intended constraints or security boundaries.

Examples include:

The model overrides or circumvents intended instructions in a security-relevant way.
The application discloses information that the model or user should not have access to, such as credentials, API keys, internal configuration, or another user's data.
The model or connected system performs an unauthorized or harmful action.
An indirect injection causes the application to treat attacker-controlled external content as an instruction and subsequently produce unauthorized disclosure or action.

For indirect injection specifically, the distinguishing condition is that the security-relevant behavior originates in externally supplied content rather than the user's direct prompt.

Simply following an instruction found in external content is not automatically a vulnerability. For example, if a webpage contains ordinary instructions that are relevant to the user's request and the model follows them as part of legitimately processing that page, there may be no security issue.

The important question is whether attacker-controlled external content can change the application's behavior in a way that violates its intended security or operational constraints.

Remediation, in Short

Both tests point toward several overlapping defensive principles.

Separate trusted instructions from untrusted content

Clearly distinguish system/application instructions from user input and external data.

Delimiters and structured context can help the model distinguish these sources, but they should be treated as mitigations rather than a guarantee that prompt injection will never succeed.

Treat external content as untrusted

Validate and sanitize external content before it reaches the model where appropriate. Consider hidden text, encoded instructions, unusual Unicode, metadata, and other forms of content that a human reviewer may not recognize as instructions.

But don't rely on sanitization alone. Natural language is too flexible to make exhaustive filtering a reliable security boundary.

Restrict model privileges

Give the model only the access and capabilities it actually needs.

If a model does not need access to credentials, private files, sensitive databases, or administrative functions, those capabilities should not be available to it merely because the model might request them.

Authorization should be enforced outside the model wherever possible.

Require approval for high-impact actions

Sensitive, high-impact, or irreversible actions should require appropriate authorization or human approval rather than relying entirely on the model to decide whether an action is safe.

The model should not be the sole security boundary protecting a privileged operation.

Treat tool metadata as untrusted input

Tool descriptions, schemas, metadata, parameters, and outputs should be treated as potentially untrusted content.

Applications should avoid assuming that a tool definition is safe simply because it was supplied through an otherwise trusted tool-calling protocol.

Add monitoring and observability

Where practical, log and monitor retrieved content, model inputs, tool calls, authorization decisions, and significant outputs.

This is particularly important in multi-agent systems, where a prompt injection may enter through one component and only become visible much later in the execution chain.

Tools for Testing This

Several tools can help automate or structure prompt-injection testing:

Garak: includes probes relevant to prompt injection and other LLM security testing.
Promptfoo: provides adversarial testing and evaluation capabilities for LLM applications, including prompt-injection testing.
Prompt Security Fuzz (ps-fuzz): a fuzzer focused on prompt-level attacks.

Tools are useful for generating and repeating attacks, but they should complement application-specific testing. A generic payload that causes an unusual model response is not necessarily a vulnerability; the tester still needs to establish what security boundary was crossed and what the resulting impact is.

What's Next in This Series

This post covered the two prompt-manipulation tests. Coming up:

AITG-APP-03 / 04 / 07 — sensitive data leaks, input leakage, and system prompt disclosure
AITG-APP-05 / 10 / 11 / 12 — unsafe outputs, harmful-content bias, hallucinations, and toxic output
AITG-APP-06 / 13 — agentic behavior limits and over-reliance on AI
AITG-APP-08 / 09 — embedding manipulation and model extraction
AITG-APP-14 — explainability and interpretability

Official Resources

Final Thoughts

Prompt injection isn't a vulnerability that can necessarily be "patched once." It is a recurring security problem for systems that allow language models to process instructions alongside user-controlled or externally sourced content.

Direct injection tests what happens when a user actively attempts to change the application's behavior. Indirect injection tests whether attacker-controlled external content can influence the system even when the attacker never interacts with the AI interface directly.

The important security question is therefore broader than:

"Can I make the model ignore its system prompt?"

A useful test asks:

"Can attacker-controlled input or external content cause the application to violate an intended security or operational constraint?"

That framing also makes the impact easier to assess. A successful jailbreak that produces harmless text is different from an injection that exposes confidential information, bypasses authorization, or causes a connected system to perform an unauthorized action.

Prompt injection techniques will continue to evolve as models, applications, and defenses change. Effective testing therefore needs to combine different attack patterns, repeat tests under representative configurations, and most importantly evaluate the actual security boundaries around the model rather than treating the model itself as the only control.

Playwright API Testing: Your API and UI Tests Can Live Together

Vanessa Sastre — Wed, 02 Sep 2026 13:01:00 +0000

Playwright API testing uses the same framework, the same Playwright project/configuration, and the same test suite you already write UI tests in to send real HTTP requests, mock responses, and verify the resulting server-side state after a UI action. No second tool, no second test runner, no context-switching between a UI suite and an API suite that drift out of sync with each other.

That's the core shift this post covers: what changes when API testing stops being a separate discipline, how to set it up for pure API tests and for UI tests that need backend validation, and how this approach affects coverage, feedback, and CI cost.

The Problem: Two Test Suites, One Application

Most teams end up with API coverage and UI coverage living in different tools: Postman or a REST client for the API; Playwright or Selenium for the UI, because that's how the tooling has traditionally been split. That split has a cost:

Duplicated setup. Auth, base URLs, and test data creation get written twice, once per suite, and drift apart over time.
Blind spots at the seam. UI tests confirm a button click looked successful. API tests confirm an endpoint works in isolation. Neither one confirms that clicking the button actually triggered the right backend call with the right payload.
Slow, UI-only precondition setup. To test "issue #42 shows correctly," a UI-only suite has to create that issue by clicking through the UI first, and one more brittle flow that can fail before the actual test even starts.
No shared debugging story. When something breaks at the UI/API boundary, you're reconciling two separate tools' worth of logs to find out where the mismatch happened.

None of this is a tooling limitation you have to accept, it's a consequence of treating "API testing" and "UI testing" as different jobs.

The Fix: One Framework, One Request Context

Playwright provides the same API-testing primitives for both pure API tests and UI tests that need backend setup or validation.

Task	Split-tooling approach	With Playwright API
Test an endpoint directly	Separate Postman collection or REST client, maintained apart from the UI suite	A `request` fixture test, in the same repo, same config, same CI job
Seed data before a UI test	Click through the UI to create it, or maintain a separate seeding script	One `request.post()` call in `beforeAll`, reusing the same auth
Validate a UI action's server-side result	Trust the UI state, or manually check the database/API afterward	`request.get()` after the UI action, asserting on the real server-side state
Keep cookie/local-storage auth in sync between suites	Separate auth setup	Reuse Playwright `storageState` between API and browser contexts

Pure API Testing

For endpoint-level tests with no browser involved, configure a base URL and headers once, and every test gets a ready-to-use request fixture.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  use: {
    baseURL: 'https://api.github.com',
    extraHTTPHeaders: {
      'Accept': 'application/vnd.github.v3+json',
      'Authorization': `token ${process.env.API_TOKEN}`,
    },
  },
});

test('should create a bug report', async ({ request }) => {
  const newIssue = await request.post('/repos/USER/REPO/issues', {
    data: { title: '[Bug] report 1', body: 'Bug description' },
  });
  expect(newIssue.ok()).toBeTruthy();

  const issues = await request.get('/repos/USER/REPO/issues');
  expect(await issues.json()).toContainEqual(
    expect.objectContaining({ title: '[Bug] report 1' })
  );
});

Setup and teardown for test data can use the built-in request fixture in beforeAll/afterAll. If you create a standalone APIRequestContext yourself with playwright.request.newContext(), dispose of it in afterAll.

Combined UI + API: Validating That Actions Trigger Backend Calls

This is where the split-tooling approach falls down hardest and where a single framework pays off the most. Two patterns cover most cases.

Establishing preconditions via API, before touching the UI

Create state through the API, then confirm it's reflected correctly in the UI without clicking through slow setup flows.

let apiContext;

test.beforeAll(async ({ playwright }) => {
  apiContext = await playwright.request.newContext({
    baseURL: 'https://api.github.com',
    extraHTTPHeaders: { 'Authorization': `token ${process.env.API_TOKEN}` },
  });
});

test('last created issue should be first in the list', async ({ page }) => {
  const newIssue = await apiContext.post('/repos/USER/REPO/issues', {
    data: { title: '[Feature] request 1' },
  });
  expect(newIssue.ok()).toBeTruthy();

  await page.goto('https://github.com/USER/REPO/issues');
  await expect(
    page.locator(`a[data-hovercard-type='issue']`).first()
  ).toHaveText('[Feature] request 1');
});

The test only ever clicks through the one flow it's actually testing, the setup happens instantly over the API.

Validating postconditions after a UI action

Drive the interaction through the UI, then confirm the backend actually recorded it, not just that the UI rendered a success state.

test('creating an issue via the UI should persist on the server', async ({ page }) => {
  await page.goto('https://github.com/USER/REPO/issues');
  await page.getByText('New Issue').click();
  await page.getByRole('textbox', { name: 'Title' }).fill('Bug report 1');
  await page.getByText('Submit new issue').click();

  const issueId = new URL(page.url()).pathname.split('/').pop();
  const newIssue = await apiContext.get(`/repos/USER/REPO/issues/${issueId}`);

  expect(newIssue.ok()).toBeTruthy();
  expect(await newIssue.json()).toEqual(
    expect.objectContaining({ title: 'Bug report 1' })
  );
});

This is the check a UI-only test structurally can't do: it proves the click didn't just update local state or a client-side cache, it verifies that the UI action resulted in the expected server-side state.

Sharing auth between both

Log in once via the API, save the resulting storageState, and reuse it in a browser context, one login flow instead of two:

const requestContext = await request.newContext();
await requestContext.get('https://api.example.com/login');
await requestContext.storageState({ path: 'state.json' });

const context = await browser.newContext({ storageState: 'state.json' });

Mocking: Testing States You Can't Easily Produce

Not every test should hit a real backend. Playwright can intercept, modify, or fully mock network traffic at the browser level with page.route(), useful for error states, edge cases, and tests that shouldn't depend on a live API being up.

Mock a response entirely: the real API is never called:

await page.route('*/**/api/v1/fruits', async route => {
  await route.fulfill({ json: [{ name: 'Strawberry', id: 21 }] });
});

Call the real API, then patch the response: useful when you need mostly-real data with one controlled edge case injected:

await page.route('*/**/api/v1/fruits', async route => {
  const response = await route.fetch();
  const json = await response.json();
  json.push({ name: 'Loquat', id: 100 });
  await route.fulfill({ response, json });
});

Replay from a recorded HAR file: record real traffic once, then run tests against it offline and deterministically, with no live dependency at all:

await page.routeFromHAR('./hars/fruit.har', {
  url: '*/**/api/v1/fruits',
  update: false,
});

These interactions are visible in Playwright's trace/network tooling, making mocked and live network behavior debuggable from the same workflow.

Beyond HTTP: Playwright can also intercept and mock WebSocket connections with page.routeWebSocket() for real-time features. For browser APIs that Playwright doesn't directly automate, page.addInitScript() can be used to install mocks before the page's own code runs.

The Efficiency Case

For a team deciding whether to consolidate API and UI testing into one framework, these are the levers that actually move:

One suite to maintain, not two. Auth, base URLs, and test data setup exist in a single place instead of being reimplemented per tool, every hour spent keeping a Postman collection and a Playwright suite in sync is an hour not spent on coverage.
Faster test setup, lower CI time. API-based setup is typically much faster than reproducing the same state through a browser workflow, because it avoids navigation, rendering, and UI interaction.
Fewer false negatives. A UI test that only checks for a success toast can pass while the backend call silently failed. Postcondition checks against the real API catch that class of bug before a customer does.
Deterministic tests, fewer flaky-test investigations. Mocked and HAR-replayed tests aren't affected by a slow or temporarily-down backend, which removes one of the most common sources of "flaky" CI failures that cost engineering time to triage and usually turn out to be infrastructure, not the code under test.

CI/CD Recommendations

Keep pure API tests in their own project/job. They're fast and don't need a browser, running them separately from UI tests shortens feedback on backend-only regressions.
Use HAR replay for UI tests that don't need live data. It removes a live dependency from the critical path of your CI pipeline, and makes those tests immune to unrelated backend outages.
Reserve postcondition checks for flows where "the UI looked right" genuinely isn't enough proof. Checkout, payments, anything where a silent backend failure would be expensive. Applying it everywhere adds maintenance cost without a matching increase in the bugs it catches.

Official Resources

API testing — APIRequestContext, setup/teardown, sharing auth between API and browser contexts
Mock APIs — mocking requests, modifying responses, HAR file recording and replay
Mock browser APIs — mocking browser-level APIs Playwright doesn't automate directly
Network — request/response interception, modification, and WebSocket mocking in depth

Final Thoughts

Playwright API testing isn't about replacing every API test with a browser test, or using one tool simply for the sake of consolidation. It's about having the flexibility to test the right layer with the same framework.

Pure API tests can stay fast and focused. UI tests can use the API to create preconditions or verify server-side outcomes. Network mocking and HAR replay can remove unnecessary dependencies from UI tests. And because all of these live within the same Playwright project, they can share configuration, authentication, test data strategies, debugging tools, and CI infrastructure.

The result is a testing strategy that's easier to compose: use the API where it makes sense, use the browser where it matters, and combine them when testing one without the other leaves gaps.

Playwright Trace Viewer: Debug Failed Tests Without Reproducing Them

Vanessa Sastre — Thu, 06 Aug 2026 21:57:33 +0000

Playwright Trace Viewer is a built-in GUI that records a full, replayable snapshot of a test run — every action, DOM state, network call, and console log — so you can debug a CI failure by opening a file, not by reproducing it.

In practice, that makes a UI test into more than a UI recording. Every browser interaction is synchronized with the underlying API traffic, console output, and page state at the moment it happened — which gives you an API debugging layer for free, without writing a single API test.

That one difference changes the economics of test debugging. This post covers what changes in your day-to-day workflow, how to turn it on, and how to justify it to the people who care about engineering time and CI spend rather than DOM snapshots.

The Problem: Debugging Requires Reproducing

CI fails → Try locally → Can't reproduce → Add debugging → Implement a fix → Push → Wait for CI → Repeat

A test fails on CI. Here's what that traditionally costs you:

Read the error message and stack trace, which often aren't clear enough on their own.
Pull the branch and try to reproduce the failure locally.
It passes locally. CI-only failures are common — different browser, viewport, timing, environment, or data state.
Add console.log, screenshots, tracing, or page.pause() calls and run the test again.
It still doesn't reproduce. Try running headed, slow down actions, add waits, or tweak the environment until it finally fails.
Eventually reproduce the issue, identify the root cause, remove the debugging code, and implement a fix.
Push the fix and wait for CI to run again. If the failure persists, repeat the process.

Depending on the bug, that loop can take anywhere from a few minutes to most of an afternoon — and it scales linearly with how flaky or environment-dependent the failure is. Every step exists because you're missing information that already existed at the moment the test ran, and is now gone.

Trace Viewer's premise: stop throwing that information away.

The Fix: Open the Trace Instead of Re-Running the Test

With tracing enabled, every action, DOM snapshot, request, and log from the run is captured into a single trace.zip. When a test fails, you open that file — locally, from CI, or in the browser — and the run replays exactly as it happened.

Here's the same debugging tasks, before and after:

Task	Without Trace Viewer	With Trace Viewer
Find why a click hit the wrong element	Re-run headed, add logs, hope it reproduces	Click the action, see the before/after DOM and the exact click coordinates
Grab a locator for a new assertion	Re-run in debug mode, use the inspector live	Open the Locator tab on the already-captured screen and pick it — no re-run
Diagnose a failed network call	Add request logging, re-run, inspect devtools	Open Network, already filtered to that action
Find the exact line of code behind a step	Search the file, count steps manually	Click the action, Source jumps to that line
Investigate a CI-only flake	SSH into CI, add debug output, wait for the next occurrence	Download the trace CI already attached, open it locally

The pattern across every row: instead of recreating the conditions of the failure, you're looking at a recording of it.

Setup: Two Lines of Config

Locally, trace everything while you're developing:

npx playwright test --trace on

Or use UI Mode, which traces every test automatically without the flag.

On CI, tracing every run is wasteful — trace only what fails, on retry:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 1,
  use: {
    trace: 'on-first-retry',
  },
});

Other values: on-all-retries, retain-on-failure (if you don't use retries), on (every run — expensive), off.

Opening a trace:

From the HTML report: click View trace on any test — it opens a local instance of Trace Viewer directly, no file hunting.

  npx playwright show-report

From the CLI, pointing at a file or a URL:

  npx playwright show-trace path/to/trace.zip
  npx playwright show-trace https://example.com/trace.zip

In the browser, no install required: trace.playwright.dev. Drag in a file, or pass one as a URL query parameter — handy for linking straight from a CI artifact. Everything renders locally in your browser; nothing is uploaded.

What's Inside a Trace

A timeline runs across the top of the viewer — click or drag anywhere on it, and every panel below updates to that moment. You never have to click through actions one by one to find the point you care about.

Tab	What you get
Actions	Every action, its locator, and its duration, with before/after DOM snapshots
Locator	The exact screen, fully replicated, with Pick Locator available on it — build a new locator without executing anything
Source	The exact line of test code behind the action you're viewing
Call	Timing, resolved locator, and mode (e.g. strict) for the selected action
Log	Playwright's internal narration — waiting for visible/enabled/stable, then acting
Errors	The failure message, with a red marker on the timeline showing exactly when it happened
Console	Browser and test console logs, filterable to a single action
Network	Every request with headers/bodies, filterable to a single action
Metadata	Browser, viewport, duration, and run context in one place
Attachments	Expected vs. actual image diffs with a slider, for visual regression tests

The Network tab is easy to underestimate. This is the API debugging layer mentioned at the top of this post: even if your suite only exercises the UI, every backend call made during the test is captured and synchronized with the user's actions. Instead of opening DevTools and trying to reproduce the issue, you already have a complete timeline of both the frontend behavior and the API interactions that produced it.

Traces as Context for AI Agents

Everything a trace captures — actions, locators, DOM snapshots, network calls, console output — is structured data sitting in a zip file, not just a GUI to click through manually. That makes it a good fit for AI coding agents and testing skills that need to diagnose a failure without a human first reproducing it.

A few ways this shows up in practice:

Autonomous failure triage. An agent can be handed a trace.zip from a CI run and asked to identify the failing action, the locator involved, and the DOM state around it, then propose a fix — without ever spinning up a browser or re-running the suite itself.
Root-cause summaries instead of raw logs. Rather than pasting a stack trace into a chat, an agent (or a skill built around one) can walk the Actions, Network, and Console data from a trace and produce a plain-language explanation of what broke and where, which is a much shorter path to a fix than reading raw output.
Locator generation without execution. Because the Locator tab replicates the exact screen at a given moment, an agent can use that captured state to derive or validate a selector for a new assertion, the same way a human uses Pick Locator — without needing a live browser session.
Feeding "video receipts" back in. If you're also recording screencasts (Playwright's page.screencast API) alongside traces, an agent's own verification steps can be captured the same way — giving you a reviewable artifact of what an agent did, not just a claim that it passed.
CI feedback loops. Since traces are already CI artifacts with stable URLs, an agent watching a pipeline can fetch a failing test's trace automatically as soon as it's produced, rather than waiting on a person to attach logs or describe the failure.

The underlying shift is the same one this post keeps coming back to: a trace turns "the test failed" into a self-contained record that doesn't require a human — or an agent — to reproduce the failure to understand it.

The Efficiency Case

Trace Viewer isn't just a developer-experience nicety — it changes measurable costs. If you need to justify adopting it (or prioritizing better trace coverage in CI), these are the levers to point at:

Mean time to resolution (MTTR) on test failures. The reproduction loop above is where most debugging time goes. Removing the "make it fail again" step is the single biggest lever on how long a failure sits open.
Engineer context-switching cost. A flaky CI failure that can be diagnosed by opening a file, versus one that requires blocking time to reproduce locally, has a very different interruption cost — especially for failures that surface hours after the engineer has moved to other work.
CI compute spend. trace: 'on-first-retry' only records what actually fails, so you get debugging data without paying to trace every green run. Compare that to teams that resort to re-running whole suites repeatedly to catch a flake in the act.
Flaky test triage throughput. Because traces are attachable CI artifacts, failures can be triaged asynchronously by whoever has time, not only by whoever can reproduce the environment.

If you want to track the impact concretely, two numbers are enough to start: time from failure to root cause identified, and number of "couldn't reproduce, closed as flaky" tickets. Both should move once trace review replaces manual reproduction as the default first step.

CI/CD Recommendations

Trace on retry, not on every run. on-first-retry (or retain-on-failure without retries) gives full debugging data for failures at a fraction of the storage and runtime cost of on.
Link traces directly from CI output. Since show-trace and trace.playwright.dev both accept URLs, wire your CI's artifact link straight into a one-click trace view instead of a manual download-and-unzip step.
Treat "couldn't reproduce" as a process gap, not bad luck. If a failure recurs without a trace attached, that's a signal to widen trace coverage, not just re-run the suite again.

Official Resources

Trace viewer documentation — full walkthrough of every tab and feature
testOptions.trace — configuration reference
trace.playwright.dev — the hosted viewer, for opening traces without the CLI
Playwright documentation — general guides, including CI configuration

Final Thoughts

Most test-debugging time isn't spent fixing bugs — it's spent recreating the conditions that exposed them. Trace Viewer removes that step by keeping a full record of the run itself, so the question shifts from "can I reproduce this?" to "what does the trace show?"

For engineering teams, the value isn't measured by prettier debugging tools — it's measured by fewer engineering hours lost to reproducing failures. Shorter debugging cycles, fewer unnecessary CI reruns, and faster root-cause analysis are improvements that show up in both developer productivity and infrastructure costs.

Stop Debugging Blind: Playwright Screencast API for Richer Test Evidence

Vanessa Sastre — Wed, 22 Jul 2026 22:13:02 +0000

When an automated test fails, the error message alone is often insufficient to diagnose the root cause.

You're usually left asking:

What actions happened before the failure?
Which element was being interacted with?
Where exactly did the app start behaving differently than expected?
Can a teammate understand the failure without re-running the test themselves?

Playwright 1.59 introduced page.screencast, a programmatic recording API that answers these questions by turning a test run into a narrated, annotated video — instead of just a pass/fail line in a report.

With screencast-based reporting, a test can produce:

A recording of the full execution
Visual annotations on every interacted element
Chapter markers that break the run into named stages
Custom HTML overlays for context or metadata
Real-time JPEG frame capture for external tooling
"Video receipts" for AI-driven test and coding agents

The goal is simple: make failures faster to understand, and make passing runs easier to trust.

Why Screencasts Matter

Traditional artifacts already carry useful information:

Artifact	What it tells you
Logs	What the framework reported
Screenshots	The app's state at one moment
Traces	DOM snapshots, network activity, console output

A screencast adds a fourth layer that none of these provide well on their own: the full user journey, as a video.

Instead of staring at this:

Expected: "Order completed"
Received: "Payment failed"

...you watch the actual sequence: checkout opens, a product is added, payment details are entered, submit is clicked, and the error appears, all annotated in place.

Screencast vs. Playwright Trace

These two tools solve different problems, and a mature framework should use both.

Trace Viewer is built for deep technical debugging:

Network activity
DOM snapshots
Console messages
Step-by-step execution inspection

Screencast is built for fast, visual understanding:

A timeline of what a user (or agent) actually did
Visible, annotated interactions
Context that non-technical stakeholders can follow without opening dev tools

In practice: reach for traces when you need to diagnose why something broke, and for screencasts when you need to show what happened quickly, and to a wider audience than just engineers.

Setting Up Screencast Recording

The cleanest way to roll this out across a suite is a shared fixture, so every test gets recording, annotations, and reporting for free — no boilerplate in individual test files.

// fixtures.ts
import { test as base } from '@playwright/test';

export const test = base.extend({
  page: async ({ page }, use, testInfo) => {
    const videoPath = testInfo.outputPath('screencast.webm');

    await page.screencast.start({ path: videoPath });
    await page.screencast.showActions({ position: 'top-right' });

    await use(page);

    await page.screencast.stop();

    await testInfo.attach('Screencast', {
      path: videoPath,
      contentType: 'video/webm',
    });
  },
});

Every test that imports test from this file now automatically records a screencast and attaches it to the report, including in CI.

Adding Narration Inside a Test

Once the fixture is in place, use showChapter() to label the stages of a test as it runs:

import { test } from './fixtures';

test('checkout flow', async ({ page }) => {
  await page.screencast.showChapter('Login', {
    description: 'User authentication flow',
  });
  await page.goto('/login');
  // ...login steps

  await page.screencast.showChapter('Checkout', {
    description: 'Complete purchase flow',
  });
  // ...checkout steps
});

The resulting video reads like a labeled walkthrough rather than raw, unexplained footage.

Screencast API Reference

`page.screencast.start(options)`

Starts a screencast session. Pass path to save a WebM recording, or onFrame to receive JPEG frames in real time — both can be used together.

await page.screencast.start({
  path: 'execution.webm',
  size: { width: 1280, height: 720 },
  quality: 90,
});

Because it's called explicitly, you control exactly when recording starts and stops, unlike the older, always-on video config option, which records for the entire test lifecycle regardless of what you actually need.

`page.screencast.stop()`

Stops the active session and finalizes the video file at the path given to start(). Typically called right after the test body finishes:

await use(page);
await page.screencast.stop();

`page.screencast.showActions(options)`

Annotates interacted elements as they're clicked, filled, or hovered, so a reviewer can see exactly which element was touched and when.

await page.screencast.showActions({
  position: 'top-right',
  duration: 1500,   // ms each annotation stays visible
  fontSize: 16,
  cursor: 'pointer', // animates a cursor between actions; use 'none' to disable
});

Valid position values: top-left, top, top-right, bottom-left, bottom, bottom-right.

To turn annotations off mid-test:

await page.screencast.hideActions();

Suite-wide alternative: if you want action annotations everywhere without touching a fixture, enable them directly in playwright.config.ts:

export default defineConfig({
  use: {
    video: {
      mode: 'on',
      show: {
        actions: { position: 'top-left' },
        test: { position: 'top-right' },
      },
    },
  },
});

`page.screencast.showChapter(title, options)`

Displays a centered, blurred-backdrop overlay to mark a new stage of the recording. Disappears automatically after duration (defaults to a couple of seconds).

await page.screencast.showChapter('Payment validation', {
  description: 'Submitting card details and confirming charge',
  duration: 4000,
});

Chaining a few of these gives you a scannable timeline:

Login → Search product → Add to cart → Checkout → Payment validation

Reviewers can jump straight to the stage they care about instead of scrubbing a long recording blindly.

`page.screencast.showOverlay(html, options)` / `showOverlays()` / `hideOverlays()`

Adds arbitrary HTML on top of the recording — handy for environment info, build numbers, or debug messages.

await page.screencast.showOverlay(
  '<div style="color:red">Running checkout validation</div>',
  { duration: 3000 }
);

Omit duration and the overlay stays until you explicitly hide or remove it. showOverlays() / hideOverlays() toggle visibility of overlays already on screen without discarding them.

Real-Time Frame Capture

Pass onFrame to start() to stream JPEG-encoded frames while the test runs, instead of (or alongside) writing a video file:

await page.screencast.start({
  onFrame: ({ data, timestamp, viewportWidth, viewportHeight }) => {
    processFrame({ data, timestamp, viewportWidth, viewportHeight });
  },
});

This opens the door to live dashboards, vision-model analysis, or custom monitoring pipelines that react to what's on screen during execution, not after.

Agentic Video Receipts

Screencasts are also a good fit for AI-driven testing and coding agents, where a plain text summary ("verified checkout — passed") is much less convincing than a video showing exactly what the agent did.

await page.screencast.start({ path: 'receipt.webm' });
await page.screencast.showActions({ position: 'top-right' });

await page.screencast.showChapter('Verifying checkout flow', {
  description: 'Validating payment completion',
});

// Agent performs its verification steps here
await page.locator('#coupon').fill('SAVE20');
await page.locator('#apply-coupon').click();
await expect(page.locator('.discount')).toContainText('20%');

await page.screencast.showChapter('Done', {
  description: 'Checkout validation completed',
});
await page.screencast.stop();

The output is a self-contained "receipt": chapter titles explain intent, action annotations show exactly what was clicked or filled, and a human can review the whole thing in the time it takes to watch a short clip, no re-running the agent required.

CI/CD Recommendations

Recording every single test run adds up fast in storage and CI time. A few practical guardrails:

Record failures, not everything. Attach the screencast only when a test doesn't match its expected status:

  if (testInfo.status !== testInfo.expectedStatus) {
    await testInfo.attach('Screencast', {
      path: videoPath,
      contentType: 'video/webm',
    });
  }

Record critical paths deliberately. Reserve always-on recording for your highest-value flows (checkout, sign-up, payment) rather than every spec.
Clean up old artifacts on a schedule so recordings don't silently pile up in CI storage.

Official Resources

This post covers the common patterns, but the official docs are the source of truth for full option lists, edge cases, and future updates:

Screencast API reference — full method and options list
Playwright release notes — see the v1.59 entry for the original page.screencast announcement, and later releases for additions like the cursor option
Playwright documentation — general guides, including video/trace configuration If you hit something this post doesn't cover, those are the pages to check first.

Final Thoughts

Test automation is moving past a simple pass/fail line. Screencast-based reporting gives teams:

Faster failure diagnosis
A visual record of execution flow anyone can follow, technical or not
Better handoffs between QA and development
A foundation for reviewing AI-driven testing and coding agents

A mature framework shouldn't just report that a test failed, it should show what happened, where it happened, and give you what you need to fix it.