DEV Community

Mohammad Bahauddin
Mohammad Bahauddin

Posted on Edited on

Understanding Playwright's Accessibility Tree: Providing Better Context for AI-Driven Locator Generation

#playwright #automationtesting #genai #agentaichallenge

#testing #automation #playwright #ai #agenticai

In the first article of this series, we explored why generic locator generation can become challenging when automating complex enterprise applications such as Salesforce Lightning.

We discussed how multiple elements can share identical accessible names and why identifying an element without understanding its surrounding context can lead to unreliable automation.

This raises an interesting question:

What if we could provide AI with a structured representation of the page instead of asking it to identify elements from an entire page?

One approach worth exploring is Playwright's Accessibility Tree and its accessibility snapshot capabilities.

1. Understanding the Accessibility Tree

A browser does not simply render HTML elements visually. It also exposes information about the interface to assistive technologies such as screen readers.

This information is represented through the Accessibility Tree.

Unlike the DOM, which describes the document structure, the Accessibility Tree focuses on how elements are presented and understood by assistive technologies.

For example, consider the following HTML:

<button aria-label="Edit customer record">
  <svg>
    <!-- Edit icon -->
  </svg>
</button>
Enter fullscreen mode Exit fullscreen mode

The DOM contains a button and an SVG element.

However, its accessibility representation can expose information such as:

  • Role: button
  • Accessible name: Edit customer record

This semantic information is particularly useful when working with Playwright.

Instead of depending entirely on CSS selectors or DOM attributes, we can use accessible roles and names to identify elements.

const editButton = page.getByRole('button', {
  name: 'Edit customer record'
});

await editButton.click();
Enter fullscreen mode Exit fullscreen mode

This is one of the reasons Playwright recommends user-facing locators such as roles and labels.

2. Exploring Accessibility Snapshots in Playwright

Playwright provides the locator.ariaSnapshot() API to retrieve an accessibility-oriented representation of a locator's contents.

Let's look at a simple example.

const snapshot = await page.locator('body').ariaSnapshot();

console.log(snapshot);
Enter fullscreen mode Exit fullscreen mode

The output is a YAML-like representation of accessible elements.

For example, a simplified representation might look like this:

- main:
  - heading "Customer Cases" [level=1]
  - region "Parent Case":
    - heading "Case #1001"
    - button "Edit"
  - region "Child Case":
    - heading "Case #1002"
    - button "Edit"
Enter fullscreen mode Exit fullscreen mode

The example above is illustrative. Actual output depends on the application's accessible markup and browser state.

Notice something interesting.

Instead of receiving only a collection of buttons named Edit, we can potentially understand the sections in which those buttons appear.

This additional information can help distinguish between otherwise similar elements.

3. Why structured accessibility information matters

Consider a Salesforce workflow involving parent and child cases.

Both records contain an Edit button.

A generic locator might look like this:

const editButton = page.getByRole('button', {
  name: 'Edit'
});
Enter fullscreen mode Exit fullscreen mode

If multiple buttons match, Playwright's strictness checks prevent the automation from arbitrarily selecting one.

We could use positional selection:

const editButton = page.getByRole('button', {
  name: 'Edit'
}).nth(0);
Enter fullscreen mode Exit fullscreen mode

However, this introduces a dependency on element ordering.

A more context-aware approach would be to identify the relevant section first.

For example:

const parentCase = page
  .getByRole('region', {
    name: 'Parent Case'
  });

const editButton = parentCase.getByRole('button', {
  name: 'Edit',
  exact: true
});

await editButton.click();
Enter fullscreen mode Exit fullscreen mode

This approach limits the search to the intended region.

The important distinction is that we are no longer identifying the button independently. We are identifying it within the context of the parent case.

Of course, this assumes the application exposes a meaningful accessible region with the appropriate name. Real Salesforce markup may require a different scoping strategy.

4. Where AI-driven automation fits into this

Now let's introduce AI into the workflow.

Imagine an AI agent receiving the following instruction:

"Open the parent case and change its status to In Progress."

The agent needs to understand:

  1. Which record represents the parent case?
  2. Where is the Edit button associated with that record?
  3. Which Status field belongs to the selected record?
  4. Which option should be selected?

If we provide only a large collection of page elements, the model needs to infer these relationships from potentially ambiguous information.

However, if we provide a structured accessibility snapshot, the model gets additional semantic information.

A simplified instruction could look like this:

Task:
Update the Parent Case status to In Progress.

Relevant page structure:

Region: Parent Case
  Heading: Case #1001
  Button: Edit

Region: Child Case
  Heading: Case #1002
  Button: Edit

Instruction:
Identify the Edit button within the Parent Case region.
Do not interact with the Child Case.
Enter fullscreen mode Exit fullscreen mode

This is not a complete AI automation implementation. It illustrates how structured context and explicit instructions can help reduce ambiguity during locator identification.

5. Accessibility snapshots are useful, but not a complete solution

There is an important limitation worth understanding.

The Accessibility Tree does not necessarily expose every element present in the DOM.

Some elements may be excluded because they are not accessible, are hidden, or do not contribute meaningful accessibility information.

Additionally, Salesforce Lightning contains dynamically rendered components, custom controls and complex interactions.

An accessibility snapshot may not always provide enough information to identify every required element.

For example:

  • Custom dropdowns may expose different accessibility structures.
  • Dynamically rendered components may change the snapshot.
  • Multiple controls can still have identical accessible names.
  • Some elements may require additional DOM-level inspection.

Therefore, accessibility snapshots should not be treated as a replacement for the DOM or as a guaranteed solution to locator ambiguity.

Instead, they can serve as another source of structured information for an automation agent.

6. What can we take away from this?

Playwright already provides powerful locator strategies.

The challenge in AI-driven automation is not simply generating a valid locator. It is generating a locator that identifies the correct element within the intended business context.

Accessibility information provides a useful foundation because it exposes semantic information such as roles, accessible names and relationships between interface elements.

When combined with contextual instructions and scoped locator strategies, this information can help an AI agent make more informed locator decisions.

However, there is still an important question:

Can we make AI identify the relevant section first, and then generate locators only within that section?

That is what I want to explore next.

What's next?

In Part 3, I'll explore a context-aware locator generation approach using Playwright, TypeScript and AI.

We'll look at how an automation agent can:

  • Understand the requested business context.
  • Identify the relevant section of the application.
  • Extract useful accessibility information.
  • Generate scoped Playwright locators.
  • Validate that the identified element matches the intended target.

The goal is to move beyond generic locator generation towards context-aware AI-driven browser automation.

Part 3: Building a Context-Aware AI Locator Generation Approach with Playwright.

#playwright #typescript #javascript #testautomation #AI #AIBasedAutomation #softwaretesting

Top comments (2)

Collapse
 
autenai profile image
Auten •

On your part 3 question: when a Lightning region has no accessible name, what will you scope by instead, the nearest heading text or a DOM attribute? One thing that helped us with duplicate "Edit" buttons: after the click, re-read the snapshot and check the expected record's heading is in the opened form before typing anything. (From the Auten team, we work on agents driving real screens.)

Collapse
 
mohd_bahauddin_afb498f213 profile image
Mohammad Bahauddin •

That's a great suggestion! I'll definitely include this in Part 3.

I'll explore heading-based scoping when accessible region names aren't available, along with stable DOM attributes as an alternative.

I also like the idea of re-reading the accessibility snapshot after an interaction and validating the expected record heading before proceeding. That's an important validation step that I hadn't covered in Part 2.

Thanks for sharing this. It gives me another practical scenario to explore in the next article!