Salesforce Lightning applications can be challenging to automate, particularly when dealing with complex page structures, dynamically rendered components and repeated UI interactions.
One challenge that becomes increasingly noticeable when working with browser automation is locator identification.
In a simple application, identifying an element by its role, accessible name or text may be sufficient. However, enterprise applications such as Salesforce often involve complex workflows where similar interactions are performed across different records and sections.
This becomes even more interesting when introducing AI into test automation.
The challenge with generic locator generation
Playwright provides several powerful ways to identify elements, including accessible roles, names, text and other locator strategies.
For example:
const editButton = page.getByRole('button', {
name: 'Edit'
});
await editButton.click();
This works when the target button is unique.
But what happens when the same button appears in multiple sections of the page?
Playwright uses strictness checks for operations that require a single element. If a locator matches multiple elements, it throws a strict mode violation rather than arbitrarily selecting one.
One possible workaround is positional selection using nth():
const editButton = page.getByRole('button', {
name: 'Edit'
}).nth(1);
await editButton.click();
Although this can work, it introduces a dependency on element ordering. If the page structure changes or another matching element is introduced, the locator may target the wrong element.
The problem is not necessarily the locator strategy itself. It is the lack of sufficient context to distinguish between elements.
Finding an element is one thing. Identifying the correct element within its intended context is another.
A practical challenge in Salesforce Lightning
Consider a Salesforce workflow involving parent and child cases.
Both records may follow a similar status-update process. The automation needs to open the relevant case, click Edit, locate the Status field, open its dropdown and select the required value, such as In Progress, Approved or Rejected.
Although the interaction pattern is similar, the automation must operate on the correct record.
For example, updating the parent case should not accidentally update the child case, and vice versa.
This is where locator identification becomes important.
When using AI-driven automation, the model needs to inspect the available page information and identify the relevant elements for each interaction.
Depending on the framework implementation, it may also need to generate or retrieve locator information as the workflow progresses.
Sometimes, the AI fails to identify the expected element or misses the relevant locator information. The interaction then fails, potentially causing the entire test case to fail.
The underlying issue is not that Playwright cannot interact with these elements. It is that the AI must consistently identify the correct element within the correct application context.
Why AI-driven automation introduces another challenge
AI-assisted automation adds another dimension to locator identification.
An LLM can inspect page information and attempt to identify the target element based on the context it receives.
However, when that information contains numerous elements with similar descriptions, the model must distinguish between them and understand their relationships.
For complex enterprise applications, providing an entire page representation can introduce a significant amount of information that may not be relevant to the requested interaction.
The model needs to determine:
- Which section or record contains the intended element?
- What is the relationship between the parent and child elements?
- Are there multiple elements with identical or similar accessible names?
- Which element belongs to the requested business context?
- Is the identified element relevant to the current step of the workflow?
For example, locating a Status field is not enough if the page contains multiple records or sections with similar fields.
The model must first understand which case is being updated and then identify the Status field belonging to that case.
This makes context an important part of AI-driven locator generation.
What if we changed the way we provide context?
Instead of asking AI to identify an element across the entire page, what if we first helped it understand the relevant section?
Imagine a page containing multiple customer records, each with its own Edit button.
Rather than identifying an Edit button globally, we can first identify the relevant customer container and then locate the button within that container.
This is the principle behind scoped locator strategies.
Playwright supports this through locator chaining and filtering:
const customerSection = page
.getByTestId('customer-card')
.filter({
has: page.getByRole('heading', {
name: 'Customer A',
exact: true
})
});
const editButton = customerSection.getByRole('button', {
name: 'Edit',
exact: true
});
await editButton.click();
Here, the first locator identifies the relevant customer container. The second locator searches for the Edit button within that container.
This establishes a parent-child relationship between the locators rather than searching for the button across the entire page.
The example uses illustrative test IDs and headings; actual selectors depend on the application's markup and accessibility structure.
We should still validate that the parent and child locators resolve to exactly one intended element.
Scoped locators provide a way to reduce ambiguity by limiting element identification to the relevant application context.
Where does this leave AI-driven automation?
Scoped locators help address element ambiguity, but how can we make an AI agent understand which scope it should use in the first place?
Can we improve AI-generated locator identification by providing structured accessibility information and explicit contextual instructions instead of asking the model to search through an entire page representation?
What if the AI could first identify the relevant scope and then search for the requested element only within that context?
This is the approach I want to explore in this series.
In the next article, I'll look at Playwright's Accessibility Tree, how semantic information can be represented in a structured format, and why it can provide useful context for AI-driven browser automation.
What's next?
This is the first article in a series exploring context-aware AI-driven locator generation with Playwright.
In the upcoming articles, I'll explore Playwright's Accessibility Tree and how structured accessibility information, natural-language instructions and scoping techniques can help AI identify elements more reliably.
Stay tuned for Part 2!
Top comments (0)