DEV Community: Antoine Dubois

Most “Flaky Tests” Are Really Architecture Tests

Antoine Dubois — Sat, 08 Aug 2026 21:26:37 +0000

There’s a point in almost every browser automation project where someone says:

“The tests are getting flaky.”

And then the team starts fixing the tests.

Longer waits. More retries. New selectors. Another helper function. Maybe a wrapper around the wrapper around click().

Sometimes that’s the right answer.

But increasingly, I think “flaky test” is becoming a catch-all phrase for something much broader:

Modern frontends have become surprisingly difficult environments to observe deterministically.

The test isn’t always the problem.

The DOM Isn’t Really the DOM Anymore

A decade ago, browser automation had a fairly straightforward mental model.

Load page. Find element. Click element.

Today you might be dealing with:

Shadow DOM
nested iframes
third-party widgets
microfrontends
virtualized lists
components that render differently based on their container size
UI updates arriving over WebSockets
client-side navigation without full page loads

That’s a different problem.

For example, testing Shadow DOM, nested iframes, and embedded widgets is less about writing the perfect selector and more about whether your automation system understands the boundaries between those different contexts.

A perfectly valid selector can still be useless if your test is searching in the wrong DOM context.

Responsive Testing Got More Complicated Too

We used to think about responsiveness in terms of viewport width.

Desktop. Tablet. Mobile.

CSS container queries changed that assumption.

A component can now change dramatically while the browser window stays exactly the same size.

That means testing only a handful of viewport resolutions can miss real regressions. There’s a useful breakdown of testing container queries, resize behavior, and breakpoint edge cases that illustrates how these failures tend to appear.

This is one of those changes that looks small from the CSS side and surprisingly large from the QA side.

Then There’s Time

A lot of UI state isn’t created by page load anymore.

It arrives later.

Consider an application consuming WebSockets or Server-Sent Events.

The test opens the page.

The page says “Connected.”

The test waits for the transaction.

Nothing happens.

Failure.

Run it again.

Pass.

This is where arbitrary sleeps become dangerous. You’re trying to fix a state synchronization problem with a stopwatch.

A better approach is understanding how WebSocket, SSE, and live-data browser tests fail and synchronizing against meaningful application state instead of elapsed time.

The same principle applies to SPAs.

Browser history, back-button behavior, restored scroll positions, and cached component state can produce bugs that don’t appear when every automated test starts from a clean URL.

That makes client-side routing and scroll restoration worth treating as first-class behavior rather than navigation trivia.

Selectors Still Matter. Just Differently.

I don’t think selectors are going away.

But I do think teams often spend too much time trying to make one clever selector survive every imaginable frontend change.

That becomes especially painful with microfrontends.

One team renames a component.

Another team introduces a wrapper.

A shared UI package changes its DOM structure.

Suddenly 87 tests fail.

A sane selector strategy for microfrontends and shared component libraries is mostly about establishing contracts between developers and tests.

That’s much more scalable than XPath archaeology.

Virtualized Interfaces Are Another Trap

Infinite scroll and virtualized tables are great examples.

An item can conceptually exist in the application while not existing in the DOM.

That distinction matters.

It’s why comparing approaches such as Endtest vs Playwright for heavily virtualized interfaces gets interesting. The question isn’t merely whether the tool can click an element.

The question is how much machinery you have to build around that click.

The Useful Question

When a browser test fails, I’d stop asking:

“How do we make this test pass?”

At least initially.

Ask:

“What assumption did this test make about the application?”

Maybe it assumed the element was already rendered.

Maybe it assumed the route caused a page load.

Maybe it assumed the item existed in the DOM.

Maybe it assumed the component responded to viewport width.

Maybe it assumed there was only one document context.

Once you find that assumption, the failure usually becomes much less mysterious.

And strangely enough, your tests tend to become much less flaky too.

Visual Regression Testing Has a Noise Problem

Antoine Dubois — Fri, 07 Aug 2026 22:07:21 +0000

Visual regression testing sounds almost unfairly simple.

Take screenshot A.

Take screenshot B.

Compare them.

If they’re different, something changed.

Unfortunately, browsers have spent the last decade making “something changed” a surprisingly complicated concept.

Fonts render differently.

Content shifts by a pixel.

Animations fire.

Responsive layouts reflow.

Container queries change components independently of the viewport.

Web Components hide half the page inside Shadow DOM.

Suddenly your visual testing system is producing hundreds of differences, and the person reviewing them develops the highly sophisticated QA technique known as:

clicking Accept repeatedly until the red badges disappear.

At that point, you technically have visual regression testing.

You just don’t have much visual regression detection.

Signal matters more than sensitivity

The instinct when evaluating screenshot tools is to ask:

How precisely can it detect pixel differences?

That’s not the question I’d start with.

A tool that detects every single pixel difference may be worse than one that ignores harmless variation.

The real question is:

How quickly can a human distinguish an actual regression from noise?

Those are very different product requirements.

A useful evaluation should include baseline management, masking, thresholds, rendering consistency, review workflows, and what happens when the expected UI legitimately changes.

This guide to evaluating browser testing tools for visual diff noise, baseline drift, and screenshot review gets into those details.

Baseline drift is especially dangerous.

If developers routinely approve new screenshots without understanding the differences, your baseline slowly stops representing “correct.”

It represents “whatever the application looked like the last time someone got tired of reviewing screenshots.”

Fonts are chaos disguised as typography

Fonts deserve their own section because they cause a ridiculous amount of visual noise.

A font loads late.

The fallback font has slightly different metrics.

A headline becomes one pixel wider.

A card becomes three pixels taller.

Everything below it moves.

Your visual comparison now tells you that half the page changed.

Technically, it’s right.

Operationally, it’s useless.

The same issue shows up with:

dark mode,
themes,
localization,
dynamic content,
personalization,
responsive typography.

There’s a good exploration of this in what to evaluate in a visual regression tool for dynamic fonts, theming, and layout shifts.

A strong visual test system needs to understand that not every difference has equal importance.

A missing checkout button and a one-pixel font shift should not create identical urgency.

Responsive testing isn’t testing three viewport widths

For years, responsive testing meant:

desktop,
tablet,
mobile.

That model is increasingly incomplete.

CSS Grid and Flexbox often fail at awkward widths that nobody specifically designed.

A navigation menu might look perfect at 1440px and 375px but wrap badly at 847px.

A product card might collapse because one translation is longer than the English version.

These are the bugs users actually see.

This walkthrough of testing CSS Grid, Flexbox, and responsive wrapping without missing layout breakpoints makes an important point: meaningful breakpoints often come from the content, not from your CSS framework.

And container queries push this even further.

A component’s layout may now depend on its parent container rather than the browser viewport.

You can have two copies of the same component on the same page behaving differently.

That makes old-school “set browser width, take screenshot” strategies increasingly inadequate.

If your frontend uses them heavily, testing CSS container queries without creating visual noise is worth a look.

Then Shadow DOM arrives

Shadow DOM solves legitimate frontend architecture problems.

It also introduces a fun testing problem:

The element you can clearly see in the browser may not exist where your automation expects it to exist in the DOM tree.

That can affect:

selectors,
interactions,
accessibility checks,
screenshots,
debugging,
reusable page objects.

If your application uses a large component library based on Web Components, this shouldn’t be an edge-case checkbox in your tool evaluation.

It should be part of the proof of concept.

There’s a useful checklist for evaluating test automation platforms for Shadow DOM and encapsulated UI libraries.

And don’t just test a single toy component.

A framework being technically capable of entering a shadow root doesn’t tell you much about how it performs against a real application containing hundreds of components.

I like the more empirical approach described in benchmarking browser test stability on Shadow DOM-heavy frontends.

Run enough representative tests to discover where the maintenance cost actually appears.

Screenshot count is a terrible KPI

One final trap: measuring visual-test maturity by the number of screenshots.

“We have 4,000 screenshot assertions” sounds impressive.

It might also mean four engineers are approving meaningless diffs every morning.

The useful metrics are closer to:

regressions caught before production,
false-positive rate,
review time,
baseline churn,
percentage of visual checks people still trust.

That last one is hard to put on a dashboard.

But it might be the most important.

Because visual regression testing is only useful as long as humans believe the red pixels mean something.

The moment they stop believing that, you have a very expensive screenshot generator.

Most CI Failures Are Environment Failures

Antoine Dubois — Wed, 05 Aug 2026 19:09:02 +0000

When a browser test passes locally and fails in CI, the first diagnosis is often “flaky test.”

That diagnosis is convenient because it does not require us to explain anything.

Rerun the job. Add a retry. Increase the timeout. Move on.

Sometimes that is enough to unblock the pipeline. But it also teaches the team to treat unexplained behavior as normal.

Most CI-only failures are not random. They are deterministic reactions to differences we have not made visible yet.

Parallelism changes the system

A test suite can behave perfectly when executed one test at a time and collapse under parallel execution.

That is not necessarily a browser problem.

Parallel jobs introduce competition for:

CPU
Memory
Network bandwidth
Shared test accounts
Database records
File names
Ports
Rate-limited APIs
Preview environments

This guide to debugging GitHub Actions browser jobs that fail under parallelism is a good starting point because it treats parallelism as a change in system behavior, not merely a faster way to run the same tests.

Two tests using the same account may invalidate each other’s sessions.

Two jobs may update the same order.

A service that responds in 300 milliseconds locally may take three seconds when eight CI workers hit it at once.

The failure may appear in the browser, but its cause can live anywhere in the environment.

Look for predictive signals

Teams often wait until the pipeline is red before collecting diagnostic information.

By then, the most useful transient evidence may already be gone.

This article on signals that predict browser-test failures before CI turns red suggests watching the conditions around the test, not only its final status.

Useful leading indicators include:

Increasing page-load time
More locator retries
Slower API responses
Rising console-error counts
Higher memory use
Longer queue times
Increased test-data conflicts
Frequent browser restarts

A test that still passes after six retries is not healthy.

It is an early warning.

Preserve enough evidence to reconstruct the run

A screenshot is helpful, but it is rarely sufficient.

The browser may show an empty state because the API failed, the session expired, the frontend crashed, or the test reached the wrong route.

A practical diagnostic package should include:

Screenshot
Current URL
Console logs
Network failures
Browser and operating-system version
Viewport
DOM or page-source snapshot
Failed locator
Timing information
Test-data identifiers

This guide on diagnosing browser failures with console logs, network timing, and DOM snapshots explains why these artifacts are more valuable together than independently.

A console error can explain the blank screenshot.

A network trace can explain the console error.

A DOM snapshot can reveal that the expected element existed but was hidden behind a loading state.

The goal is not to collect everything. It is to preserve the shortest path to an explanation.

Playwright is not immune to CI drift

Playwright does a lot to make browser automation more reliable, including automatic waiting and strong browser tooling.

But it still runs inside an environment.

This guide to debugging Playwright tests that fail only in CI covers the differences that still matter:

Headless versus headed behavior
Browser versions
Fonts
Locale and timezone
Missing secrets
Viewport dimensions
CPU throttling
Network conditions
Container permissions

The right question is not, “Why is Playwright flaky?”

It is, “Which assumption in this test is different in CI?”

That wording changes the investigation.

Real-time flows expose weak waits

WebSocket and real-time applications are particularly good at revealing timing assumptions.

There may be no navigation to await. The UI changes when an event arrives, and that event may depend on another service completing work.

This guide on testing WebSocket and real-time UI flows without phantom failures recommends waiting for meaningful state rather than arbitrary time.

Do not sleep for two seconds and hope the message arrives.

Wait until:

The status changes
The expected item appears
The notification contains the relevant event
The backend confirms the state
The UI stops showing an intermediate condition

A fixed delay is a guess disguised as synchronization.

Sessions and tabs are environment state too

Multi-tab and cross-domain flows create additional failure modes.

A test may open a payment provider, authenticate in a second tab, return to the original application, and expect the session to persist.

Locally, the browser may already contain cookies or cached permissions. In CI, it starts from a clean environment.

This guide to evaluating browser tools for multi-tab, session-persistence, and cross-domain workflows highlights why these workflows should be part of tool evaluation.

A platform may handle simple single-page tests well and still struggle when browser context becomes part of the scenario.

Build a failure narrative

At Endtest, we have been working toward results that show the failed step, the surrounding logs, screenshots, and the information needed to investigate what happened. AI-assisted failure analysis can help suggest a likely cause, but it should always point back to the evidence.

That is the standard I would use for any test system.

A failure result should tell a coherent story:

What did the test attempt?

What did the browser observe?

Which environment conditions mattered?

What changed from the last successful run?

Who should investigate next?

Retries can keep a pipeline moving.

Evidence makes the system better.

Your CI Failure Probably Is Not Random

Antoine Dubois — Tue, 04 Aug 2026 22:20:28 +0000

There is a particular kind of browser test failure that can consume an entire afternoon.

The test passes on your laptop.

It passes when you rerun it locally.

It may even pass when you SSH into the CI machine and execute it manually.

But inside the real pipeline, it fails.

The easiest explanation is that the test is flaky. Sometimes that is true. More often, “flaky” is just a word we use when we have not identified the environmental difference yet.

Start by assuming the environments are different

Your laptop and the CI runner are not the same environment.

Even when they use the same browser version, they may differ in:

CPU availability
Network latency
Screen dimensions
Font availability
Locale
Timezone
Cached assets
Test data
Browser permissions
Feature flags
Secrets and authentication state

This checklist for E2E tests that fail only in CI is a useful place to begin because it separates timing, data, and environment drift instead of treating every failure as another waiting problem.

Adding a larger timeout can make the test pass. It does not necessarily fix the cause.

Sometimes it simply turns a five-second mystery into a thirty-second mystery.

Time is test data

Date-sensitive applications are especially good at producing failures that look random.

A billing screen may behave differently near the end of the month. A calendar may calculate the first day of the week from the browser locale. A promotion may expire according to the server timezone rather than the browser timezone.

The problem becomes more confusing when the developer, CI service, and production environment are in three different regions.

This guide on testing locale, timezone, and calendar-dependent interfaces recommends treating time configuration as an explicit part of the test rather than an ambient property of the machine.

That means recording the timezone and locale in the test result. It also means using fixed dates when the workflow permits it.

A screenshot showing “March 31” is not enough evidence if you do not know which timezone produced it.

Real-time interfaces need state-based waits

WebSocket-driven interfaces create a different class of timing problem.

A user performs an action, the server processes an event, and the browser updates when a message arrives. There may be no navigation and no obvious network request for the automation framework to wait for.

Sleeping for two seconds works until the server needs three.

This article on testing WebSocket-driven UI flows without chasing race conditions makes the stronger approach clear: wait for the product state you actually care about.

Do not wait because “the UI probably finished.”

Wait until:

The expected notification appears
The status changes
The item enters the correct list
The relevant event has been reflected in the DOM
A specific backend state can be verified

A test should synchronize with meaning, not time.

Rendering problems often masquerade as assertion failures

A visual assertion can fail even when the underlying product behavior is correct.

Fonts may load later in CI. Images can be decoded at a different speed. Cookie banners may alter the viewport. A container may shift after a client-side component initializes.

The guide to debugging layout shift before it becomes visual test flakiness is relevant even for teams that do not run screenshot comparisons.

Layout shifts can cause ordinary click failures too. The test identifies the correct element, but another component moves it between locating and clicking.

Before blaming the automation framework, check whether the page is visually stable.

Performance budgets should be selective

Performance checks belong in CI, but running a complete performance audit on every change can make developers resent the pipeline.

That usually leads to one of two outcomes:

The checks are ignored.
The checks are removed.

A better approach is described in this article on enforcing frontend performance budgets without slowing every merge.

Use lightweight checks for every pull request and reserve expensive measurements for relevant changes, scheduled builds, or release candidates.

The objective is not to collect every possible metric. It is to catch meaningful regressions while developers can still act on them.

Pull requests are their own environment

One of the more confusing CI patterns is a test that fails on pull requests but succeeds after the same code reaches the main branch.

This is often caused by configuration rather than application behavior:

Forked pull requests may not receive secrets.
Preview environments may use different URLs.
Pull-request workflows may have different permissions.
The branch may be tested before dependent assets are deployed.
Cache keys may differ.
Conditional workflow steps may run only on the default branch.

This analysis of GitHub Actions browser tests that fail only on pull requests is a reminder to compare workflow execution paths, not only test code.

The same test command does not guarantee the same test environment.

Build a failure package, not just a failure message

“Element not found” is barely useful as a diagnostic result.

A useful CI failure should preserve enough evidence to reconstruct what the browser experienced:

Screenshot
Current URL
Browser and operating system
Viewport size
Console errors
Relevant network failures
Page source or DOM snapshot
Locale and timezone
Test data identifiers
The exact failed step

This is where modern automation platforms can save significant time. Whether you use a managed platform such as Endtest or your own Playwright or Selenium infrastructure, the run should produce an investigation package.

The goal is not merely to know that the test failed.

The goal is to reduce the distance between failure and explanation.

Most CI-only failures are not random. They are deterministic reactions to variables the team has not made visible yet.

The best debugging improvement is often not another retry.

It is better evidence.

Your CI Pipeline Does Not Need More Tests. It Needs Better Decisions

Antoine Dubois — Mon, 03 Aug 2026 21:25:34 +0000

It is easy to measure the size of a test suite.

You can count test cases, browsers, devices, assertions, executions, and minutes of runtime.

It is much harder to measure whether the suite helps the team make better decisions.

That is the real job of CI.

A pipeline should help answer a practical question:

Is this change safe enough to move forward?

When the pipeline cannot answer that question clearly, teams compensate by adding more tests, rerunning failures, and asking developers to inspect increasingly large logs.

The result is slower CI without greater confidence.

Stop running everything because it feels safer

Running every automated test for every code change sounds responsible.

It is also frequently wasteful.

A documentation update probably does not need the same validation as a change to authentication, billing, or shared application state. Yet many pipelines treat them identically because the organization has no reliable way to estimate impact.

This creates two problems.

First, feedback becomes slow. Developers wait for unrelated tests before learning whether their change is usable.

Second, important failures become harder to notice because they are mixed into a much larger volume of routine execution.

Test impact analysis attempts to solve this by mapping changes to the areas, dependencies, and user flows most likely to be affected.

A practical guide to building a test impact analysis workflow for faster CI/CD decisions explains how to approach this without pretending the mapping will be perfect from day one.

You do not need a flawless prediction engine.

You need a process that is better than “run absolutely everything and hope someone reads the result.”

A failure without context is barely useful

A browser test fails on step 17.

The screenshot shows a loading spinner.

What happened?

Perhaps the API was slow. Perhaps a JavaScript exception prevented rendering. Perhaps the previous click never registered. Perhaps the test reached the page with an expired session.

A single screenshot rarely answers the question.

This is where session replay becomes useful. A good replay can show the sequence of interactions, DOM changes, console errors, network requests, redirects, and timing leading up to a failure.

But simply recording video is not enough.

A useful debugging workflow should connect the replay to the failed step and make it easy to compare the expected path with the actual one.

This article on building a browser session replay debugging workflow for flaky UI tests offers a practical framework.

The important shift is to design replay around diagnosis rather than archiving.

Nobody benefits from terabytes of test recordings that are difficult to search and rarely opened.

Webhooks need first-class evidence

Webhook failures are a perfect example of CI ambiguity.

The application performs an action. A remote service is supposed to receive an event. The pipeline times out.

Now the investigation begins.

Was the webhook generated? Was it sent? Did DNS fail? Did the receiving endpoint reject the signature? Was the event delivered twice? Did the endpoint accept it but fail during processing?

Without structured evidence, each pipeline run becomes a small forensic project.

A reliable webhook test should preserve:

The event identifier
The payload
The signature headers
Delivery attempts
Response codes
Retry timing
The receiving system’s processing result
Correlation identifiers across services

The guide to testing webhooks in CI without turning every pipeline run into a mystery covers the mechanics in more detail.

The broader lesson applies beyond webhooks: distributed behavior needs distributed evidence.

A browser screenshot cannot explain what happened inside an asynchronous backend workflow.

Flakiness should be detected as a pattern

Most teams notice flaky tests informally.

Someone says, “That one fails sometimes.”

Another person says, “Just rerun it.”

Eventually the workaround becomes part of the culture.

The problem with this approach is that human memory is poor at recognizing gradual changes. A test that failed once every fifty runs may now fail once every twelve, but nobody notices because each failure is handled independently.

Anomaly detection can help by treating test behavior as a time series.

You can monitor:

Changes in failure frequency
Sudden increases in runtime
Browser-specific instability
Failures concentrated on certain workers
New retry dependence
Correlation with deployments or infrastructure changes

This does not require an elaborate machine-learning system. Even a rolling baseline can reveal meaningful changes.

The article on adding flaky-test anomaly detection before developers start ignoring failures provides a useful implementation path.

The earlier you identify deterioration, the cheaper it is to fix.

Once the team has normalized rerunning failures, the technical problem has become a cultural problem too.

Sometimes the test is fine and the grid is slow

Not every inconsistent browser test is caused by a weak locator or missing wait.

Distributed browser infrastructure introduces its own failure modes.

A Selenium Grid may experience latency between the test runner, hub, browser node, application server, and external dependencies. Each individual delay may be small. Combined across hundreds of commands, they can change the timing enough to expose failures.

Signs of grid-related latency include:

Commands becoming slow across unrelated tests
Failures clustering on specific nodes
Timeouts increasing during parallel execution
Large differences between local and remote execution
Browser startup delays
Long gaps between a command and its corresponding application activity

This guide to spotting Selenium Grid network bottlenecks before they become flaky tests explains what to measure.

The important point is diagnostic discipline.

Do not rewrite a stable test because the network is overloaded. Do not increase every timeout because one node is unhealthy. Fix the layer that is actually failing.

CI should reduce uncertainty

A fast pipeline is useful.

A comprehensive pipeline is useful.

But neither matters if the result does not guide action.

The best CI systems are designed around decisions:

Which tests are relevant to this change?
What exactly happened when one failed?
Is the failure new or part of an existing pattern?
Did the application fail, or did the test infrastructure fail?
Can the team proceed safely?

When CI answers those questions, testing becomes a development tool.

When it does not, the pipeline becomes a ceremony everyone waits for and nobody fully trusts.

Your Browser Tests May Be Passing in a World Your Users Never See

Antoine Dubois — Wed, 29 Jul 2026 09:11:06 +0000

A browser test can pass perfectly while testing something that barely resembles the user’s experience.

This is not usually fraud or negligence. It is a side effect of how test environments evolve.

The test runner starts with a clean browser, a fixed viewport, a predictable location, a known account, and a URL pointing to a stable environment. Real users arrive with old cookies, narrow screens, unusual locale settings, browser extensions, consent choices, interrupted sessions, and devices your team may not own.

The more controlled the test environment becomes, the easier it is to forget what has been controlled away.

Headless mode is not merely Chrome without a window

Headless browsers have improved enormously.

For many applications, there is little practical difference between a headless and headed run. But “little difference” is not the same as “no difference.”

Problems can still emerge from:

Font availability
GPU acceleration
Media permissions
Window dimensions
Focus behaviour
Download handling
Animation timing
Browser visibility APIs
Clipboard access
Popups and new windows

This overview of why web applications break in headless mode provides a useful checklist.

The mistake is not using headless mode. Headless execution is practical and often necessary.

The mistake is assuming that one successful headless configuration represents every browser context that matters.

A sensible strategy is to run the large regression suite efficiently, then preserve a smaller group of high-risk flows for headed browsers, real devices, or both.

Responsive testing is not a screenshot exercise

Teams often treat responsive testing as a set of screenshots at three widths:

Desktop
Tablet
Mobile

That is better than nothing, but responsive failures are not limited to visual differences.

A navigation element may remain visible but stop responding to touch. A modal may fit the viewport but place its close button beneath the browser chrome. A sticky footer may cover the final form field only when the mobile keyboard opens.

This guide to testing responsive breakpoints in real browsers explores the failures that appear between Chrome, Safari, and mobile viewports.

The useful assertion is rarely “the screenshot matches exactly.”

It is more likely:

The user can reach the primary navigation.
The checkout button remains visible and clickable.
Text does not overlap interactive controls.
The focus order remains usable.
The page does not scroll horizontally.
The virtual keyboard does not block the next action.

These assertions survive minor design changes while still protecting the user journey.

Preview environments are built to drift

Preview deployments are excellent for moving quickly. They also create a messy testing surface.

A preview may contain:

A new frontend connected to an old API
A partial database migration
Different feature flags
Missing background workers
Temporary hostnames
Expired credentials
Stale test data
A branch that has not received the latest configuration changes

When a browser test fails in that environment, the application code may not be the cause.

The article on building a browser QA checklist for feature flags, rollbacks, and environment drift is useful because it treats the environment as part of the test result.

Before debugging selectors, confirm what was actually deployed.

Record the commit, feature-flag state, service versions, test account, region, hostname, and relevant configuration. Without those details, a failure from yesterday may be impossible to reproduce today.

The first page is often determined before your test starts

Geo routing, cookie consent, language selection, and regional entry paths can change the application before the first automated click.

A user in France may see a consent banner that a user in the United States never sees. A request from the United Kingdom may be routed to a different domain. A returning user may bypass the region selector because a preference already exists in a cookie.

This comparison of Playwright and Selenium for testing cookie consent, geo routing, and region-specific entry paths highlights how much setup can be required before the visible workflow begins.

The most reliable regional tests usually control several variables together:

IP location
Browser locale
Timezone
Existing cookies
Local storage
Accepted consent state
Application account region

Changing only the browser language may not be enough. Changing only the IP address may not be enough either.

The application may use several signals and resolve conflicts according to rules that are not documented anywhere.

Cross-domain authentication is a system, not a page

OAuth and SSO flows are where clean browser-test abstractions go to die.

The test leaves your application, reaches an identity provider, encounters a consent screen, opens a popup or redirect, and eventually returns with a new session.

Any part of that sequence can vary by account, browser policy, region, or previous consent.

This practical look at testing OAuth redirects, consent screens, and cross-domain login handoffs with Endtest outlines the real workflow more accurately than a simple “click Sign in” example.

The test should distinguish between several failures:

The identity provider never opened.
The user was already authenticated.
The consent screen appeared unexpectedly.
The callback URL was incorrect.
The application received the callback but failed to create a session.
The session was created but the frontend failed to update.

These are different bugs owned by different systems. A single timeout error hides that distinction.

Shadow DOM and accessibility can disagree with the screenshot

Modern component libraries frequently encapsulate controls inside Shadow DOM.

That creates a familiar situation: the button is clearly visible to a human, but the test cannot locate it using the selector strategy applied elsewhere in the application.

The reverse can happen too. A selector finds an element that appears correct visually, but its accessible name is missing or incorrect.

This guide to browser-testing platforms for Shadow DOM, accessible names, and component-library regressions is a reminder that the DOM tree, accessibility tree, and rendered interface are related but not identical.

A robust test may need to verify all three.

That is especially important when components are updated centrally. A small change to a shared button, modal, or input can create regressions across dozens of screens.

Editable grids expose browser differences quickly

Editable grids combine almost every difficult browser behaviour:

Keyboard navigation
Focus management
Inline validation
Virtualized rows
Copy and paste
Selection ranges
Scrolling containers
Delayed saves
Optimistic updates

A grid can look correct while losing keystrokes or saving values into the wrong row.

This comparison of Endtest and Playwright for editable grids and keyboard-driven data entry provides a practical framework for deciding what to automate.

Do not test only the happy path of clicking a cell and entering text.

Test tabbing, escaping, invalid data, row changes, sorting after edits, interrupted saves, and the persistence of values after a reload.

Passkeys force the test outside the webpage

WebAuthn passkeys make authentication better for users and more complicated for browser automation.

The flow can depend on:

The operating system
The physical device
A platform authenticator
Browser policy
Managed-enterprise settings
Biometrics
Cross-device QR flows
Previously registered credentials

A simulated browser context can test some of this, but not everything.

This comparison of Playwright and Selenium for testing WebAuthn passkeys explains why real-device coverage still matters.

The goal is not to run every test on expensive physical infrastructure. It is to identify which parts of the authentication journey cannot be represented faithfully by a generic browser session.

Add reality deliberately

You cannot reproduce every user environment.

You should not try.

Instead, identify the variables most likely to alter important behaviour:

Browser engine
Device class
Viewport
Location
Language
Consent state
Authentication state
Feature flags
Deployment configuration
Accessibility mode

Then build a small, deliberate matrix around those risks.

A thousand identical headless Chrome runs do not provide the same confidence as a carefully chosen set of environments.

The purpose of browser automation is not to create the cleanest possible laboratory.

It is to discover whether the product still works when the outside world reaches it.

Build an Evidence Supply Chain for Tests

Antoine Dubois — Mon, 27 Jul 2026 21:33:38 +0000

Most teams think of automated tests as decision machines.

Green means ship. Red means stop.

That model works right up until the first ambiguous failure.

Then someone opens the CI job, scrolls through 8,000 lines of logs, downloads a screenshot with an unhelpful filename, discovers the video has already expired, and asks in Slack whether anybody changed the login page.

At that moment, the test is no longer making a decision.

It is generating a mystery.

The difference between a mature test operation and a noisy one is often not the quality of the assertions. It is the quality of the evidence pipeline surrounding them.

A failed test is the beginning of work

A browser test failure creates a small investigation.

The investigator needs to know:

what ran;
where it ran;
which application version was tested;
which data and feature flags were active;
what the browser saw;
what the network returned;
what changed since the last successful run;
who owns the next action.

If that information is scattered across CI, S3, chat threads, dashboards, and somebody’s local machine, the failure becomes expensive regardless of how quickly the test itself executed.

A practical starting point is an explicit test artifact pipeline using AWS S3 for upload fixtures, screenshots, and failure evidence. The important word is pipeline.

Artifacts should not be random files saved because a framework happened to support screenshots. They should move through a defined lifecycle with predictable names, metadata, access controls, retention rules, and links back to the exact execution that produced them.

You are not storing screenshots.

You are preserving evidence.

Evidence without ownership becomes archaeology

Many teams can tell you where test results are stored.

Fewer can tell you who is expected to act on each class of failure.

That gap creates the familiar ping-pong thread:

QA says it looks like a frontend issue.

Frontend says the API returned something unexpected.

Backend says the test data is invalid.

DevOps says the environment was healthy.

Eventually someone reruns the job and it passes, allowing everyone to stop discussing it without learning anything.

A better model assigns ownership based on failure categories and gives the first responder enough evidence to make a useful classification. QA leaders assigning ownership for browser test failures is an operating-system problem, not a Jira-label problem.

Ownership should answer three questions:

Who performs the first classification?
What evidence must be attached before reassignment?
When is a failure allowed to be marked as test noise?

Without those rules, every ambiguous failure becomes a social negotiation.

Logs are not a substitute for a workflow

CI systems are very good at collecting output.

They are less good at deciding which output matters.

Over time, teams add more logging because previous failures were difficult to diagnose. Then the logs become so large that nobody reads them. The system technically contains the answer but functionally hides it.

This is why a useful flake triage workflow should not turn CI logs into a dumping ground.

A good failure record is layered.

The first layer should answer the obvious questions in seconds:

test name;
failure step;
expected and actual state;
screenshot;
environment;
browser;
commit;
retry history.

The second layer can include network traces, console output, DOM snapshots, videos, and raw logs.

The third layer can preserve deep diagnostic data for the rare investigation that actually needs it.

When everything is displayed at once, nothing is prioritized.

Retention is part of release governance

Artifact retention is often treated as a storage-cost setting.

Delete videos after seven days. Keep screenshots for 30. Archive reports for 90.

But retention also determines whether you can reconstruct a release decision later.

Suppose a customer reports a serious regression three weeks after deployment. Can you prove which browser journeys passed before that release? Can you inspect the evidence? Can you see whether a flaky test was manually overridden? Can you identify who approved the exception?

These are not merely compliance questions. They are management questions.

When evaluating automation with AI capabilities, audit trails, artifact retention, and release sign-off matter because AI introduces another actor into the process.

If a model changed a locator, generated an assertion, or classified a failure, the record should show that.

“AI-assisted” should never mean “historically untraceable.”

Fixtures and evidence belong to the same system

Upload tests expose a common architectural mistake.

Teams store test code in Git, fixtures in a shared folder, generated files on a CI worker, downloads in temporary storage, and screenshots in a reporting service. The test passes locally because everything is nearby. CI fails because the file path, permissions, lifecycle, or cleanup behaviour differs.

The comparison between Endtest and Playwright for browser tests depending on AWS S3 fixtures, uploads, and downloaded artifacts points to a broader decision: how much artifact plumbing does your team want to own?

With a code framework, you can design exactly what you need.

You also inherit every detail:

presigned URL generation;
bucket policies;
fixture versioning;
cleanup jobs;
naming conventions;
parallel-run isolation;
malware scanning;
secrets;
retention;
report linking.

A platform may remove some of that flexibility while reducing the amount of infrastructure your team maintains.

Neither approach is universally better. The costly approach is pretending those responsibilities do not exist.

Configuration drift corrupts evidence

A screenshot is only useful if you know what conditions produced it.

Feature flags, experiment buckets, and release toggles complicate this because the same URL can render several legitimate versions of the application. A failure may reflect a defect, an expected variant, or a test running under a configuration nobody realized was active.

That is why browser test suites become fragile when feature flags and experiment buckets drift apart.

Every execution record should capture relevant configuration state.

Not every flag in the company—just the flags that materially change the tested journey.

Otherwise, you are preserving a picture without preserving the scene.

Design backward from the release decision

The easiest way to improve a test evidence system is to stop designing it around what the tools can emit.

Start with the decision someone must make.

For a failed checkout test, that decision might be:

Is checkout actually broken?
Is the failure limited to one browser?
Did a release toggle create a known variant?
Is the test data reusable?
Can we safely ship?
Who owns the next step?

Now ask which evidence is required to answer those questions quickly.

That is your artifact specification.

The test runner, S3 bucket, dashboard, and CI job are implementation details.

This way of thinking produces smaller logs, more useful screenshots, better retention policies, and clearer ownership because every artifact has a purpose.

The boring system wins

There is no impressive demo for a clean evidence pipeline.

Nobody applauds a consistent object key in S3. Nobody posts a screenshot of a well-designed retention policy. Nobody gets excited about a failure record that includes the correct feature-flag state.

But these are the details that turn automation into operational leverage.

A test that catches a problem is useful.

A test that catches a problem, preserves the right evidence, routes it to the right owner, and supports a fast release decision is much more valuable.

Build for that second outcome.

The Hardest Browser Tests Live Outside the Browser

Antoine Dubois — Fri, 24 Jul 2026 22:11:07 +0000

The easiest browser test is the one that never leaves the browser.

Open a page. Click a button. Check a message.

Real customer journeys are rarely that polite.

A user signs up, receives an email, opens a verification link, enters an SMS code, uploads a file, downloads a CSV, switches between applications, and retries after an expired session. Meanwhile, the test runs against a preview environment whose URL disappears tomorrow.

At that point, the browser driver is no longer the difficult part.

The difficult part is orchestration.

This distinction matters because teams often choose their test stack based on recorder quality, selector syntax, or how concise a “click” command looks. Those features matter. But the long-term cost of an end-to-end suite is usually determined by everything surrounding the click.

Verification flows are distributed systems

Email and SMS verification are often described as simple test steps:

submit the form;
retrieve the code;
continue.

Each step hides failure modes.

The email might be delayed, duplicated, placed in spam, or generated for a previous test. The SMS provider may throttle requests. The link may expire. Multiple parallel tests may share the same inbox or phone number. A timestamp comparison may fail because systems use different clocks.

The practical guide to building a reliable email and SMS verification test harness with Mailgun, IMAP, and Twilio shows why the integration layer deserves first-class design.

A good verification harness needs correlation, not just retrieval.

Instead of “fetch the newest email,” use a unique value created for that test:

a generated email alias;
a transaction identifier;
a unique subject token;
a phone number reserved for one worker;
a known timestamp boundary.

Then poll with a deadline and preserve the raw message when the test fails.

The same logic applies whether you use Mailgun, IMAP, Twilio, an internal service, or a testing platform with built-in email and SMS actions. The goal is to make the external event attributable to one test run.

Test the channel transition, not just the code

It is tempting to bypass an email or SMS step by reading the code directly from the database.

That can be useful for lower-level tests. But an end-to-end test exists to validate the complete journey.

A database shortcut will not reveal:

broken email templates;
malformed links;
incorrect domains;
SMS delivery configuration errors;
encoding problems;
expired or reusable tokens;
links that open the wrong environment.

This framework for evaluating browser testing platforms for email links, SMS codes, and cross-channel verification flows makes the trade-off clear: the more channels a user journey crosses, the more important correlation and observability become.

You do not need every test to exercise the real provider. A sensible strategy often includes:

fast component or API tests for token rules;
a smaller set of full delivery tests;
production-like templates and domains in staging;
explicit negative cases for expired and reused codes.

The mistake is pretending that one layer replaces all the others.

Downloads are outputs, not click events

A browser automation tool can click “Export CSV” successfully while the generated file is empty, corrupted, badly encoded, or populated with the wrong rows.

That means the click is setup. The file is the assertion.

The guide to testing downloaded files, CSV exports, and post-export validation in CI covers the workflow most suites omit:

identify the download event;
store the file in a known location;
verify its name and type;
parse the content;
check the expected schema and values;
clean up after the run.

For CSV files, avoid comparing the entire file as raw text unless ordering and formatting are part of the contract. Parse the rows and assert the business data.

For PDFs, spreadsheets, and archives, validate enough structure to prove the export is usable. A non-zero file size is not a meaningful assertion.

CI adds another complication: downloads may use temporary names, isolated browser directories, or operating-system paths that differ from local machines. Treat those paths as configuration, not constants.

Preview environments change the definition of stable

Ephemeral deployments are excellent for delivery speed. Each branch or pull request can have its own URL, database, and configuration.

They also make naïve test suites fall apart.

A fixed base URL no longer works. The environment may not be ready when the pipeline starts. Authentication callbacks may reject the temporary domain. Test data may be seeded asynchronously. The deployment may disappear before a rerun begins.

This guide to selecting browser testing for preview deployments, ephemeral URLs, and expiring environments focuses on the operational capabilities that matter:

dynamic environment URLs;
readiness checks before test execution;
per-run variables and secrets;
clear separation of environment failures from product failures;
automatic cleanup;
useful artifacts when the environment no longer exists.

A test platform does not need to own your deployment process. It does need to accept that the target is temporary.

The test should receive the environment URL from the deployment pipeline, verify that the application is healthy, and only then begin the browser flow.

Otherwise, “element not found” may really mean “the container was still starting.”

Microfrontends turn navigation into integration testing

Microfrontend architectures let teams deploy parts of the interface independently. They also create boundaries that browser tests need to understand.

A shell may load remote modules, preserve authentication, share navigation state, and handle failures when one module is unavailable. The visible page can be assembled from assets with different release cycles.

The comparison of Endtest and Playwright for microfrontend shells, AI-assisted navigation, and shared session state raises a useful question: how much framework logic does your team want to own?

With a code-first approach, you can build precise helpers for remote-module readiness, shared storage, cross-origin transitions, and shell-level errors. That flexibility is valuable when the team has the time and expertise to maintain it.

A managed platform can reduce the amount of glue code, particularly when tests need to reuse the same browser session across multiple workflows or allow less technical team members to contribute.

The right choice depends less on whether one tool can click a microfrontend and more on who will maintain the surrounding abstractions two years from now.

Recovery paths are where products reveal themselves

Happy paths are predictable. Recovery flows are where integrations collide.

Consider a user who:

uploads an invalid file;
receives a validation error;
corrects the file;
waits for processing;
confirms the result through email;
follows a link back into the application;
resumes the same session.

That single journey may involve browser state, object storage, an API, a background worker, an email provider, and a temporary token.

The article on where Endtest fits in CI pipelines that need email verification, file uploads, and multi-step recovery flows is useful because it evaluates the workflow as a whole rather than as a collection of isolated clicks.

When evaluating any tool, ask whether it can preserve and expose the state you need across the full journey:

browser cookies and local state;
uploaded test files;
generated messages;
API responses;
screenshots and logs;
retry boundaries;
secret handling.

The fewer custom bridges you need to write, the lower the maintenance surface. But managed convenience is only useful when it covers the actual workflow.

AI applications need more than browser tests

AI products often combine a browser interface, APIs, streaming responses, queues, model providers, and usage limits. Testing only the UI can be slow and ambiguous. Testing only the API can miss rendering and interaction failures.

A practical approach is described in building an API and load testing stack for AI apps without overengineering it.

Start with the risks, not the tool list.

You may need to measure:

API correctness and schema stability;
time to first token;
total response duration;
cancellation and retry behaviour;
concurrency limits;
provider errors;
browser rendering during streaming;
cost per completed workflow.

Not every team needs a large performance engineering platform. A modest combination of API checks, targeted load tests, and a few realistic browser journeys can reveal most early-stage problems.

Complexity should be earned by evidence.

External QA partners need operational context

Some teams outsource testing during large releases or when internal capacity is limited. That can work well, but feature-flag rollouts and gradual exposure require more than a generic regression checklist.

The guide on choosing a QA partner for feature flag rollouts, kill switches, and gradual exposure testing points to the real requirement: the partner must understand release operations.

They should be able to test:

multiple flag combinations;
targeted user segments;
rollback behaviour;
stale cached values;
analytics and monitoring signals;
emergency kill switches;
partial exposure across regions or accounts.

A team that only verifies the fully enabled state is not testing the rollout. It is testing the destination.

The architecture around the test matters most

Browser automation discussions often focus on syntax:

click()
fill()
expect()

Those commands are necessary, but they are rarely the source of the hardest engineering work.

The hard problems are identity, timing, correlation, environment lifecycle, external services, and recovery.

A durable end-to-end strategy treats those concerns as architecture:

create unique data for every run;
make asynchronous systems observable;
pass environment information explicitly;
validate outputs, not just interactions;
preserve evidence across channel transitions;
separate infrastructure failures from product defects;
keep the number of custom integrations proportional to their value.

The browser is the visible part of the journey.

The reliability lives everywhere else.

AI Test Automation Needs Review Gates, Not Blind Trust

Antoine Dubois — Thu, 23 Jul 2026 20:12:49 +0000

AI has made it much easier to create test steps.

It has not made it easier to know whether those steps are good.

That distinction matters. A tool can generate a large test suite in minutes and still miss the behavior that would actually hurt users. It can produce plausible assertions, convincing locators, and neat summaries while quietly encoding the wrong assumptions.

The new bottleneck is not test generation. It is test judgment.

Teams adopting AI-assisted testing need review gates that answer three questions:

What did the AI decide?
What evidence did it use?
Who or what is allowed to approve the result?

Without those answers, faster generation simply creates a larger maintenance surface.

AI workflows are rarely one-step interactions

Simple demos make AI testing look like this:

Enter a prompt.
Wait for a response.
Assert that the answer contains a phrase.

Real AI products are more complicated.

A customer-support workflow may classify a request, retrieve account data, search a knowledge base, draft an answer, request approval, update a ticket, and escalate to a human when confidence is low.

The article on using Endtest for multi-step AI support workflows is useful because it discusses workflow fit and tradeoffs rather than pretending every AI flow can be validated with a single text assertion.

Each transition needs its own observable contract:

Was the correct tool called?
Did the interface show the right intermediate state?
Was sensitive information hidden?
Did the workflow stop for approval?
Was the escalation path available?
Did the final action match the user's request?

Testing only the final text skips most of the product.

Generated tests need a human-readable plan

Before an AI system creates steps, it should expose the plan it intends to follow.

That plan might include:

Preconditions
User role
Test data
Navigation path
Expected checkpoints
Failure conditions
Cleanup actions

A reviewer should be able to reject the plan before the system generates twenty detailed steps around a bad assumption.

The same is true for AI-generated UI flows. A platform should support human review gates, version history, and clear ownership. This guide on what to check in a browser testing platform for AI-generated UI flows provides a practical evaluation framework.

The best review gate is not necessarily a manual approval for every change. It can be policy-driven:

Auto-approve low-risk wording updates.
Require review when a test deletes, purchases, or changes permissions.
Require review when generated steps introduce new domains.
Require review when confidence is low.
Require review when the AI changes an assertion rather than a locator.

This keeps human attention focused on consequential changes.

More generated code is not always more coverage

Playwright, Selenium, and Cypress can all be paired with AI code generation. The interesting question is not which one can produce code. All of them can.

The question is what happens after the first draft.

This comparison of Playwright, Selenium, and Cypress for AI-assisted test generation examines where code volume helps and where it becomes a maintenance burden.

Generated code can create the illusion of progress:

More files
More assertions
More test cases
More lines changed in each pull request

But coverage depends on behavior, not volume.

Ten generated tests that repeat the same happy path are less useful than one test that checks a permission boundary, a recovery path, and a meaningful business rule.

A review process should therefore compare generated tests against a coverage model:

Which user risks are covered?
Which states are exercised?
Which integrations are touched?
Which failure modes remain untested?
Which tests are duplicates in disguise?

RAG applications need evidence-aware assertions

Retrieval-augmented generation creates a special testing problem. The answer can sound correct while citing the wrong source, omitting a required citation, or using retrieved context that should not have been accessible.

A browser test for a RAG chatbot may need to verify:

Citation panel visibility
Correct mapping between claims and sources
Permission-aware retrieval
Empty-result behavior
Escalation to a human
Recovery after a failed retrieval
Streaming updates without duplicated citations

This evaluation guide for browser testing platforms used with RAG chatbots highlights the UI and workflow details that generic chatbot assertions miss.

Do not reduce these tests to “the answer contains expected words.” That kind of assertion can pass even when the product is dangerously wrong.

Prompt changes are application changes

Teams version source code, schemas, and infrastructure.

Prompts often get edited directly in a dashboard.

That is a problem because a small prompt change can alter:

Tool selection
Refusal behavior
Output structure
Tone
Citation style
Escalation decisions
Sensitivity to ambiguous input

The article on why AI test suites miss regressions after prompt changes explains why visual stability is not behavioral stability.

Prompt versions should appear in test evidence just like deployment versions. When a failure begins, the team should be able to answer:

Which prompt was active?
What changed?
Which test cases were affected?
Were model settings changed at the same time?
Did retrieved data also change?

Without this context, triage becomes guesswork.

Streaming output breaks traditional UI assumptions

A streaming response is not one render. It is a sequence of renders.

Chunks can arrive in unexpected boundaries. The interface may reflow, buttons may move, citations may appear late, and an early assertion may inspect a temporary state.

This guide to debugging AI UI tests after streaming chunks reorder the screen describes a problem that becomes common as applications rely on incremental output.

Tests should wait for a semantic completion signal, not simply a pause in network traffic.

Useful completion signals include:

A “generating” indicator disappears.
A final response container receives a completed state.
A stop button changes back to send.
A stream-complete event is emitted.
Required citations finish rendering.
An action button becomes enabled.

These signals are more reliable than guessing how long generation should take.

Test the human handoff, not only the AI

Many AI-heavy products are designed to involve people:

A support agent approves a draft.
A compliance reviewer checks a response.
A user confirms a destructive action.
A supervisor handles low-confidence cases.
A developer approves a generated test update.

Those handoffs are part of the system.

Browser automation fits many parts of these workflows, but some decisions remain genuinely ambiguous. Recognizing that boundary is more useful than claiming AI can fully validate AI.

A robust test should verify that the human can:

See why review is required
Inspect the evidence
Edit the proposed action
Reject it safely
Approve it intentionally
Audit what happened later

Build a layered confidence model

AI test automation works best when confidence comes from multiple layers:

Deterministic checks

Use exact assertions for permissions, routing, API status, data persistence, and required UI controls.

Structured-output checks

Validate schemas, required fields, allowed values, and tool-call arguments.

Semantic checks

Use carefully constrained AI evaluation for meaning, relevance, or policy compliance.

Human review

Reserve people for ambiguous, high-risk, or novel cases.

Production signals

Track real user corrections, escalations, abandoned flows, and incident patterns.

No single layer is enough.

Choose tools by operational fit

Lists of AI testing tools can help with discovery, but they should be treated as starting points. This practical shortlist of AI test automation tools in 2026 offers one useful market view.

A separate video discussion about test automation can also help you compare approaches before adopting one.

The final evaluation should focus on your workflow:

Can reviewers understand generated changes?
Can you version prompts and tests together?
Can you reproduce a failed run?
Can you restrict risky actions?
Can you inspect intermediate AI decisions?
Can you keep test maintenance below the value the suite produces?

AI can accelerate test creation.

Review gates are what keep that acceleration pointed in the right direction.

Browser Testing Gets Hard When the Browser Becomes Part of the Product

Antoine Dubois — Wed, 22 Jul 2026 20:50:09 +0000

The easiest browser tests live entirely inside one tab.

Open a page. Fill in a form. Click a button. Check the result.

Unfortunately, many valuable user journeys do not stay inside that neat boundary.

The browser itself participates in the workflow. It stores sessions, asks for permissions, handles downloads, opens deep links, restores tabs, applies location settings, and moves keyboard focus. Email may become part of authentication. A push notification may become the entry point to the application.

At that point, you are not merely testing a web page. You are testing an agreement between the application, the browser, the operating environment, and several external systems.

That is where otherwise respectable automation suites start to become brittle.

Permissions are stateful, not just modal dialogs

Permission testing is often reduced to one scenario:

Click Allow and verify that the feature works.

Real users can allow, block, dismiss, ignore, or previously revoke a permission. Browsers can remember the decision. The application may show its own educational prompt before triggering the native prompt. Different environments may start with different permission state.

A useful permissions matrix includes:

permission not yet requested;
permission granted;
permission denied;
prompt dismissed;
permission changed outside the application;
unsupported browser or environment;
permission granted but the downstream service unavailable.

Notification flows add another complication: the action may happen after the user leaves the original page. The browser can receive a message, the user can click it, and the application may open a specific route with encoded context.

Testing Notification Permissions and Deep-Link Journeys with Endtest is a helpful reference for thinking through that full path instead of testing the permission prompt in isolation.

Location-aware experiences have similar branches. Region selection, browser geolocation, IP-based defaults, and account-level preferences can disagree. This guide to using Endtest for geolocation, permission prompts, and region-aware UX flows covers several of those combinations.

Authentication increasingly leaves the tab

Email verification, magic links, and password resets look simple from the user’s perspective. Behind the scenes, they connect several systems:

The application generates a token.
An email provider accepts and delivers the message.
The test obtains the correct email.
The link opens with the correct token and environment.
The application validates expiration, reuse, and account state.
The browser establishes a new session.

Teams often build a quick inbox script, then spend months maintaining it.

The harness becomes brittle because it assumes subject lines never change, messages arrive in order, only one environment uses the mailbox, HTML structure stays stable, and delivery is immediate.

A better design treats the inbox as structured test infrastructure. It uses unique addresses or correlation IDs, filters deterministically, verifies the sender and timestamp, and extracts the intended link rather than the first URL in the message.

How to Test Email Verification, Magic Links, and Password Reset Flows Without Building a Brittle Inbox Harness goes deeper into those tradeoffs.

The key is to test both sides of the boundary. Confirm that the application produced the correct message, but also confirm that the browser journey created by the message behaves correctly.

Browser storage is application state

Cookies, local storage, session storage, IndexedDB, and browser cache are often treated as setup details. In modern SaaS products, they are part of the product’s behavior.

They can determine:

whether the user is logged in;
which workspace opens;
whether onboarding appears;
which feature flags are active;
whether unsaved work is restored;
what theme is selected;
whether a dismissed message returns;
which cart or draft is recovered.

Tests become unreliable when they inherit state accidentally.

One run leaves a token behind. Another test reuses the browser profile. A failed cleanup skips logout. The next scenario starts on the dashboard even though it expects the login page.

A stable suite makes state ownership explicit. Each test should know which storage it requires and which storage it must remove.

Endtest vs Selenium for testing session restoration, browser storage, and logout edge cases discusses these scenarios from a tool-comparison perspective.

Regardless of tooling, include adversarial cases:

expired access token with a valid refresh token;
valid cookie but revoked server session;
two tabs logging out at different times;
storage cleared while the page remains open;
restored tab after a deployment;
account switched while stale cached data remains.

Those are the cases that reveal whether “logout” actually means logout.

Downloads are not finished when the button is clicked

A test that clicks Export and sees no error has not verified a download.

The file could be empty, incorrectly named, generated for the wrong account, encoded incorrectly, or based on stale data. A blob URL may work in one browser and fail in another. A download can start before generation is complete. Multiple files can appear with suffixes such as (1) and cause the test to inspect the wrong one.

A complete download test may verify:

a download was triggered;
the filename matches the expected pattern;
the file completed within a reasonable interval;
the MIME type or extension is correct;
the file is not empty;
the contents contain the expected record set;
temporary blob URLs are handled correctly;
the test cleans up the downloaded artifact.

What to Check in a Browser Testing Platform for Downloaded Files, Filename Assertions, and Blob URL Handling provides a practical checklist.

This is a good example of why “browser automation” often needs access to more than the DOM.

Keyboard behavior exposes invisible defects

A page can look perfect and still be difficult or impossible to use without a mouse.

Keyboard-only testing catches issues that ordinary click-based scripts miss:

focus skips an interactive element;
focus becomes trapped in a modal;
the visual focus indicator disappears;
pressing Enter submits the wrong form;
Escape closes the wrong layer;
focus returns to the top of the document after a dialog closes;
a custom control does not implement arrow-key behavior;
a hidden element remains in the tab order.

Accessibility checks should not be limited to running an automated scanner. Static rules are useful, but focus order and interaction behavior require an actual journey.

What to Look for in a Browser Testing Tool for Accessibility Regressions, Focus Order, and Keyboard-Only Navigation explains what the automation platform itself needs to support.

A practical technique is to create a small set of keyboard-first smoke tests for the most important flows. Do not translate mouse tests mechanically. Start with the user’s intended sequence of focus and actions.

Selectors still matter, but context matters more

Even in complex browser workflows, selectors remain foundational. A bad locator can make every state-management problem look like a timing problem.

The Selenium Selector Cheatsheet is a useful reference for choosing among IDs, CSS selectors, XPath, attributes, text, and other locator strategies.

The more important rule is to select elements according to their role in the interface.

Prefer stable attributes, accessible roles, labels, and intentionally exposed test hooks. Avoid coupling the test to layout depth or styling classes unless the layout or styling is what you are testing.

Also remember that a locator can be technically correct and still be ambiguous during transitions. If two matching buttons briefly exist, “find the first button” is not a stable contract.

Flake triage needs evidence, not piles of screenshots

Once tests cover permissions, email, storage, downloads, and multiple tabs, failures gain more possible causes.

The answer is not to attach ten screenshots to every failed run.

A useful failure record should help someone distinguish among:

application defect;
environment problem;
test data collision;
browser-state leak;
selector ambiguity;
timing or rendering instability;
external service delay;
automation defect.

That usually requires structured context: active URL, recent browser events, relevant network calls, storage state, selected DOM details, console errors, and the exact action being attempted.

How to Build a Flake Triage Dashboard for Browser Tests Without Turning CI Into a Screenshot Graveyard provides a sensible model for organizing that evidence.

Screenshots are still valuable. They are simply much more useful when paired with enough context to explain what happened before and after the image.

Test the browser contract

The browser is no longer a neutral window around the application.

It remembers decisions. It blocks capabilities. It restores sessions. It downloads artifacts. It routes users from external entry points. It controls focus and mediates access to device features.

Reliable browser testing begins when these behaviors stop being treated as inconvenient exceptions.

Model them as explicit state. Create clean setup and cleanup rules. Verify the external side effect, not only the click that initiated it. Capture evidence that distinguishes product defects from automation defects.

Once you do that, the hardest browser tests become less mysterious. They are still complex, but the complexity is visible—and visible complexity can be designed for.

AI Test Automation Needs Guardrails, Not More Confidence

Antoine Dubois — Tue, 21 Jul 2026 09:56:01 +0000

AI can generate a test, repair a selector, summarize a failed run, and propose reproduction steps before a human has finished reading the ticket.

That speed is useful. It is also exactly why teams need stronger review systems.

The biggest mistake in AI-assisted QA is treating an output as trustworthy because it is fluent, detailed, or accompanied by a high confidence score. A plausible test repair can still weaken coverage. A polished reproduction guide can still describe a path that never happened. A passing AI feature test can still be validating yesterday’s model behavior.

The answer is not to reject automation. It is to make the automation produce evidence.

Autonomous test fixes should arrive as reviewable changes

A self-healing system can repair a broken locator in seconds. But a locator change is not always a repair.

Suppose a test originally clicks the “Delete project” button and the AI changes the locator to the first visible button in the dialog. The test passes again, but it may now click “Cancel.” From the pipeline’s point of view, the fix worked. From the product’s point of view, the test stopped testing the feature.

An autonomous fix should therefore include:

the original step and locator
the proposed replacement
the DOM evidence used to choose it
a screenshot before and after the action
the observed outcome
a summary of why the behavior is considered equivalent
the scope of tests affected by the change

The proposed change should enter the same kind of review gate used for code. Low-risk repairs can be approved quickly, while semantic changes should require a human decision.

The article on building a review gate for autonomous test fixes in CI/CD offers a practical model for separating harmless maintenance from coverage-changing edits.

Measure reproduction quality, not writing quality

AI-generated bug reproduction steps often sound authoritative even when they are assembled from incomplete logs.

A useful reproduction sequence must satisfy several conditions:

another person can follow it
the sequence reaches the same failure
required data and account state are identified
timing assumptions are explicit
irrelevant actions are removed
the observed result matches available evidence

You can score these dimensions separately. Reproduction success rate is more useful than a generic confidence score. So is the percentage of steps supported by logs, screenshots, network events, or recorded user actions.

The guide on what to measure before trusting AI-generated bug reproduction steps is valuable because it shifts the question from “Does this explanation look good?” to “Can we verify it?”

AI copilots need state-based tests

A copilot that edits forms, tables, or inline content does not behave like a deterministic button.

It may choose a different field order, rewrite only part of a record, produce a preview before applying changes, or ask the user for approval. The exact wording can vary while the product behavior remains correct.

Tests for these interfaces should focus on state transitions:

What data existed before the action?
What did the user ask the copilot to change?
What proposed change was shown?
What did the user approve or reject?
What data was finally persisted?
Was an audit trail created?

This avoids brittle assertions against every sentence the model produces. You still need content checks, but they should be tied to product rules: required values were preserved, prohibited fields were not changed, totals remain valid, and the final state matches the approved proposal.

For a broader evaluation framework, see what to check in a browser testing platform for AI copilots that edit forms, tables, and inline content.

Search tests need ranked-result tolerances

AI-powered search introduces another trap: assuming the same query must always return the same ordered list.

Traditional search assertions often compare exact result positions. That becomes fragile when the product uses embeddings, reranking, query rewriting, personalization, or a model that changes over time.

A better test model separates invariants from tolerances.

Invariants might include:

prohibited results never appear
exact identifier matches remain highly ranked
filters are respected
tenant boundaries are not crossed
result links are valid

Tolerances might include:

a relevant result appears within the top five rather than exactly first
the top results meet a minimum relevance score
ranking drift stays within an accepted threshold
alternative but equivalent results are allowed

The article on testing AI-powered search, reranking, and result-drift validation explains why ranked systems need evaluation sets and drift monitoring, not just fixed expected arrays.

A stable staging environment does not mean stable AI behavior

Teams frequently validate an AI feature in staging and assume the same test will protect production. That assumption breaks when production uses a different model version, prompt template, retrieval index, safety policy, temperature, or tool configuration.

The UI may be identical while the decision system behind it has changed.

Every AI feature test run should record the configuration that produced the result:

model and version
system prompt or prompt revision
retrieval index version
tool definitions
relevant feature flags
sampling settings
safety or moderation configuration

Without that metadata, a failed test after rollout is difficult to explain and a passing test is difficult to reproduce.

This breakdown of why AI feature tests pass in staging but fail after model or prompt rollouts is a strong reminder that the model configuration is part of the deployed application.

Put browser automation in the larger AI testing stack

Browser tests are important because they observe the product from the user’s perspective. They can verify approval screens, tool calls, retries, persisted changes, permissions, and visible error states.

But browser tests should not carry the whole AI quality strategy.

A mature stack usually includes several layers:

offline evaluation against curated examples
API-level tests for model and tool behavior
security and abuse testing
browser tests for complete user workflows
production monitoring for drift and regressions
human review for ambiguous or high-impact decisions

The article on where Endtest fits in an AI testing stack for fast-changing product interfaces gives one practical view of how browser automation can complement, rather than replace, the other layers.

Evidence should travel with every AI decision

The common theme is simple: AI output should not be accepted because it sounds right.

A repaired test should show why the new step is equivalent. A reproduction guide should be executable. A copilot test should compare approved and persisted state. A search test should distinguish invariants from acceptable ranking drift. A rollout should record the model configuration that produced the result.

AI makes it possible to automate more of the QA workflow. Guardrails make that automation safe enough to trust.

How to Test AI-Powered Web Apps Without Treating the Model Like a Normal API

Antoine Dubois — Fri, 17 Jul 2026 21:23:36 +0000

AI-powered web applications look familiar on the surface.

They have text boxes, buttons, menus, loading indicators, and API calls. That makes it tempting to test them like any other web application: submit an input, wait for a response, and compare the output with an expected string.

That approach breaks quickly.

Model output is variable. Safety behavior depends on context. A response can be semantically correct but displayed in the wrong conversation. An agent can produce a convincing final message after calling the wrong tool. A prompt-injection defense can block obvious attacks while failing when malicious instructions arrive through a webpage, document, image, or previous message.

Testing these applications requires two kinds of evidence at the same time:

Deterministic product evidence: the UI, state, permissions, tool calls, and workflow behaved correctly.
Probabilistic model evidence: the output stayed within an acceptable range across repeated and adversarial inputs.

Prompt injection is a workflow problem

Prompt injection testing is often reduced to pasting “ignore previous instructions” into a chat box. That is a useful smoke test, but it does not represent how browser-based agents encounter untrusted content.

An agent may read instructions from:

A webpage.
A support ticket.
A PDF.
A hidden DOM node.
An email.
A retrieved knowledge-base entry.
A tool response.
A previous conversation turn.

The guide on testing prompt injection defenses in AI-powered browser workflows provides a good foundation.

The test should verify more than the final sentence. It should inspect whether the agent:

Treated external content as data rather than authority.
Attempted a prohibited tool call.
Exposed secrets in an intermediate step.
Navigated to an unapproved domain.
Changed its goal after reading untrusted content.
Requested confirmation before a sensitive action.
Preserved the original user instruction.

A safe final answer does not prove that the workflow was safe. The agent may have attempted a dangerous action that happened to fail.

Evidence and replay matter more than a single pass/fail label

When an AI test fails, the first question is often: “What exactly happened?”

Traditional browser automation can usually answer with a screenshot, stack trace, and failed assertion. AI workflows need additional context:

The complete conversation.
System and developer instructions.
Retrieved content.
Model and configuration.
Tool calls and tool results.
Safety decisions.
Intermediate UI state.
The final visible output.

The article on evaluating AI testing tools for prompt injection evidence, conversation replay, and unsafe output triage explains why replayability is central.

A useful replay package should preserve enough information to investigate the failure without depending on the original environment still existing. Redact secrets, but do not remove the context that determined the model's behavior.

For nondeterministic systems, one failed sample may be insufficient. Store repeated runs and compare the distribution of outcomes. A defense that succeeds nine times and fails once is not equivalent to a deterministic pass.

The UI around the model is still normal software—and it still breaks

AI output applications often include controls such as:

Regenerate.
Retry.
Stop generation.
Copy to clipboard.
Edit prompt.
Switch model.
Continue response.
Rate output.
Restore a previous version.

These controls are deterministic enough to test carefully, even when the generated text is variable.

The comparison of Endtest and Playwright for testing AI output UIs with regenerate, retry, and copy-to-clipboard actions highlights the practical browser-automation concerns.

Useful checks include:

Regenerate creates a new response under the correct prompt.
Retry does not duplicate the user's message.
Stopping generation leaves the conversation in a recoverable state.
Copy uses the final content rather than hidden streaming fragments.
Buttons remain associated with the correct response after new messages arrive.
Scrolling does not cause actions to target the wrong message.
A failed response can be retried without losing conversation context.
The interface distinguishes old and regenerated versions.

Do not assert the entire generated paragraph unless the application promises exact output. Assert structure, safety, required facts, prohibited content, and the relationship between UI actions and conversation state.

Agentic systems must be tested at every tool boundary

An agentic workflow can produce the correct final result through an unsafe or inefficient process.

For example, an assistant may successfully schedule a meeting but:

Invite the wrong person first.
Read a calendar it was not authorized to access.
Create two events and delete one.
Ignore a conflict.
Expose private event details in the response.

That is why AI testing platforms for agentic workflows, tool calls, and multi-step recovery paths need more than final-output assertions.

Test every tool boundary:

Was the correct tool selected?
Were the arguments valid and authorized?
Did the agent interpret the result correctly?
Did it retry safely after failure?
Did it avoid repeating side effects?
Did it ask for confirmation where required?
Did the UI accurately reflect the action?

Inject realistic failures:

Tool timeout.
Partial result.
Permission denial.
Stale data.
Conflicting data.
Rate limit.
Side effect succeeds but acknowledgement fails.
User changes the goal midway through the workflow.

Recovery behavior is part of the product, not an edge case.

Frequent UI and copy changes punish brittle assertions

Teams building AI products tend to change their interfaces quickly. Labels, model names, helper text, output formatting, and streaming behavior may evolve every week.

This makes exact text assertions expensive. It also makes tool selection important.

The comparison of Endtest and Autify for teams testing AI-driven web apps with frequent UI and copy changes is useful as a way to think about maintenance trade-offs.

Regardless of platform, separate assertions into categories:

Stable product contracts

The prompt is submitted once.
A response belongs to the correct conversation.
The user can stop generation.
Unsafe actions require confirmation.
Tool execution status is visible.

Flexible presentation details

Exact helper text.
Minor button-label changes.
Markdown formatting.
Response phrasing.
Nonessential layout changes.

Not every text change should break the suite. But not every text assertion should be removed either. Security warnings, consent language, prices, permissions, and destructive-action labels may require exact verification.

Multimodal applications combine several sources of truth

A multimodal application may process text, images, audio, and live screen state in one workflow. Testing only the final transcript or response misses the alignment between those inputs.

The AI testing market report for multimodal apps describes how the test surface changes when modalities interact.

Consider a support assistant that listens to a call, reads a screenshot, and suggests the next action. The system can fail in several distinct ways:

Audio transcription is wrong.
The screenshot is associated with the wrong customer.
The model describes an element that is not on screen.
The UI shows an older frame than the model analyzed.
The assistant combines correct facts from different sessions.
The final recommendation is correct but based on prohibited private data.

A multimodal test should preserve timestamps and associations between inputs. Verify that the model processed the correct image, audio segment, browser state, and conversation.

Useful test cases include:

Contradictory text and image input.
Silent or corrupted audio.
Images with embedded prompt injection.
Rapidly changing screen state.
Delayed modality arrival.
The same content presented in different modalities.
Missing accessibility text.
Inputs belonging to different users or sessions.

Use layered assertions

AI testing works best when assertions are layered rather than reduced to one exact answer.

A practical stack looks like this:

Layer 1: Deterministic workflow

Verify routes, controls, messages, tool calls, permissions, retries, and data ownership.

Layer 2: Structural output

Check required sections, data types, citations, links, or JSON schema.

Layer 3: Semantic requirements

Evaluate whether required facts and instructions are present.

Layer 4: Safety constraints

Detect prohibited disclosure, unsafe instructions, policy violations, or unauthorized actions.

Layer 5: Statistical behavior

Repeat adversarial and ambiguous cases to estimate the rate of unacceptable outcomes.

This structure keeps deterministic bugs separate from model-quality failures. A broken copy button should not be classified as an LLM hallucination. A correct button flow should not excuse an unsafe tool call.

Final thought

An AI-powered application is not just a model endpoint with a chat interface.

It is a system made of prompts, retrieved content, browser state, tools, permissions, UI controls, and sometimes several input modalities. The final response is only the visible end of that chain.

Reliable testing follows the complete chain. It verifies what the user saw, what the model received, which tools the agent used, what state changed, and whether the same scenario remains safe across repeated runs.

That is how teams move beyond “the answer looked good” and start testing AI applications as real production systems.