DEV Community: Markus Gasser

Your Browser Tests Don’t Need to Be Perfect. They Need to Explain Themselves.

Markus Gasser — Wed, 19 Aug 2026 21:57:26 +0000

There’s a strange goal that creeps into test automation projects.

We start by wanting confidence that the product works.

Then, somewhere along the way, the goal quietly changes to:

Make the tests pass.

That sounds similar. It isn’t.

If a browser test fails once every 50 runs and nobody knows why, the problem isn't merely that the test is flaky. The problem is that every failure creates a small investigation.

And investigations are expensive.

Someone opens CI. Someone reruns the job. Someone looks at a screenshot. Someone asks whether Chrome updated. Someone wonders if the test account was already logged in.

Twenty minutes later, the test passes on a rerun and everybody moves on.

Nothing was actually learned.

Do that across 500 tests, several browsers, and a few releases per week, and you've accidentally built a small tax system for your engineering team.

A failed test is only useful if it leaves evidence

I think one of the most underrated characteristics of a test automation system is how well it explains failure.

Consider a login test that unexpectedly lands on the dashboard instead of the login page.

There are at least two very different explanations.

The application might have a bug.

Or the previous test might have left behind a cookie.

Those require completely different fixes.

This is why understanding whether browser test failures come from application bugs or leaked browser state matters more than simply adding another retry.

Retries can make dashboards greener.

They don't necessarily make the system more reliable.

A better test setup preserves enough information to answer questions such as:

What URL was actually loaded?
What did the DOM look like?
Which cookies and local storage values existed?
Which browser and version ran the test?
What happened immediately before the failure?
Was this a fresh browser session?
Did the network request succeed?

The more of those questions you can answer without rerunning the test, the cheaper failures become.

AI doesn't remove this requirement

AI testing tools are making test creation dramatically easier.

That's useful.

But generating a test is usually not the expensive part of its lifecycle.

The expensive part comes six months later when the test fails after somebody changes authentication, redesigns a component, upgrades Chrome, modifies test data, or introduces a new asynchronous request.

So when evaluating an AI testing product, I would spend less time watching the "create a test from a prompt" demo and more time investigating what happens after something breaks.

A useful starting point is this guide on what to check in an AI-native test automation platform for low-maintenance browser regression.

The important questions aren't just:

"Can the AI create a test?"

They're things like:

"Can I understand what it created?"

"Can I edit it?"

"Can I see why it changed something?"

"Can I debug the result without asking the AI to try again?"

That distinction becomes even more important as systems become more agentic.

If an agent performs several actions before something goes wrong, you need to reconstruct its decisions. Traceability, replay controls, screenshots, logs, and failure evidence stop being nice debugging features and become part of the testing model itself.

There's a useful breakdown of this in how to evaluate AI testing platforms for traceability, replay, and failure evidence.

AI can reduce maintenance.

It shouldn't turn test execution into a black box.

Sometimes the browser isn't actually the thing you're testing

Downloads are a great example.

I've seen teams build surprisingly elaborate automation around the browser's native download dialog.

That often creates more trouble than value.

The requirement usually isn't:

Verify that Chrome rendered its Save File dialog correctly.

The requirement is closer to:

Verify that clicking Export produces the correct file.

Those are different tests.

A more robust approach is often to trigger the download, wait for the browser to report it, inspect the resulting file, and validate what actually matters: filename, type, existence, size, or contents.

That's why testing browser file downloads without relying on fragile native dialog assertions is a useful pattern.

It illustrates a broader rule:

Automate the business outcome, not every implementation detail between the user and that outcome.

The closer your assertions are to what the user actually cares about, the less likely your suite is to break because a browser vendor moved a button around.

Then Chrome updates

Of course, even a well-designed suite still runs inside a browser.

And browsers move fast.

An application can remain completely unchanged while a test that passed yesterday suddenly fails because the execution environment changed underneath it.

That can come from:

browser updates,
driver changes,
security restrictions,
rendering behavior,
permission changes,
timing changes,
deprecated APIs.

This creates one of the more frustrating categories of automation failures because the team naturally starts debugging its own code first.

Sometimes for hours.

Keeping track of browser version drift, pinning strategies, compatibility exceptions, and upgrade checks can save a surprising amount of time.

Pinning everything forever isn't the answer either.

Eventually your users will run newer browsers.

A healthier strategy is usually controlled change: know which versions you're running, upgrade intentionally, and have a small compatibility suite that tells you whether a browser update is about to become your problem.

The metric I would actually watch

Teams love test counts.

"We have 2,400 automated tests."

Fine.

But 2,400 tests that produce 40 ambiguous failures every morning might be worse than 700 tests that reliably tell you when the product is broken.

So I'd pay attention to something else:

How much human time does a failed automated test consume before we understand what happened?

Call it mean time to diagnosis if you want to make it sound respectable.

But the concept is simple.

If a failure takes two minutes to understand, automation is doing its job.

If it takes 30 minutes, several reruns, a Slack thread, and the person who originally wrote the framework three years ago, you don't have a test coverage problem.

You have an observability problem.

And adding another 500 tests will probably make it worse.

Reliability is useful. Explainability is what makes it scale.

No serious browser test suite will have zero failures unrelated to product defects.

Networks fail. Data changes. Browsers update. Environments drift. Authentication expires. Third-party services behave strangely.

Trying to eliminate every one of those variables is probably impossible.

The more practical goal is to make unexpected behavior cheap to understand.

That means preserving state.

Recording evidence.

Knowing your browser versions.

Testing outcomes instead of incidental UI.

And making sure that when AI participates in test creation or maintenance, humans can still see what happened.

A good automation system doesn't merely tell you:

Red.

It tells you enough that five minutes later you can say:

Ah. That's why.

That difference sounds small.

At scale, it may be one of the most important differences between a test suite people trust and one they quietly learn to ignore.

AI Can Write Tests Faster Than Your Team Can Understand Them

Markus Gasser — Sat, 08 Aug 2026 21:18:01 +0000

AI coding tools have solved one problem remarkably well:

They can produce code extremely quickly.

That sounds obviously good.

And most of the time, it is.

But software development has never really been constrained by how fast we can type.

The expensive part comes later.

Understanding the code.

Reviewing it.

Debugging it.

Changing it six months later when the person—or model—that wrote it has forgotten why it exists.

Test automation is where this becomes especially interesting.

Generating the Test Is the Cheap Part

You can ask an AI coding assistant:

Write Playwright tests for our signup, login, checkout, password reset, dashboard, invoices, settings, and admin pages.

And a few minutes later you might have hundreds or thousands of lines of test code.

It feels like incredible leverage.

Until the suite starts failing.

That’s the argument behind looking at the hidden cost of AI-generated test code.

Generation cost has collapsed.

Maintenance cost hasn’t.

In some cases, AI actually increases it because you now have more code than your team would have written manually.

AI Pull Requests Need Different Review

There’s another subtle problem.

Humans tend to judge large AI-generated pull requests differently.

When someone on your team writes 80 lines, you probably read them.

When an AI assistant generates 1,800 lines?

You skim.

You look at the filenames.

You check whether CI is green.

Merge.

That’s dangerous for normal application code and potentially worse for test code because a bad test can happily pass for months.

There are good ideas in this guide to testing AI coding assistant pull requests, but the bigger principle is simple:

AI-generated tests need validation just like AI-generated product code.

“Generated successfully” does not mean “tests the right thing.”

Agents Add Another Failure Mode

Now we’re moving from AI that writes test code to AI that actually decides what actions to take.

That introduces a new question:

What if the model chooses the wrong tool?

An agent might have access to:

browser interaction
API calls
database queries
file operations
search
internal tools

The action itself may work perfectly.

It’s just the wrong action.

Testing tool-selection failures in AI agents therefore becomes part of testing the product.

And dynamic web applications make this harder.

Partial renders, changing state, asynchronous updates, and stale context can cause an agent to confidently interact with yesterday’s version of the page.

That combination of tool selection, state drift, and partial renders is probably going to become a much bigger QA category over the next few years.

Self-Healing Can Heal the Wrong Thing

AI test repair is another feature that sounds almost universally good.

Selector changed?

Fix it automatically.

But imagine the application used to have:

Submit Order

And now has:

Submit Quote

The test can no longer find its element.

An AI repair system finds the new button and updates the test automatically.

Technically, it repaired the selector.

Semantically, it may have changed what the test means.

That’s why I like the idea of evaluating AI-generated test repairs before trusting automatic fixes.

Self-healing should reduce maintenance.

It shouldn’t quietly rewrite your specification.

Measure Trust, Not Just Pass Rate

Traditional test dashboards love pass rates.

97.4% passed.

Great.

But with AI-generated or AI-repaired testing, I think teams need another category of metrics:

How trustworthy was the conclusion?

That includes things like:

Was the failure reproducible?
Was a selector repaired?
Did the test behavior change after repair?
Was the failure caused by infrastructure?
Did AI classify the failure correctly?
How often do humans override AI conclusions?

That’s why these ideas around measuring AI test failures in CI matter.

The next generation of QA dashboards probably won’t just say PASS or FAIL.

They’ll tell you why the system believes the result.

AI Should Reduce Abstraction, Not Increase It

This is where I think test automation is heading in an interesting direction.

The worst outcome would be:

AI generates enormous automation frameworks that humans barely understand, then another AI maintains those frameworks because humans can no longer understand them.

That’s technically impressive.

It’s also a strange place to end up.

The better outcome is AI generating tests into representations humans can still inspect, understand, and modify.

Use AI to remove work.

Don’t use it to create a larger black box.

Because the real productivity metric isn’t:

How much test code did the AI generate?

It’s:

How much less testing infrastructure does your team have to think about?

Your Browser Tests Aren’t Flaky. Your Product Is Stateful.

Markus Gasser — Fri, 07 Aug 2026 22:02:53 +0000

There’s a point in almost every end-to-end test suite where the team starts blaming the tests.

The login test fails once every 40 runs.

The onboarding flow occasionally opens on step three instead of step two.

A notification appears before the assertion is ready.

Someone adds a two-second wait.

Problem solved.

Until next Tuesday.

What we call “flaky tests” are often something more interesting: state problems that the test suite happens to expose.

Modern web applications have a lot more state than they did ten years ago. There’s browser state, server state, authentication state, feature-flag state, WebSocket state, local storage, cookies, background jobs, third-party services, extensions, and whatever your frontend framework is currently keeping alive in memory.

The test runner is often the least mysterious thing in the system.

The hardest bugs happen between steps

Consider a five-step onboarding process.

Most teams test it like this:

Fill out step one.
Continue.
Fill out step two.
Continue.
Finish.

That’s useful.

But customers don’t behave like a test script.

They click Back.

They reload halfway through.

They open another tab.

They submit twice.

Their session expires.

They abandon the flow and return tomorrow.

That’s why I liked this breakdown of testing multi-step onboarding flows without missing state leaks, validation problems, and back-button bugs.

The important thing isn’t adding 30 more assertions.

It’s testing the transitions.

Stateful applications tend to break in the gaps between the happy-path steps.

This applies well beyond onboarding. If your application contains long-lived sessions, dynamic forms, approval processes, or workflows that stretch across multiple screens, this guide on evaluating automation platforms for multi-step workflows and session-heavy flows covers many of the same issues.

Real-time applications make this worse

Then you add WebSockets.

Now the browser can change without the browser doing anything.

A notification arrives.

A status becomes “Approved.”

A dashboard updates.

A chat message appears.

The naive approach is to sprinkle sleeps everywhere:

wait 2 seconds
check status

But if the event normally arrives in 200 milliseconds and occasionally takes 2.2 seconds, you haven’t fixed the timing problem.

You’ve created a slower flaky test.

A better model is to wait for a meaningful application condition.

There’s a useful discussion of this in how to test WebSocket and real-time UI updates without creating flaky browser suites.

The distinction matters:

Time passing is not an application state.

“Wait until the order becomes processed” is meaningful.

“Wait three seconds” is guessing.

That sounds obvious when you write it down, yet huge automation suites are held together with exactly those guesses.

Mock less than you think — but more than zero

Another place teams get stuck is deciding whether E2E tests should hit real services.

I don’t think there’s a universal answer.

If every browser test calls seven third-party APIs, your suite can become a distributed systems monitoring tool instead of a product test.

But if you mock everything, you can build a beautiful green test suite for an application that doesn’t actually work.

A good compromise is to ask what the test is supposed to prove.

If you’re testing:

how your UI handles an error response, stub it;
a complicated edge case that is expensive to reproduce, mock it;
the actual integration between two systems, use the real service.

This article on when to mock, stub, or hit real services in Playwright E2E tests goes deeper into that tradeoff.

The mistake is turning this into ideology.

“E2E tests must always use real services” sounds principled until Stripe has an incident and your deployment pipeline becomes unusable.

Browser state is part of your application

Cookies are another great example.

A consent banner looks trivial until you realize that accepting it might affect:

analytics initialization,
advertising scripts,
local storage,
accessibility,
page layout,
tracking events,
subsequent sessions.

Testing whether the button disappears is not enough.

The more useful question is whether the application enters the correct state afterward.

There’s a good practical checklist in how to test cookie consent banners without breaking analytics, tracking, or accessibility.

Browser extensions can create an even stranger version of the same problem.

Password managers, ad blockers, developer extensions, privacy extensions, or your own product extension can inject DOM nodes, scripts, styles, and event handlers.

Suddenly a test behaves differently on one machine and everyone blames Selenium.

If that sounds familiar, debugging Chrome extension side effects without polluting your main test suite is worth reading.

Don’t page people because a selector blinked

There’s one last consequence of all this.

If your test suite has uncertain state, be very careful before connecting every failure to Slack, Teams, or PagerDuty.

Alerts create an implicit promise:

This deserves your attention.

Break that promise often enough and people stop paying attention.

Then the one important failure gets ignored along with the 43 harmless ones.

The better approach is to classify failures first.

Is this:

a confirmed product regression?
an infrastructure problem?
an environment outage?
a test maintenance issue?
a transient retry?
an unknown failure worth investigating?

Only some of those deserve an interruption.

If you’re wiring browser automation into incident channels, this guide on what to check before adding browser test alerts to Slack, Teams, or PagerDuty is a useful sanity check.

Flakiness is information

A flaky test is annoying.

But it’s also telling you something.

Sometimes the selector is bad.

Sometimes the test is badly designed.

And sometimes the test is exposing the fact that your application has five sources of state nobody has clearly modeled.

Deleting the test fixes only one of those problems.

The more productive question isn’t:

Why did the automation fail?

It’s:

What assumption did this test make that wasn’t true this time?

That question tends to lead somewhere useful.

The Framework Tax Is a Product Decision

Markus Gasser — Wed, 05 Aug 2026 19:07:15 +0000

A few years ago, the default test automation decision was mostly technical.

Do we use Selenium? Which language? Where do we run the grid?

Today, the menu is much larger. You can write Playwright tests, ask Claude to generate them, use an AI-first testing platform, or combine several of those approaches.

That sounds like progress, and it is. But more choices have also made it easier to optimize for the wrong thing.

Teams compare how quickly the first test can be created. They should be comparing what happens to the hundredth test after six months of product changes.

The initial script is cheap.

Ownership is expensive.

The framework is not the whole cost

Selenium is open source. Playwright is open source. An AI coding assistant may already be included in a developer’s subscription.

That can make a code-first setup look nearly free.

But the license is only one line in the budget. The rest is scattered across engineering time:

Building and updating fixtures
Managing browsers and drivers
Maintaining CI workflows
Investigating flaky failures
Reviewing generated test code
Keeping shared utilities understandable
Training people who did not design the framework

This comparison of Endtest and Selenium for lower-maintenance browser regression suites gets to the heart of the tradeoff.

Selenium gives teams a tremendous amount of control. But control creates responsibility. Someone has to own the framework, the browser infrastructure, the reporting, and the conventions that keep the suite from becoming a collection of unrelated scripts.

A second analysis of Endtest versus Selenium without owning grid infrastructure makes the same point from an infrastructure angle.

The question is not whether your team can maintain a Selenium Grid.

The question is whether maintaining it is a good use of your team’s attention.

AI-generated code still becomes your code

AI coding tools have made it dramatically faster to produce Playwright tests.

That is useful. It also creates a new kind of maintenance illusion.

A generated test may look complete, but somebody still has to understand:

Why the locator was chosen
Which waits are meaningful
What data assumptions are hidden in the setup
Whether the assertions represent the actual user outcome
How the test should change when the application changes

This comparison of Claude-generated Playwright tests and editable Endtest automation frames the choice well.

Code generation accelerates authorship.

Editable automation changes who can maintain the result.

That difference matters for teams where QA owns browser coverage but does not have a dedicated SDET maintaining a custom framework.

I am not arguing that one model is universally better. A team with strong test engineering skills may prefer code because it offers maximum flexibility. A smaller product team may prefer a managed system because it keeps tests readable and execution infrastructure out of the critical path.

The right answer depends on the team you actually have, not the team architecture looks impressive on a slide.

AI-generated frontends make framework ownership harder

AI is changing the application side too.

Teams are generating components, layouts, and entire frontend flows faster than before. That can increase product velocity, but it also means the interface may churn more frequently.

The article on testing AI-generated frontends with Endtest and Playwright describes an increasingly common problem: the test framework becomes a second system that must keep pace with a rapidly changing first system.

If the UI changes weekly, the team needs one of two things:

Enough engineering capacity to maintain the framework continuously.
A testing model designed to reduce that maintenance burden.

The worst option is pretending the burden does not exist.

Compare churn, not just features

AI-first testing tools often look similar in feature lists.

They may all mention test generation, self-healing, browser coverage, dashboards, and integrations.

A more useful comparison is how they handle churn.

This review of Endtest and Mabl for teams testing AI-generated UI changes is valuable because it focuses on what happens after the interface changes.

When evaluating any platform, I would ask:

Are generated tests understandable?
Can a human edit every important step?
Are locator changes visible?
Can AI suggestions be rejected?
Does the system preserve evidence when a repair is attempted?
Can the team separate product failures from test maintenance?

The tool that creates the prettiest demo is not necessarily the one that survives twelve months of product development.

Pricing should include maintenance

People naturally compare subscription prices because they are visible.

Engineering time is harder to measure, so it often disappears from the comparison.

The Endtest pricing breakdown for browser coverage, support, and maintenance is a useful reminder to calculate total operating cost, not just the invoice.

For a code-first framework, include:

Initial setup
Monthly framework maintenance
CI and browser infrastructure
Failure investigation time
Onboarding
Opportunity cost when engineers are pulled away from product work

For a managed platform, include:

Subscription cost
Migration effort
Platform limitations
Vendor dependency
Training and workflow changes

Neither side should get a free pass.

The decision is really about ownership

At Endtest, we have always believed that tests should remain human-readable and editable, even when AI helps create them. Features such as the AI Test Creation Agent are useful because they compress the tedious part without turning the result into a black box.

But the larger principle applies beyond any single product.

Choose the automation model that matches your ownership model.

Use a custom framework when the flexibility is worth maintaining.

Use a managed platform when browser infrastructure and framework upkeep are not strategic advantages.

Use AI generation when someone can still review and understand the result.

The most expensive test suite is not the one with the highest subscription price.

It is the one everybody depends on and nobody truly owns.

The Test Automation Tax Nobody Budgets For

Markus Gasser — Tue, 04 Aug 2026 22:18:06 +0000

Most teams calculate the cost of test automation by looking at how long it takes to create the first version of the test suite.

That is usually the least expensive part.

The real cost appears six months later, when the application has changed, the original test author has moved to another project, and nobody is entirely sure whether the failing test represents a product bug or a stale assumption.

I have seen teams spend far more time maintaining automation than they originally spent building it. The problem is rarely a single bad technology decision. It is usually a collection of small decisions that compound.

The wrong locator here. A test with no clear owner there. A self-healing system that quietly changes behavior. Eventually, the test suite becomes another product that the company must maintain.

Difficult interfaces reveal weak abstractions

A simple login form can make almost any automation tool look good.

The more useful evaluation happens when you point the tool at the awkward parts of your application:

Nested iframes
Shadow DOM components
Dynamic tables
Custom controls
Reusable web components
Elements rendered differently across browsers

A good starting point is this guide on evaluating automation tools for Shadow DOM, iframes, and other hard-to-test surfaces.

The goal is not to find a tool that claims to support every frontend technology. Nearly every vendor can create a convincing demo.

The question is what happens when the component changes.

Can someone on the QA team understand the generated locator? Can they replace it without rewriting the test? Does the failure report explain which DOM boundary caused the problem?

Testing modern components also requires a more deliberate approach to selectors. This guide to testing web components and slot-based interfaces without fragile selectors highlights an important principle: tests should target stable product meaning, not incidental implementation details.

That distinction sounds academic until a frontend refactor breaks 80 tests overnight.

Code generation does not eliminate maintenance

Playwright has made browser automation considerably more approachable. But approachable is not the same as maintenance-free.

The hidden maintenance cost of Playwright tests is rarely the initial setup. It is the ongoing engineering work:

Updating fixtures
Diagnosing timing problems
Managing test data
Reviewing generated code
Maintaining CI infrastructure
Teaching new team members how the framework works

None of these are arguments against Playwright. They are arguments against calling the software “free” simply because the license costs nothing.

The same principle applies to AI-generated tests. Generating code quickly is valuable, but speed at creation can hide fragility.

Small wording changes are a common example. This analysis of why AI-generated tests fail after minor UI copy changes explains what happens when an agent uses visible text as both the intent and the locator.

A human understands that “Continue,” “Next,” and “Proceed” may represent the same action. A brittle test sees three unrelated strings.

Self-healing can create silent failures

Self-healing sounds like the perfect solution to locator maintenance.

A button moves, the old selector fails, and the system finds the new button automatically.

But there is a dangerous edge case: the test can heal itself by selecting the wrong element.

That is worse than a normal failure.

A normal failure stops the run and asks someone to investigate. An incorrect repair can allow the test to pass while validating the wrong behavior.

This article on preventing self-healing agents from rewriting the wrong locator outlines the controls I would expect from any serious implementation:

A visible explanation of what changed
The old and new locator
Confidence information
Screenshots or DOM evidence
Human approval for ambiguous replacements
A way to roll back the change

Self-healing should reduce repetitive work. It should not remove accountability.

The owner matters more than the framework

The most reliable test suites I have seen have one thing in common: ownership is obvious.

Someone knows why each important test exists. Someone decides whether a failure should block a release. Someone removes redundant tests instead of letting the suite grow forever.

When ownership is vague, automation slowly turns into an archaeological project.

This guide on auditing test suite ownership before an automation program stalls provides a useful exercise. For each major area, identify:

Who owns the product behavior?
Who owns the automated coverage?
Who investigates failures?
Who approves major test changes?
Who decides when a test should be deleted?

If the answer to all five questions is “the QA team,” you probably do not have ownership. You have a queue.

Optimize for understandable automation

This is one reason I prefer automation systems that keep tests readable after AI has helped create them.

For example, the Endtest AI Test Creation Agent can help translate a natural-language instruction into test steps, while the resulting test remains visible and editable. AI Assertions can evaluate behavior that is difficult to capture with a rigid text comparison, but the broader workflow still needs to be understandable by the team.

The AI is an accelerator, not the owner.

That is the standard I would use for any automation platform or framework:

Can another person understand the test, diagnose its failure, and safely change it without asking the original author?

If the answer is no, the team has not eliminated maintenance.

It has merely postponed it.

Your Test Suite Is Not Reliable If Nobody Trusts Its Failures

Markus Gasser — Mon, 03 Aug 2026 21:14:49 +0000

Most teams do not have a test automation problem.

They have a trust problem.

The distinction matters because you can add more tests, increase browser coverage, buy another dashboard, and parallelize the entire suite without fixing the underlying issue.

When a test fails, does the team believe it?

Or does someone rerun the pipeline and hope the red box disappears?

Once developers stop treating failures as useful information, the test suite becomes expensive background noise. It still consumes infrastructure, engineering time, and attention. It just no longer changes decisions.

That is a dangerous place to end up.

Pass rate is a comforting but incomplete metric

A dashboard showing a 96% pass rate looks reassuring.

But what does it actually tell you?

It does not tell you whether the failing 4% caught meaningful product defects. It does not tell you whether the passing tests covered the risky parts of the release. It does not tell you how many tests passed only after being retried three times.

Pass rate is easy to calculate, which is why teams overvalue it.

More useful measurements connect test behavior to actual release risk. That may include failure recurrence, defect-detection rate, time to diagnosis, retry dependence, escaped regressions, and the percentage of failures that require human investigation.

This breakdown of AI testing metrics that predict production risk offers a useful starting point. The core lesson is that testing metrics should help you make release decisions, not merely decorate a dashboard.

A test suite that passes 99% of the time but misses checkout failures is worse than one that passes 92% of the time and reliably catches revenue-impacting regressions.

The number is not the goal. Better judgment is the goal.

Maintenance cost is part of reliability

Teams often evaluate an automated test by asking whether it works today.

That is the easiest point in the test’s life.

The expensive part begins later:

The interface changes.
The test data becomes invalid.
An API response changes shape.
The application introduces a loading state.
The original test author leaves.
Nobody remembers why a particular assertion exists.

A test that takes ten minutes to create and three hours per month to maintain is not a ten-minute test.

This is especially important when evaluating AI-assisted testing. Generating a test quickly is useful, but generation speed tells you almost nothing about the long-term economics of the suite.

A better evaluation looks at how often generated tests require editing, how clearly failures are explained, how much context a person needs to repair them, and how sensitive they are to ordinary product changes.

The article on AI test evaluation metrics that predict maintenance cost explores exactly this issue. Maintenance burden is not an unfortunate side effect. It is one of the primary outputs of your testing architecture.

Ignore it, and the suite will eventually become too expensive to trust.

Observability is not the same as having more logs

When a test fails, most tools can produce information.

The problem is that information is not automatically useful.

A screenshot, browser console dump, network archive, DOM snapshot, video, and 4,000-line execution log may technically contain the answer. But if someone needs forty minutes to reconstruct what happened, the system has poor observability.

Good observability reduces the distance between a failure and a plausible explanation.

It should help answer:

What did the test expect?
What actually happened?
What changed immediately before the failure?
Is this likely a product problem, test problem, data problem, or infrastructure problem?
Has the same failure pattern appeared before?

This is why teams should be careful when comparing tools based on dashboard screenshots. More widgets do not necessarily mean faster diagnosis.

A practical guide to evaluating AI test observability without getting lost in dashboard noise makes this distinction well. The best system is not the one that records the most data. It is the one that helps a person reach the correct conclusion with less effort.

AI agents need recovery tests, not just happy-path tests

Testing an AI agent introduces another layer of uncertainty.

A conventional test follows a known sequence. An agent may choose a tool, interpret the result, update its working state, and decide what to do next.

That means a successful final answer can hide a fragile process.

Suppose an agent is asked to update a customer record. It selects the wrong tool, receives an error, retries with a second tool, partially changes the record, and then reports success.

A test that checks only the final response may miss the entire problem.

Agent testing needs to examine:

Tool selection
Argument construction
Memory updates
Recovery behavior
Retry limits
Partial side effects
Escalation when confidence is low

The guide on testing AI agents for tool use, memory, and recovery paths is worth reading because it treats the agent’s decision process as part of the product.

That is the right mental model.

An AI agent is not reliable merely because it reaches the correct answer during a demo. It is reliable when it behaves safely and predictably after something goes wrong.

Maintenance needs an operating routine

Even a well-designed test suite degrades without ownership.

Tests accumulate. Product behavior changes. Old assumptions survive in assertions long after the corresponding feature has been redesigned.

You need a recurring maintenance routine.

That does not have to become a giant quarterly initiative. A lightweight process is usually better:

Remove tests that no longer influence decisions.
Review tests with frequent retries.
Identify the most expensive failures to diagnose.
Consolidate duplicated setup logic.
Update test data before it causes widespread failures.
Assign ownership for unstable areas.
Track whether maintenance work reduces future interruptions.

This test automation maintenance checklist for scaling QA teams provides a practical structure.

The key is to treat maintenance as normal product work rather than evidence that the automation project failed.

Software changes. Tests must change with it.

The real objective

The goal of test automation is not to produce the largest possible collection of green checkmarks.

It is to help a team ship changes with more confidence and less wasted effort.

That requires more than execution.

It requires meaningful metrics, diagnosable failures, sensible recovery behavior, and a maintenance model the team can sustain.

The test suite earns trust one useful failure at a time.

Once it loses that trust, adding more tests usually makes the problem worse.

AI Can Generate a Test Suite. That Does Not Mean You Have One.

Markus Gasser — Wed, 29 Jul 2026 09:08:16 +0000

Generating browser tests with AI has become almost comically easy.

You paste a requirement into Claude, ChatGPT, Copilot, or another coding assistant. A few seconds later, you have a folder full of Playwright tests, fixtures, page objects, and configuration files.

The tests may even pass.

This creates a dangerous moment because the output looks much closer to a finished system than it really is.

A generated test suite is not valuable because it contains many tests. It is valuable when the team can understand it, trust it, maintain it, and use its results to make release decisions.

Those are very different things.

The demo is not the product

Most AI test-generation demos follow the same path:

Give the model a user story.
Ask it to generate tests.
Run the tests.
Show several green checkmarks.

That proves the model can produce executable code. It does not prove that the resulting suite covers the right risks.

Before trusting a generated suite, QA teams should measure much more than the number of tests or the initial pass rate. This guide on what QA teams should measure before trusting a test suite generated by Claude or another coding assistant covers the kinds of signals that matter: useful coverage, false confidence, maintainability, failure clarity, and the suite’s ability to detect real regressions.

A suite with 300 generated tests may be less useful than 30 carefully chosen ones.

The larger suite can create more noise, more duplicated setup, more brittle selectors, and more opportunities for failures that nobody understands.

AI expands whatever process you already have

AI does not automatically fix a weak testing strategy.

It scales it.

When your requirements are vague, the generated tests are vague. When your application has inconsistent test data, AI generates more code around inconsistent data. When your team cannot agree on what should be tested, the model fills in the gaps with assumptions.

This is why hallucinations in test automation are not just a model-quality problem. They are often an input-quality and architecture problem.

The practical guide to reducing AI hallucinations in test automation makes an important distinction: the more context you send and the less structured that context is, the more opportunities the model has to invent details.

The obvious response is to improve the prompt. That helps, but only up to a point.

A better long-term approach is to reduce how much the model must infer.

Instead of repeatedly asking an AI assistant to reconstruct a large Playwright framework, teams can store tests in a structured, human-readable format. AI can help create or modify individual actions without having to regenerate the entire implementation.

That is one of the more useful ideas in this practical guide to AI test automation: use AI where it reduces work, but keep the resulting tests visible and editable.

Logging matters more when the system is autonomous

A human-written browser test generally performs a known sequence of actions.

An AI agent may inspect the page, choose an action, fail, reinterpret the screen, retry, select a different locator, and continue.

That flexibility can make the agent more resilient. It can also make failures much harder to understand.

Suppose an agent tries to click a button three times. On the fourth attempt, it chooses a text link with a similar label and reaches the next page.

Did the test recover intelligently?

Or did it stop testing the intended path?

Without the right evidence, you cannot know.

At a minimum, the run should preserve:

The original goal
The chosen action
The locator or target
The page state before the action
The error returned by the browser
Every retry and the reason for it
The alternative strategy selected
Screenshots and DOM evidence
The final outcome

This article about what to log when an AI test agent retries a browser step and still fails provides a useful starting point.

For more complex systems, ordinary text logs are rarely enough. Instrumenting AI test agents with OpenTelemetry spans, structured logs, and replayable artifacts gives the team a better way to reconstruct the agent’s decisions.

The important word is reconstruct.

When an autonomous test fails, the person investigating should not have to guess what the agent believed it was doing.

Prompt drift becomes test drift

Traditional browser tests usually fail because the application changes.

AI-assisted tests introduce another source of change: the prompt.

A small prompt edit can alter which paths are selected, how assertions are interpreted, and what the system considers a successful result.

The same thing happens when the underlying model changes, even when the prompt does not.

That means teams need versioning for more than test code. They may need to preserve:

Prompt versions
Model or agent configuration
Input datasets
Generated steps
Human approvals
Expected outputs
Screenshots and evidence
The reason a generated change was accepted

The article on prompt replay, human review, and evidence retention explains why these capabilities matter once AI-generated behavior enters a release process.

Similarly, this guide to choosing an AI testing platform when prompt changes make UI signals harder to trust highlights a problem many teams discover late: a passing result is not useful unless you understand what remained consistent between runs.

AI interfaces create new categories of assertions

Testing an AI feature is not the same as testing a deterministic form.

A traditional application might return a fixed validation message. An AI assistant may produce several valid answers with different wording.

This changes the question from:

Did the output exactly match this string?

to:

Did the output satisfy the user’s goal without violating important constraints?

Consider AI-generated copy. The text may be semantically correct while still breaking the interface through unusually long sentences, unsupported characters, missing accessible labels, or poor localization. The article on testing AI-generated copy in web interfaces covers these less obvious failure modes.

Form assistants create similar complications. You need to test not just whether a suggestion appears, but whether the user can reject it, edit it, recover from an invalid suggestion, and reset the state. This practical look at testing AI-powered form assistants goes deeper into those flows.

Floating copilots and AI sidebars introduce another set of problems. They can obscure controls, preserve stale context, or behave differently when reopened. This comparison of Endtest and Playwright for testing AI sidebars and floating command panels illustrates how quickly a seemingly simple chat panel turns into a state-management problem.

RAG testing needs evidence, not confidence

Retrieval-augmented generation applications are especially easy to test badly.

A chatbot provides a polished answer, the answer sounds reasonable, and the test passes.

But the real questions are:

Were the correct documents retrieved?
Did the answer reflect those documents?
Were citations attached to the correct claims?
Did the system ignore outdated or unauthorized sources?
Did a ranking change alter the answer?
Could the result be reproduced?

A practical review of testing RAG chatbots, retrieval drift, and source-citation flows shows why testing only the final text is insufficient.

At a broader level, this market map of browser-testing platforms for AI search and reranking validation provides a useful way to think about the available tooling.

The system needs to record enough evidence to distinguish a model problem from a retrieval problem, a ranking problem, a permissions problem, or a frontend problem.

Otherwise, every failure becomes “the AI gave a bad answer,” which is not actionable.

The affordable option is not always the free framework

Open-source libraries are inexpensive to download.

The system built around them may not be inexpensive at all.

There is test generation, framework design, code review, CI integration, debugging, reporting, test-data management, browser infrastructure, retries, artifact storage, and ongoing maintenance.

AI can reduce some of that work. It can also produce more code than the team can realistically review.

That is why the discussion around affordable AI test automation should focus on total cost rather than licence cost.

A generated Playwright repository may be free in the narrowest possible sense. If several engineers spend every sprint repairing it, explaining it, and rebuilding its infrastructure, it is not free in any business sense.

Start with the decision the test should support

The most useful question is not:

How many tests can AI generate for us?

It is:

What release decision will these tests help us make?

Once that is clear, the rest becomes easier.

You can choose the smallest useful scope, define the evidence required, limit agent autonomy, preserve the generated steps, and decide where human review belongs.

AI can dramatically reduce the work required to create tests.

It cannot decide what your organization should trust.

That remains the team’s job.

The Test Framework Is Not the Product

Markus Gasser — Mon, 27 Jul 2026 21:29:05 +0000

A few years ago, the hardest part of building a browser test framework was getting started.

You had to choose a runner, configure browsers, create page objects, wire up reporting, add retries, manage secrets, connect it to CI, and convince someone else on the team to learn how the whole thing worked.

Today, you can open an AI assistant and ask it to generate most of that before lunch.

That sounds like a dramatic improvement. In some ways, it is.

But it also moves the bottleneck.

The question is no longer, “Can we create a framework?”

The question is, “Can we operate what was created?”

That distinction matters more than it appears.

Generation cost is not ownership cost

A generated framework feels cheap because the first version arrives quickly. The code compiles, a few tests pass, and the pull request looks more complete than anything you could have written in an afternoon.

Then reality starts applying pressure.

The application changes. Authentication behaves differently in staging. A shared helper starts hiding failures. Parallel workers collide over test data. Someone upgrades a dependency and three reporters stop agreeing with one another.

The initial generation was fast. The ownership cost was merely deferred.

This is the central problem described in what actually breaks when Claude generates a large Playwright framework. Large generated systems often fail in the seams: fixtures, abstractions, environment assumptions, test data, and conventions that were never explicitly agreed upon.

The code may be readable line by line while the system remains difficult to reason about as a whole.

That is a dangerous form of complexity because it looks productive.

More code can hide less understanding

Teams sometimes evaluate AI-generated automation by counting output:

number of test files;
number of scenarios;
number of passing checks;
number of prompts completed;
number of lines added.

Those numbers are easy to produce and easy to report.

They are also weak proxies for confidence.

A suite with 500 generated tests can be less useful than a suite with 40 deliberately chosen journeys. The larger suite may validate superficial states repeatedly while missing the handful of transitions that actually put revenue, customer trust, or data integrity at risk.

That is why AI test coverage breaks down when teams optimize for prompt pass rate instead of user journey risk. A prompt can succeed while the resulting test strategy remains badly shaped.

The goal is not to prove that the AI followed instructions.

The goal is to reduce the probability of an expensive surprise.

Those are not the same thing.

The stack keeps expanding quietly

A common setup now looks like this:

Playwright runs the browser.
Claude generates or modifies test code.
GitHub Actions runs the suite.
A reporting service stores results.
A visual tool compares screenshots.
A test data service creates accounts.
Slack receives alerts.
Someone maintains prompts, conventions, and guardrails.

Every component can be reasonable on its own.

The problem is the integration surface.

When a test fails, the team has to determine whether the issue came from the product, generated code, browser timing, fixture state, environment configuration, a model assumption, or the reporting layer.

This is the point where Playwright plus Claude starts feeling like too many moving parts. The burden is not necessarily that either tool is bad. The burden is that your team has effectively become the vendor responsible for assembling, documenting, and supporting the combined system.

That can be a good trade for some companies.

It is not automatically a good trade for yours.

Generated frameworks inherit generated inconsistencies

Ask an AI to add ten tests over several weeks and you may receive several competing ideas about architecture.

One test uses page objects. Another uses fixtures directly. One helper waits for network idle. Another waits for a locator. One file creates data through an API. Another drives the setup through the UI. Naming conventions shift with the wording of the prompt.

Each individual decision can look defensible.

Together, they create entropy.

The same concern applies whether the output is Playwright or Selenium. What to watch for when Claude generates a large Playwright or Selenium framework is not merely syntax quality. It is whether the generated system develops a coherent internal model that humans can consistently extend.

Without a strong architecture owner, AI often accelerates local decisions faster than the team can establish global consistency.

You get more automation and less standardization at the same time.

The real test is the second year

The first month of a new framework is unusually flattering.

The original author remembers everything. The application has not drifted much. Dependencies are current. The test count is manageable. Failures still feel novel enough to investigate.

The second year is where the economics become visible.

Can a new engineer understand why a helper exists?

Can QA modify a business flow without rewriting TypeScript?

Can you identify unused fixtures?

Can you upgrade the runner without a migration project?

Can you distinguish a product defect from a brittle assertion in ten minutes?

A useful evaluation should focus on operational questions like these. This is also why guidance on choosing a browser testing tool for stable runs on fast-changing frontends should be read as an organizational decision, not a feature checklist.

Fast-changing products punish unclear ownership and fragile abstractions.

They reward systems that remain legible under change.

Lightweight is a promise that needs testing

“Lightweight” sounds good because nobody wants another platform rollout.

But lightweight can mean several different things:

fewer features;
less configuration;
a smaller runtime;
a simpler interface;
less vendor involvement;
more work delegated to your own team.

The last meaning is often omitted.

Before adopting a small AI runner, compare what is included and what you will need to build around it. What to compare before adopting a lightweight AI test runner is less about raw capability than about the boundary between the product and your internal engineering work.

A tool can have a tiny installation footprint and a very large organizational footprint.

That is not necessarily wrong. It just needs to be priced honestly.

Buy versus build is now build versus continuously regenerate

The old debate was straightforward:

Should we buy a testing platform or build our own framework?

AI has introduced a third option that feels different but often behaves similarly:

Continuously regenerate and patch an internal framework with AI.

This can reduce the labour required for individual changes. It does not remove the need for architecture, review, debugging, security decisions, test data management, release policies, and maintenance ownership.

AI changes the speed of implementation.

It does not eliminate the consequences of implementation.

The better question is not, “How quickly can we generate this?”

It is:

What permanent responsibility are we creating for the team?

That question is boring, which is usually a sign that it is useful.

A generated test framework can absolutely be the right choice. But the framework is not the product your company sells. It is infrastructure supporting the product.

Treat it accordingly.

Optimize for confidence, comprehensibility, and maintenance cost—not for the excitement of watching a model produce 4,000 lines of code in one sitting.

Your Test Suite Isn't Slow. It's Accumulating Decisions

Markus Gasser — Fri, 24 Jul 2026 22:09:41 +0000

Most browser test suites do not collapse in one dramatic moment.

They get a little slower on Monday.

A little noisier on Tuesday.

Someone adds a retry on Wednesday.

By Friday, the pipeline takes 28 minutes, three tests fail for reasons nobody can reproduce, and the team has quietly learned to merge anyway.

That is how reliability debt works. It rarely looks urgent while you are creating it.

The common explanation is that browser tests are inherently flaky. That explanation is convenient because it makes the problem feel unavoidable. But many of the failures we call “flakiness” are really the accumulated result of dozens of small technical decisions:

an animation that behaves differently in CI;
a feature flag that changes the DOM after the test starts;
a performance threshold that ignores natural variance;
more parallel workers than the environment can actually support;
an assertion that checks implementation details instead of user-visible outcomes.

The test suite is not betraying you. It is reporting the architecture you gave it.

The environment is part of the product

A test that passes locally and fails in CI is often treated as a tooling problem. Sometimes it is. More often, the two environments are not equivalent.

One subtle example is CSS motion. A developer machine may use normal motion preferences while a CI browser reports reduced motion, or vice versa. That can change transition duration, animation timing, element visibility, and even which branch of a component renders.

The result is a test that appears to fail randomly even though it is responding consistently to different inputs. This article on why browser tests fail when CSS motion preferences differ between local and CI environments is a good reminder that browser configuration is test data.

The same principle applies to locale, timezone, colour scheme, viewport size, available fonts, GPU behaviour, network conditions, and feature flags.

Teams often spend hours debugging the final assertion when the real difference was established before the first line of the test ran.

A useful rule is this:

If an environment setting can change the user experience, make it explicit in the test configuration.

Do not depend on whatever default the laptop, container, or hosted runner happens to provide.

Feature flags create multiple applications

A feature flag is not just a Boolean variable. It creates another version of your product.

Five independent flags can theoretically create 32 combinations. Most teams do not test all of them, nor should they. But many teams also fail to define which combinations matter.

That is where browser tests become confusing. The same test name may execute against different UI structures depending on rollout state, account assignment, cached configuration, or timing.

The practical guide on debugging frontend tests that fail after feature flag changes highlights the first thing to verify: what state did the application actually render?

That question sounds obvious. Yet many test reports preserve screenshots and logs without preserving the active flag set.

When a failure is tied to gradual rollout logic, capture at least:

the evaluated flag values;
the user or account segment;
the application version;
the relevant API response;
the DOM or screenshot at the moment the branch appeared.

Without that context, you are not debugging a test. You are reconstructing a missing environment.

Modern UI patterns require different assertions

React Server Actions and optimistic interfaces make applications feel faster by showing the expected result before the server confirms it.

That is good product design. It also creates several states that browser tests can accidentally confuse:

the original state;
the optimistic state;
the confirmed state;
the rollback state after an error.

A test that sees the optimistic update and immediately passes may miss a server failure. A test that waits only for a network request may ignore a rollback bug. A test that asserts every transitional DOM detail becomes brittle whenever the implementation changes.

The article on testing React Server Actions and optimistic UI offers a more durable approach: assert the state transitions that matter to the user, not every intermediate implementation detail.

For a “save” action, that might mean:

the user sees the immediate optimistic update;
the server request succeeds;
the state remains correct after a reload;
an error produces a clear rollback or recovery path.

This is a broader lesson. Browser automation should validate product promises. The closer your assertions are to internal mechanics, the more maintenance you purchase.

Parallelism has a ceiling

When a suite becomes slow, the reflex is to add more workers.

That works—until it does not.

Parallel test execution competes for CPU, memory, browser processes, network bandwidth, database connections, test accounts, and shared environments. Once one of those resources saturates, adding workers can make the suite slower rather than faster.

This framework for understanding why CI test suites get slower when parallelism increases is worth using before paying for larger runners or increasing concurrency again.

The important metric is not the number of workers. It is throughput.

Run a simple experiment:

execute the same representative group with 2 workers;
repeat with 4, 8, and 16;
record total duration, failure rate, CPU, memory, and external-service latency;
stop increasing concurrency when throughput stops improving reliably.

Most teams have an efficient range, not an efficient maximum.

And beware of shared state. Two tests using the same account, inbox, cart, project, or database record are not truly independent. Parallelism merely makes the collision occur sooner.

Dynamic interfaces punish vague synchronization

Tables, filters, and infinite scrolling are common sources of false confidence because the first visible state looks complete before the application is finished.

A filter click can trigger debouncing, multiple requests, a loading placeholder, a DOM replacement, and finally a stable result. Waiting for “the table to be visible” proves almost nothing.

The comparison of Playwright, Cypress, and Selenium for dynamic tables, filters, and infinite scroll shows that tooling matters, but the test model matters more.

Reliable tests usually wait for a business-level condition:

the loading indicator disappears;
the result count changes;
a known row appears;
the final page cursor updates;
the API response corresponding to the action completes.

“Sleep for two seconds” is not synchronization. It is a bet.

Sometimes the bet wins for months. Then the CI environment becomes slightly slower, and the suite suddenly looks haunted.

Performance budgets need tolerance, not wishful thinking

Performance checks are valuable because regressions can be invisible in functional tests. But a strict threshold without an understanding of variance produces alert fatigue.

A page that normally loads between 900 ms and 1.2 seconds should not fail every time it reaches 1.21 seconds. At the same time, using a generous fixed ceiling can hide a gradual decline.

A better model is explained in this guide to building a CI gate for frontend performance budgets without flagging every normal fluctuation.

Useful performance gates often combine:

an absolute maximum;
a percentage regression from a baseline;
several samples instead of one;
separate budgets for different page types;
a warning range before a hard failure.

The goal is not to produce a perfectly stable number. The goal is to detect meaningful degradation early enough to act.

Healthy load tests can still describe an unhealthy product

A load test can show low server response times while real users experience slow pages.

This happens because the test measures the backend request but not the full browser experience: JavaScript execution, hydration, third-party scripts, image decoding, layout shifts, client-side rendering, and long tasks.

The article on why load test results can look healthy while users still experience slow pages makes the distinction clear: infrastructure health and user experience overlap, but they are not identical.

You need both views.

Load tests answer questions such as:

Can the service handle 5,000 concurrent requests?
Where does database latency increase?
When do queues and connection pools saturate?

Browser performance tests answer different questions:

When can the user interact?
Did the main thread freeze?
Did the page shift while loading?
Did a third-party dependency delay the critical path?

A green load test is useful evidence. It is not a certificate of speed.

Measure the cost before adding more coverage

Teams track how many tests they have, but fewer track what those tests cost to own.

That is a mistake because test count is an input. Maintenance burden is the business outcome.

This guide on measuring test suite maintenance cost before it eats your sprint suggests looking beyond execution time.

Track:

engineering hours spent fixing tests;
failures caused by product defects versus test defects;
median time to diagnose a failed run;
repeated failures by component;
tests ignored or retried;
percentage of the suite that has not caught a defect in the last year.

The last metric is uncomfortable, which is why it is useful.

Some tests protect critical workflows and should survive years of product changes. Others were created because coverage looked good in a planning document and now provide very little signal.

Deleting a low-value test is not reducing quality. It can increase quality by making failures credible again.

The operating principle

A reliable test suite is not the one with the most sophisticated framework or the highest test count.

It is the one the team still believes.

That belief comes from boring fundamentals:

explicit environments;
observable feature states;
meaningful synchronization;
realistic concurrency;
outcome-based assertions;
performance thresholds that understand variance;
continuous measurement of maintenance cost.

When a suite becomes slow and noisy, do not begin with retries.

Begin with the decisions.

Your CI Is Not Flaky. Your Failure Triage Is.

Markus Gasser — Thu, 23 Jul 2026 20:08:37 +0000

A red CI build is not a diagnosis.

It is a notification that something happened. That “something” could be a product regression, an unreliable test, a broken test environment, a stale fixture, a browser update, a third-party outage, or a timing issue that only exists under shared infrastructure.

Yet many teams still react to every failure in exactly the same way:

Open the failed job.
Re-run it.
Hope it turns green.
Merge when it does.

That workflow feels fast because it avoids investigation. In reality, it transfers the cost downstream. The same failure returns later, confidence in the suite declines, and people eventually stop treating red builds as meaningful.

The real problem is rarely “too many flaky tests.” It is usually the absence of a dependable failure-triage system.

The three buckets every failure should enter

A useful CI process starts by classifying failures into three broad categories.

1. Product failures

These are the failures you actually want the suite to find:

A button no longer submits a form.
A permission rule exposes the wrong action.
A search result is missing.
A checkout flow breaks after a backend change.
A UI component renders but cannot be interacted with.

The key signal is repeatability. The failure normally appears under the same product state and can be reproduced outside the original CI job.

2. Test failures

These happen when the application is working but the test is not:

A locator depends on brittle DOM structure.
An assertion checks an intermediate state.
A fixed wait is shorter than the real loading time.
A test leaks state into the next test.
A screenshot baseline includes unstable content.

A good starting point is this CI failure triage checklist, which treats product bugs, test noise, and environment drift as separate operational problems instead of one generic “automation failure.”

3. Environment failures

These are often the hardest to identify because they imitate product and test failures:

The test environment was partially deployed.
Seed data was missing.
A shared account was locked.
DNS, storage, email, or a third-party API responded slowly.
The browser version changed underneath the suite.
A worker ran out of memory or disk space.

Environment failures are especially common when browser tests run on shared CI infrastructure. A practical evaluation of Endtest for shared CI browser testing explores the connection between infrastructure stability, coverage, and triage speed.

Stop using retries as your first diagnostic tool

Retries are useful, but only when they generate evidence.

A blind retry answers one question: did the test pass the second time?

It does not tell you why the first run failed.

A better retry captures a comparison set:

First-run screenshot and retry screenshot
Browser console output
Network failures
DOM snapshot or trace
Test data identifiers
Browser and operating system versions
Deployment version
Exact step where timing diverged

This turns a retry from an eraser into an experiment.

When the original and retry runs fail in different places, the problem is likely broader than one selector. When both fail at the same step with the same application state, the odds of a product issue rise. When the retry passes after a much longer load time, the environment or synchronization strategy deserves attention.

Build a CI gate around risk, not perfection

Many teams want a strict rule: one failed test blocks the deployment.

That sounds disciplined. It becomes counterproductive when the suite contains known instability.

The opposite rule—ignore failures and investigate later—is worse.

The useful middle ground is a risk-based gate. This article on building a CI gate for flaky tests without slowing every deployment describes the kind of tradeoff teams need to make.

A practical gate can treat failures differently:

A repeatable failure in a critical checkout or authentication flow blocks immediately.
A first-time failure in a low-risk area triggers one diagnostic retry.
A known flaky test does not silently pass; it creates a tracked reliability event.
Multiple unrelated failures in the same worker flag an environment problem.
A sudden increase in suite-wide duration triggers investigation even if tests pass.

The goal is not to make CI green. The goal is to make CI trustworthy.

Reliability needs a benchmark

Teams often say a suite is “mostly stable,” but that description is too vague to operate.

Start with a few numbers:

First-run pass rate
Retry recovery rate
Failures per 100 executions
Median time to classify a failure
Percentage of failures with enough evidence to diagnose
Failure concentration by test, environment, browser, and application area
Number of quarantined tests and average time spent in quarantine

Modern frontends make these metrics more important. Lazy loading, Suspense boundaries, and route-level code splitting create legitimate intermediate states that tests can easily misread. This guide to benchmarking browser test reliability on lazy-loaded applications is useful because it focuses on repeatable measurement rather than intuition.

A 98% pass rate may sound strong. If you run 2,000 tests per day, it still produces 40 failures. If most of those require manual inspection, the suite is expensive even when its percentage looks impressive.

Flaky-test triage should reduce future work

A flaky-test process fails when it becomes a place to store unresolved failures.

The output of triage should be one of four actions:

Fix the product.
Fix the test.
Fix the environment.
Remove or redesign a test that provides less value than it costs.

The workflow described in this flaky-test triage guide is centered on reducing recurring noise, which is the right objective. Counting flaky tests is not progress. Preventing the same failure pattern from returning is progress.

A useful triage record should include:

Failure category
Evidence
Owner
Temporary containment
Permanent corrective action
Deadline
Similar tests that may share the same weakness

That last field matters. A brittle locator found in one test is often present in twenty others.

AI failures need additional context

AI-assisted workflows add another layer to failure analysis. The UI may be unchanged while a prompt, model response, citation order, or streaming sequence has changed.

For those systems, capture:

Prompt version
Model configuration
Input data
Retrieved context
Response chunks in arrival order
Final assembled response
Screenshots and traces
Human-review decision

This AI test failure triage workflow shows why screenshots alone are insufficient when the behavior depends on prompts and generated output.

The economics are easy to ignore

Teams tend to compare test tools by license price while ignoring the cost of investigation, maintenance, infrastructure, and engineering attention.

That is why the more useful comparison is return on investment, not syntax. The argument is developed in Playwright vs Selenium in 2026: The Real Question Is ROI, Not Syntax.

The same principle applies to CI reliability. A free framework can still support an expensive testing operation. A paid platform can still be poor value. What matters is the total effort required to produce dependable release information.

For another perspective on browser automation strategy, this video discussion is worth adding to your research list.

A red build should create knowledge

The best CI systems do more than approve or reject a commit.

They teach the team:

Which product areas are fragile
Which tests are expensive to maintain
Which environments drift most often
Which failures lack evidence
Which types of regressions escape until production
Where engineering effort will improve confidence fastest

A red build is valuable when it creates a clear decision.

Without classification, evidence, ownership, and follow-through, it is just another notification everyone learns to ignore.

Modern Frontends Don’t Have One “Ready” State

Markus Gasser — Wed, 22 Jul 2026 20:46:41 +0000

A lot of browser tests are still written around a simple mental model:

Open the page.
Wait for it to load.
Interact with the final UI.
Assert that the expected result appears.

That model worked reasonably well when pages arrived as complete documents and JavaScript added a few interactions afterward.

Modern frontends are different.

The page can be visible before it is interactive. A component can render three times before it settles. Text can arrive before the buttons around it. A skeleton can disappear while the real content is still being measured. A route transition can keep the previous screen in the DOM for a few hundred milliseconds. A theme preference can be applied after hydration and briefly produce the wrong colors.

There is no longer one obvious moment when the page is “ready.”

That is why some test suites look healthy in CI while users still report flickering controls, broken keyboard focus, stale content, or clicks that land on elements that are about to disappear.

The intermediate UI is part of the product

Teams often treat loading states as temporary implementation details. Users do not experience them that way.

A user on a slower device may spend several seconds looking at a skeleton screen. Someone opening a server-rendered page may try to click before hydration finishes. A user with reduced-motion preferences may receive a completely different transition path. A returning user may see the wrong theme for half a second before local storage is read.

These are real product states, even if they are short-lived.

A useful starting point is to stop asking only:

Did the final screen appear?

Instead, ask:

What states did the user pass through before the final screen appeared?

That shift immediately changes what you test.

For server-rendered applications, it is worth tracking whether your suite can actually detect client/server divergence rather than merely waiting until the browser repairs it. This guide on measuring whether a frontend test suite catches hydration mismatches provides a useful way to think about coverage.

The same problem appears in streaming interfaces. Content may be inserted in chunks, replaced, or reordered as additional data arrives. An automation agent that assumes the first plausible element is the final element can act too early. The article Why AI Test Agents Break on Streaming UIs, Skeleton States, and Incremental Renders explores why this is especially difficult for AI-driven automation.

“Visible” is not the same as “stable”

A visible element can still be unsafe to interact with.

It may be:

moving because a font has just loaded;
covered by a fading transition layer;
attached to a component that is about to re-render;
a placeholder that will be replaced;
part of the previous route;
visually complete but not yet connected to event handlers.

This is one reason fixed sleeps are so seductive. A two-second pause seems to make the problem disappear.

Until CI gets slower. Or the application gets faster. Or an animation duration changes. Or a third-party request takes 2.3 seconds.

A stronger test waits for a meaningful application condition. That might be the disappearance of a loading marker, the presence of the final record count, the completion of a network request, or a stable element that remains attached across consecutive checks.

Skeleton screens deserve particular attention because they can create both false positives and false negatives. A test may mistake a skeleton row for a real record, or wait for all placeholders to disappear even though infinite scrolling intentionally keeps one visible. This practical guide to testing skeleton screens, loading shimmers, and progressive rendering covers the problem in more detail.

Animation changes the meaning of timing

CSS View Transitions and animated route changes make applications feel smoother, but they also blur the boundary between two screens.

During a transition:

the old screen may still be present;
the new screen may already be present;
both can match the same selector;
focus may move before the animation completes;
screenshots may capture a blended frame;
clicks may hit an element that is technically visible but not usable.

The wrong response is usually to disable all animation in tests. That can be useful for a narrow set of visual checks, but it also means the test environment no longer exercises the behavior users receive.

A better strategy is to separate functional checks from motion-specific checks. Most tests can wait for a final route marker or stable state. A smaller set should deliberately verify transitions, reduced-motion behavior, focus continuity, and interruption handling.

How to Test CSS View Transitions, Route Animations, and Motion-Safe UI Changes offers a good framework for doing that without filling the suite with arbitrary delays.

Persisted preferences create hidden branches

Theme switching looks simple until persistence enters the picture.

A complete test may need to cover:

the default theme for a first-time visitor;
the operating system preference;
a manually selected theme;
persistence after refresh;
persistence in a new tab;
behavior after logout;
synchronization across sessions;
contrast and icon changes;
the brief state before stored preferences are applied.

This is not just a visual regression problem. It is also a browser-state problem.

The comparison in Endtest vs Playwright for testing theme switching, persisted preferences, and dark mode regression risk is useful because it looks beyond the obvious “click the theme toggle” scenario.

Whichever tool you use, the important part is making state explicit. A test should know whether it is starting with clean storage, seeded storage, or a previous session. Otherwise, failures become dependent on execution order.

High-churn interfaces expose maintenance problems quickly

Fast-changing products amplify every weakness in a test suite.

A selector tied to button text breaks when the copy team runs an experiment. A screenshot assertion fails after a spacing adjustment. A locator based on a generated class disappears after a framework upgrade. A test that expects a modal becomes invalid when the flow moves to an inline panel.

This is where maintainability matters more than how quickly the first test was recorded or coded.

A Practical Look at Endtest for Fast-Changing Frontends With Frequent Copy and Selector Drift examines that exact pressure. Two related evaluations—testing dynamic SaaS interfaces with less maintenance and using Endtest for high-churn web apps—are also useful when comparing maintenance approaches.

The broader lesson is tool-independent: design tests around stable product intent rather than incidental markup.

A test should care that the user can submit an invoice, not that the third nested <div> contains a button with exactly the same text forever.

That does not mean avoiding precise assertions. It means being precise about outcomes while being deliberate about which implementation details deserve to become dependencies.

A better model for frontend readiness

For each important workflow, define readiness at three levels:

1. Render readiness

Is the expected structure present?

This catches missing components, server failures, and major rendering problems.

2. Interaction readiness

Can the user actually operate the interface?

This includes event handlers, focus behavior, overlays, enabled controls, and elements that are no longer moving or being replaced.

3. Business readiness

Has the application reached the state that matters?

For example, the order appears in the account, the preference survives refresh, or the newly created record is available from another screen.

Many flaky tests stop at render readiness and immediately perform a business action. The unstable space between those levels is where failures hide.

Test the journey, not just the screenshot at the end

Modern frontend testing is increasingly about transitions between states.

A robust suite observes:

what appears first;
what changes next;
what can be interacted with at each point;
which state is persisted;
which state is temporary;
what happens when rendering is interrupted;
whether accessibility preferences produce a different path.

The final screen still matters. It is just no longer the entire story.

When a team starts treating hydration, streaming, animation, skeletons, and preference restoration as first-class behavior, a surprising number of “random” failures become understandable. More importantly, the suite starts catching the same awkward moments that users notice before those moments become support tickets.

Why Browser Tests Fail Everywhere Except Your Laptop

Markus Gasser — Tue, 21 Jul 2026 09:54:30 +0000

A browser test that fails everywhere is usually easy to diagnose.

A browser test that passes on your laptop, passes when you open DevTools, and then fails in CI is much more dangerous. It encourages the team to blame timing, rerun the job, add another sleep, and move on.

That is how flaky suites become permanent infrastructure.

The problem is rarely that CI is “random.” More often, CI is exposing a difference that your local workflow hides: a different build, a different dependency tree, a different network sequence, a different browser lifecycle, or a different rendering path.

Here is how I approach those failures without immediately reaching for longer timeouts.

Start by proving which environment you are actually testing

Preview deployments often look identical to production while behaving differently underneath.

They may use:

a different API base URL
feature flags tied to branch names
temporary authentication callbacks
incomplete seed data
stricter cookie policies
edge caching that has not warmed up
environment variables injected by a different build pipeline

Before debugging the test, capture the page URL, build identifier, commit SHA, enabled flags, API host, browser version, and viewport. A screenshot is helpful, but it is not enough. Two pages can look the same while loading different JavaScript bundles or talking to different services.

This guide on debugging browser tests that fail only in preview environments is a useful checklist because it treats the preview environment as its own system rather than a smaller copy of production.

A simple rule helps: if the environment can differ, log the difference before the test starts.

Compare the production build, not just the source code

Developers commonly reproduce a failure by running the application locally in development mode. That can be misleading.

Development builds usually preserve readable function names, include detailed error overlays, skip aggressive minification, and generate source maps differently. CI may be exercising a production bundle where stack traces are compressed, chunks are loaded in a different order, and an exception is swallowed by an error boundary.

This is why a test can appear stable while DevTools is open yet fail in a headless CI run. DevTools changes timing, keeps more diagnostics available, and sometimes makes the failure easier for a human to interpret without changing the underlying cause.

A better reproduction workflow is:

Build the exact artifact produced in CI.
Serve that artifact locally.
Use the same environment variables and feature flags.
Run the same browser version in the same mode.
Preserve console errors and unhandled promise rejections.

The article on tests that pass in DevTools but fail in CI because of source maps, minification, and error stacks goes deeper into this class of failure.

Stop treating live UI updates like normal page loads

Server-sent events, streaming responses, and live notification systems create a different synchronization problem.

The page may be loaded, the button may be visible, and the application may still be waiting for a message that arrives later. A test that asserts immediately after an action is not necessarily flaky; it may simply be observing the wrong state transition.

Avoid waiting for arbitrary delays. Instead, wait for evidence that the application reached the state you care about:

a specific notification ID appears
a counter changes from one known value to another
a row receives a final status
a loading marker disappears
a network stream produces a recognizable event

When possible, record the event payload or a correlation ID alongside the UI artifact. That gives you a way to distinguish “the server never sent the update” from “the browser received it but the UI did not render it.”

For more patterns, see testing server-sent events, live notifications, and partial UI updates without flaky assertions.

Hydration failures are state mismatches, not selector problems

Modern React applications can render usable HTML before the client-side application has fully taken control. Suspense boundaries, streaming server rendering, and hydration make the page feel faster, but they also create brief periods where the DOM is present without being stable.

A test may find a button and click it while React is replacing that exact node. The resulting error looks like a detached element, an intercepted click, or a selector that suddenly stopped matching.

The wrong response is usually to create a more complicated selector.

The better response is to identify an application-level readiness signal. Examples include:

a root element gains a hydrated attribute
a skeleton disappears
a client-side event handler becomes active
a streaming region reaches its completed state
the same node remains stable across two observations

This overview of browser testing for React hydration, Suspense, and streaming UI explains why these applications need more than generic “element visible” checks.

Verify the dependency graph used by CI

A lockfile is supposed to make builds reproducible, but teams still end up with differences caused by package manager versions, optional dependencies, platform-specific packages, cached modules, or install commands that do not enforce the lockfile strictly.

When a browser test starts failing after a dependency update, record:

the package manager and version
the lockfile checksum
the runtime version
the resolved version of the suspected package
whether the dependency cache was restored
whether the install modified the lockfile

Do not limit the comparison to direct dependencies. A transitive update in a router, date library, UI component, or polyfill can change browser behavior without appearing in the application code diff.

The checklist in what to log when browser tests fail only in CI after a dependency lockfile change is especially useful when the test failure begins immediately after routine dependency maintenance.

Treat third-party scripts as asynchronous dependencies

Analytics, support widgets, consent managers, fraud tools, payment SDKs, and experimentation platforms can all change the page after your application considers itself ready.

They may inject iframes, move focus, modify the DOM, register global event handlers, delay the main thread, or place overlays above interactive elements. Worse, their behavior can vary by geography, cookie state, account, or time of day.

When one of these scripts is involved, capture more than a screenshot. Log which third-party resources loaded, their response status, load duration, and whether they created new frames or overlays.

This article on what QA teams should log when a browser test fails only after a third-party script loads provides a practical baseline.

Build a failure package, not a failure message

“Element not clickable” is not a diagnosis.

A useful CI failure should preserve enough context for someone to investigate without rerunning the test five times. At minimum, keep:

screenshot and video
browser console output
relevant network failures
application build ID
browser and operating system versions
feature flags
dependency lockfile checksum
current URL and viewport
the last meaningful user action
a DOM snapshot or page source around the failure

The goal is not to collect everything forever. The goal is to make the first failure actionable.

Once you compare environments, production bundles, live update states, hydration readiness, dependency resolution, and third-party behavior, many “random CI failures” stop being random. They become ordinary engineering problems with observable causes.