DEV Community: Markus Gasser

Your Test Didn't Flake. Your UI Moved While Nobody Was Looking.

Markus Gasser — Wed, 14 Oct 2026 21:07:37 +0000

A test clicks Save. The app saves the record. A green toast appears, hangs around for two seconds, and blocks the next button.

The test fails.

Someone adds a three-second sleep. The test passes. Everyone moves on.

Two weeks later, the toast takes four seconds to disappear on a busy CI machine. Same failure. Different Monday.

I think this is one of the most expensive habits in frontend testing: treating every unpredictable UI event as a timing problem. Sometimes the timing is fine. The application has changed which element is actually available for interaction.

A small UI change can be a big state change

Modern apps throw a surprising amount of stuff at a browser: notifications, loading masks, drawers, dropdowns, permission prompts, confirmation dialogs, and third-party widgets. Any of these can alter where focus goes or which control is on top.

Imagine this sequence:

A user changes a billing address.
A confirmation dialog opens.
Clicking Confirm triggers a loading overlay.
A toast slides across the bottom of the page.
The test tries to click Continue.

If Continue is behind the toast, clicking it isn't a legitimate user action at that moment. Retrying the same coordinates is no better. A useful test should wait for an actionable state, then locate the button again.

This is why I like the idea behind this benchmark for testing toast interference and modal stacking. It separates completing the flow from recovering sensibly when the interface gets noisy. Those are different abilities.

A test that passes after twenty silent retries isn't automatically better than one that fails immediately with a screenshot showing the obstruction.

The DOM can be ready while the experience isn't

React and Next.js introduce a variation of the same problem. The server sends HTML. The browser paints it. Hydration begins. Something changes.

The button you found on the first render might be replaced, disabled, or wired to behavior only after hydration. This creates a particularly irritating failure: everything looks fine in the screenshot, but the click happens at the wrong stage.

Blanket rules such as "fail the build for every hydration warning" are tempting. They also make it hard to distinguish a harmless discrepancy from a broken purchase button.

A more useful approach is to reproduce the mismatch, identify the affected component, and assert the interaction the user actually needs. There's a focused walkthrough of testing React and Next.js hydration mismatches without treating every warning as fatal.

The key question isn't whether the console is immaculate. It's whether the page ended up usable and correct.

Focus bugs are the quiet cousin of click bugs

Custom comboboxes are a great trap for overconfident tests. You type into the box, see an option, click it, and assume the widget works.

Now try the same thing without touching the mouse.

Does Arrow Down move the active option? Does Escape close the list? Does focus remain where it should? Does aria-activedescendant point to a real option? Does Enter choose the highlighted item rather than the first item in the DOM?

A click-only test can happily pass through a component that fails for keyboard users. The combobox and typeahead testing guide is worth using as a checklist when you're dealing with custom widgets.

I'd add one rule: check the resulting value and focus state, not merely whether a dropdown was visible.

Please stop screenshotting every pixel

The same mistaken obsession with implementation details shows up in layout testing.

A grid can reflow because content grows, a font changes, or the viewport narrows. That doesn't necessarily mean the user experience is broken. Meanwhile, a footer button can become inaccessible with only a tiny visible change.

For CSS subgrid and intrinsic sizing, I'd rather assert things like:

Key elements don't overlap.
Important controls remain inside the viewport or reachable by scrolling.
Text isn't clipped.
Reading and tab order still make sense.

Use screenshots for visually meaningful regressions, not as the only definition of correctness. The practical examples in this piece on testing CSS subgrid and layout reflows make that distinction nicely.

A better way to debug the next "flaky" failure

When the next intermittent click failure appears, resist the immediate urge to increase a timeout. First find out what was actually on screen. Was the button covered? Was it disabled? Did the page re-render? Where was keyboard focus? Which state was the app in immediately before the action?

Then create a small, repeatable version of the problem. Force the toast to appear. Delay the hydration step. Stack the dialogs. Navigate with the keyboard.

A test that fails consistently under a known bad state is easier to fix than a test that passes inconsistently after a random sleep.

The funny part is that teams often spend more time arguing about locator syntax than asking whether the element was genuinely available to the user. The second question is usually where the bug lives.

10 No-Code Test Automation Tools in 2026, Compared on Real-World Scenarios

Markus Gasser — Tue, 06 Oct 2026 21:56:11 +0000

There are plenty of lists of no-code testing tools.

Most are easy to write.

Open ten vendor websites. Copy the feature lists. Note that everyone now has AI, self-healing tests, and CI/CD integration. Put the products into a table. Call it a comparison.

The problem is that this tells you very little about what happens when you try to automate a difficult application.

A login form is easy.

What about uploading a file inside an iframe?

What about a signup flow that requires opening an email?

What happens when the application opens another browser tab halfway through the test?

Can the tool handle variables, API calls, PDFs, and drag-and-drop interactions without forcing the tester to write JavaScript?

Those are much better questions.

For this comparison, those awkward cases matter more than the size of a vendor's feature list.

TL;DR

Rank	Tool	Best fit
1	Endtest	Best overall no-code test automation platform
2	mabl	Strong low-code alternative for modern QA teams
3	BrowserStack Low Code Automation	Teams that care heavily about browser and device coverage
4	Katalon	Teams mixing manual testers and automation engineers
5	Testsigma	AI-first test creation across several application types
6	ACCELQ	Large enterprise testing programs
7	testRigor	Teams that like writing tests in plain English
8	Testim	Web and Salesforce testing with optional JavaScript
9	Leapwork	Visual business-process automation
10	Shiplight	Developer teams using coding agents and Playwright

If the goal is simply to choose one platform to try first, Endtest would be the pick.

That recommendation comes from what happened once the test cases became difficult, rather than from counting AI features on product pages.

A small problem with the term "no-code"

"No-code" has become a loose category.

Some platforms let non-technical users build most tests visually but still expose JavaScript or another scripting layer for difficult cases. Others try to keep the whole workflow in natural language.

Neither model is automatically better.

Having an escape hatch is useful.

The better question is how often you need it.

For this article, a no-code tool is one where the normal way of building and maintaining a useful end-to-end suite does not require programming.

Optional code is fine.

Needing custom code every time the application does something unusual is where the label starts getting weaker.

The scenarios that matter

Rather than comparing products with a basic search or login test, the hands-on portion of the evaluation used scenarios such as:

Uploading a file from a form located inside an iframe
Creating an account with email verification
Dragging cards between columns
Moving between multiple browser tabs
Entering and leaving iframes
Working with PDF files
Creating and reusing variables
Calling an API during an end-to-end test
Changing the application's language and verifying the result

They are not exotic laboratory tests.

Sooner or later, most large end-to-end suites contain things like these.

They also expose an important difference between platforms. A tool may be excellent at creating straightforward tests and still become surprisingly awkward once a workflow stops being linear.

1. Endtest

Endtest came out on top because it handled the full set of benchmark scenarios while staying inside the same test-authoring model.

It supports several ways of building tests. A tester can record a browser session, add steps manually, or give its AI Test Creation Agent a scenario in plain English.

The resulting test remains visible and editable rather than disappearing into an opaque AI workflow.

That matters more than it initially sounds.

AI test generation is becoming common. The question is what you have after the AI finishes.

With Endtest, the result is still a normal test made of readable steps. A tester can change a locator, insert an assertion, use a variable, add conditional logic, call an API, or inspect exactly what the agent created.

The platform also supports:

AI assertions
AI variables
self-healing locators
visual testing
API testing
email and SMS flows
PDF testing
accessibility checks
web and mobile automation
cross-browser execution

The benchmark result was the biggest reason for putting it first: Endtest completed all nine scenarios.

There was another difference that showed up during execution.

In the test suite used for this comparison, Endtest ran at approximately 1.21 steps per second. mabl averaged around 0.38 steps per second in the corresponding cloud runs.

Those numbers should not be treated as a universal performance benchmark. Application speed, waits, geography, and test design can all change the result.

Still, the gap was large enough to matter in this particular test set.

There is also something refreshingly simple about the pricing.

The Starter plan is currently listed at $175 per month. Endtest publishes its pricing and includes unlimited test creation, unlimited executions, and unlimited users in its standard plans.

The weak point is ecosystem size.

BrowserStack has a much larger device-testing ecosystem. Katalon has a larger community around mixed code and no-code automation. A company that already standardized on one of those platforms may have a good reason to stay there.

For a team starting from scratch, though, Endtest required the fewest compromises in this comparison.

Best fit: teams that want serious web and mobile automation without building a traditional automation framework or hiring a large team to maintain it.

2. mabl

mabl was the closest competitor.

The authoring experience is polished, and the product is clearly built around the idea that testers and developers may work on the same automation suite.

Tests can be created with point-and-click interactions or natural-language instructions. Developers who want more control can add JavaScript or Appium snippets and work alongside Playwright tests.

In the benchmark used here, mabl handled six of the nine scenarios directly. The remaining three could be completed with workarounds.

That is a respectable result.

It also illustrates why feature matrices can be deceptive.

Two platforms may both have a checkmark next to "file uploads" or "iframes," yet one might handle the exact workflow naturally while the other requires restructuring the test.

mabl is especially appealing for organizations where QA engineers work closely with developers and nobody is particularly ideological about whether a test is no-code or low-code.

The cloud execution speed was the main disappointment in this comparison. The runs used here averaged about 0.38 steps per second.

Again, that is one test environment, so it would be a mistake to convert it into a general claim that every Endtest suite runs three times faster than every mabl suite.

It was still noticeable.

If Endtest were removed from this list, mabl would probably be the next platform to evaluate.

3. BrowserStack Low Code Automation

BrowserStack has something most specialized automation vendors cannot easily reproduce: its testing infrastructure.

Its Low Code Automation product uses a recorder for test creation, supports natural-language AI interactions, provides self-healing capabilities, and runs through BrowserStack's cloud infrastructure.

Reports can include video, screenshots, console information, and network logs.

For cross-browser work, that is a strong package.

BrowserStack is particularly attractive when the difficult part of your testing problem is the environment matrix.

You may need Chrome and Edge on Windows, Safari, several mobile browsers, and a long list of real phones.

That has been BrowserStack's territory for years.

There are a couple of details worth noticing during an evaluation.

Some advanced workflows can still lead back to JavaScript. That makes the product feel more accurately described as low-code than completely no-code.

There is nothing wrong with that. It just matters if the reason you are shopping for a platform is that your QA team specifically wants to avoid programming.

Best fit: teams where browser and real-device infrastructure matters just as much as the test editor.

4. Katalon

Katalon is harder to place on a list like this because it can be several different products depending on who is using it.

A manual tester can work with record-and-playback and visual tooling.

More technical QA engineers can go deeper.

Developers can write code.

Katalon has also been adding AI features around requirements, test generation, execution, failure analysis, and bug reporting.

That range is useful inside a large QA organization.

It can also make the product feel heavier than a focused no-code platform.

Katalon makes the most sense when a company does not want to choose between codeless testing and traditional automation.

Different people can work at different levels of abstraction.

The tradeoff is that there is more to learn.

A small team that simply wants to describe a scenario and turn it into a maintainable end-to-end test may find Endtest or mabl more direct.

A larger QA department with manual testers, automation engineers, and existing testing processes may prefer Katalon's breadth.

5. Testsigma

Testsigma has become more interesting as it has moved further into agent-based testing.

Its newer AI workflows can generate tests from plain English, requirements, and other artifacts.

The platform covers web, mobile, APIs, and several enterprise application types.

The appeal is straightforward: the starting point for automation can be material the team already has.

A company with Jira stories, requirements, and manual cases does not necessarily have to rebuild everything as automation by hand.

The part worth testing carefully is what happens after generation.

Ask to see the generated test.

Change it.

Break the application intentionally.

Change an element.

Add a conditional branch.

See what the maintenance workflow feels like.

The first AI-generated test is the easy part now.

Best fit: teams that want AI involved early in test design and already have a lot of written requirements or manual test cases.

6. ACCELQ

ACCELQ has been pushing codeless automation for much longer than the latest wave of generative AI products.

Its approach is built around natural-language test logic and reusable application models.

The platform supports web interfaces, APIs, databases, and other enterprise systems, including end-to-end flows that cross more than one technical layer.

This is one of the products where "enterprise" really does describe the problem it is trying to solve.

Imagine an order that begins in a browser, triggers backend services, writes to a database, and eventually appears in another internal system.

ACCELQ is comfortable in that kind of environment.

That depth comes with more concepts and more setup than lighter tools.

For a five-person SaaS company that wants to automate its regression suite, ACCELQ would probably be more platform than necessary.

For an organization testing large business processes across multiple systems, it deserves a serious look.

7. testRigor

testRigor takes the plain-English idea further than most products.

Instead of building tests around conventional selectors, users write instructions closer to how a human would describe the action.

For example:

click "Add to cart"

That is easier to read than a long CSS selector or XPath expression.

Readable tests matter.

They are easier for someone new to the project to understand, and they reduce the gap between the person defining the requirement and the person automating it.

The concern with any natural-language system is ambiguity.

Humans are good at understanding imprecise instructions because we bring a large amount of context with us.

Automation software still has to decide what those instructions mean.

So when evaluating testRigor, skip the demo-shop examples and feed it workflows from your own application where two controls have similar names or where the page changes dynamically.

If the natural-language model matches the way your application is structured, testRigor can be a very attractive no-code option.

8. Testim

Testim is a mature product with a recorder, a visual editor, and AI-powered Smart Locators.

It also gives users a real JavaScript editor when the visual tooling is not enough.

Tests can contain conditions, loops, reusable groups, and data-driven parameters.

That makes Testim flexible.

It also puts it firmly on the low-code side of the spectrum.

The hands-on benchmark used for this comparison ran into problems with the specific iframe and multi-tab scenarios being tested.

A custom-step workaround did not solve those particular flows.

That should not be read as "Testim cannot use iframes or tabs."

It means the tested workflows did not work cleanly in that evaluation.

That distinction matters.

A vendor can support a capability in general while still struggling with your version of the problem.

Testim becomes more interesting if Salesforce is involved. Tricentis has a dedicated Testim Salesforce product aimed specifically at Lightning applications and Salesforce workflows.

For conventional web applications, Testim remains capable.

For Salesforce-heavy companies, its position on the shortlist gets stronger.

9. Leapwork

Leapwork looks different from most of the tools above.

Tests are built visually as flows made from connected building blocks. There is less emphasis on a traditional test-script metaphor.

That can work particularly well when the thing being automated is already thought of as a business process.

A tester can look at the flow and understand how one action leads to the next without reading code.

The same approach can cover applications beyond a normal browser-testing scenario, which is one reason Leapwork often appears in larger enterprise automation programs.

The downside is that visual flows have their own learning curve.

"No code" does not automatically mean "no complexity."

A large visual workflow can become difficult to understand in the same way that a large codebase can.

Best fit: organizations that prefer visual process modeling and need non-developers to understand or maintain automation.

10. Shiplight

Shiplight is the odd one on this list.

"Agent-native" is probably a more accurate category than traditional no-code.

Shiplight installs into coding-agent environments such as Claude Code, Cursor, and Codex.

The coding agent operates the browser and writes readable YAML tests. Those tests are ultimately executed through Playwright and can be converted into regular Playwright files.

That is a very different workflow from giving a manual tester a visual automation editor.

For a development team already spending its day inside an AI coding agent, the model makes sense.

The test lives closer to the codebase.

The coding agent can create it while working on the product.

Playwright remains underneath.

For a manual QA team looking for a browser recorder and a visual editor, Shiplight would not be the first recommendation.

This is less a criticism than a category distinction.

The interesting question over the next couple of years will be whether agent-native testing becomes a separate market or gets absorbed into existing test platforms.

Why Endtest ranked first

The answer is fairly mundane.

It ran the scenarios.

That sounds obvious, but it is easy to lose sight of when evaluating software from landing pages.

Endtest did well in the areas that matter after the demo:

Tests remained readable and editable after AI creation.
The difficult browser scenarios in the benchmark worked.
API, email, PDF, and data operations could stay inside the same test.
Cross-browser execution did not require a separate infrastructure product.
The measured cloud runs were fast.
Pricing was public and relatively straightforward.

Endtest also allows loops, conditions, variables, database operations, and custom JavaScript from its editor, so there is still an escape hatch when a team wants one.

The difference is that the escape hatch did not become the normal path during the scenarios used here.

That is what a good no-code platform should aim for.

The feature that matters more than AI test generation

Test generation is getting cheap.

Almost every serious platform on this list now has some combination of a recorder, natural-language generation, or an AI agent.

A year or two ago, seeing a prompt turn into an automated browser test felt impressive.

In 2026, it should be expected.

Maintenance is harder.

Create 500 tests and then change the application.

Rename buttons.

Move elements.

Replace a checkout component.

Change the navigation.

Introduce a new authentication step.

Now see how much work is required to get the suite green again.

That exercise tells you far more about the long-term cost of a test automation platform than watching an agent create a checkout test.

Self-healing deserves the same scrutiny.

Ask what the product actually heals.

Does it simply find an element after its CSS selector changes?

Can it understand that an interaction changed?

Does it silently modify the test?

Can a tester review what changed?

There is a substantial difference between recovering a locator and understanding a modified workflow.

Take your worst tests into the trial

This is probably the most useful advice in this article.

Do not spend a product trial automating your homepage.

Pick the test cases your current QA team hates.

The ones everyone postpones.

Maybe there is an iframe inside an iframe.

Maybe clicking a button opens another tab.

Maybe the flow waits for an email, extracts a link, downloads a PDF, and checks something inside it.

Try those first.

A platform that handles your five ugliest tests will almost certainly handle the easy 200.

The reverse is not necessarily true.

Which tool should you choose?

For a general-purpose no-code platform, start with Endtest.

mabl would be the next product to evaluate if you want a polished low-code workflow and close collaboration between QA and engineering.

BrowserStack deserves extra attention when browser and real-device coverage dominate the buying decision.

Katalon fits organizations that want manual, no-code, and coded automation living relatively close together.

ACCELQ makes sense when testing crosses several enterprise systems.

Testsigma and testRigor are both worth evaluating if plain-English creation is central to how the team wants to work.

Testim becomes more compelling when Salesforce is part of the application stack.

Leapwork suits organizations that think visually in terms of processes.

Shiplight is interesting for developer teams that want the coding agent itself to own more of the testing workflow.

There is no reason to narrow the decision down from screenshots and feature grids.

Pick three.

Give all three the same unpleasant test cases.

Then keep the one that causes the least swearing.

FAQ

What is the best no-code test automation tool in 2026?

Endtest is the strongest general-purpose option in this comparison.

It completed all nine scenarios in the hands-on benchmark and combines agentic test creation with an editable no-code test model, cross-browser execution, mobile testing, APIs, email/SMS testing, PDFs, and self-healing capabilities.

What is the difference between no-code and low-code testing?

A no-code platform expects the main test workflow to be created and maintained without programming.

Low-code products do the same for common cases but deliberately expose scripting or programming features for more advanced situations.

There is no clean industry boundary between the two terms, so it is more useful to ask how often your own tests require the coding layer.

Can no-code tools handle complex end-to-end testing?

Some can.

The useful test is whether they can deal with things such as iframes, tabs, file uploads, emails, APIs, dynamic data, and non-trivial application state without requiring a custom script every few steps.

That is why evaluating real workflows matters more than checking whether a product has a recorder.

Can no-code testing replace Playwright or Selenium?

For some teams, yes.

A QA team that mainly needs maintainable end-to-end coverage may get better results from a managed no-code platform because browser infrastructure, reporting, maintenance, and authoring are already provided.

A development team that wants tests stored entirely as source code, custom fixtures, unusual integrations, or complete control over execution may still prefer Playwright or Selenium.

Neither approach is automatically better.

They solve different organizational problems.

What should you test during a free trial?

Start with five difficult cases from your existing regression suite.

Include dynamic elements, multiple tabs, iframes, uploads, external messages, or documents if your application uses them.

Also change something in the application after the tests are created.

Creating the test tells you about authoring.

Breaking it tells you about the product.

What If Your Test Automation Strategy Is Optimizing the Wrong Thing?

Markus Gasser — Tue, 06 Oct 2026 10:01:57 +0000

Most test automation strategies are built around a few familiar goals:

create more tests
run them faster
increase coverage
reduce manual QA
catch bugs before production

All reasonable.

But I think a lot of teams are optimizing the wrong part of the system.

Because once your test suite gets large enough, the expensive part isn't necessarily writing the test.

And it isn't always running the test.

It's this:

Something failed. Now how long does it take before a human actually understands why?

That number is usually invisible.

Nobody puts it on the dashboard.

Nobody brags about it in a product demo.

And yet it might be one of the biggest costs in your entire automation strategy.

Test execution is cheap. Confusion is expensive.

Imagine your CI runs 8,000 browser tests overnight.

Forty fail.

The dashboard looks bad, but the actual problem starts the next morning.

Somebody now has to answer:

Is this a real bug?

Is this a test bug?

Is this a browser issue?

Did a third-party script interfere?

Did the environment change?

Did the page load differently?

Did a consent banner appear?

Did a WebSocket reconnect at the wrong moment?

Did the AI "repair" something incorrectly?

If each failure takes 20 minutes to understand, you've burned more than 13 engineering hours before anyone has fixed anything.

The actual test run may have taken 30 minutes.

So which number matters more?

We measure the wrong things because they're easier to measure

Test automation dashboards love clean numbers.

Execution time: 18m 42s
Pass rate: 98.7%
Coverage: 74%
Tests created this month: 312

Those are easy to count.

But what about:

Average time to understand a failed test: 17 minutes

Or:

Percentage of failures another engineer can reproduce
without talking to the test author: 41%

Those metrics are much harder to measure.

They're also much closer to the real cost of maintaining automation.

There's an interesting benchmark plan for measuring how quickly teams can reconstruct a failed browser test from evidence packs, logs, and replay data.

I think that's a much more interesting benchmark than shaving another three seconds off a test run.

The browser is no longer just your application

This is where things get messy.

People still talk about browser tests as if the browser is loading your page.

It's not.

It's loading something more like:

Your application
+ analytics
+ tag manager
+ cookie consent
+ feature flags
+ chat widget
+ monitoring
+ payment scripts
+ A/B testing
+ whatever marketing added last Thursday

Now imagine one of those scripts loads slowly.

Or not at all.

Or in a different order.

Or injects a banner.

Or moves the layout.

Or blocks a click.

Your test fails with:

Element not clickable

And everybody starts looking at the selector.

The selector may be perfectly fine.

The button may simply have a consent banner sitting on top of it.

This guide on investigating browser test failures caused by third-party scripts, tag managers, and consent banners covers this category of failure in much more detail.

The important lesson is bigger than cookie banners.

A modern browser test is often testing an ecosystem, not just your frontend.

"Page loaded" is becoming a meaningless concept

Here's another assumption that quietly breaks automation.

The page finished loading.

So the page is ready.

Not necessarily.

A modern application might do this:

Navigation complete
    ↓
React hydration
    ↓
Feature flags loaded
    ↓
User state fetched
    ↓
WebSocket connected
    ↓
Live data arrives
    ↓
UI updates

At which point is the application actually ready?

That depends on what you're testing.

If your test starts asserting against the UI between steps four and six, you can get failures that look random but are actually completely deterministic.

You just happened to observe the application in a valid intermediate state.

Real-time applications make this much worse

WebSocket-heavy applications are a good example.

Suppose your dashboard says:

Open incidents: 12

Then a WebSocket event arrives.

Now it says:

Open incidents: 13

Fine.

Now imagine this sequence:

Socket disconnects
Reconnect begins
Two events arrive
UI renders one
Another event arrives
React rerenders
Assertion runs

You now have a race that may happen only occasionally.

The test fails.

Someone reruns it.

It passes.

"Flaky."

Done.

Except nothing was random.

Your test just didn't model the application lifecycle correctly.

There's a useful guide on testing WebSocket-driven UI updates, reconnect states, and live data races in Playwright and Selenium.

These are the kinds of scenarios I'd use to evaluate an automation stack.

Not whether it can fill in a login form.

More AI-generated tests may actually make this problem worse

This is the part I think a lot of teams are going to discover over the next couple of years.

AI makes test creation cheaper.

That's useful.

But if you make something cheaper, people usually do more of it.

So instead of:

500 tests

you get:

5,000 tests

Great.

Now what happens when 83 fail after a deploy?

The bottleneck moves.

You didn't remove the maintenance problem.

You moved it from:

How do we write enough tests?

to:

How do we understand what all these tests are telling us?

That's why platform capabilities around failure evidence, CI stability, ownership, and handoff become much more important as test volume grows.

This rubric on choosing a testing platform when you need stable CI gates, shareable failure evidence, and low-maintenance ownership looks at the problem from that angle.

That sounds much less exciting than "AI writes your tests."

But it's probably closer to the problem mature teams eventually have.

A passing test can be more dangerous than a failing one

This is another uncomfortable part of AI-assisted testing.

Let's say you have this step:

Click "Delete Account"

The UI changes.

The locator breaks.

An AI system repairs the step by pointing it at:

Deactivate Account

The test runs.

Everything passes.

Your dashboard is green.

Except the test is no longer testing the same thing.

A failed test tells you:

Something changed.

A wrongly repaired test tells you:

Nothing changed.

That's worse.

So if your platform supports automatic repair, I want answers to questions like:

What changed?
Why was this replacement selected?
What was the original locator?
What is the new one?
Can I review the repair?
Can I reject it?
Can I see the first failure before the rerun?

If the answer is no, then "self-healing" might be reducing visible failures while increasing invisible risk.

That's not a trade I'd make blindly.

This is why "best testing tool" is usually the wrong question

People love asking:

What's the best test automation platform?

There usually isn't one.

Different teams are optimizing for different things.

One team may care about:

fast authoring
easy onboarding
low maintenance
fewer QA specialists

Another may care about:

governance
permissions
reporting
reusable components
centralized administration
broader QA workflows

That's why a comparison like mabl vs Katalon is more useful when it frames the question as faster authoring versus broader governance.

That's an actual tradeoff.

A score like this:

Tool A: 9.1
Tool B: 8.8

is almost meaningless without knowing what the team cares about.

Software teams don't buy scores.

They buy operating models.

Feature matrices are becoming less useful too

Most modern testing platforms can claim some version of this:

AI ✓
Cross-browser ✓
CI/CD ✓
Assertions ✓
Reports ✓
Self-healing ✓

Okay.

Now show me what happens when:

a third-party script blocks the page
the WebSocket disconnects
the UI hydrates slowly
an iframe changes origin
the browser opens another tab
test data changes
the locator repair picks the wrong element
the test fails after 37 successful steps

That tells me much more.

There are some broader comparisons trying to test these tools against more realistic scenarios. For example, this independent review of 10 AI-powered test automation tools in 2026 looks at practical flows including iframes, file uploads, multiple tabs, email validation, variables, API calls, and language changes.

That's the kind of evaluation I find more useful than another grid of checkmarks.

Here's the question I'd ask your QA team tomorrow

Not:

How many tests do we have?

Not:

What's our pass rate?

Not even:

How fast does CI run?

Ask this:

When a browser test fails, how long does it usually take us to know what actually happened?

And then ask:

Could someone who didn't write the test figure it out from the artifacts alone?

If the answer to the second question is no, your automation strategy has a scaling problem.

Because every failure now depends on tribal knowledge.

That's fine with 50 tests.

It's painful with 500.

It's expensive with 5,000.

And AI can get you from 500 to 5,000 surprisingly quickly.

A better automation benchmark

If I were evaluating a testing platform today, I'd build one deliberately nasty scenario.

Something like:

1. Start with an authenticated user
2. Trigger a consent banner
3. Load a third-party script
4. Connect to a WebSocket
5. Receive a live data update
6. Change network conditions
7. Disconnect the socket
8. Reconnect
9. Trigger a layout change
10. Intentionally cause a failure

Then I'd stop looking at the test runner.

I'd give the output to someone else.

Someone who didn't create the test.

No walkthrough.

No hints.

No rerunning it five times.

And I'd ask:

What broke?

Then I'd start the clock.

That number might tell you more about the value of the platform than most of the metrics you're tracking today.

Maybe the goal isn't more automation

This is the part I think is worth questioning.

A lot of teams treat "more automation" as automatically better.

But automation isn't the goal.

The goal is reducing uncertainty.

A test that fails and creates 30 minutes of confusion isn't necessarily saving you much.

A test that fails and immediately explains:

The checkout button became inaccessible
after the consent layer loaded.
Here is the screenshot.
Here is the DOM.
Here are the console logs.
Here is the network activity.
Here is the exact sequence that reproduced it.

is incredibly valuable.

So maybe your next test automation project shouldn't be:

How do we generate 2x more tests?

Maybe it should be:

How do we make every failure 5x easier to understand?

Because once test creation gets cheap enough, that's probably where the real bottleneck moves.

And if your current strategy doesn't account for that, you may be optimizing a number that stopped mattering a while ago.

The Boring Part of Test Automation That Matters More Than AI

Markus Gasser — Fri, 02 Oct 2026 17:51:15 +0000

Every few months, test automation gets a new shiny object.

First it was Selenium.

Then Cypress.

Then Playwright.

Now everything has AI attached to it.

AI-generated tests. AI self-healing. AI agents clicking around your application. AI deciding what should be tested. AI deciding whether a test actually failed.

And some of this stuff is genuinely useful.

But after working with test automation for long enough, I've become increasingly convinced that one of the biggest problems isn't creating tests.

It's answering a much less exciting question:

When a test fails, can another human figure out what actually happened?

That sounds boring.

It is boring.

It's also where a surprising amount of engineering time disappears.

A failed test is not the same thing as useful information

Imagine your CI pipeline tells you:

Checkout test failed.

Element not found:
#place-order-button

Okay.

Now what?

Was the button missing?

Was the checkout page still loading?

Was the user redirected?

Did an experiment change the DOM?

Did a cookie banner cover the button?

Did the test end up inside the wrong iframe?

Did the application return a 500?

Did the browser resize and trigger an entirely different layout?

Was there a JavaScript error?

Or did the test simply click too early?

The red FAILED label is usually the least interesting part of the failure.

The interesting part is the evidence around it.

Test automation has an evidence problem

A good failure should ideally give you enough context that you don't need to immediately rerun the test locally.

At minimum, I want some combination of:

the exact failed step
screenshots
browser console output
network information
page source or DOM state
browser and operating system
timestamps
previous steps
test data
video when useful
enough environment information to reproduce the run

This becomes even more important once the person investigating the failure isn't the person who created the test.

That's common in larger teams.

It's basically guaranteed if you're outsourcing QA.

There's a good breakdown of this problem in The Evidence Standard That Makes Outsourced QA Bugs Reproducible.

The central idea is simple: a bug report shouldn't require a Zoom call to become useful.

The same should be true for an automated test failure.

AI actually makes this problem more important

Here's the funny part.

As AI makes test creation cheaper, we're probably going to create more tests.

Which means:

More executions.

More failures.

More flaky failures.

More unusual edge cases.

And therefore more things somebody has to investigate.

Generating 500 tests sounds impressive until 37 of them fail overnight and your engineering team spends Tuesday morning figuring out which four failures are real.

This is why I'd pay attention to how AI testing platforms handle evidence and review workflows, not just how impressive their test-generation demo looks.

This benchmark plan for AI testing platforms takes an interesting approach by looking at audit-pack completeness, evidence exports, and whether another person can actually reproduce what happened.

That's a much more useful benchmark than:

"We asked the AI to test a todo app and it found three bugs."

Modern web apps are making failures weirder

Web applications aren't getting simpler either.

Consider something as ordinary as an embedded payment form.

It may be:

Your application
    ↓
iframe
    ↓
cross-origin payment widget
    ↓
dynamic authentication flow

When automation fails somewhere inside that stack, "element not found" tells you almost nothing.

Cross-origin frames and embedded widgets deserve their own testing strategy. There's a detailed browser testing benchmark focused specifically on cross-origin iframes, embedded widgets, and frame-switch failure evidence.

These are exactly the kinds of scenarios I'd include when evaluating a testing platform.

Not another login form.

Layout is becoming state

Here's another category of failures that can waste hours.

A test works perfectly.

Then someone introduces:

ResizeObserver
container queries
hydration
responsive rendering
lazy-loaded components

And suddenly the button that was clickable 100 milliseconds ago is somewhere else.

Or the DOM node gets recreated.

Or a component appears before hydration finishes and then disappears.

If you've encountered tests that mysteriously fail after responsive or hydration changes, this guide on debugging browser tests after ResizeObserver, container queries, or hydration reflow is worth reading.

The broader lesson is that screenshots alone aren't always sufficient.

You need enough evidence to reconstruct how the page got into that state.

The edge cases are where automation platforms separate themselves

Almost every browser automation product can:

open Google
type something
click a button

That's no longer an interesting benchmark.

I want to know what happens when the browser is doing something annoying.

For example, what happens under bandwidth constraints?

Testing prefers-reduced-data and slow connections introduces a completely different set of timing problems. This guide covers testing slow-network and bandwidth-sensitive interfaces without turning the suite into a collection of timing flakes.

Or consider browser storage.

What happens when this happens?

localStorage.setItem("data", veryLargeValue);

And the browser replies:

QuotaExceededError

Does your application recover gracefully?

Does your automated test even know what happened?

There's a useful walkthrough on testing QuotaExceededError, browser storage limits, and graceful fallbacks.

These aren't exotic theoretical scenarios.

They're the exact cases where "works on my machine" starts becoming expensive.

Infinite lists are another great example

Virtualized lists are fantastic for application performance.

They're also wonderfully annoying to automate.

The test may believe it saw 20 rows.

The user thinks there are 2,000.

Then the framework recycles those 20 DOM elements while you scroll.

Now you're testing a UI where:

DOM element ≠ permanent record

That changes how you should think about assertions, scrolling, selectors, and screenshots.

There's a solid guide to testing virtualized infinite lists without missing recycled rows, loading gaps, or offscreen click bugs.

Again, these are the tests I'd run before choosing a tool.

Not whether its landing page says "AI-powered."

And then there's state

State is responsible for an absurd number of test failures.

Cookies.

Local storage.

Accounts.

Feature flags.

Experiments.

Cached resources.

Previous executions.

A test that leaves state behind isn't really an independent test anymore.

Feature flags make this especially interesting.

Suppose you test:

Flag OFF → old checkout
Flag ON  → new checkout
Flag OFF → old checkout again

That third state matters.

You aren't just verifying that a feature flag works.

You're verifying that it rolls back cleanly.

This walkthrough on testing feature-flag rollbacks without carrying state between browser automation runs covers the issue in more depth.

It's a small detail until your release process depends on rolling a bad feature back at 2:00 AM.

Then it stops being a small detail.

Test results should help you make a decision

This is ultimately where I think QA tooling gets evaluated incorrectly.

People compare:

recorder quality
AI generation
number of integrations
supported browsers
pricing
whether there's a nice dashboard

Those matter.

But your testing system ultimately exists to answer:

Can we ship this?

And when the answer is "maybe," it needs to show you why.

A useful failure should be able to move through something like:

Automated test
      ↓
Failure evidence
      ↓
Developer / QA review
      ↓
Jira / GitHub / Slack
      ↓
Decision

If information gets lost between those steps, the automation isn't saving as much time as you think.

There's a useful overview of what to look for in QA platforms used for release sign-off, Jira escalations, and Slack-friendly triage.

I like this framing because it evaluates testing software as part of an engineering process rather than as an isolated test runner.

Autonomy vs. control is another real tradeoff

AI testing products are also starting to split into different philosophies.

Some aim for:

Give us your application and we'll figure everything out.

Others are closer to:

We'll automate a lot, but you still control the scenarios, environments, and browser coverage.

Neither approach is automatically better.

They're solving slightly different problems.

For example, this QA.tech vs Octomind comparison frames the decision around autonomy versus control over browser coverage.

That's the kind of tradeoff teams should actually discuss.

If you're running a small SaaS with three engineers, maximum autonomy may be exactly what you want.

If you're testing a mature application with payments, iframes, multiple tabs, complicated state, permission levels, and several supported browsers, control may matter a lot more.

My boring test for testing tools

So here's the evaluation I'd use.

Take a genuinely ugly flow.

Something with:

an iframe
multiple browser contexts
dynamic content
an API call
a responsive layout change
persisted state
a deliberately introduced failure

Then run it.

Don't watch the successful execution.

Break it.

Now hand the failure report to someone who didn't create the test.

And ask:

Can you tell me what happened?

No screen sharing.

No rerunning it five times.

No asking the person who wrote the test.

Just the evidence.

If they can understand the failure and reproduce it, you've got something useful.

If they can't, it doesn't really matter how quickly the AI created the test.

Because test creation is only the first few minutes of a test's life.

You'll be debugging and maintaining it for the next three years.

And that's the boring part nobody puts in the demo.

AI Test Automation Is Easy to Demo. Reliability Is the Hard Part.

Markus Gasser — Sat, 19 Sep 2026 21:49:06 +0000

There is a version of AI test automation that looks almost magical.

You open a website.

You tell an AI agent:

Test the checkout flow.

It clicks around for a few minutes, creates some tests, finds a broken button, and hands you a report.

If you're evaluating testing tools, this is an impressive demo.

But there is an awkward question that usually doesn't get answered during the demo:

What happens after you run those tests 500 times?

Because creating an automated test has never really been the difficult part.

Keeping it useful is.

The cost of creating tests is collapsing

Ten years ago, creating a serious end-to-end test suite usually meant writing a lot of Selenium code.

Then came better frameworks.

Then low-code and no-code tools.

Then recorders.

Now we have AI agents that can inspect an application, understand a natural-language instruction, identify elements, generate assertions, and sometimes even repair tests when the UI changes.

That is real progress.

A QA engineer who previously spent an afternoon creating a complicated workflow might now create it in twenty minutes.

A developer who doesn't know much about Selenium locators can describe what they want tested in plain English.

And a small startup that never had enough QA resources can potentially automate workflows that previously remained entirely manual.

All of this is good.

But it also changes where the bottleneck lives.

When generating tests becomes cheap, maintaining trust in those tests becomes expensive.

A test that passes once tells you almost nothing

Imagine you build a test that does this:

Create an account.
Verify the email address.
Add a product to the cart.
Apply a discount code.
Complete checkout.
Verify the confirmation email.

You run it.

Green.

Great.

Then you run it tomorrow.

Green again.

Still great.

Now put it in your CI/CD pipeline and run it hundreds of times across Chrome, Firefox, Safari, different screen resolutions, and different test environments.

Things start getting interesting.

The application takes two seconds longer to load.

An API rate limit kicks in.

A third-party authentication provider responds slowly.

A cookie banner appears only in one geography.

A browser update changes timing behavior.

The test account already exists.

A download takes seven seconds instead of three.

The application opens a link in another tab.

An iframe loads after the test already tried to interact with it.

Suddenly your elegant AI-generated test isn't really testing the application anymore.

It's testing your ability to keep the test alive.

This is why flaky tests are more dangerous than missing tests

A missing test is obvious.

Everyone knows the workflow isn't covered.

A flaky test is worse because it slowly trains the team to ignore the test suite.

It usually happens gradually.

At first, one test fails occasionally.

Someone reruns it.

Green.

Then three tests behave that way.

Eventually someone says:

CI is red, but it's probably just the tests.

That sentence is the beginning of the end for an automation suite.

Once developers stop believing that a failure represents a real problem, your tests become expensive background noise.

This is why I think one of the most important metrics for an AI testing platform isn't how quickly it can generate tests.

It's how effectively it helps you understand and maintain them after generation.

AI-generated tests still need evidence

Suppose an AI agent tells you:

Checkout failed.

That's useful.

But the next question is obvious.

Why?

A serious testing system should give you enough evidence to answer that question without recreating the entire execution manually.

That can include:

screenshots
video recordings
browser console logs
network information
page source
timestamps
test data
environment configuration
the exact step that failed
the locator or element that was used
execution history

This becomes even more important when AI is involved.

If an AI agent automatically changes a locator, retries a step, or decides that another element is equivalent to the original one, you need visibility into what happened.

Otherwise self-healing can quietly turn into self-hiding.

A test that remains green by changing its own interpretation of the expected behavior is not necessarily a healthy test.

Self-healing should repair implementation details, not requirements

There is an important distinction here.

Imagine your application changes this:

<button id="checkout-button">

into:

<button data-testid="complete-order">

The behavior is identical.

A testing system finding the new element automatically can be extremely useful.

Now imagine the product team removes the checkout confirmation message entirely.

If the AI decides:

I can't find the confirmation message, but I found the order history page, so I'll consider that equivalent.

That's very different.

The first example repairs a locator.

The second example changes the meaning of the test.

Good AI-assisted automation needs a boundary between the two.

AI can help maintain how a test interacts with the application.

It should be much more conservative about changing what the test is proving.

The best test isn't necessarily the smartest test

There is another trap with AI-generated automation.

Because AI can generate complicated scenarios quickly, teams are tempted to create complicated scenarios everywhere.

But boring tests are often better tests.

A small, deterministic workflow that validates one important business function can be much more valuable than a huge autonomous agent exploring half the application.

You want tests where a failure produces a useful signal.

If a 45-step AI-generated journey fails on step 38, you now need to understand:

whether step 38 is broken
whether step 12 corrupted state
whether the AI chose the wrong path
whether test data changed
whether the environment caused the failure

That's a lot of uncertainty.

Sometimes five smaller tests would have been better.

AI changes the economics of test creation.

It doesn't repeal the engineering principles behind good automation.

Evaluate test automation tools after the honeymoon period

When teams evaluate testing platforms, the first evaluation is usually something like:

"Can this tool automate our login and checkout flows?"

That's reasonable.

But almost every mature testing platform can automate a login form.

The more revealing questions come later.

What happens when:

the UI changes?
a locator matches three elements?
the application becomes slower?
the test needs a dynamically generated email address?
a file has to be uploaded?
a PDF gets downloaded?
the workflow opens another browser tab?
authentication expires halfway through the test?
the same suite runs in CI instead of someone's laptop?
Chrome ships an update?
a test fails only on Windows?
twenty tests fail at once?

Those scenarios tell you much more about a testing platform than watching an AI agent successfully click a Sign In button.

If you're currently comparing platforms, the report Top Test Automation Tools in 2026 - A Performance-Based Comparison Review provides another useful perspective on the current test automation landscape.

But whatever comparison you use, I would spend less time asking:

"How quickly can I create my first test?"

And more time asking:

"What does test number 500 look like?"

CI/CD is the real maturity test

One of the fastest ways to discover whether your automated tests are actually reliable is to put them in a pipeline.

Running a test manually is forgiving.

You can see what's happening.

You know whether the environment is having a bad day.

You can rerun something.

You can wait another few seconds.

CI doesn't care.

The test either works or it doesn't.

That exposes assumptions very quickly.

You discover that your tests depended on:

a specific screen resolution
cached authentication
manually prepared test data
a file sitting on someone's computer
an unusually fast network
a particular browser version
tests running in a specific order

This is painful, but useful.

A test suite that reliably survives CI execution is much closer to something you can trust in production.

AI should reduce debugging time, not just authoring time

This might be the bigger opportunity for AI in testing.

Generating tests gets most of the attention because it demos well.

But consider the economics.

Suppose AI reduces the creation time of a test from 30 minutes to 5 minutes.

You saved 25 minutes.

Good.

Now suppose that test runs 2,000 times during its lifetime and fails unexpectedly 30 times.

If each failure takes someone 15 minutes to investigate, you've spent seven and a half hours debugging one test.

This is why AI-assisted failure analysis, intelligent log summaries, locator diagnosis, screenshot comparison, and root-cause suggestions may ultimately provide more value than test generation itself.

The expensive part of automation isn't always writing the first version.

It's everything that happens afterward.

The boring future of AI testing is probably the useful one

I suspect AI testing will gradually become less impressive to watch.

And that's a good thing.

Today, the demos emphasize autonomous agents moving around websites.

A few years from now, the valuable AI functionality may be much less visible.

It will quietly:

choose better locators
recognize when the application is genuinely broken
distinguish product failures from infrastructure failures
explain why an assertion failed
detect repeated flaky patterns
generate appropriate test data
adapt to harmless UI changes
summarize hundreds of execution logs
highlight unusual behavior across browsers

In other words, AI will stop being the star of the demo.

It will become infrastructure.

That's usually what happens with useful technology.

The novelty disappears.

The reliability remains.

And in test automation, reliability was the thing we actually needed all along.

The Best Tests Aren’t the Ones That Pass

Markus Gasser — Tue, 25 Aug 2026 07:47:31 +0000

There’s a trap in test automation that’s surprisingly easy to fall into:

You optimize for getting tests to pass instead of optimizing for learning something useful when they fail.

Those sound similar. They aren’t.

A good test doesn’t just turn green in CI. It survives the messy parts of a real software team: changing environments, stale state, UI refactors, prompt changes, preview deployments, and the occasional modal that decides scrolling is optional.

I’ve been reading a few interesting pieces lately that approach this from different angles.

For AI systems, this look at prompt change governance makes an important point: once prompts become part of production behavior, reviewing and approving their changes starts looking a lot like reviewing code.

The same operational thinking applies to APIs. This guide to choosing API testing tools focuses on the boring-but-important stuff: environment switching, sharing collections, and getting useful CI output.

Preview environments are another good example. Spinning one up is easy. Trusting the smoke test that runs against it is harder. This benchmark plan looks at setup friction, evidence quality, and whether cleanup is actually safe.

Even something as mundane as overlays can expose weak automation. Testing scroll lock, body freezing, and overscroll containment is a nice example of testing behavior instead of hard-coding implementation details.

And as AI test agents become more common, I suspect one metric will matter more than almost anything else: what happens after something goes wrong?

This benchmark for recovery-path quality explores exactly that.

A test that works on the happy path is a demo.

A test system that gives you useful evidence, adapts to ordinary change, and fails in understandable ways is infrastructure.

That’s a much higher bar.

Your Browser Tests Don’t Need to Be Perfect. They Need to Explain Themselves.

Markus Gasser — Wed, 19 Aug 2026 21:57:26 +0000

There’s a strange goal that creeps into test automation projects.

We start by wanting confidence that the product works.

Then, somewhere along the way, the goal quietly changes to:

Make the tests pass.

That sounds similar. It isn’t.

If a browser test fails once every 50 runs and nobody knows why, the problem isn't merely that the test is flaky. The problem is that every failure creates a small investigation.

And investigations are expensive.

Someone opens CI. Someone reruns the job. Someone looks at a screenshot. Someone asks whether Chrome updated. Someone wonders if the test account was already logged in.

Twenty minutes later, the test passes on a rerun and everybody moves on.

Nothing was actually learned.

Do that across 500 tests, several browsers, and a few releases per week, and you've accidentally built a small tax system for your engineering team.

A failed test is only useful if it leaves evidence

I think one of the most underrated characteristics of a test automation system is how well it explains failure.

Consider a login test that unexpectedly lands on the dashboard instead of the login page.

There are at least two very different explanations.

The application might have a bug.

Or the previous test might have left behind a cookie.

Those require completely different fixes.

This is why understanding whether browser test failures come from application bugs or leaked browser state matters more than simply adding another retry.

Retries can make dashboards greener.

They don't necessarily make the system more reliable.

A better test setup preserves enough information to answer questions such as:

What URL was actually loaded?
What did the DOM look like?
Which cookies and local storage values existed?
Which browser and version ran the test?
What happened immediately before the failure?
Was this a fresh browser session?
Did the network request succeed?

The more of those questions you can answer without rerunning the test, the cheaper failures become.

AI doesn't remove this requirement

AI testing tools are making test creation dramatically easier.

That's useful.

But generating a test is usually not the expensive part of its lifecycle.

The expensive part comes six months later when the test fails after somebody changes authentication, redesigns a component, upgrades Chrome, modifies test data, or introduces a new asynchronous request.

So when evaluating an AI testing product, I would spend less time watching the "create a test from a prompt" demo and more time investigating what happens after something breaks.

A useful starting point is this guide on what to check in an AI-native test automation platform for low-maintenance browser regression.

The important questions aren't just:

"Can the AI create a test?"

They're things like:

"Can I understand what it created?"

"Can I edit it?"

"Can I see why it changed something?"

"Can I debug the result without asking the AI to try again?"

That distinction becomes even more important as systems become more agentic.

If an agent performs several actions before something goes wrong, you need to reconstruct its decisions. Traceability, replay controls, screenshots, logs, and failure evidence stop being nice debugging features and become part of the testing model itself.

There's a useful breakdown of this in how to evaluate AI testing platforms for traceability, replay, and failure evidence.

AI can reduce maintenance.

It shouldn't turn test execution into a black box.

Sometimes the browser isn't actually the thing you're testing

Downloads are a great example.

I've seen teams build surprisingly elaborate automation around the browser's native download dialog.

That often creates more trouble than value.

The requirement usually isn't:

Verify that Chrome rendered its Save File dialog correctly.

The requirement is closer to:

Verify that clicking Export produces the correct file.

Those are different tests.

A more robust approach is often to trigger the download, wait for the browser to report it, inspect the resulting file, and validate what actually matters: filename, type, existence, size, or contents.

That's why testing browser file downloads without relying on fragile native dialog assertions is a useful pattern.

It illustrates a broader rule:

Automate the business outcome, not every implementation detail between the user and that outcome.

The closer your assertions are to what the user actually cares about, the less likely your suite is to break because a browser vendor moved a button around.

Then Chrome updates

Of course, even a well-designed suite still runs inside a browser.

And browsers move fast.

An application can remain completely unchanged while a test that passed yesterday suddenly fails because the execution environment changed underneath it.

That can come from:

browser updates,
driver changes,
security restrictions,
rendering behavior,
permission changes,
timing changes,
deprecated APIs.

This creates one of the more frustrating categories of automation failures because the team naturally starts debugging its own code first.

Sometimes for hours.

Keeping track of browser version drift, pinning strategies, compatibility exceptions, and upgrade checks can save a surprising amount of time.

Pinning everything forever isn't the answer either.

Eventually your users will run newer browsers.

A healthier strategy is usually controlled change: know which versions you're running, upgrade intentionally, and have a small compatibility suite that tells you whether a browser update is about to become your problem.

The metric I would actually watch

Teams love test counts.

"We have 2,400 automated tests."

Fine.

But 2,400 tests that produce 40 ambiguous failures every morning might be worse than 700 tests that reliably tell you when the product is broken.

So I'd pay attention to something else:

How much human time does a failed automated test consume before we understand what happened?

Call it mean time to diagnosis if you want to make it sound respectable.

But the concept is simple.

If a failure takes two minutes to understand, automation is doing its job.

If it takes 30 minutes, several reruns, a Slack thread, and the person who originally wrote the framework three years ago, you don't have a test coverage problem.

You have an observability problem.

And adding another 500 tests will probably make it worse.

Reliability is useful. Explainability is what makes it scale.

No serious browser test suite will have zero failures unrelated to product defects.

Networks fail. Data changes. Browsers update. Environments drift. Authentication expires. Third-party services behave strangely.

Trying to eliminate every one of those variables is probably impossible.

The more practical goal is to make unexpected behavior cheap to understand.

That means preserving state.

Recording evidence.

Knowing your browser versions.

Testing outcomes instead of incidental UI.

And making sure that when AI participates in test creation or maintenance, humans can still see what happened.

A good automation system doesn't merely tell you:

Red.

It tells you enough that five minutes later you can say:

Ah. That's why.

That difference sounds small.

At scale, it may be one of the most important differences between a test suite people trust and one they quietly learn to ignore.

AI Can Write Tests Faster Than Your Team Can Understand Them

Markus Gasser — Sat, 08 Aug 2026 21:18:01 +0000

AI coding tools have solved one problem remarkably well:

They can produce code extremely quickly.

That sounds obviously good.

And most of the time, it is.

But software development has never really been constrained by how fast we can type.

The expensive part comes later.

Understanding the code.

Reviewing it.

Debugging it.

Changing it six months later when the person—or model—that wrote it has forgotten why it exists.

Test automation is where this becomes especially interesting.

Generating the Test Is the Cheap Part

You can ask an AI coding assistant:

Write Playwright tests for our signup, login, checkout, password reset, dashboard, invoices, settings, and admin pages.

And a few minutes later you might have hundreds or thousands of lines of test code.

It feels like incredible leverage.

Until the suite starts failing.

That’s the argument behind looking at the hidden cost of AI-generated test code.

Generation cost has collapsed.

Maintenance cost hasn’t.

In some cases, AI actually increases it because you now have more code than your team would have written manually.

AI Pull Requests Need Different Review

There’s another subtle problem.

Humans tend to judge large AI-generated pull requests differently.

When someone on your team writes 80 lines, you probably read them.

When an AI assistant generates 1,800 lines?

You skim.

You look at the filenames.

You check whether CI is green.

Merge.

That’s dangerous for normal application code and potentially worse for test code because a bad test can happily pass for months.

There are good ideas in this guide to testing AI coding assistant pull requests, but the bigger principle is simple:

AI-generated tests need validation just like AI-generated product code.

“Generated successfully” does not mean “tests the right thing.”

Agents Add Another Failure Mode

Now we’re moving from AI that writes test code to AI that actually decides what actions to take.

That introduces a new question:

What if the model chooses the wrong tool?

An agent might have access to:

browser interaction
API calls
database queries
file operations
search
internal tools

The action itself may work perfectly.

It’s just the wrong action.

Testing tool-selection failures in AI agents therefore becomes part of testing the product.

And dynamic web applications make this harder.

Partial renders, changing state, asynchronous updates, and stale context can cause an agent to confidently interact with yesterday’s version of the page.

That combination of tool selection, state drift, and partial renders is probably going to become a much bigger QA category over the next few years.

Self-Healing Can Heal the Wrong Thing

AI test repair is another feature that sounds almost universally good.

Selector changed?

Fix it automatically.

But imagine the application used to have:

Submit Order

And now has:

Submit Quote

The test can no longer find its element.

An AI repair system finds the new button and updates the test automatically.

Technically, it repaired the selector.

Semantically, it may have changed what the test means.

That’s why I like the idea of evaluating AI-generated test repairs before trusting automatic fixes.

Self-healing should reduce maintenance.

It shouldn’t quietly rewrite your specification.

Measure Trust, Not Just Pass Rate

Traditional test dashboards love pass rates.

97.4% passed.

Great.

But with AI-generated or AI-repaired testing, I think teams need another category of metrics:

How trustworthy was the conclusion?

That includes things like:

Was the failure reproducible?
Was a selector repaired?
Did the test behavior change after repair?
Was the failure caused by infrastructure?
Did AI classify the failure correctly?
How often do humans override AI conclusions?

That’s why these ideas around measuring AI test failures in CI matter.

The next generation of QA dashboards probably won’t just say PASS or FAIL.

They’ll tell you why the system believes the result.

AI Should Reduce Abstraction, Not Increase It

This is where I think test automation is heading in an interesting direction.

The worst outcome would be:

AI generates enormous automation frameworks that humans barely understand, then another AI maintains those frameworks because humans can no longer understand them.

That’s technically impressive.

It’s also a strange place to end up.

The better outcome is AI generating tests into representations humans can still inspect, understand, and modify.

Use AI to remove work.

Don’t use it to create a larger black box.

Because the real productivity metric isn’t:

How much test code did the AI generate?

It’s:

How much less testing infrastructure does your team have to think about?

Your Browser Tests Aren’t Flaky. Your Product Is Stateful.

Markus Gasser — Fri, 07 Aug 2026 22:02:53 +0000

There’s a point in almost every end-to-end test suite where the team starts blaming the tests.

The login test fails once every 40 runs.

The onboarding flow occasionally opens on step three instead of step two.

A notification appears before the assertion is ready.

Someone adds a two-second wait.

Problem solved.

Until next Tuesday.

What we call “flaky tests” are often something more interesting: state problems that the test suite happens to expose.

Modern web applications have a lot more state than they did ten years ago. There’s browser state, server state, authentication state, feature-flag state, WebSocket state, local storage, cookies, background jobs, third-party services, extensions, and whatever your frontend framework is currently keeping alive in memory.

The test runner is often the least mysterious thing in the system.

The hardest bugs happen between steps

Consider a five-step onboarding process.

Most teams test it like this:

Fill out step one.
Continue.
Fill out step two.
Continue.
Finish.

That’s useful.

But customers don’t behave like a test script.

They click Back.

They reload halfway through.

They open another tab.

They submit twice.

Their session expires.

They abandon the flow and return tomorrow.

That’s why I liked this breakdown of testing multi-step onboarding flows without missing state leaks, validation problems, and back-button bugs.

The important thing isn’t adding 30 more assertions.

It’s testing the transitions.

Stateful applications tend to break in the gaps between the happy-path steps.

This applies well beyond onboarding. If your application contains long-lived sessions, dynamic forms, approval processes, or workflows that stretch across multiple screens, this guide on evaluating automation platforms for multi-step workflows and session-heavy flows covers many of the same issues.

Real-time applications make this worse

Then you add WebSockets.

Now the browser can change without the browser doing anything.

A notification arrives.

A status becomes “Approved.”

A dashboard updates.

A chat message appears.

The naive approach is to sprinkle sleeps everywhere:

wait 2 seconds
check status

But if the event normally arrives in 200 milliseconds and occasionally takes 2.2 seconds, you haven’t fixed the timing problem.

You’ve created a slower flaky test.

A better model is to wait for a meaningful application condition.

There’s a useful discussion of this in how to test WebSocket and real-time UI updates without creating flaky browser suites.

The distinction matters:

Time passing is not an application state.

“Wait until the order becomes processed” is meaningful.

“Wait three seconds” is guessing.

That sounds obvious when you write it down, yet huge automation suites are held together with exactly those guesses.

Mock less than you think — but more than zero

Another place teams get stuck is deciding whether E2E tests should hit real services.

I don’t think there’s a universal answer.

If every browser test calls seven third-party APIs, your suite can become a distributed systems monitoring tool instead of a product test.

But if you mock everything, you can build a beautiful green test suite for an application that doesn’t actually work.

A good compromise is to ask what the test is supposed to prove.

If you’re testing:

how your UI handles an error response, stub it;
a complicated edge case that is expensive to reproduce, mock it;
the actual integration between two systems, use the real service.

This article on when to mock, stub, or hit real services in Playwright E2E tests goes deeper into that tradeoff.

The mistake is turning this into ideology.

“E2E tests must always use real services” sounds principled until Stripe has an incident and your deployment pipeline becomes unusable.

Browser state is part of your application

Cookies are another great example.

A consent banner looks trivial until you realize that accepting it might affect:

analytics initialization,
advertising scripts,
local storage,
accessibility,
page layout,
tracking events,
subsequent sessions.

Testing whether the button disappears is not enough.

The more useful question is whether the application enters the correct state afterward.

There’s a good practical checklist in how to test cookie consent banners without breaking analytics, tracking, or accessibility.

Browser extensions can create an even stranger version of the same problem.

Password managers, ad blockers, developer extensions, privacy extensions, or your own product extension can inject DOM nodes, scripts, styles, and event handlers.

Suddenly a test behaves differently on one machine and everyone blames Selenium.

If that sounds familiar, debugging Chrome extension side effects without polluting your main test suite is worth reading.

Don’t page people because a selector blinked

There’s one last consequence of all this.

If your test suite has uncertain state, be very careful before connecting every failure to Slack, Teams, or PagerDuty.

Alerts create an implicit promise:

This deserves your attention.

Break that promise often enough and people stop paying attention.

Then the one important failure gets ignored along with the 43 harmless ones.

The better approach is to classify failures first.

Is this:

a confirmed product regression?
an infrastructure problem?
an environment outage?
a test maintenance issue?
a transient retry?
an unknown failure worth investigating?

Only some of those deserve an interruption.

If you’re wiring browser automation into incident channels, this guide on what to check before adding browser test alerts to Slack, Teams, or PagerDuty is a useful sanity check.

Flakiness is information

A flaky test is annoying.

But it’s also telling you something.

Sometimes the selector is bad.

Sometimes the test is badly designed.

And sometimes the test is exposing the fact that your application has five sources of state nobody has clearly modeled.

Deleting the test fixes only one of those problems.

The more productive question isn’t:

Why did the automation fail?

It’s:

What assumption did this test make that wasn’t true this time?

That question tends to lead somewhere useful.

The Framework Tax Is a Product Decision

Markus Gasser — Wed, 05 Aug 2026 19:07:15 +0000

A few years ago, the default test automation decision was mostly technical.

Do we use Selenium? Which language? Where do we run the grid?

Today, the menu is much larger. You can write Playwright tests, ask Claude to generate them, use an AI-first testing platform, or combine several of those approaches.

That sounds like progress, and it is. But more choices have also made it easier to optimize for the wrong thing.

Teams compare how quickly the first test can be created. They should be comparing what happens to the hundredth test after six months of product changes.

The initial script is cheap.

Ownership is expensive.

The framework is not the whole cost

Selenium is open source. Playwright is open source. An AI coding assistant may already be included in a developer’s subscription.

That can make a code-first setup look nearly free.

But the license is only one line in the budget. The rest is scattered across engineering time:

Building and updating fixtures
Managing browsers and drivers
Maintaining CI workflows
Investigating flaky failures
Reviewing generated test code
Keeping shared utilities understandable
Training people who did not design the framework

This comparison of Endtest and Selenium for lower-maintenance browser regression suites gets to the heart of the tradeoff.

Selenium gives teams a tremendous amount of control. But control creates responsibility. Someone has to own the framework, the browser infrastructure, the reporting, and the conventions that keep the suite from becoming a collection of unrelated scripts.

A second analysis of Endtest versus Selenium without owning grid infrastructure makes the same point from an infrastructure angle.

The question is not whether your team can maintain a Selenium Grid.

The question is whether maintaining it is a good use of your team’s attention.

AI-generated code still becomes your code

AI coding tools have made it dramatically faster to produce Playwright tests.

That is useful. It also creates a new kind of maintenance illusion.

A generated test may look complete, but somebody still has to understand:

Why the locator was chosen
Which waits are meaningful
What data assumptions are hidden in the setup
Whether the assertions represent the actual user outcome
How the test should change when the application changes

This comparison of Claude-generated Playwright tests and editable Endtest automation frames the choice well.

Code generation accelerates authorship.

Editable automation changes who can maintain the result.

That difference matters for teams where QA owns browser coverage but does not have a dedicated SDET maintaining a custom framework.

I am not arguing that one model is universally better. A team with strong test engineering skills may prefer code because it offers maximum flexibility. A smaller product team may prefer a managed system because it keeps tests readable and execution infrastructure out of the critical path.

The right answer depends on the team you actually have, not the team architecture looks impressive on a slide.

AI-generated frontends make framework ownership harder

AI is changing the application side too.

Teams are generating components, layouts, and entire frontend flows faster than before. That can increase product velocity, but it also means the interface may churn more frequently.

The article on testing AI-generated frontends with Endtest and Playwright describes an increasingly common problem: the test framework becomes a second system that must keep pace with a rapidly changing first system.

If the UI changes weekly, the team needs one of two things:

Enough engineering capacity to maintain the framework continuously.
A testing model designed to reduce that maintenance burden.

The worst option is pretending the burden does not exist.

Compare churn, not just features

AI-first testing tools often look similar in feature lists.

They may all mention test generation, self-healing, browser coverage, dashboards, and integrations.

A more useful comparison is how they handle churn.

This review of Endtest and Mabl for teams testing AI-generated UI changes is valuable because it focuses on what happens after the interface changes.

When evaluating any platform, I would ask:

Are generated tests understandable?
Can a human edit every important step?
Are locator changes visible?
Can AI suggestions be rejected?
Does the system preserve evidence when a repair is attempted?
Can the team separate product failures from test maintenance?

The tool that creates the prettiest demo is not necessarily the one that survives twelve months of product development.

Pricing should include maintenance

People naturally compare subscription prices because they are visible.

Engineering time is harder to measure, so it often disappears from the comparison.

The Endtest pricing breakdown for browser coverage, support, and maintenance is a useful reminder to calculate total operating cost, not just the invoice.

For a code-first framework, include:

Initial setup
Monthly framework maintenance
CI and browser infrastructure
Failure investigation time
Onboarding
Opportunity cost when engineers are pulled away from product work

For a managed platform, include:

Subscription cost
Migration effort
Platform limitations
Vendor dependency
Training and workflow changes

Neither side should get a free pass.

The decision is really about ownership

At Endtest, we have always believed that tests should remain human-readable and editable, even when AI helps create them. Features such as the AI Test Creation Agent are useful because they compress the tedious part without turning the result into a black box.

But the larger principle applies beyond any single product.

Choose the automation model that matches your ownership model.

Use a custom framework when the flexibility is worth maintaining.

Use a managed platform when browser infrastructure and framework upkeep are not strategic advantages.

Use AI generation when someone can still review and understand the result.

The most expensive test suite is not the one with the highest subscription price.

It is the one everybody depends on and nobody truly owns.

The Test Automation Tax Nobody Budgets For

Markus Gasser — Tue, 04 Aug 2026 22:18:06 +0000

Most teams calculate the cost of test automation by looking at how long it takes to create the first version of the test suite.

That is usually the least expensive part.

The real cost appears six months later, when the application has changed, the original test author has moved to another project, and nobody is entirely sure whether the failing test represents a product bug or a stale assumption.

I have seen teams spend far more time maintaining automation than they originally spent building it. The problem is rarely a single bad technology decision. It is usually a collection of small decisions that compound.

The wrong locator here. A test with no clear owner there. A self-healing system that quietly changes behavior. Eventually, the test suite becomes another product that the company must maintain.

Difficult interfaces reveal weak abstractions

A simple login form can make almost any automation tool look good.

The more useful evaluation happens when you point the tool at the awkward parts of your application:

Nested iframes
Shadow DOM components
Dynamic tables
Custom controls
Reusable web components
Elements rendered differently across browsers

A good starting point is this guide on evaluating automation tools for Shadow DOM, iframes, and other hard-to-test surfaces.

The goal is not to find a tool that claims to support every frontend technology. Nearly every vendor can create a convincing demo.

The question is what happens when the component changes.

Can someone on the QA team understand the generated locator? Can they replace it without rewriting the test? Does the failure report explain which DOM boundary caused the problem?

Testing modern components also requires a more deliberate approach to selectors. This guide to testing web components and slot-based interfaces without fragile selectors highlights an important principle: tests should target stable product meaning, not incidental implementation details.

That distinction sounds academic until a frontend refactor breaks 80 tests overnight.

Code generation does not eliminate maintenance

Playwright has made browser automation considerably more approachable. But approachable is not the same as maintenance-free.

The hidden maintenance cost of Playwright tests is rarely the initial setup. It is the ongoing engineering work:

Updating fixtures
Diagnosing timing problems
Managing test data
Reviewing generated code
Maintaining CI infrastructure
Teaching new team members how the framework works

None of these are arguments against Playwright. They are arguments against calling the software “free” simply because the license costs nothing.

The same principle applies to AI-generated tests. Generating code quickly is valuable, but speed at creation can hide fragility.

Small wording changes are a common example. This analysis of why AI-generated tests fail after minor UI copy changes explains what happens when an agent uses visible text as both the intent and the locator.

A human understands that “Continue,” “Next,” and “Proceed” may represent the same action. A brittle test sees three unrelated strings.

Self-healing can create silent failures

Self-healing sounds like the perfect solution to locator maintenance.

A button moves, the old selector fails, and the system finds the new button automatically.

But there is a dangerous edge case: the test can heal itself by selecting the wrong element.

That is worse than a normal failure.

A normal failure stops the run and asks someone to investigate. An incorrect repair can allow the test to pass while validating the wrong behavior.

This article on preventing self-healing agents from rewriting the wrong locator outlines the controls I would expect from any serious implementation:

A visible explanation of what changed
The old and new locator
Confidence information
Screenshots or DOM evidence
Human approval for ambiguous replacements
A way to roll back the change

Self-healing should reduce repetitive work. It should not remove accountability.

The owner matters more than the framework

The most reliable test suites I have seen have one thing in common: ownership is obvious.

Someone knows why each important test exists. Someone decides whether a failure should block a release. Someone removes redundant tests instead of letting the suite grow forever.

When ownership is vague, automation slowly turns into an archaeological project.

This guide on auditing test suite ownership before an automation program stalls provides a useful exercise. For each major area, identify:

Who owns the product behavior?
Who owns the automated coverage?
Who investigates failures?
Who approves major test changes?
Who decides when a test should be deleted?

If the answer to all five questions is “the QA team,” you probably do not have ownership. You have a queue.

Optimize for understandable automation

This is one reason I prefer automation systems that keep tests readable after AI has helped create them.

For example, the Endtest AI Test Creation Agent can help translate a natural-language instruction into test steps, while the resulting test remains visible and editable. AI Assertions can evaluate behavior that is difficult to capture with a rigid text comparison, but the broader workflow still needs to be understandable by the team.

The AI is an accelerator, not the owner.

That is the standard I would use for any automation platform or framework:

Can another person understand the test, diagnose its failure, and safely change it without asking the original author?

If the answer is no, the team has not eliminated maintenance.

It has merely postponed it.

Your Test Suite Is Not Reliable If Nobody Trusts Its Failures

Markus Gasser — Mon, 03 Aug 2026 21:14:49 +0000

Most teams do not have a test automation problem.

They have a trust problem.

The distinction matters because you can add more tests, increase browser coverage, buy another dashboard, and parallelize the entire suite without fixing the underlying issue.

When a test fails, does the team believe it?

Or does someone rerun the pipeline and hope the red box disappears?

Once developers stop treating failures as useful information, the test suite becomes expensive background noise. It still consumes infrastructure, engineering time, and attention. It just no longer changes decisions.

That is a dangerous place to end up.

Pass rate is a comforting but incomplete metric

A dashboard showing a 96% pass rate looks reassuring.

But what does it actually tell you?

It does not tell you whether the failing 4% caught meaningful product defects. It does not tell you whether the passing tests covered the risky parts of the release. It does not tell you how many tests passed only after being retried three times.

Pass rate is easy to calculate, which is why teams overvalue it.

More useful measurements connect test behavior to actual release risk. That may include failure recurrence, defect-detection rate, time to diagnosis, retry dependence, escaped regressions, and the percentage of failures that require human investigation.

This breakdown of AI testing metrics that predict production risk offers a useful starting point. The core lesson is that testing metrics should help you make release decisions, not merely decorate a dashboard.

A test suite that passes 99% of the time but misses checkout failures is worse than one that passes 92% of the time and reliably catches revenue-impacting regressions.

The number is not the goal. Better judgment is the goal.

Maintenance cost is part of reliability

Teams often evaluate an automated test by asking whether it works today.

That is the easiest point in the test’s life.

The expensive part begins later:

The interface changes.
The test data becomes invalid.
An API response changes shape.
The application introduces a loading state.
The original test author leaves.
Nobody remembers why a particular assertion exists.

A test that takes ten minutes to create and three hours per month to maintain is not a ten-minute test.

This is especially important when evaluating AI-assisted testing. Generating a test quickly is useful, but generation speed tells you almost nothing about the long-term economics of the suite.

A better evaluation looks at how often generated tests require editing, how clearly failures are explained, how much context a person needs to repair them, and how sensitive they are to ordinary product changes.

The article on AI test evaluation metrics that predict maintenance cost explores exactly this issue. Maintenance burden is not an unfortunate side effect. It is one of the primary outputs of your testing architecture.

Ignore it, and the suite will eventually become too expensive to trust.

Observability is not the same as having more logs

When a test fails, most tools can produce information.

The problem is that information is not automatically useful.

A screenshot, browser console dump, network archive, DOM snapshot, video, and 4,000-line execution log may technically contain the answer. But if someone needs forty minutes to reconstruct what happened, the system has poor observability.

Good observability reduces the distance between a failure and a plausible explanation.

It should help answer:

What did the test expect?
What actually happened?
What changed immediately before the failure?
Is this likely a product problem, test problem, data problem, or infrastructure problem?
Has the same failure pattern appeared before?

This is why teams should be careful when comparing tools based on dashboard screenshots. More widgets do not necessarily mean faster diagnosis.

A practical guide to evaluating AI test observability without getting lost in dashboard noise makes this distinction well. The best system is not the one that records the most data. It is the one that helps a person reach the correct conclusion with less effort.

AI agents need recovery tests, not just happy-path tests

Testing an AI agent introduces another layer of uncertainty.

A conventional test follows a known sequence. An agent may choose a tool, interpret the result, update its working state, and decide what to do next.

That means a successful final answer can hide a fragile process.

Suppose an agent is asked to update a customer record. It selects the wrong tool, receives an error, retries with a second tool, partially changes the record, and then reports success.

A test that checks only the final response may miss the entire problem.

Agent testing needs to examine:

Tool selection
Argument construction
Memory updates
Recovery behavior
Retry limits
Partial side effects
Escalation when confidence is low

The guide on testing AI agents for tool use, memory, and recovery paths is worth reading because it treats the agent’s decision process as part of the product.

That is the right mental model.

An AI agent is not reliable merely because it reaches the correct answer during a demo. It is reliable when it behaves safely and predictably after something goes wrong.

Maintenance needs an operating routine

Even a well-designed test suite degrades without ownership.

Tests accumulate. Product behavior changes. Old assumptions survive in assertions long after the corresponding feature has been redesigned.

You need a recurring maintenance routine.

That does not have to become a giant quarterly initiative. A lightweight process is usually better:

Remove tests that no longer influence decisions.
Review tests with frequent retries.
Identify the most expensive failures to diagnose.
Consolidate duplicated setup logic.
Update test data before it causes widespread failures.
Assign ownership for unstable areas.
Track whether maintenance work reduces future interruptions.

This test automation maintenance checklist for scaling QA teams provides a practical structure.

The key is to treat maintenance as normal product work rather than evidence that the automation project failed.

Software changes. Tests must change with it.

The real objective

The goal of test automation is not to produce the largest possible collection of green checkmarks.

It is to help a team ship changes with more confidence and less wasted effort.

That requires more than execution.

It requires meaningful metrics, diagnosable failures, sensible recovery behavior, and a maintenance model the team can sustain.

The test suite earns trust one useful failure at a time.

Once it loses that trust, adding more tests usually makes the problem worse.