DEV Community: Simon Gerber

The Second Browser Tab Is Where Good Tests Go to Die

Simon Gerber — Wed, 14 Oct 2026 21:11:57 +0000

The sign-in test passes locally.

Then you run it in CI and the browser sits on a blank tab until the job times out.

The application is fine. The identity provider is fine. The browser opened another window, and the automation is still trying to click a button in the first one.

Welcome to one of the less glamorous parts of browser automation: keeping track of which browser context you're actually controlling.

OAuth is easy until the provider changes the route

Consider a typical third-party login:

Click "Continue with Google".
A popup opens on another origin.
Enter credentials in the provider's window.
The provider redirects or closes the popup.
The original application receives the result and renders the signed-in state.

It's tempting to automate that as a series of clicks. But the test really has two responsibilities: interact with the correct window during authentication, and prove that control returned to the app afterward.

That distinction matters when the popup sometimes redirects inside itself and sometimes closes. A hardcoded assumption about the "second tab" can break as soon as a browser, account state, or provider configuration changes.

The discussion in this guide to cross-origin popups, OAuth returns, and window-handle swaps is a useful reminder to follow window lifecycle events, not only tab positions.

The wrong assertion can make a broken login look successful

Here's a suspiciously common test:

Click Sign in.
Enter username and password.
Click Continue.
Assert that the login popup disappeared.

Congratulations. You've verified that a popup disappeared.

It might have closed because login succeeded. It might have crashed. It might have been blocked. The user might still be signed out.

The assertion should live in the application: a signed-in session exists, the right account is shown, and a protected page behaves as expected. If the purpose of the test is specifically to inspect OAuth errors, that's a different scenario.

This sounds obvious until you browse a test suite with 300 steps and discover that half of the "success" checks only verify that a modal closed.

Tool choice matters, but not in the usual way

Framework comparisons often start with syntax and speed. I care more about how visible the browser state is when something goes wrong.

Can the test identify the new page or handle without guessing? Can it work across origins? Can it show you what happened in each window? Can the person debugging it understand the sequence without replaying the failure on their laptop?

The Cypress vs WebdriverIO comparison explores some of the tradeoffs around browser control and debugging. The right choice depends on the flows you actually need to support, especially if your app relies on popups, multiple tabs, or external authentication.

Don't decide based solely on the prettiest demo. Give both candidates the same annoying real-world login flow.

Repeatable tests start with boring session hygiene

A second issue often gets mixed up with window handling: test state leaks.

The test opens with an already-authenticated cookie. Or the provider remembers the previous account. Or the registration email is already in use. Now the flow takes a different path from yesterday, and somebody labels it flaky.

Before each run, define the desired starting state. Is this a first-time login or a returning user? Are cookies preserved intentionally? What happens to the user record afterward? If the test creates data, who deletes it?

This evaluation guide to disposable test data and session resets asks a surprisingly valuable question: does the testing platform survive a rerun without someone manually cleaning up?

A great first run followed by a broken second run is not a great test.

Capture the handoff, not just the ending

If a multi-window test fails, a screenshot of the last visible page is often insufficient. You want the sequence:

Which window was active before the click?
Which new window appeared, and when?
What URL did each context reach?
When was the original context restored?
What did the application show after return?

That's also why I find this multi-window testing benchmark plan more useful than a generic pass-rate comparison. It's interested in handoffs and the evidence left behind when they fail.

You don't need a giant framework to think this way. Even a simple test can explicitly record the active context, verify the return, and reset its session.

The irony is that a one-line login assertion can depend on five systems: your frontend, your backend, a browser, an identity provider, and the test runner's context management. When it fails, "add a sleep" is rarely the best investigative strategy.

How to Make Playwright Autonomous (Without Letting an AI Agent Run Wild)

Simon Gerber — Sat, 10 Oct 2026 21:14:46 +0000

Playwright is very good at following instructions.

Tell it to click a button, it clicks the button. Tell it to fill an input, it fills the input. Give it a test with 40 steps and, assuming the page behaves, it'll run all 40.

The problem is that someone has to write those steps.

What if you gave Playwright a goal instead?

Add two items to a shopping cart, apply the discount code, and check that the total is correct.

The browser would need to inspect the page, work out what to click, notice when something changes, and decide what to do next. That's a different kind of automation. And it's a fun engineering problem, provided you don't confuse the agent completed its plan with the application passed a test.

Let's build a small version. No giant agent framework. Just Playwright, an LLM API, and a loop we can inspect.

What we're actually building

A normal Playwright script is a list of instructions:

await page.getByRole('button', { name: 'Add notebook' }).click();
await page.getByRole('button', { name: 'Checkout' }).click();

Our agent will do something closer to this:

Observe the current page using an accessibility snapshot.
Ask a language model for exactly one next action.
Validate that the action belongs to a small set of permitted operations.
Execute it with Playwright.
Observe the new state and repeat.
Separately assert the result in code.

The important word is separately. An agent is useful for finding a path through the interface. It shouldn't get the final vote on whether the system works.

A tiny app to test

We'll use a local checkout page so this tutorial doesn't depend on somebody else's website or an account with real payment details.

Create a directory and install the dependencies:

mkdir autonomous-playwright
cd autonomous-playwright
npm init -y
npm install playwright dotenv
npx playwright install chromium

You'll also need an API key for an OpenAI-compatible chat completions endpoint. The example below uses OpenAI's endpoint and defaults to gpt-4.1-mini. You can select a different compatible model with an environment variable.

Create index.html:

<!doctype html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Tiny Shop</title>
</head>
<body>
  <main>
    <h1>Tiny Shop</h1>
    <p>Notebook: $12</p>
    <p>Pen: $3</p>
    <button id="notebook">Add notebook</button>
    <button id="pen">Add pen</button>
    <p id="cart" aria-live="polite">Cart: 0 items</p>
    <label for="code">Discount code</label>
    <input id="code" />
    <button id="apply">Apply discount</button>
    <p id="discount">Discount: $0</p>
    <button id="checkout">Checkout</button>
    <p id="receipt" role="status"></p>
  </main>
  <script>
    let notebookCount = 0;
    let penCount = 0;
    let discount = 0;
    const $ = id => document.getElementById(id);
    function renderCart() {
      $('cart').textContent = `Cart: ${notebookCount + penCount} items`;
    }
    $('notebook').onclick = () => { notebookCount++; renderCart(); };
    $('pen').onclick = () => { penCount++; renderCart(); };
    $('apply').onclick = () => {
      discount = $('code').value.trim() === 'SAVE5' ? 5 : 0;
      $('discount').textContent = `Discount: $${discount}`;
    };
    $('checkout').onclick = () => {
      const total = notebookCount * 12 + penCount * 3 - discount;
      $('receipt').textContent = `Order total: $${total}`;
    };
  </script>
</body>
</html>

This is deliberately boring. Boring fixtures are good. If the experiment goes sideways, we want to know whether the problem is in the agent, not the store.

Run the page in one terminal:

python3 -m http.server 4173

You should now have a checkout page at http://127.0.0.1:4173.

Give Playwright a brain, but very small hands

Put your API key in .env:

OPENAI_API_KEY=your_api_key_here
MODEL=gpt-4.1-mini

Don't commit this file to Git.

Now create agent.mjs:

import 'dotenv/config';
import { chromium } from 'playwright';
import fs from 'node:fs/promises';

const BASE_URL = 'http://127.0.0.1:4173';
const MAX_STEPS = 12;
const MODEL = process.env.MODEL || 'gpt-4.1-mini';
const GOAL =
  'Add exactly one notebook and one pen, apply code SAVE5, ' +
  'then checkout. The correct final total is $10.';

if (!process.env.OPENAI_API_KEY) {
  throw new Error('Set OPENAI_API_KEY in .env');
}

// The model can choose only one of these operations.
// It cannot submit raw JS, shell commands, or arbitrary URLs.
const SYSTEM_PROMPT = `
You operate a local demo shopping page through a restricted browser API.
Return ONLY a JSON object with one of these shapes:
{"action":"click","role":"button","name":"visible accessible name"}
{"action":"fill","role":"textbox","name":"visible accessible name","value":"text"}
{"action":"finish","reason":"brief explanation"}

Choose ONE action per response. Base it on the current snapshot and history.
Do not invent elements. Do not repeat an action that already succeeded.
Only use the role and accessible name shown on the page.
When the checkout receipt shows the requested total, choose finish.
`;

async function chooseAction(snapshot, history) {
  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: MODEL,
      temperature: 0,
      response_format: { type: 'json_object' },
      messages: [
        { role: 'system', content: SYSTEM_PROMPT },
        {
          role: 'user',
          content: JSON.stringify({ goal: GOAL, history, snapshot })
        }
      ]
    })
  });

  if (!response.ok) {
    throw new Error(`Model API returned ${response.status}: ${await response.text()}`);
  }

  const data = await response.json();
  const content = data.choices?.[0]?.message?.content;
  if (!content) throw new Error('Model returned no content');
  return JSON.parse(content);
}

function validateAction(a) {
  if (!a || typeof a !== 'object' || Array.isArray(a)) {
    throw new Error('Invalid action');
  }
  if (a.action === 'finish') {
    if (typeof a.reason !== 'string') throw new Error('Missing finish reason');
    return;
  }
  if (a.action !== 'click' && a.action !== 'fill') {
    throw new Error(`Unsupported action: ${a.action}`);
  }
  if (a.role !== (a.action === 'click' ? 'button' : 'textbox')) {
    throw new Error('Role not permitted for this action');
  }
  if (typeof a.name !== 'string' || !a.name || a.name.length > 100) {
    throw new Error('Invalid accessible name');
  }
  if (a.action === 'fill') {
    if (typeof a.value !== 'string' || a.value.length > 100) {
      throw new Error('Invalid input value');
    }
  }
}

async function applyAction(page, action) {
  validateAction(action);
  if (action.action === 'finish') return;

  const locator = page.getByRole(action.role, {
    name: action.name,
    exact: true
  });

  // Do not guess when a locator is ambiguous.
  const count = await locator.count();
  if (count !== 1) throw new Error(`Expected one match, found ${count}`);

  if (action.action === 'click') await locator.click({ timeout: 3000 });
  if (action.action === 'fill') await locator.fill(action.value, { timeout: 3000 });
}

const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();
const page = await context.newPage();
const history = [];
let agentFinished = false;
let failure;

try {
  await context.tracing.start({ screenshots: true, snapshots: true });
  await page.goto(BASE_URL, { waitUntil: 'domcontentloaded' });

  for (let step = 1; step <= MAX_STEPS; step++) {
    // The snapshot is a compact description of the accessible UI.
    const snapshot = (await page.locator('body').ariaSnapshot()).slice(0, 12000);
    const action = await chooseAction(snapshot, history);
    validateAction(action);

    console.log(`Step ${step}:`, action);
    history.push(action);

    if (action.action === 'finish') {
      agentFinished = true;
      break;
    }

    // Prevent the agent from navigating out of our local test target.
    if (new URL(page.url()).origin !== new URL(BASE_URL).origin) {
      throw new Error('Agent left the approved origin');
    }
    await applyAction(page, action);
  }

  if (!agentFinished) throw new Error(`Agent exceeded ${MAX_STEPS} steps`);

  // Here is the actual test oracle, independent of the model's opinion.
  const cart = await page.locator('#cart').textContent();
  const discount = await page.locator('#discount').textContent();
  const receipt = await page.locator('#receipt').textContent();

  if (cart?.trim() !== 'Cart: 2 items') throw new Error(`Wrong cart: ${cart}`);
  if (discount?.trim() !== 'Discount: $5') {
    throw new Error(`Wrong discount: ${discount}`);
  }
  if (receipt?.trim() !== 'Order total: $10') {
    throw new Error(`Wrong receipt: ${receipt}`);
  }

  console.log('PASS: correct cart, discount, and total');
} catch (error) {
  failure = error;
  console.error('FAIL:', error.message);
  await page.screenshot({ path: 'failure.png', fullPage: true }).catch(() => {});
} finally {
  await fs.writeFile('agent-history.json', JSON.stringify(history, null, 2));
  await context.tracing.stop({ path: 'trace.zip' }).catch(() => {});
  await browser.close();
}

if (failure) process.exitCode = 1;

Run it in a second terminal:

node agent.mjs

On a successful run, the model will choose actions corresponding to adding a notebook, adding a pen, entering SAVE5, applying it, checking out, and finishing. The exact order and number of model calls can vary. At the end, the script checks three concrete facts: two cart items, a $5 discount, and a $10 receipt.

If it fails, you'll still get agent-history.json and trace.zip; failures also attempt a screenshot. Open the trace with:

npx playwright show-trace trace.zip

A note about the API: response_format: { type: 'json_object' } asks the model for JSON, not for a guaranteed valid browser command. That's why validateAction() still exists. The code uses the raw Playwright library rather than Playwright Test, so the trace contains browser activity but not Playwright Test's assertion metadata.

The architecture is more important than the model

Notice how little authority we gave the LLM.

It can click a named button or fill a named textbox. That's it. The model can't execute JavaScript, visit another website, delete files, call internal APIs, or decide that a failing assertion should be ignored.

It also doesn't receive the entire DOM. We use Playwright's locator.ariaSnapshot() to send a readable view of the accessible interface. This API is available in Playwright 1.49 and later. It's often enough for buttons, links, forms, and page text. It isn't magic: poorly labeled controls and visual-only interfaces remain difficult.

Three design choices are doing most of the work:

One action at a time. If the model proposes a 12-step plan in advance, step four may be wrong because step two opened a dialog. Observe again after every action.

A fixed budget. Autonomous doesn't mean infinite. Twelve actions is plenty for our checkout. For a production app, give each scenario a time limit, a step limit, and a clear failure state.

An independent oracle. We don't ask, "Did you test checkout successfully?" and accept "Yes". We inspect application state and compare it with expected values. In a real system you might check an order API, a database record, or a known fixture instead of trusting the page alone.

That last point is where a lot of impressive autonomous testing demos quietly stop being convincing.

Let's break it deliberately

Change this line in index.html:

const total = notebookCount * 12 + penCount * 3 - discount;

to:

const total = notebookCount * 12 + penCount * 3;

Now the checkout ignores the discount. The agent may complete all the clicks and even report that it's finished. The script should still fail because the receipt says $15, not $10.

That's what you want from a test: a failure that doesn't depend on how confident the agent sounds.

Put the original line back before continuing.

Why this gets expensive faster than you expect

For our small page, a model call after every action is manageable. For a typical enterprise flow, the numbers look different.

Say you have a signup journey with 25 browser actions. At one model call per action, that's 25 model requests for one attempt. Run it across four browsers and you've got up to 100 requests, before retries. Add 200 scenarios and you start caring about latency, context size, model availability, and cost.

And the more interesting the app, the worse the edge cases become:

A button opens a new tab, but the agent is still observing the old one.
A checkout step runs inside an iframe.
A file upload requires interacting with a hidden input.
An email verification step depends on another system.
A model chooses a plausible but wrong "Continue" button.
A slow UI update makes a successful action look like a failure, so the agent clicks twice.

These aren't criticisms of Playwright. Playwright gives you APIs for popups, frames, uploads, and waiting. But now your agent needs policies for all of them, plus recovery logic, reporting, retries, and a way for a human to correct its decisions.

You can build it. The question is whether maintaining that platform is the best use of your team's time.

What I'd change before using this in CI

I wouldn't point the script above at a production account. It's a learning example with intentionally narrow permissions, not a hardened security boundary. Its origin check happens between steps; real deployments should also restrict network access and use isolated test accounts.

Before a serious rollout, I'd make a few changes:

Use deterministic assertions and fixtures. Keep the LLM in charge of exploration, not the definition of success.
Limit the accessible environment. Run in disposable browser contexts with test data, restricted credentials, and network controls. An application's page content can contain instructions aimed at the model.
Treat action history as data, not proof. Save traces and screenshots, but redact secrets before sending page content to an external model or storing artifacts.
Handle tabs, frames, and downloads explicitly. Don't assume page is always the right page or that accessibility snapshots expose every useful target.
Keep successful paths for review. Once an agent discovers a stable checkout flow, consider converting it into an ordinary Playwright test. There is no prize for paying an LLM to rediscover the same five clicks every morning.

The last one is especially useful. Autonomous discovery and deterministic regression testing solve slightly different problems. You don't have to choose only one.

When I'd use something other than a homemade agent

If you're learning how browser agents work, this project is worth doing. You'll understand the observation/action loop, the failure modes, and why good test oracles matter.

If you're responsible for a team's regression suite, the calculation changes.

There are platforms that already package autonomous browser exploration with test creation and managed execution. Endtest is one example. Its Endtest Bot is designed to explore a web application and generate test scenarios, while the broader platform handles running and managing web tests. That's much closer to the outcome most QA teams want than maintaining their own model prompts and browser orchestration code.

I'd evaluate a product like that on the difficult flows, not on a login demo: iframes, multiple tabs, uploads, conditional paths, and what happens when an element changes. Also ask to inspect and edit the generated steps. If you can't understand what an autonomous test did, you have a maintenance problem disguised as an AI feature.

A custom Playwright agent can still be the better choice if you need full control over models, infrastructure, or unusual browser behavior. You're trading platform fees for engineering time. Neither option is free.

The part worth keeping

You can make Playwright autonomous with surprisingly little code. The browser library was never the hard part.

The hard part is deciding what the agent is allowed to do, how to tell whether it did the right thing, and what evidence you have when it didn't.

Start with one business-critical flow. Make the failure deterministic. Then add autonomy where it saves you work, rather than where it makes for the flashiest demo.

That's a much better test automation strategy than building a robot that can click anything and trusting it when it says everything is fine.

References: Playwright accessibility snapshots, Playwright locator API, Playwright tracing, Playwright browser contexts.

TestDino Is Useless If You Already Use Playwright

Simon Gerber — Thu, 08 Oct 2026 21:41:17 +0000

I keep seeing TestDino pop up around Playwright.

The pitch sounds reasonable.

You already have Playwright tests.

TestDino sits on top of them and gives you:

centralized reporting
flaky-test detection
failure grouping
AI analysis
historical trends
screenshots and traces
PR integrations
an MCP server

Basically:

Playwright
    ↓
TestDino
    ↓
More dashboards

I like testing tools.

I like Playwright.

I particularly like tools that remove annoying QA work.

So I went into this expecting there to be something interesting here.

And there is.

Just not enough for me to understand why most Playwright teams would pay for it.

The more I looked, the more TestDino started to feel like SaaS rent for looking at your own Playwright results.

First, what does TestDino actually do?

This is important because the website throws around phrases like:

AI-native test observability

and:

cloud companion for Playwright

Underneath that language, the product is much easier to explain.

You install their Playwright reporter:

reporter: [
  ['@testdino/playwright', {
    token: process.env.TESTDINO_TOKEN
  }]
]

Your Playwright runs get uploaded to TestDino.

TestDino then stores, organizes, analyzes, and displays the results.

Its own documentation describes it as a platform for Playwright test reporting, failure analysis, flaky-test detection, integrations, analytics, and historical results.

That's useful.

But let's be clear about what you're buying.

You're not replacing Playwright.

You're not getting a new browser automation engine.

You're not getting some magical testing infrastructure that suddenly executes things Playwright couldn't execute.

You're mostly adding a SaaS layer around Playwright.

And that's where I start having questions.

Playwright already gives me a ridiculous amount of debugging information

Playwright isn't exactly starving us for evidence.

A failing Playwright test can already give you:

screenshots
videos
traces
DOM snapshots
network activity
console output
source information
action timelines
retries

TestDino's own trace documentation explains how its trace viewer exposes the actions, DOM, network, console, source, and timing information captured by Playwright traces.

That's nice.

But I kept having this thought while looking at the product:

How much am I paying TestDino to organize information that Playwright already generated?

The answer isn't "nothing."

Centralized history is genuinely useful.

Cross-run flake detection is useful.

Grouping failures across hundreds of tests is useful.

But this is a narrower value proposition than the marketing makes it sound.

If your Playwright suite is 150 tests and your team can already diagnose failures from CI artifacts, TestDino feels like bringing Salesforce into a lemonade stand.

And it only works with Playwright

This one is pretty brutal.

TestDino is explicitly built around Playwright.

If your company has:

Playwright
+
Selenium
+
Cypress
+
mobile automation
+
API suites

you don't really have a central testing platform.

You have a Playwright dashboard.

Everything else still lives somewhere else.

That may be completely fine for a young company standardized on Playwright.

It's much less appealing for a mature QA organization.

The weird part is that the more sophisticated your testing organization becomes, the more likely you are to have multiple types of automation.

Which makes TestDino's narrowness increasingly noticeable.

The pricing looks reasonable until you do the math

At first glance, pricing actually looks pretty good.

Current published pricing is approximately:

Free     $0
Pro      $49/month
Team     $99/month

with lower prices if billed annually.

Free gets 5,000 executions.

Pro gets 10,000.

Team gets 40,000.

Not crazy.

Then you read the important part.

TestDino says:

One test across 3 browsers = 3 results.

That's where this gets interesting.

Imagine a fairly modest suite:

500 tests
×
3 browsers
×
30 daily runs

That's:

45,000 test results/month

Congratulations.

You've exceeded the published Team allowance.

With 500 tests.

Running once per day.

That's not remotely unusual for a serious automation team.

And if you're running the suite:

per PR
+
nightly
+
staging
+
production

those numbers can climb very quickly.

Usage-based pricing isn't inherently bad.

But TestDino's value increases when you run lots of tests, while the pricing also increases with how many tests you run.

That's an awkward incentive.

The user limits are weird too

The Free plan allows one user.

Pro allows three.

Team allows 30.

So suppose you're a tiny QA team with:

2 QA engineers
2 developers

You may have very low execution volume.

Doesn't matter.

If all four need proper access, you're already outside Pro's three-user allowance.

That jumps you toward Team.

For what is still fundamentally a Playwright reporting dashboard.

This is the kind of SaaS packaging that makes me tired.

Then there's artifact retention

This part bothers me more.

According to TestDino's pricing table, Pro and Team have:

21 days artifact retention

while analytics may be retained much longer.

Think about what TestDino is selling.

Historical context.

Debugging.

Failure analysis.

Understanding recurring issues.

Now imagine:

"Didn't we have this exact checkout failure last month?"

Great question.

The analytics may remember it.

But the screenshot, video, or artifact you actually want may already be gone.

Three weeks feels short for a product whose entire purpose is to help me understand historical test failures.

I can obviously keep my artifacts elsewhere.

But now we're back to:

Why am I paying another service for this?

The pricing page itself confused me

Here's something I didn't expect.

On the pricing page, the Pro plan says it includes:

AI classification
AI test analysis

as additions over Free.

But lower on the same page, the FAQ says every plan includes AI failure classification.

Maybe there's a perfectly reasonable distinction between exactly which AI features are included.

But I shouldn't have to reverse engineer the pricing page to understand it.

This is a test analytics company.

I expect the test results to be clear.

I'd also appreciate the pricing being clear.

Uploading all this debugging data is not a trivial decision

Remember what we're sending.

Potentially:

screenshots
videos
traces
logs
DOM state
network information
test names
branch information
commit information
failure data

TestDino advertises ISO 27001, SOC 2 Type II, GDPR support, and EU hosting options, which is reassuring.

But its pricing page lists data redaction under Enterprise features.

That makes me uncomfortable.

For a reporting product ingesting screenshots, traces, logs, and potentially application data, redaction isn't some exotic Fortune 100 feature.

It can be basic hygiene.

Maybe your test environment contains nothing sensitive.

Great.

But I'd check that assumption before uploading every Playwright artifact your company produces.

The AI part doesn't move me much

TestDino groups failures and categorizes issues.

The product talks about classifications such as timing, environment, network, assertion, bugs, UI changes, and flaky behavior.

That sounds useful.

But I don't think:

Timeout → probably timing issue

is enough to justify another platform.

What I'd want to know is:

Was the classification correct?

How often is it wrong?

Can it distinguish a flaky test from a real race condition?

Can it identify the commit that caused the regression?

Can it propose a fix that actually preserves test intent?

Does it save enough investigation time to justify uploading and storing everything elsewhere?

Those are much harder questions.

A dashboard saying:

NETWORK

with a nice badge doesn't automatically mean the debugging problem has been solved.

The MCP server is probably the most interesting part

This is where I finally found something I genuinely like.

TestDino exposes test information to AI agents over MCP.

That means your coding agent can ask things like:

What failed in the latest run?

Which tests are flaky?

Show the historical failures for this test.

What changed around this failure?

without you copying logs into Claude, Cursor, or another tool manually.

That's actually interesting.

I can see real value there.

But again:

I'm uploading my test data into a separate SaaS product so an AI can query it.

That's useful if TestDino becomes the place where my testing history lives.

It's much less compelling if I already have that information structured elsewhere.

MCP doesn't magically make mediocre data valuable.

It just makes it easier for an agent to access it.

Public reviews are surprisingly positive

Here's where I have to be fair.

People who publicly review TestDino mostly seem to like it.

Reviewers frequently praise:

easy setup
clean dashboards
flaky-test visibility
centralized reporting
support

So maybe I'm the grumpy one here.

But there are hints of the same concerns I had.

Some public feedback has called for deeper debugging, more customization, stronger documentation for advanced workflows, better linking between manual and automated tests, and more flexible repository handling.

Those are not catastrophic problems.

They just reinforce my overall impression:

TestDino looks polished at the surface.

I'm less convinced about depth.

An earlier hands-on review found exactly that problem

An independent review from 2025 had a similar experience.

The reviewer liked the polish, ease of navigation, screenshots, videos, and support.

But they also complained that dashboard metrics couldn't be drilled into sufficiently, AI insights weren't linked cleanly to the underlying test run or code, and PR information wasn't well connected to test runs.

To TestDino's credit, its current documentation describes much better PR-to-run navigation, timelines, code diffs, failure clusters, logs, screenshots, and traces.

So some of those problems appear to have been addressed since that review.

That's good.

It also tells you the product is still evolving quickly.

Whether you interpret that as:

Great, they're shipping fast.

or:

I'm paying to participate in the maturation process.

is up to you.

Then I read the Terms of Service

And this may be my favorite part.

TestDino's current Terms say you may not use the platform to compile a:

comparison
evaluation
benchmark
competitive assessment

without TestDino's prior written consent.

They say consent will not be unreasonably withheld for a good-faith factual comparison if the party identifies itself before requesting access.

Read that again.

A testing tool.

A product literally designed around measuring whether software behaves correctly.

Doesn't want you evaluating the product through platform access without asking permission first.

That's certainly one approach.

The terms also define competing products and people connected to them as "Restricted Parties" and say they aren't eligible to access the platform without a written agreement.

I'm not a lawyer, so I'm not going to speculate about how broadly any of that would ultimately apply.

But as a buyer?

I don't love it.

If I'm selecting engineering infrastructure, independent benchmarking is exactly what I want people doing.

I want:

test it
break it
compare it
measure it
publish the results

If a vendor wants people to request permission before using the product for an evaluation or benchmark, that's a yellow flag for me all by itself.

Ironically, this makes an "honest hands-on review" harder

This is also why I wouldn't pretend I spent a week hammering the private product and publish invented observations.

TestDino's publicly available showcase gives a detailed look at streaming results, run summaries, PR coverage, flaky detection, error grouping, test management, test explorer, tags, and imports/exports.

Its documentation is extensive.

There are third-party tutorials and reviews.

But there's a difference between:

I inspected the product's public surfaces, documentation, pricing, demos, independent reviews, and technical architecture.

and:

I personally ran 10,000 tests through it.

I'm not going to pretend those are the same thing.

Any review that does should immediately lose credibility.

So who actually needs TestDino?

I can think of one very specific team.

You have:

a large Playwright-only test suite

+

lots of CI runs

+

multiple engineers investigating failures

+

persistent flaky-test problems

+

poor historical reporting

+

a desire to expose test history to AI agents

Then yes.

I can understand TestDino.

You're buying organization.

You're buying history.

You're buying triage.

You're buying convenience.

Fine.

But if you're a small or medium Playwright team already keeping useful CI artifacts?

I'd have a hard time recommending it.

You already have a very capable testing framework.

You already have traces.

You already have screenshots.

You already have videos.

You already have CI history.

Now you're adding:

another account
another reporter
another dashboard
another API key
another SaaS bill
another usage limit
another retention policy
another copy of your test data

to make those things look nicer.

My biggest problem with TestDino

It's not that TestDino appears completely useless.

That would actually make the review easier.

There are useful ideas here.

The MCP integration is interesting.

Cross-run flaky-test analytics are valuable.

Centralized evidence can save time.

The product looks polished.

The problem is that the value proposition feels incredibly thin.

It feels like someone looked at Playwright's reporting output and said:

What if this was a startup?

And now we have pricing tiers, AI classification, usage quotas, retention windows, team-member limits, enterprise redaction, and a Terms of Service section about benchmarking.

I wanted to find the killer feature.

The thing where I went:

Oh. That's why I'd install this.

I didn't find it.

My verdict

For a large Playwright-only organization drowning in CI results:

Maybe.

For everyone else:

I'd skip it.

My rough score:

UI and polish:        8/10
Setup:                8/10
Reporting:            7/10
AI differentiation:   4/10
Framework coverage:   2/10
Pricing model:        5/10
Artifact retention:   4/10
Need to exist:        3/10

The harshest thing I can say about TestDino isn't that it looks badly built.

It doesn't.

It's that after going through everything it offers, I kept coming back to the same question:

Why isn't this just part of my existing Playwright and CI workflow?

And when a developer tool makes me ask that question this many times, adding another SaaS subscription is probably not the answer.

The Real Test of Test Automation Is What Happens After Something Changes

Simon Gerber — Wed, 19 Aug 2026 22:03:29 +0000

Test automation demos tend to begin at the nicest possible moment.

The application is stable.

The environment is clean.

The test data is ready.

Someone creates a test, clicks Run, and everything turns green.

Useful? Sure.

But that isn't the difficult part.

The difficult part starts after the system changes.

A browser gets updated.

A prompt is modified.

An AI agent gets access to a new tool.

A developer changes the upload flow.

A selector disappears.

A test gets "healed."

Now the real questions begin:

Which tests should run?
Who should review what changed?
Is the failure caused by the product or the environment?
Did the AI fix the test correctly, or merely make it pass?
Can somebody reproduce what happened tomorrow?

That's where test automation either becomes infrastructure people trust or another dashboard people learn to ignore.

Changes should trigger evidence, not hope

This is especially important for AI-powered products.

Traditional software releases usually have a visible code change.

AI systems can change behavior because of code, but also because someone changed:

a prompt,
a model,
a system instruction,
a retrieval source,
a tool definition,
a tool policy,
an orchestration rule.

Any of those can alter what users experience.

Yet many teams still treat those changes as if they were configuration tweaks rather than releases.

That's risky.

If changing a prompt can alter a customer-facing workflow, then a prompt change should be capable of triggering the same kind of regression checks as a code change.

One practical example is triggering Endtest runs from an AI release pipeline after prompt, model, or tool-policy changes.

The broader idea matters more than the specific tool:

Attach regression testing to the thing that can change behavior.

If a model switch can change checkout assistance, run checkout tests.

If a prompt change can alter how an agent uses a CRM, run those workflows.

If a tool-policy update can change which actions are allowed, verify the important permission boundaries.

Otherwise you're deploying behavior changes and hoping your existing CI pipeline notices.

It probably won't.

The best automation removes fake complexity

A lot of flaky automation comes from testing the wrong layer.

File uploads are a good example.

A human clicks an upload control and sees an operating-system file picker.

So the first instinct is often to automate the picker.

But your application doesn't own that dialog.

The browser doesn't fully own it either.

The operating system does.

Trying to automate it as if it were part of your web application introduces a dependency that usually adds no meaningful product coverage.

In Playwright, a better pattern is to work with the browser's upload primitives directly. This guide on testing browser file uploads in Playwright without native dialog flakiness shows the basic principle.

The interesting lesson is broader than Playwright:

Don't automate an implementation detail simply because a user can see it.

Test the behavior your product is responsible for.

Did the file get selected?

Did the application validate it?

Was it uploaded?

Did the server accept it?

Did the UI show the correct result?

Those are product behaviors.

Whether macOS happened to render a particular file dialog is usually not.

This sounds obvious when written down.

It is surprisingly easy to forget when building a test suite.

AI-generated tests still need a human operating model

AI can now generate tests, update selectors, suggest fixes, and sometimes recover from failures.

That can remove a lot of repetitive work.

But it also creates a new question:

Who owns the change?

Imagine an AI system modifies twelve tests after a frontend refactor.

Maybe all twelve changes are correct.

Maybe eleven are correct and one quietly changes the intent of an assertion.

That one matters.

The more autonomy a testing platform has, the more important review and approval become.

A useful evaluation framework is to look at AI testing platforms through test authoring review, approval handoffs, and ongoing maintenance.

I would ask questions such as:

Can I see exactly what the AI changed?
Can I approve or reject those changes?
Is there an audit trail?
Are generated steps editable?
Can different team members review tests?
Can we distinguish an application change from a test change?
What happens when the AI is uncertain?

This is not bureaucracy for its own sake.

It's the same reason we review code.

Automation becomes more valuable as more people trust it.

And trust usually requires visibility.

Sometimes the test is right and the browser changed

One of the fastest ways to waste an afternoon is to assume every browser test failure is an application regression.

Browsers are software too.

They ship changes.

Rendering changes.

Security behavior changes.

Timing changes.

Driver behavior changes.

Deprecated features disappear.

A test can fail on Tuesday even though your application hasn't changed since Friday.

When that happens, the worst debugging sequence is:

Assume the app broke.
Rewrite the test.
Add waits.
Add retries.
Discover three hours later that Chrome updated.

A more disciplined approach is to first establish what changed in the execution environment.

This guide to debugging browser test failures caused by browser engine drift rather than app regressions covers the kinds of signals worth checking.

At minimum, capture:

browser name,
browser version,
operating system,
automation library version,
relevant driver/runtime version,
screenshots,
console errors,
network failures.

Then compare the failing run with the last known good run.

If the application commit is identical but the browser changed, that's a useful clue.

It doesn't prove the browser is responsible.

But it changes where you look first.

That can save a lot of expensive guessing.

Benchmarks should include failure, not just success

Tool evaluations often suffer from the same problem as demos.

Everything is tested under ideal conditions.

A team creates five workflows.

All five pass.

The tool gets a high score.

That tells you surprisingly little.

If I were comparing AI-native browser testing platforms, I would deliberately introduce failure.

Change an element label.

Add latency.

Move a button.

Modify test data.

Break an iframe locator.

Return an unexpected API response.

Change the browser version.

Then measure what happens.

A reproducible framework like this benchmark plan for AI-native browser testing platforms is much closer to how these tools should be compared.

The useful metrics aren't just "time to create first test."

I'd care about:

setup time,
repeatability,
flake rate,
recovery behavior,
evidence quality,
time to diagnose failure,
human intervention required,
maintenance after application changes.

Those numbers tell you what living with the tool might actually feel like.

And that matters because you're not buying the demo.

You're buying the next two years.

The hidden KPI is time-to-understanding

Most QA dashboards measure execution.

Number of tests.

Pass rate.

Failure rate.

Duration.

Coverage.

Those are useful.

But I think one of the most valuable metrics is rarely shown:

How long does it take a human to understand a failed test?

Imagine two systems.

System A has a 98.5% pass rate.

System B has a 99.3% pass rate.

At first glance, System B looks better.

But suppose a failure in System A takes three minutes to diagnose because it includes the screenshot, page state, browser version, logs, and exact test change.

A failure in System B takes 25 minutes because somebody has to rerun it twice and reconstruct what happened.

Which one is actually cheaper?

I'd probably take System A.

Reliability matters.

But diagnosability compounds.

Every unclear failure creates another interruption.

Every interruption pulls somebody away from product work.

Over months, those small interruptions become a meaningful engineering cost.

Treat testing as part of the change system

The strongest testing setups I've seen tend to share a simple property:

Tests are connected to change.

Code changes trigger the relevant tests.

Browser upgrades trigger compatibility checks.

AI prompt and model changes trigger workflow regression.

AI-generated test modifications go through review.

Failures preserve enough evidence to explain themselves.

Tool evaluations intentionally introduce breakage rather than only measuring happy paths.

None of this is particularly glamorous.

It doesn't make for a 30-second product demo.

But it's the difference between having automated tests and having an automated quality system.

Creating the test is step one.

What happens after something changes is the part that determines whether the test is still useful six months later.

The Hardest E2E Tests Live at the Boundaries

Simon Gerber — Sat, 08 Aug 2026 21:28:36 +0000

The easiest automated test in the world looks something like this:

Open page.
Click button.
Assert text.

Unfortunately, that’s not where expensive bugs usually live.

They live at boundaries.

The moment your application has to interact with another system, another browser context, another identity provider, another clock, or another human team, automation gets much more interesting.

Authentication Is Basically a Collection of Edge Cases

“Test login” sounds simple.

Then you ship a real enterprise application.

Now you have:

SSO
MFA
OTP codes
expiring sessions
idle logout
refresh tokens
re-authentication
multiple tabs
redirects to identity providers

Your clean little login test becomes a distributed systems test wearing a browser costume.

Session timeout is a perfect example.

You could literally wait 30 minutes during every test run.

Please don’t.

The better approaches in testing session timeout, idle logout, and re-authentication are about controlling the state instead of making CI experience time at human speed.

The same goes for evaluating a platform that needs to handle MFA, SSO, OTP, and expiring sessions.

These aren’t exotic requirements anymore.

They’re normal SaaS requirements.

Payments Are Even Worse

Checkout combines nearly every annoying browser automation problem.

You have:

payment iframes
cross-origin content
redirects
3DS authentication
external payment providers
popups
asynchronous callbacks

And sometimes the browser leaves your domain entirely.

That makes comparisons like Endtest vs Playwright for payment iframes, 3DS, and redirect checkout more meaningful than comparing how each tool handles a simple button click.

The difficult part of automation is rarely syntax.

The difficult part is surviving reality.

AI-Driven Interfaces Add Another Boundary

Now add AI.

Traditional applications are approximately deterministic.

Input A usually produces Output B.

AI applications introduce probability.

The interface may be deterministic while the output isn’t.

So your test strategy has to change.

An assertion like:

response === expected_response

may become almost useless.

Instead you care about:

intent
required facts
prohibited output
schema
confidence
behavioral drift

That’s why monitoring AI output drift in production test gates is becoming an interesting problem.

The product might not “break” in the traditional sense.

It may slowly become worse.

That’s harder to catch.

Tools Have Different Costs at These Boundaries

This is also where automation tool comparisons get more nuanced.

If your application changes aggressively and AI is involved in the UI itself, the question isn’t:

“Which testing tool has more features?”

It’s:

Which tool creates less operational overhead for the way my team actually ships software?

That’s the interesting part of comparisons such as Endtest vs Functionize for AI-driven UI changes.

Two products can theoretically automate the same workflow while creating very different amounts of maintenance.

And maintenance compounds.

Saving ten minutes once is irrelevant.

Saving ten minutes on something your team does 500 times is a product feature.

People Are Part of the System Too

There’s one boundary engineers sometimes forget:

Humans.

You can have an excellent test suite and still ship slowly because nobody knows who investigates failures.

Or the offshore QA team discovers problems eight hours after engineering finishes for the day.

Or the test team can execute a release but doesn’t understand what changed.

That’s why evaluating offshore QA partners based on overlap hours and release handoffs isn’t simply procurement trivia.

Coordination latency is part of your QA system.

A bug found at 9:05 AM by someone sitting beside the developer is very different from the same bug reported at 1:00 AM through a spreadsheet.

This Is Where Automation ROI Is Won

Basic tests are important.

You should absolutely test that users can log in and buttons work.

But once those tests exist, I’d spend disproportionate effort on the boundaries:

Authentication.

Payments.

External services.

Session state.

AI behavior.

Redirects.

Asynchronous state.

Human handoffs.

Those are the places where assumptions collide.

And they’re the places where a good automated test can save you from the kind of bug that doesn’t produce a cute little red unit test.

It produces a Slack message from a customer.

Those are considerably more expensive.

AI Features Need a Different Kind of Test Suite

Simon Gerber — Fri, 07 Aug 2026 22:09:02 +0000

A lot of teams are adding AI to existing products the way SaaS companies added chat widgets ten years ago.

There’s the normal application.

Then there’s the AI box.

You type something into it.

Magic happens.

Ship it.

The problem is that the AI box rarely stays isolated.

Soon it can fill forms.

Suggest values.

Change application state.

Create records.

Recommend actions.

Trigger workflows.

Hand something off to a human.

Now you don’t merely have an LLM feature.

You have an unpredictable participant inside your application.

And that changes what “testing the UI” means.

The dangerous failure is often a believable UI

Traditional UI testing is good at binary failures.

The button is missing.

The API returned 500.

The page didn’t load.

AI introduces something nastier:

the interface works perfectly while showing the wrong thing.

Imagine an AI assistant generating an insurance quote.

The page renders.

Every element is present.

The interaction succeeds.

But the model inferred the wrong coverage level and the frontend confidently displays it.

From the perspective of a classic browser test, everything worked.

From the perspective of the business, you may have a serious problem.

This is why I find the emerging category of tools for detecting hallucinated UI states in LLM-powered applications interesting.

AI testing has to care about the semantic relationship between what the model produced and what the application ultimately did.

Form assistants are a good example

AI-powered forms look deceptively easy to test.

Enter a prompt.

Watch fields populate.

Assert that the fields contain values.

But what happens if:

the model fills a disabled field?
it overwrites data the user manually entered?
server-side validation rejects its suggestion?
the AI changes a dependent dropdown but not the value below it?
the user corrects one field and the assistant “helpfully” changes it back?
the submission fails and the AI retries incorrectly?

Now you have three sources of truth:

the user,
the model,
the application.

That’s why testing form assistants requires considerably more than testing whether generated text appeared.

There’s a practical breakdown in how to test AI form assistants before they corrupt validation, autofill, or submission state.

And if you’re evaluating an automation platform specifically for these workflows, this guide to testing AI-generated form assistants and recovery paths with Endtest looks at the tooling side.

The interesting test cases live at the boundary between AI output and deterministic application rules.

That boundary deserves disproportionate attention.

Streaming responses break traditional waits

Chat interfaces introduce another common failure mode.

A normal browser test might wait for:

div.response

That works beautifully until responses stream token by token.

The element exists almost immediately.

The answer does not.

So the test reads half a sentence and proceeds.

You can increase the timeout, but now a fast response waits unnecessarily.

You can wait for network silence, except modern apps often have analytics, telemetry, or background requests that mean “network silence” never really happens.

You need an application-level definition of completion.

Maybe the Stop button disappears.

Maybe a status changes.

Maybe a streaming flag becomes false.

Whatever it is, it should correspond to something the user would recognize as “the answer is finished.”

There’s a good treatment of this problem in testing streaming AI responses without flaky timing assertions.

Again, the theme is the same:

waiting longer is not the same as waiting intelligently.

Handoffs are where AI becomes a workflow

The most interesting AI applications aren’t fully autonomous.

They hand control back and forth between humans and software.

AI prepares something.

A human approves it.

AI continues.

Someone escalates.

The workflow gets reassigned.

An audit event is recorded.

That creates a much richer test surface than a chat response.

You have to validate both the model behavior and the surrounding product workflow.

This review of testing AI agent handoff flows, approvals, and escalation UI with Endtest is a useful example of the kinds of scenarios worth exercising.

The key word there is workflow.

AI products still need boring deterministic software around the AI.

Permissions need to work.

Buttons need to work.

Approvals need to work.

Audit logs need to work.

The model doesn't get to excuse those failures.

“97% accurate” tells you almost nothing

AI vendors love accuracy numbers.

97.2%.

98.7%.

99%.

The problem is that a single percentage can hide almost everything that matters.

Accurate at what?

On whose dataset?

Under which prompt?

Which languages?

What happens on ambiguous inputs?

Was the evaluation performed once six months ago?

Are failures clustered around one particularly important use case?

Before treating an AI testing tool's accuracy score as a meaningful comparison, read something like what to check before trusting reported AI testing accuracy scores.

Averages are especially dangerous when the cost of individual errors is asymmetric.

Being wrong about the color of a generated icon is not equivalent to being wrong about whether a financial transaction succeeded.

AI testing needs observability

When a deterministic test fails, you usually ask:

What happened?

With AI systems, you also need:

Why did the system reach that result?

That may require:

prompt versions,
model versions,
retrieved context,
tool calls,
token streams,
traces,
screenshots,
browser state,
retries.

That’s why AI test observability is becoming its own category.

There’s a useful overview of evaluating AI test observability platforms for prompt replays, traces, and failure triage.

Without this context, reproducing an AI failure becomes forensic archaeology.

And eventually somebody asks about the data

Once you're saving prompts and replaying conversations, another question arrives:

What exactly are we storing?

AI test data can contain:

customer information,
internal documents,
production prompts,
retrieved content,
personal data,
secrets inadvertently entered by users.

At that point, synthetic data and masking stop being enterprise checkboxes and start becoming part of your testing architecture.

This market map of AI test data governance tools provides a good overview of that side of the ecosystem.

AI doesn’t eliminate deterministic testing

There’s a weird idea floating around that AI will eventually make normal test automation obsolete.

I suspect the opposite happens.

As applications become less deterministic, teams will want more deterministic evidence around them.

Did the correct field change?

Was the approval actually persisted?

Was the right user notified?

Did the application prevent an invalid submission?

Can we reproduce what happened?

AI can generate wildly different language while the product still has very specific rules.

Those rules are where your test suite earns its keep.

AI Regression Testing Needs Guardrails

Simon Gerber — Wed, 05 Aug 2026 19:10:35 +0000

Traditional browser testing assumes the product is mostly deterministic.

Click this button. See that page. Submit this form. Confirm the success message.

AI products break that comfortable model.

The wording may change while the result remains correct. A prompt update can improve one workflow and quietly damage another. Retrieval, caching, and tool use can introduce failure paths that are invisible in a simple UI assertion.

The answer is not to abandon regression testing.

It is to test the system at several layers and preserve enough evidence to understand why behavior changed.

Rollbacks are a feature, not an emergency procedure

Teams often test the new AI behavior and forget to test the rollback.

But AI features usually depend on more than one deployable component:

Prompt versions
Model configuration
Retrieval indexes
Embedding models
Caches
Tool definitions
Feature flags
UI behavior

Rolling back only the prompt may leave the rest of the system in a newer state.

This guide on testing LLM feature rollbacks across prompts, caches, and retrieval paths explains why rollback testing should be treated as a complete user flow.

A useful rollback test should confirm that:

The intended version is active
Cached responses do not preserve the broken behavior
Retrieval uses compatible data
Tool calls still match the older contract
The frontend presents the correct controls
Audit logs identify the active configuration

The worst time to discover a rollback is incomplete is during an incident.

Prompt changes are product changes

A one-line prompt adjustment can alter tone, structure, tool selection, refusal behavior, and the data included in a response.

That makes prompt changes deceptively large.

This article on testing AI-powered UI changes without turning every prompt tweak into a regression fire drill suggests separating what must remain exact from what can vary.

For example:

Exact requirements:

The correct tool is called
Restricted data is not shown
A confirmation step appears before a destructive action
The user reaches the expected workflow outcome

Flexible requirements:

Sentence wording
Response length within a reasonable range
Ordering of non-critical explanations
Stylistic differences

If everything is asserted exactly, the tests become noisy.

If nothing is asserted exactly, the tests become meaningless.

Multistep agents need traceability

A multistep browser agent may interpret a request, inspect the page, select an element, fill a form, call a tool, and verify the result.

A final “passed” status hides all of those decisions.

This guide on evaluating AI test agents for multistep browser flows without sacrificing debuggability makes a crucial point: every important decision should leave evidence.

At minimum, I would want to see:

The user instruction
The interpreted goal
Each selected action
The locator or element evidence
Tool inputs and outputs
Retries
Confidence or ambiguity
The final assertion

Without that trace, a successful run may still be untrustworthy.

The agent could have completed the wrong workflow correctly.

The market is splitting into categories

“AI testing platform” is becoming too broad to be useful.

Some products focus on generating tests. Others focus on evaluating model output, monitoring production behavior, testing security, or maintaining browser automation.

This market map of AI regression-testing platforms is helpful because it separates categories by the problem being solved.

Before comparing vendors, define the job:

Do you need browser workflow coverage?
Prompt regression evaluation?
Tool-use validation?
Production monitoring?
Security testing?
Human review and governance?
Cross-browser execution?

A product can be excellent in one category and still be the wrong choice for another.

Admin consoles deserve first-class tests

AI products increasingly include internal consoles for traces, overrides, approvals, and audit history.

These interfaces are not secondary.

They are how operators understand and control the system when automation behaves badly.

This practical review of using Endtest to validate AI-agent admin consoles, trace views, and override panels highlights workflows that are easy to overlook:

Filtering traces
Inspecting tool calls
Applying human overrides
Replaying failed runs
Confirming audit entries
Restricting actions by role

A product that can perform powerful autonomous actions needs equally reliable controls.

Safety tests must include the browser flow

Prompt injection and data leakage are often treated as model-evaluation problems.

But the final risk appears in the product workflow.

A copilot may receive malicious content from a document, call an internal tool, and display restricted information in the browser.

This guide on testing AI copilots for data leakage, prompt injection, and unsafe tool use describes why the test must cover both model behavior and application controls.

Useful scenarios include:

A document instructs the agent to ignore policy
A user asks for data outside their permissions
The model attempts a destructive tool call
Sensitive values appear in logs
A confirmation is bypassed
The UI exposes hidden system instructions

The browser is where permissions, confirmations, and evidence meet.

Keep the result editable and reviewable

At Endtest, AI Assertions can evaluate outcomes that are difficult to express as exact strings, while the AI Test Creation Agent can accelerate the creation of browser workflows. But the resulting tests still need to be understandable and editable by people.

That is the guardrail I keep coming back to.

AI can help create tests.

AI can help analyze failures.

AI can even suggest repairs.

But a human team should still be able to see what changed, why it changed, and how to reverse it.

The goal is not autonomous testing at any cost.

The goal is faster confidence without losing control.

Do Not Hire an AI Test Agent From Its Demo

Simon Gerber — Tue, 04 Aug 2026 22:22:08 +0000

AI testing demos are almost unfairly persuasive.

You describe a workflow in plain English. The agent opens a browser, finds the correct elements, completes the flow, and produces a passing test.

Five minutes later, it feels as if test maintenance has been solved.

The problem is that a demo shows the happy path for the agent, not the operating model for your team.

The real evaluation begins when the application changes, the locator is ambiguous, the environment is partially broken, or the agent makes the wrong decision with high confidence.

Evaluate the agent like a junior operator

I would not give a new employee broad production access simply because they completed one task successfully in an interview.

I would first want to know:

How they explain their decisions
What they do when information is missing
Whether they escalate uncertainty
How easily their work can be reviewed
Whether mistakes can be reversed

An AI agent deserves the same scrutiny.

This guide on evaluating AI test agents without losing browser-test debug visibility focuses on an essential requirement: automation should become easier to maintain, not harder to inspect.

An agent that produces a passing result without showing how it arrived there creates a trust problem.

Do not test it only on your cleanest workflow

Vendor demos usually use predictable applications:

Stable IDs
Simple forms
Obvious button labels
No iframes
No duplicate elements
No delayed events
No permission prompts
No uncertain outcomes

Your evaluation should do the opposite.

Give the agent a workflow where two elements have similar labels. Change a piece of copy after the test has been generated. Remove an element. Slow down an API response. Cause the browser to crash. Return an unexpected validation message.

The article on evaluating browser-flow agents without trusting the demo offers a useful principle: measure recovery behavior, not just task completion.

When the agent cannot confidently determine the right action, it should say so.

A clarifying question is better than an impressive-looking mistake.

Measure more than pass rate

Pass rate is a poor standalone metric for AI-generated or AI-maintained tests.

A test can pass because the product works. It can also pass because the agent selected the wrong element, skipped a failed assertion, modified the workflow, or healed the test into a different scenario.

This guide to measuring AI test runs before trusting their pass rate suggests looking at the decisions behind the result.

Useful metrics include:

Percentage of runs requiring agent intervention
Number of locators changed automatically
Frequency of ambiguous element matches
Human acceptance rate for suggested changes
False repair rate
Time saved during investigation
Number of silently changed test steps

I would also measure how frequently the agent requests clarification. A low number is not necessarily good. It may mean the system is guessing.

Procurement questions are product questions

AI testing purchases often begin as feature comparisons and end as security reviews.

That transition should happen earlier.

The AI testing procurement scorecard covers questions that become important once the proof of concept is over:

What application data is sent to the model?
Are screenshots or page sources retained?
Can sensitive values be masked?
Which employees can enable AI features?
Are model actions logged?
Can generated changes require approval?
Where is the data processed?
What happens when an AI feature is disabled?

An AI feature is not isolated from the rest of the product. Its governance model becomes part of your testing infrastructure.

Test the product on genuinely uncertain UI

AI-powered form assistants and guided checkout flows are good evaluation targets because they combine normal browser interaction with probabilistic behavior.

The assistant may change its wording while still giving the correct answer. It may suggest different products based on small changes in input. A rigid text assertion can fail even when the experience is acceptable.

This review of Endtest for AI-powered form assistants and guided checkout flows illustrates where AI Assertions can be more useful than exact string comparisons.

For example, instead of asserting that the assistant displays one precise sentence, you may want to validate that:

The response answers the customer’s question.
The recommendation respects the selected constraints.
The assistant does not claim the payment succeeded before confirmation.
The checkout flow still reaches the expected outcome.

The assertion needs flexibility, but the workflow still needs objective boundaries.

Account for the people who will own the tests

A technically impressive agent can still be the wrong product for a team.

This comparison of Endtest and Playwright for QA teams without a dedicated SDET highlights a factor that feature matrices often ignore: who will operate the system after implementation?

A developer-led team may prefer code, custom fixtures, and complete framework control.

A QA-led team may benefit more from readable steps, managed execution, built-in reporting, and an interface where tests can be updated without editing a framework.

Neither approach is universally correct.

The mistake is choosing an architecture for the team you hope to hire rather than the team you currently have.

AI should compress work, not hide it

At Endtest, features such as the AI Test Creation Agent are intended to accelerate test creation while keeping the resulting test understandable and editable.

That distinction matters.

The most useful AI systems do not replace the test with a mysterious generated artifact. They compress repetitive work while preserving review, ownership, and control.

Before adopting an AI test agent, ask one final question:

When it makes a mistake—and eventually it will—how quickly will my team understand what happened?

That answer matters far more than the demo.

Many “Flaky Tests” Are Really Frontend Architecture Problems

Simon Gerber — Mon, 03 Aug 2026 21:26:36 +0000

Browser tests are often blamed for exposing instability they did not create.

A test fails after a page loads. Someone adds a wait.

It fails again. Someone changes the locator.

It passes locally but fails in CI. Someone adds another retry.

Six months later, the test contains four waits, three fallback selectors, and a comment warning future developers not to touch anything.

At that point, the team says the test is flaky.

Sometimes it is.

But sometimes the test is accurately revealing that the frontend has become difficult to observe, predict, and operate.

Hydration mismatches are not harmless console noise

Server-rendered applications can display HTML before the client-side framework takes control.

That improves perceived performance, but it creates a contract: the client must hydrate a structure compatible with what the server rendered.

When the structures differ, frameworks may warn, discard parts of the DOM, recreate components, or attach events in unexpected ways.

A human user may not notice. The page still appears.

A browser test often notices immediately.

An element exists, disappears, and is recreated. A click lands during the transition. Text changes after the assertion begins. A locator points to a node that is no longer connected to the document.

The result looks like test instability, but the root cause is a rendering inconsistency.

The guide on debugging hydration mismatches before they break browser tests explains the common causes and the signals worth capturing.

A second useful perspective is this breakdown of hydration mismatch bugs that turn into false frontend failures.

The lesson is not to make tests tolerate every hydration problem.

It is to use the test failure as evidence that the interface has unstable intermediate states.

Dynamic interfaces need stable contracts

Modern frontends change frequently.

Design systems evolve. Components are restructured. Text is personalized. Forms add conditional fields. Lists reorder themselves based on live data.

A test that depends on incidental HTML structure will suffer.

For example, this locator may work today:

div:nth-child(3) > div:nth-child(2) > button

But it does not describe the user’s intent. It describes the current DOM arrangement.

A stronger contract might be a stable identifier, accessible name, field label, or explicit test attribute.

The best browser-testing tools cannot completely compensate for a frontend that offers no stable way to identify important elements.

Still, tools differ significantly in how they handle dynamic states, reusable components, debugging, waiting behavior, and test maintenance.

This comparison of browser testing tools for design systems and dynamic frontends outlines the capabilities that matter.

The correct choice is rarely the tool with the longest feature page.

It is the one whose assumptions match the way your interface actually changes.

Weekly UI changes alter the tool-selection equation

Many testing evaluations are based on a short proof of concept.

A team automates five stable flows. Everything passes. The tool appears successful.

But the real question is what happens after twelve weeks of product development.

How many tests break when components are refactored?

How quickly can someone understand and repair them?

Does changing a shared flow require editing one reusable component or twenty individual scripts?

Can product managers and QA engineers understand what the test does without reading framework code?

Teams shipping frequent UI changes should evaluate tools under change, not just under execution.

This guide to choosing an AI testing platform when the UI changes every week focuses on that difference.

A demo proves that a test can be created.

A change-heavy trial proves whether the testing approach can survive your product roadmap.

Dynamic forms are a useful stress test

Forms expose many of the weaknesses in browser automation.

They can contain:

Conditional sections
Client-side validation
Server-side validation
Asynchronous suggestions
Repeating fields
Custom dropdowns
Date pickers
File uploads
Auto-save behavior
Values populated from previous answers

A test may need to interact with elements that are visible only after a specific sequence of inputs. A small UI change can alter timing, structure, or labels.

This makes dynamic forms a useful comparison scenario for different automation approaches.

The analysis of Endtest versus Playwright for dynamic-form testing highlights the underlying tradeoff.

A code-first framework may offer deep flexibility and direct control. A higher-level platform may reduce the amount of framework code the team must create and maintain.

Neither advantage is imaginary.

The correct choice depends on who will own the tests, how customized the application is, how often flows change, and whether the team wants to maintain an automation framework as an internal software project.

Do not solve every frontend problem in the test layer

There is a temptation to make the test absorb all application uncertainty.

If the component loads unpredictably, add a longer wait.

If the DOM changes during hydration, retry the click.

If the list has no stable identifiers, find the third matching item.

These changes may make the test pass, but they can hide product weaknesses.

A better debugging sequence is:

Determine whether the application has reached a stable user-visible state.
Check for hydration, rendering, and console errors.
Confirm that interactive elements expose stable identities.
Inspect whether network and state transitions are observable.
Only then decide whether the test needs a different wait or locator.

Tests should handle legitimate asynchronous behavior.

They should not be forced to guess what the application is doing.

The test is part of the feedback loop

Browser automation is often discussed as a layer placed on top of the product.

In practice, it also provides feedback about the product’s architecture.

A UI that is easy to test usually has clearer states, better accessibility, more stable contracts, and more predictable transitions.

A UI that is extremely difficult to test may be difficult for users and developers too.

So the next time someone calls a browser test flaky, inspect the test.

But also inspect the frontend.

The test may be telling you something important.

The Hardest Browser Tests Are Systems Tests in Disguise

Simon Gerber — Wed, 29 Jul 2026 09:15:24 +0000

A browser test looks simple from the outside.

Open the page. Enter some data. Click a button. Confirm the result.

But many important user journeys extend far beyond the browser.

An SMS must arrive. An email provider must accept a message. A webhook must be processed. A file must be scanned. A background job must finish. An identity provider must redirect the user. A session must survive several transitions.

At that point, the browser is only one participant in a distributed system.

Treating the workflow as a sequence of clicks usually creates slow, mysterious tests.

OTP tests fail in the spaces between systems

Consider a one-time-password flow.

The browser requests a code. The application creates the code. A messaging provider accepts it. A webhook reports delivery. The message reaches the device or inbox. The user retrieves the code and enters it before it expires.

Any of those steps can fail independently.

A generic error such as “OTP field not found” tells you almost nothing.

This guide to browser-testing tools for webhook-driven SMS verification and time-sensitive OTP flows explains the capabilities needed to test the complete process.

The test should preserve timestamps and identifiers across the entire chain:

When the code was requested
The request or transaction ID
When the provider accepted the message
Whether a retry occurred
When the message became retrievable
Which code was entered
When the code expired
What the application returned

Without that information, every timeout looks the same.

Build the harness before expanding the test count

Teams sometimes create one successful email-verification test and immediately duplicate it across dozens of scenarios.

Then the inbox becomes unreliable, messages are retrieved out of order, test accounts collide, and the whole suite begins to fail intermittently.

A reliable harness needs isolation.

This guide to building an SMS and email verification harness with Twilio, Mailgun, and IMAP describes the plumbing that tends to be overlooked.

Each execution should use a unique address or phone-routing strategy. The retrieval logic should match the message using a correlation value, not merely “the latest email.” Messages should be cleaned up or expired predictably.

Polling should also have boundaries.

Checking every 100 milliseconds for five minutes does not make the system reliable. It creates traffic and hides a delivery problem behind a long timeout.

Upload testing begins before the upload

File-upload tests are often reduced to selecting a file and waiting for a success message.

Real upload flows contain many more states:

Drag enter
Drag leave
Drop
Local preview
Client-side validation
Chunking
Progress
Cancellation
Retry
Server processing
Final availability

This article on testing drag-and-drop uploads, previews, and retry states in real browsers shows why the interaction deserves more than one happy-path test.

A useful suite should include invalid files, oversized files, interrupted uploads, duplicate names, and failures after the bytes reach the server.

The system may display 100% progress while a background service is still scanning or transforming the file.

That is not necessarily a bug, but the test must understand which completion state matters to the user.

Validation and transport are different layers

When an upload fails, it is useful to know where it failed.

Client-side validation might reject the file before any request is sent. The browser may send the request and receive a server error. The server may accept the file but fail during processing.

This practical review of using Endtest for file uploads, drag-and-drop, and client-side validation separates those layers.

That separation makes failures actionable.

“Upload failed” is not a diagnosis.

“Client rejected the MIME type before sending a request” is.

Downloads need validation after the click

Download tests have a similar blind spot.

The browser clicks Export, a file appears, and the test passes.

But the file might be empty, malformed, truncated, encoded incorrectly, or filled with stale data.

A better workflow confirms:

The expected download started
The correct filename was used
The file completed
The file is not empty
The format can be parsed
Required columns or values exist
Sensitive data was not included unexpectedly

This guide to browser tests for upload, download, and file-validation flows covers the full lifecycle.

The key is to avoid making the browser test responsible for every byte-level assertion.

The browser can trigger the export and capture the artifact. A specialized validation step can inspect the resulting CSV, PDF, image, or archive.

Recovery paths deserve more attention than checkout completion

Most checkout automation focuses on the straightforward purchase:

Add an item.
Enter payment details.
Submit the order.
Confirm success.

Real users experience interruptions.

The session expires. The payment requires additional authentication. Inventory changes. The browser is refreshed. The user navigates backward. A provider returns an ambiguous result.

This guide to testing multi-step checkout recovery, session expiry, and reauthentication focuses on the paths that often produce the most expensive bugs.

Recovery tests should verify more than whether the user can continue.

They should check that:

The cart remains accurate
Discounts are preserved correctly
Payment is not submitted twice
Inventory is not reserved indefinitely
The user receives a clear status
An uncertain transaction can be reconciled

A recovery flow that creates duplicate orders is worse than a checkout flow that fails visibly.

Authentication recovery is different from first-time login

That imbalance is strange because recovery flows often combine email delivery, expiring tokens, password rules, session invalidation, and security controls.

This practical look at testing authentication recovery flows rather than only first-run logins highlights the scenarios teams frequently miss.

For example:

Can a reset link be used twice?
Does requesting a second link invalidate the first?
Are existing sessions revoked after a password change?
What happens when the link expires while the page is open?
Does the user return to the intended destination after authenticating?
Can the flow leak whether an account exists?

These tests protect both usability and security.

Load changes the frontend too

Load testing is often treated as a backend concern.

The API slows down, latency rises, and errors appear. But frontend behaviour also changes under load.

Skeleton states remain visible longer. Polling requests overlap. Buttons are clicked twice because the first response appears stalled. Notifications arrive out of order. A route transition begins before the previous request finishes.

This guide to load-testing tools for frontend-heavy applications with spiky traffic makes a good case for connecting load results with browser evidence.

A useful strategy is to run a small number of browser journeys while a separate load test creates pressure on the system.

The browser tests then answer a different question:

What does degraded performance actually feel like to a user?

An API returning in four seconds may be technically within a threshold while still creating a broken interface.

Build observability into the workflow

Complex browser tests become reliable when they stop depending exclusively on what is visible on the screen.

The browser remains important, but it should be connected to evidence from the surrounding system:

API responses
Webhooks
Message-provider events
Background-job status
Stored artifacts
Correlation IDs
Server timestamps
Processing states

This does not mean bypassing the user experience.

It means giving the test enough context to explain what happened.

A screenshot of a spinner is weak evidence.

A screenshot of the spinner combined with a job ID, queue status, provider response, and elapsed time is a diagnosis.

Test the boundaries, not just the clicks

The most valuable browser tests often live at system boundaries:

Browser to identity provider
Application to messaging provider
Upload client to storage service
Checkout to payment processor
Export request to background worker
Recovery link to session manager

Those boundaries are where ownership becomes unclear and failures become difficult to reproduce.

They are also where automated evidence provides the greatest leverage.

When a user journey crosses several systems, do not force the entire explanation into a single timeout message.

Design the test so that every system leaves a trace.

Test the User-Visible Contract

Simon Gerber — Mon, 27 Jul 2026 21:35:53 +0000

Modern frontends have made a simple idea surprisingly difficult:

When is the page finished?

With server-rendered HTML, client hydration, React Server Components, Suspense boundaries, streamed responses, background requests, variable fonts, live regions, and data visualizations, a screen may be usable before it is technically settled.

It may also continue changing after the user has started interacting with it.

This creates a trap for browser automation.

The test observes every intermediate state. The user only cares whether the experience eventually becomes correct and remains usable.

When those two perspectives are confused, teams spend a lot of time fixing tests that are accurately detecting implementation details nobody promised would be stable.

The DOM is an implementation detail more often than we admit

Browser automation naturally gravitates toward the DOM because that is where elements can be located and asserted.

But the DOM is not the product.

The user-visible contract is the product.

For a streamed account page, that contract might be:

the heading appears;
the account summary becomes available;
controls are enabled when the required data arrives;
loading states do not block unrelated actions;
focus does not jump unexpectedly;
errors are announced accessibly.

The exact number of renders is usually irrelevant.

This is the right lens when deciding what to look for in a browser testing tool for React Server Components, streaming UI, and partial re-renders. A tool needs to handle elements that appear, disappear, or update without forcing the test author to encode every transient state.

The more your assertions mirror implementation timing, the more expensive ordinary frontend evolution becomes.

Assert outcomes, not rendering choreography

Imagine a product page with three regions:

basic product information is rendered immediately;
availability streams in;
recommendations load later.

A brittle test waits for all network activity to stop, checks the entire page, and fails because the recommendation service takes an extra second.

A useful test checks that the product can be understood and purchased once the required state is available. Recommendations can be validated separately because they represent a different user promise.

This is the practical distinction in testing React Server Components and partial re-renders with Endtest: identify which transitions are meaningful and which are rendering noise.

The test should not ignore instability.

It should classify it.

A price changing after initial display may be a serious defect. A skeleton being replaced by content is expected. A button briefly disabled while its permissions load may be acceptable—or it may be a product problem if the state is confusing.

Those are product decisions, not framework defaults.

Hydration noise is not automatically harmless

There is a temptation to solve flaky tests by waiting longer.

Sometimes that works.

It can also hide real defects.

A hydration mismatch may only flicker for 100 milliseconds, but if it clears an input, moves focus, changes an accessible name, or causes a button to receive the wrong click, the user impact is real.

The goal is not to suppress every transient observation. The goal is to distinguish harmless implementation transitions from changes that break the user-visible contract.

A strong guide to testing React Server Components and streaming updates without chasing hydration noise should therefore include explicit invariants:

user-entered values persist;
focus remains predictable;
interactive elements do not move under the pointer;
accessible labels remain stable;
committed actions are not duplicated;
errors are recoverable.

These invariants survive framework upgrades better than assertions about component internals.

Fonts can change behaviour, not just appearance

Font loading bugs are a perfect example of why visual details and functional behaviour overlap.

A fallback font can make a label wider. That pushes a button onto another line. The layout shift moves the target after the test has located it. An emoji fallback changes line height. A variable-font axis produces different measurements in CI than on a developer laptop.

The test appears flaky.

The underlying issue is a race between layout and interaction.

The debugging techniques for browser tests that fail after font loading, emoji fallbacks, or variable fonts change the layout are useful beyond typography. They reinforce a broader principle: a visible element is not necessarily a stable interaction target.

Tests should wait for the condition that matters.

That may be:

fonts are ready;
layout has stopped shifting;
the element is enabled;
the target remains in the same position long enough to interact;
the application exposes an explicit ready state.

“Visible” is often too weak.

“Network idle” is often too broad.

The right condition is usually product-specific.

Accessibility exposes hidden timing defects

Sighted users can often infer that a changing region is loading or that a modal has appeared.

Assistive technology depends on the application communicating those transitions correctly.

ARIA live regions, focus traps, status messages, and dynamic announcements introduce a second user-visible contract: what the accessibility tree communicates over time.

This is why accessibility testing platforms should handle ARIA live regions, focus traps, and dynamic content announcements.

A page can look correct in a screenshot while being unusable with a keyboard or screen reader.

Useful browser checks include:

focus moves into a modal and remains trapped appropriately;
closing the modal returns focus to a sensible element;
validation errors are announced;
loading completion is communicated;
repeated partial renders do not generate a flood of duplicate announcements;
hidden content is not exposed as interactive.

These are temporal behaviours. A static accessibility scan will not catch all of them.

Dynamic charts need semantic assertions

Charts and dashboards create another temptation: screenshot everything.

Visual comparison can be valuable, but dynamic charts contain variability. Timestamps change. Data labels move. Anti-aliasing differs. Tooltips appear based on pointer position. A heatmap may be correct even when individual pixels are not identical.

A practical evaluation of Endtest for visual checks on dynamic charts, heatmaps, and data-rich dashboards highlights the need to combine visual checks with semantic ones.

Instead of asking only, “Does this image match?”, ask:

Is the chart present?
Is the expected series available?
Does the legend contain the correct labels?
Does the tooltip expose the expected value?
Is the empty state correct?
Can a keyboard user reach the data?
Are obviously invalid values absent?

The screenshot is evidence. It should not always be the sole assertion.

Separate stable contracts from unstable presentation

A useful test design exercise is to divide page behaviour into three layers.

Stable business contract

These should rarely change without intentional product work:

the user can submit an order;
permissions restrict access;
totals are correct;
errors explain recovery;
saved data persists.

Interaction contract

These can evolve, but users still depend on them:

focus behaviour;
enabled and disabled states;
loading feedback;
keyboard navigation;
announcements;
control labels.

Presentation details

These change frequently:

spacing;
animation duration;
component nesting;
skeleton shape;
exact render count;
decorative icons.

Test heavily at the first layer, selectively at the second, and deliberately at the third.

The mistake is not testing presentation. Visual regressions matter.

The mistake is assigning every implementation detail the same release-blocking weight as a broken payment flow.

Modern UI testing is an exercise in restraint

Browser tools are becoming more capable. They can observe more states, capture more traces, compare more pixels, and generate more assertions.

That does not mean every observable state deserves a test.

The hard work is choosing the few contracts that should remain true while the implementation underneath them changes.

That is what makes a suite stable without making it shallow.

Test what the user is promised.

Observe the rest when it helps you diagnose failures.

Do not turn every intermediate render into a permanent obligation.

AI Can Write the Test. Your Team Owns the Maintenance

Simon Gerber — Fri, 24 Jul 2026 22:12:16 +0000

AI has made the first hour of test automation dramatically cheaper.

Describe a workflow, paste a requirement, or point an agent at an application, and it can produce a respectable first draft. The code may include page objects, fixtures, assertions, and comments. It looks like progress because it is progress.

But creation was never the full cost.

The expensive part begins after the test joins the repository.

Now the team must decide whether the assertions are meaningful, whether the selectors are durable, whether the data is safe, whether the test fails for the right reasons, and who will fix it six months later when the interface changes.

AI lowers the cost of producing tests. It can also increase the number of tests you are responsible for.

That is the trade-off teams need to understand.

Generated code can fail before generated tests even run

AI-generated frontend changes frequently work in a local environment and fail in CI. The problem is not always the quality of the generated code. It is the missing context around it.

The CI environment may use a different Node version, package lock state, browser build, environment variable set, feature flag, operating system, or resource limit.

This analysis of why AI-generated frontend changes fail in CI even when local runs pass highlights an important principle: AI tends to optimise for the context it can see.

If the model sees the component but not the pipeline, it can create a locally valid solution that violates an invisible constraint.

The same thing happens with generated tests. A model may assume:

a clean database;
one browser worker;
stable test IDs;
immediate API responses;
permission to create arbitrary users;
access to secrets that do not exist in CI;
a desktop viewport;
English copy.

A useful generation prompt should include operational constraints, not only the user journey.

Tell the model where the test runs, how data is created, what can execute in parallel, which selectors are preferred, and what evidence should be captured on failure.

The prompt is part of the test architecture.

Claude can write Playwright. That is not the decision

A modern coding model can generate Playwright tests quickly. It can also refactor helpers, interpret error logs, and suggest alternate selectors.

The interesting question is not whether it can write the code. It can.

The better question is what changes in maintenance, review, and signal quality after generation becomes abundant.

The article When Claude Writes Your Playwright Tests describes the shift well. Reviewers need to evaluate more than syntax.

A generated test can be perfectly valid TypeScript and still be a poor test because it:

asserts incidental copy;
uses a selector that matches multiple elements;
hides an unreliable action behind retries;
duplicates coverage that already exists;
skips negative behaviour;
passes after an optimistic update but before server confirmation;
creates data it never cleans up.

Code review for generated tests should begin with intent:

What product risk is this test reducing?
What failure would it catch?
Why is the browser the right layer?
Does it fail when the product is broken?
Will the team understand the failure report?

Only then should the reviewer care about formatting and helper reuse.

The test count can become a vanity metric

When generation is expensive, teams are selective. When generation becomes almost free, restraint becomes more important.

It is easy to produce 200 tests from requirements, tickets, and recorded sessions. The number looks impressive. Then the product changes, 47 tests fail, and nobody knows which failures matter.

This framework on what to measure before trusting a Playwright test suite generated with Claude recommends measuring the suite as an operating system rather than a code output.

Useful metrics include:

percentage of failures that represent real defects;
median diagnosis time;
maintenance hours per month;
repeated failures from the same cause;
coverage of critical workflows;
tests that have never failed meaningfully;
retries required for a green pipeline;
reviewer effort per generated test.

That final metric matters. A model can produce code in seconds, but a senior engineer may spend 20 minutes proving that the code is safe and useful.

AI does not eliminate review cost. It moves the work.

AI testing platforms also contain human work

A platform may advertise AI-generated tests, self-healing, natural-language instructions, or autonomous maintenance. Those capabilities can reduce effort. They can also introduce new operating tasks:

reviewing generated steps;
validating prompts;
inspecting traces;
approving suggested fixes;
resolving ambiguous assertions;
monitoring model usage;
handling false repairs.

The article on estimating the real cost of an AI testing platform when prompt reviews, traces, and human approvals add up offers a healthier cost model.

Do not compare only subscription price with an open-source licence price.

Compare the full workflow:

Total cost = platform cost
           + test creation time
           + review time
           + maintenance time
           + infrastructure
           + failure diagnosis
           + training and adoption

The same equation should be used for a home-grown framework.

“Free” software can be expensive when specialised engineers spend a significant part of every sprint maintaining it. A paid platform can also be expensive when it requires constant human supervision.

The correct unit is not licence cost. It is cost per trustworthy result.

Build versus buy is mostly a staffing decision

The comparison between Endtest and hand-rolled Playwright for fast-changing frontends frames the decision around ownership.

A custom Playwright framework can provide excellent control. Your team can define fixtures, abstractions, reporters, environment management, and CI behaviour exactly as needed.

But somebody owns all of it.

That ownership includes:

browser and dependency upgrades;
authentication helpers;
test-data utilities;
parallel execution rules;
retry strategy;
screenshots, videos, traces, and logs;
integrations with email, SMS, and storage;
onboarding documentation;
support for less technical contributors.

A managed system such as Endtest trades some framework-level freedom for a smaller internal maintenance surface and broader accessibility.

Neither model is automatically better.

The mistake is selecting a code-first framework because the initial proof of concept was easy, then discovering that the team unintentionally created an internal product.

A useful question is:

Do we want to build tests, or do we also want to build and operate a testing platform?

Some organisations should answer yes. Many should not.

Reliability is different for streaming AI applications

AI applications introduce behaviours that traditional CRUD tests do not encounter as often:

streamed text arrives incrementally;
the UI changes while the response is still in progress;
intermediate state may be incomplete but valid;
model outputs vary;
cancellations and retries matter;
network latency changes the rendering sequence.

A brittle test may wait for exact text, assume a fixed number of updates, or interact before the stream is complete.

The guide to benchmarking browser test reliability for AI apps that stream responses and update state incrementally recommends testing the state machine rather than the precise output.

For example, verify that:

a response begins within an acceptable period;
the loading or streaming state is visible;
content grows rather than being replaced incorrectly;
controls are enabled or disabled at the right times;
cancellation stops the stream;
the final state is persisted correctly;
an error offers a recovery action.

For non-deterministic content, assert structure and constraints instead of a complete sentence.

You may check that the response is non-empty, contains required facts, follows a schema, avoids prohibited content, or receives an acceptable evaluation score.

The more variable the product, the more deliberate the oracle must be.

Stable coverage does not require a heavy framework

Some teams need deep framework control. Others mainly need dependable coverage of business-critical journeys without dedicating several engineers to test infrastructure.

This practical review of Endtest for teams that need stable coverage without a heavy framework represents the second group.

The value proposition is not that code is bad. It is that infrastructure work has an opportunity cost.

A team that avoids building its own reporting, test editor, execution grid, collaboration layer, and integrations can spend that time improving product coverage.

The trade-off is accepting the platform's model and limits. That is why a proof of concept should test the awkward workflows, not only login.

Try:

multi-step authentication;
file uploads and downloads;
dynamic tables;
cross-domain navigation;
email and SMS verification;
failure diagnosis;
role-based collaboration;
CI execution under realistic parallelism.

Any tool looks good when the demo is “open page, click button.”

Human approval should focus on risk

AI-generated tests should not require a committee meeting. But they should receive review proportional to their impact.

A low-risk visual check for an internal page may need a quick inspection. A payment, access-control, deletion, or regulatory workflow deserves deeper review.

A lightweight approval checklist can ask:

Is the workflow worth testing at the browser layer?
Are secrets and personal data handled safely?
Can the test run in parallel?
Are assertions tied to user outcomes?
Does failure produce enough evidence?
Is cleanup defined?
Who owns future maintenance?

This prevents the two common extremes: blindly accepting generated code or making AI assistance so bureaucratic that nobody uses it.

The goal is not perfect generation.

The goal is controlled leverage.

The new bottleneck is judgement

For years, the bottleneck in browser automation was writing the code.

AI is removing part of that bottleneck.

What remains is judgement:

choosing the right risks;
deciding the right test layer;
defining stable assertions;
controlling test data;
evaluating failures;
deleting low-value coverage;
choosing which infrastructure to own.

That is good news. These are more valuable problems than remembering the syntax for a locator.

But they are not automatic.

AI can write the test.

Your team still owns whether anyone should trust it.