DEV Community: Antoine Dubois

Your Smoke Tests Are Crying Wolf. That's a Release Problem.

Antoine Dubois — Wed, 14 Oct 2026 21:09:42 +0000

It's 4:45 p.m. The deployment finishes. The smoke suite goes red.

Nobody panics. Nobody rolls back. Someone asks in Slack whether the test failed again because the sample account already existed.

That reaction is the problem.

A smoke test has one job: give the team a fast, credible answer to "Did this release break something we care about?" If everybody assumes the answer is probably wrong, the suite has become another dashboard to ignore.

This isn't solved by adding fifty more assertions.

A release check has to earn the right to stop a release

There's a difference between a scheduled regression run and a post-deploy decision. A regression run can take time, collect lots of detail, and investigate edge cases. A post-deploy check usually has a much narrower window.

It needs to identify the deployed version, confirm that a handful of critical customer journeys work, and report a failure in language someone can act on.

For example:

Bad result: Login flow failed. Timeout exceeded.

Useful result: Production build 2026.10.15-3: Login succeeded, but account dashboard never displayed the billing summary. Screenshot and browser logs attached. Previous build passed the same check.

The second message doesn't guarantee that rollback is appropriate. It gives the release owner a real starting point.

This guide to API-triggered post-deploy smoke checks is useful because it treats the API trigger as part of a larger release workflow, not the entire solution.

The retry trap

Say a smoke check fails immediately after deployment. The team retries it. It passes.

What happened?

Maybe the app needed another ten seconds to warm up. Maybe a replica was unhealthy. Maybe the first test left data behind. Maybe the product really does fail one in ten times.

Those possibilities have wildly different implications. A retry should give you more information, not erase the first result.

I'd record the first failure, the retry result, the app version, the environment, and whether any state was modified between attempts. Most importantly, I'd keep the artifacts from both runs.

A helpful starting point is this benchmark plan for API-triggered smoke checks. Its emphasis on retry safety and rollback signal quality gets at the real question: can someone trust the evidence enough to make a production decision?

If your suite's success rate improves only because you discard the first attempt, you haven't necessarily improved reliability. You may have improved the reporting.

The test runner is part of the release system

When people evaluate QA tooling, they tend to count supported browsers, integrations, and test authoring features. Fair enough. But those details don't tell you what happens at 4:45 p.m. when the deployment is red and the engineer who wrote the test has gone home.

I would ask a vendor or internal platform team to demonstrate four things with a deliberately failing flow:

Trigger the run through the same API the deployment pipeline will call.
Correlate results with the exact release or commit.
Preserve screenshots, step history, and relevant logs.
Deliver a result the release owner can interpret without opening five different tools.

That's the practical lens used by this review of QA platforms for smoke checks, failure evidence, and release handoffs.

Notice what's missing: an argument over which product has the prettiest green checkmark.

Preview environments can make this worse

Branch previews are wonderful until test runs point at the wrong one.

Picture two pull requests. Both create a user called test@example.com. One changes the checkout form. The other changes the account page. The test infrastructure shares a backend database, and one run deletes the other's test user.

Now you've got a failure that looks like a code regression but is actually an isolation bug.

Treat the preview URL, build identifier, test accounts, and cleanup policy as inputs to the run. Never assume the environment from yesterday still exists or still belongs to the same branch.

If you're comparing AI testing services, this look at preview environments and branch isolation provides a useful set of questions beyond "Can it generate tests?"

What I would actually put in a smoke suite

Not everything. Definitely not everything.

I'd start with a tiny set of paths that have a clear business consequence when broken: sign in, load the main customer screen, perform one harmless read/write action, and verify a critical integration response. The exact list depends on the product. An ecommerce store and an internal admin dashboard shouldn't have identical release gates.

Each check should have a known cleanup path, a tight timeout based on normal behavior, and a named owner. If a check is too unreliable to block a release, don't quietly pretend it blocks one. Fix it or classify it as informational until it earns that role.

The real win isn't a permanently green dashboard. It's the day a smoke test goes red and nobody asks, "Is this another test problem?"

Looking for a Cheaper mabl Alternative? Here's What I'd Compare Before Renewing

Antoine Dubois — Sat, 10 Oct 2026 21:24:04 +0000

I have a fairly uncontroversial opinion that still seems to annoy people in software procurement: a good product can be a bad purchase.

That's where I put mabl for a particular kind of team.

It's a legitimate test automation platform. Its browser recorder is approachable, its automatic healing can save maintenance time, and the product extends well beyond clicking around a web page. It offers API, accessibility, performance, visual, email, and PDF testing. More recently, mabl has invested heavily in AI-assisted test authoring and recovery.

The problem isn't that mabl doesn't do enough. The problem is that you might be paying for a much larger testing platform than you actually need.

If I were reviewing a mabl renewal today, Endtest would be near the top of my shortlist. Not because it has a flashier AI pitch. Because it offers much of the test automation functionality most teams buy mabl for, with published entry pricing that's easier to budget for.

But you shouldn't switch testing platforms because of a blog post. You should switch because the numbers and the test results work out.

The expensive part isn't always the subscription

Think about an engineering team with eight developers, two QA engineers, and a SaaS product that ships every weekday.

They probably want five fairly ordinary things:

Someone should be able to create a browser test without spending an afternoon writing selectors.
Tests should run in CI and on a schedule.
When a test fails, the team should be able to tell whether the application is broken or the test is broken.
Important flows should run on more than one browser.
Adding coverage shouldn't mean hiring another automation engineer.

That's not a modest list. But it isn't especially exotic, either.

The subscription line on the budget is only the start. The real cost of testing includes people debugging failures, rewriting tests after UI changes, maintaining environments, and waiting for results. A cheaper tool that wastes three hours a week isn't cheaper for very long.

This is why I don't think a price-only comparison is useful. And it's also why mabl deserves credit: its automatic waits, healing, and reporting address real operational costs.

First, understand what you're paying for

mabl's pricing page currently directs buyers to request a custom quote. It describes a credit-based approach for cloud testing, beginning with 500 credits per month, alongside unlimited local runs and cloud concurrency at no additional concurrency charge. The exact economics depend on your contract and usage.

Endtest publishes a Starter plan at $175/month and a Pro plan at $450/month, plus custom enterprise pricing. Its pricing page lists unlimited test creation, executions, and users, with limits and entitlements that vary by plan, including parallel slots and AI features.

Those aren't directly comparable billing models. Don't confuse an unlimited-executions line with unlimited simultaneous capacity. And don't treat a credit as equivalent to an Endtest execution.

Still, published pricing gives you something useful: a concrete starting point for a negotiation or pilot.

Here's how I'd model it.

Annual software spend	Amount
mabl renewal quote	Enter your actual quote
Endtest Starter at $175/month	$2,100
Endtest Pro at $450/month	$5,400
Difference	Your quote minus the relevant Endtest plan

For example, if your mabl quote is $1,500 per month and the $450 Endtest configuration covers the same workload, the arithmetic is $18,000 versus $5,400 annually. That's a 70% reduction in subscription cost, or $12,600 per year.

That's a scenario, not an estimate of what mabl charges every customer. Your real quote might be higher or lower; you might also need more Endtest parallel capacity or an enterprise arrangement.

That distinction matters. An apparently cheap replacement isn't a bargain if you discover the missing requirement after migration.

Feature parity is the wrong question

People love feature grids. I understand why. They're easy to forward to a manager.

But the question, "Does Endtest have everything mabl has?" is too broad to be useful. Modern testing products change quickly, and both vendors have capabilities that may not map neatly onto the other's offering.

A better question is: Can the replacement execute our critical workflows, with the controls and evidence our team needs?

Here's the shortlist I'd start with:

Requirement	mabl	Endtest	What to verify in a trial
No-code browser test creation	Yes	Yes	How readable and editable are generated steps?
AI-assisted creation	Yes	Yes	Can it complete a real multi-page business flow?
Self-healing	Yes	Yes	Does it recover correctly, not just quietly continue?
Cross-browser execution	Yes	Yes	Browser versions and combinations you actually use
API testing	Yes	Yes	Authentication, variables, response assertions
Accessibility checks	Yes	Yes	Rules, scope, and report format
Email and PDF workflows	Yes	Yes	Your actual email and PDF scenarios
Native mobile testing	Yes	Yes	Devices, builds, and plan entitlements
CI/CD and scheduling	Yes	Yes	Triggering, result retrieval, and failure behavior
Performance testing	Yes	Evaluate separately	Don't assume equivalent performance-testing depth
Pricing	Custom quote	Published entry tiers; enterprise by quote	Total cost at your required concurrency

For mabl's own capability breakdown, see its product overview. For Endtest, see its feature and plan breakdown.

Notice the last two rows. I would not claim blanket feature parity. mabl has an integrated performance-testing offering, and that could be a deciding factor if your QA team relies on it. Conversely, Endtest has capabilities such as browser-based recording, AI-driven test creation, and advanced browser execution options that may be more than sufficient for teams primarily focused on end-to-end regression testing.

A spreadsheet doesn't make that judgment. A trial does.

The test that tells me more than a sales demo

Don't build a demo around the login form. Both tools should pass it, and neither should get much credit for doing so.

Take a workflow that's been mildly irritating your team for the past year. Something like:

Sign up with a new email address.
Open the verification email and follow the link.
Create a project.
Upload a PDF to that project.
Open a details panel embedded in an iframe.
Edit a setting and confirm the result persists after a refresh.
Assert that the expected record exists through an API call.

It's a better test than a pretty dashboard because it forces the product to deal with state, email, files, frames, and assertions. It also exposes whether the test output is something an engineer can inspect and fix.

I'd put the same workflow into both tools and score it on four things:

Coverage: How much can the tool do without custom code or brittle workarounds?

Creation time: How long until the test genuinely works, including assertions, not just until it looks plausible in an editor?

Repeatability: Does it survive repeated cloud runs, including after a small UI change?

Failure diagnosis: When it breaks intentionally, can an engineer locate the problem quickly?

A test that passes once tells you very little. A test that fails clearly is often more valuable.

A small experiment worth running

Here's a simple benchmark sheet I would use in a proof of concept. It takes less than a minute to understand and avoids the usual vendor-demo theater.

Run each scenario at least 20 times on each platform, using the same application version, data setup, and comparable execution environments.

Metric	How to measure it
Scenario coverage	Fully automated scenarios / total scenarios
Stable-run rate	Successful runs / total runs, excluding confirmed app defects
Median runtime	Median of measured run durations
Debugging time	Minutes from failure notification to identified cause
Authoring time	Engineer minutes until a reviewed, passing test
Monthly cost	Actual quoted monthly bill for your target workload

A lightweight script can summarize the results from a CSV file exported or assembled from your run logs:

import csv
from collections import defaultdict
from statistics import median

# benchmark.csv columns:
# tool,scenario,passed,duration_seconds,debug_minutes
# passed must be 1 for a successful run, 0 otherwise.

runs = defaultdict(list)

with open("benchmark.csv", newline="", encoding="utf-8") as file:
    for row in csv.DictReader(file):
        runs[row["tool"]].append(row)

for tool, records in sorted(runs.items()):
    successes = sum(int(r["passed"]) for r in records)
    durations = [float(r["duration_seconds"]) for r in records]
    debugging = [float(r["debug_minutes"]) for r in records
                 if r["debug_minutes"].strip()]

    print(f"\n{tool}")
    print(f"  Stable-run rate: {successes / len(records):.1%}")
    print(f"  Median runtime: {median(durations):.1f}s")
    if debugging:
        print(f"  Median debug time: {median(debugging):.1f} min")

Keep failed-run durations in the runtime calculation, or report successful and failed runtimes separately. Otherwise a tool that fails quickly can look deceptively fast. Also record the root cause of every failure. A legitimate application bug is not an automation flake.

Most importantly, don't cherry-pick the best run of each tool. That's how you end up with an impressive benchmark and an unhappy engineering team.

Migrating is where the savings can disappear

Even if Endtest wins a proof of concept, I wouldn't immediately rewrite 400 existing tests.

I'd use a staged migration:

Week one: Inventory the suite. Identify critical user journeys, duplicate tests, long-running tests, flaky tests, and the features that require custom JavaScript or unusual integrations.

Week two: Rebuild a small but representative group in Endtest. Include one straightforward flow and several deliberately ugly ones. Run them in parallel with mabl against the same staging release.

Week three: Compare failures and operating effort. Track who diagnosed each failure, how long it took, whether an assertion was too weak, and whether your CI pipeline received a reliable pass/fail signal.

Week four: Decide what to migrate, what to retire, and what should stay where it is. Some tests may not be worth porting at all.

Your goal isn't to reproduce the historical test count. Your goal is to preserve useful coverage.

Migrating from other tools to Endtest is fairly easy, since you can just use their AI Test Import feature.

Where I'd still choose mabl

There are sensible reasons to stay.

If your team uses mabl's integrated performance testing, relies on its existing workflows across several departments, or has years of working tests with very little maintenance overhead, switching may be a distraction. Procurement friction and migration risk are real costs.

And mabl's unlimited local runs and cloud concurrency model might be particularly attractive at your usage level. Don't assume the lowest published competitor price will be the cheapest configuration for a large suite.

I'd also be cautious about any vendor, including Endtest, that promises AI will eliminate test maintenance. It won't. Your application still changes, your data still expires, and your test oracles still need to be correct.

AI can reduce the labor. It doesn't remove responsibility.

Where Endtest becomes hard to ignore

Now consider the more common case: a small or midsize engineering team mostly needs dependable end-to-end web testing, mobile coverage, API checks, cross-browser execution, and reasonable failure reports.

They don't have a dedicated automation-infrastructure team. They don't want to build their own browser grid. And every year somebody has to explain the QA tooling bill to finance.

For that team, Endtest is a serious alternative. You get an AI-enabled, no-code testing platform with substantial overlap in the workflows that drive a typical regression suite. You can inspect its pricing without starting with a sales call, and you can use a trial to find out whether its capabilities fit your application.

The cost difference can be substantial when your actual mabl quote is well above the Endtest plan that meets your needs. But that's something to establish with your own quote and your own tests, not a percentage copied from somebody else's benchmark.

One other useful question to ask before signing: If we cancel in two years, how easily can we take our test definitions and results with us? Data portability isn't exciting when you're buying a tool. It matters a great deal when you're replacing one.

The decision I'd make

I wouldn't recommend dumping mabl because it's a bad product. I don't think it is.

I'd recommend questioning whether it's the right-sized product for the job, especially if the next renewal is forcing your team to choose between broader test coverage and the budget to build actual product features.

Give Endtest the same hard scenarios. Compare creation time, real pass rates, debugging overhead, and the total annual bill. If it delivers the same useful coverage for your application at a fraction of the cost, the decision becomes pretty straightforward.

And if it doesn't, you've spent a few days validating that your mabl subscription is earning its keep.

Either way, you've replaced a sales argument with an engineering decision. That's a worthwhile result.

Most Test Suites Only Work in a World That Doesn't Exist

Antoine Dubois — Thu, 08 Oct 2026 21:25:52 +0000

A lot of automated tests begin in an unusually pleasant universe.

The user is logged in.

The session is valid.

The database contains exactly the right data.

The feature flags are known.

The browser has no weird state.

The network works.

Nothing unexpected is covering the screen.

And the deployment that happened five minutes ago is assumed to be fine because CI passed.

Then the test runs.

Green.

Fantastic.

Unfortunately, real users don't live in that universe.

They leave tabs open overnight.

Their access tokens expire halfway through a workflow.

They get assigned to experiments.

They lose network connectivity.

A modal opens while another component still has focus.

A deployment changes one service but not another.

And sometimes the application quietly logs them back in while they're doing something important.

If your test automation strategy assumes those things don't happen, you might have excellent coverage of a product that doesn't actually exist.

The clean-session assumption is particularly dangerous

Here's a very normal test:

Login
Open Dashboard
Create Project
Invite User
Verify Invitation

Looks good.

Now try this:

Login
Open Dashboard
Wait until access token expires
Start creating project
API returns 401
Application silently refreshes token
Request retries
Continue workflow

Did the project get created once?

Twice?

Not at all?

Did the UI recover?

Did the application dump the user back onto the login page?

Did it preserve what they had typed?

Did the test framework even notice that authentication changed underneath it?

This is one of those situations where a test can pass hundreds of times simply because you're always starting with a fresh session.

There's a useful benchmark plan for session expiry recovery, silent re-login, and token-refresh evidence that treats this as something browser testing platforms should be evaluated against.

I agree.

Authentication isn't just a setup step.

It's application state.

Users don't politely expire between tests

Most automation frameworks encourage isolation.

That's generally a good thing.

Fresh context.

Fresh cookies.

Fresh test data.

Predictable environment.

But isolation can accidentally remove the exact states that make real applications difficult.

A real user might:

09:00  Log in
09:15  Open a form
09:22  Switch tabs
10:04  Come back
10:07  Submit

Your automated test might do the entire thing in 14 seconds.

Those are technically the same workflow.

They're not the same conditions.

If your product has session expiration, refresh tokens, idle timeouts, or role changes, time itself becomes part of the test.

Feature flags create a similar illusion

Feature flags are great.

They let you roll things out gradually.

Test with internal users.

Run experiments.

Disable broken features quickly.

But they also create a new dimension of application state.

The same URL can now represent two different products.

checkout_v2 = false

vs.

checkout_v2 = true

And if your test suite handles this by copying flag setup into 80 scenarios, you eventually get something like:

Set checkout_v2 = true
Set new_nav = false
Set pricing_test = variant_b
Set recommendations = true

Repeated everywhere.

Then somebody changes the feature configuration and half the suite becomes archaeological evidence.

This guide on testing feature flag overrides without hardcoding the same flag values into every scenario tackles that exact problem.

The larger lesson is useful even if you're not using feature flags heavily.

Test setup is part of your architecture.

If every test independently recreates the world it needs, maintenance eventually becomes expensive.

UI state is more than "element visible"

Here's another situation I think browser automation often oversimplifies.

Modals.

Testing a modal usually looks something like:

Click "Delete"
Verify confirmation modal is visible
Click "Confirm"

Okay.

But accessibility changes what "working" actually means.

If a modal is open, what happens to the rest of the page?

Can keyboard focus escape?

Can you tab into controls behind the modal?

Can screen-reader users interact with background content that visually appears locked?

The relatively new inert attribute gives browsers a cleaner way to make parts of a page non-interactive.

But testing it properly requires more than:

modal.isVisible() === true

This walkthrough on testing the inert attribute, modal background locking, and tabbable leakage is a good example of how much behavior can hide behind something that looks visually correct.

A screenshot can show you a perfect modal.

The keyboard experience can still be completely broken.

This is why I don't love "end-to-end coverage" as a metric

Teams will say:

We have 80% of our critical user journeys automated.

That sounds great.

But what does it mean?

If you test:

Login → Dashboard → Checkout → Confirmation

do you also test:

Login
    ↓
Session expires
    ↓
Checkout begins
    ↓
Token refreshes
    ↓
Feature flag changes rendering
    ↓
Modal opens
    ↓
Keyboard navigation continues
    ↓
Purchase completes

Probably not.

And I'm not saying you should automate every imaginable combination.

That would be ridiculous.

The point is that "journey coverage" can give you false confidence if your journeys are modeled as clean sequences instead of changing states.

AI-generated tests can amplify this problem

AI makes it easier to generate a lot of automation.

Type:

Test checkout.

And an agent can potentially create the flow for you.

That's useful.

But the first generated version will often represent the obvious path.

Open checkout
Enter card
Click Pay
Verify success

The interesting part is what happens later.

Can you edit the steps?

Can you add recovery behavior?

Can another person understand why something failed?

Can you attach notes explaining why a strange step exists?

Can the test survive being handed to a different team six months from now?

Those questions matter much more once AI starts generating a significant percentage of the suite.

This rubric on editable steps, recovery notes, and team handoffs in AI testing platforms focuses on exactly that.

I think "editable" is an underrated feature in AI systems.

Autonomy sounds cooler.

Editability is what saves you when reality gets complicated.

Fully autonomous QA sounds great until you ask what happens at 3 AM

There's a lot of interest now in autonomous QA agents.

And I understand why.

The idea is compelling:

Give agent application
    ↓
Agent explores
    ↓
Agent creates tests
    ↓
Agent runs continuously
    ↓
Agent finds failures

That's a very different model from manually maintaining hundreds or thousands of scripts.

There's a demonstration of that approach in Endtest Bot: Your Autonomous QA Agent.

I think agentic testing will become a meaningful part of QA.

But the important question isn't whether the agent can click around your product.

It's:

What happens when the application behaves unexpectedly?

An autonomous system becomes much more valuable when it can recognize:

Session expired
Token refreshed
Expected state restored
Continue scenario

rather than simply reporting:

Step failed

That difference is huge.

Recovery may be more important than self-healing

"Self-healing" became one of the favorite phrases in test automation.

Usually it means:

Locator broke
    ↓
Tool found another locator
    ↓
Test continued

Useful.

But that's a pretty narrow definition of healing.

Real applications need recovery from things like:

expired sessions
intermittent APIs
reauthentication
WebSocket reconnects
temporary loading states
feature changes
interrupted transactions
background refreshes

Changing #submit-button to [aria-label="Submit"] is the easy version.

Recovering application state without changing the meaning of the test is much harder.

That's where I think the next generation of testing tools will either become genuinely useful or just generate impressive demos.

And then we deploy

There's another weird boundary in a lot of testing strategies.

Everything before deployment gets enormous attention.

Everything after deployment gets:

We'll check Sentry.

But a successful deployment pipeline doesn't mean the product actually works.

You can still have:

Bad environment variable
Broken CDN asset
Missing migration
Expired production credential
Misconfigured feature flag
Third-party API failure
DNS issue
Regional outage

CI may have been completely green.

Which raises a practical question:

What exactly should we run after deployment?

There isn't one answer.

Sometimes a fast API check is enough.

Sometimes you need a real browser.

Sometimes an important customer journey should be monitored continuously.

This decision guide for API smoke checks, full browser flows, and synthetic monitoring gives a useful framework for deciding where each belongs.

I'd think about them as different layers.

API Smoke Check
"Is the service basically alive?"

Browser Flow
"Can a user actually complete this?"

Synthetic Monitoring
"Does it keep working after we leave?"

Those are related questions.

They're not interchangeable.

Your production test strategy probably deserves more attention than your CI strategy

Suppose your checkout service deploys successfully.

CI is green.

Your API smoke check passes.

But the production cookie-consent configuration changed.

Now the checkout button is covered for European users.

The backend is healthy.

The API is healthy.

The deployment is healthy.

Checkout is broken.

This is why I increasingly think the traditional line between:

Testing

and:

Monitoring

is becoming artificial.

If an automated browser checks checkout before deployment, we call it a test.

If the same browser checks checkout every five minutes after deployment, we call it synthetic monitoring.

The user doesn't care what we call it.

They care whether checkout works.

Try auditing your suite for assumptions instead of tests

Here's an exercise that may be more useful than counting test cases.

Take ten important automated tests.

For each one, write down its assumptions.

Something like:

User has valid session
Feature X enabled
Feature Y disabled
Network stable
No third-party overlay
API responds within 5 seconds
User role unchanged
No background token refresh
Desktop viewport
English locale

Now ask:

Which of these assumptions are actually guaranteed?

You might find that your tests aren't really validating a workflow.

They're validating a workflow under one carefully curated version of reality.

That's still useful.

But it's important to know what you're getting.

I wouldn't try to test every combination

There's an obvious objection.

If we start considering:

sessions
flags
roles
locales
networks
devices
browser state
accessibility state
deployment state

the number of combinations explodes.

Absolutely.

You shouldn't test all of them.

The solution isn't combinatorial madness.

It's risk-based selection.

Ask:

Which state changes have broken before?

Which ones affect money?

Which ones affect authentication?

Which ones affect permissions?

Which ones are difficult to recover from?

Which ones differ between staging and production?

Which ones would be embarrassing if a customer found them first?

Test those.

The uncomfortable question

So here's the question I'd ask your team:

If our application behaved exactly like a normal production application instead of a freshly reset test environment, how much of our automation would still work?

Expired sessions.

Changing flags.

Half-finished interactions.

Browser history.

Modal focus.

Background refreshes.

Post-deployment configuration.

Real user state.

If the answer is "not much," that's worth knowing.

Because maybe your biggest automation gap isn't another missing test case.

Maybe your suite is simply making assumptions that production never agreed to.

Your Test Suite Probably Covers Screens, Not Behavior

Antoine Dubois — Tue, 06 Oct 2026 10:44:35 +0000

Most teams can tell you how many automated tests they have.

Some can tell you their pass rate.

A few can tell you their code coverage.

But ask this:

How many meaningful state transitions in your product are actually tested?

And the room usually gets quiet.

That's a much harder question.

It's also probably a better one.

Because users don't experience your product as a collection of pages.

They experience it as a sequence of state changes.

Logged out
    ↓
Logged in
    ↓
Trial
    ↓
Paid
    ↓
Payment failed
    ↓
Grace period
    ↓
Canceled

Or:

Draft
    ↓
Submitted
    ↓
Approved
    ↓
Rejected
    ↓
Edited
    ↓
Resubmitted

The bugs that hurt usually aren't:

The Settings page doesn't exist.

They're more like:

A user who downgraded, then upgraded, then changed teams still has the wrong permissions.

That's not a page problem.

That's a state-transition problem.

A lot of "coverage" is really screenshot coverage

Here's how browser automation often grows.

Someone automates login.

Then signup.

Then checkout.

Then settings.

Then the admin panel.

Eventually you have a few hundred tests and a dashboard that says:

742 automated tests
97.8% passing

Looks healthy.

But those numbers don't tell you whether you've tested the dangerous combinations.

For example:

Trial user → paid user
Paid user → canceled user
Canceled user → reactivated user
Admin → downgraded member
Invited user → expired invitation
Feature OFF → feature ON → feature OFF

Those transitions are where systems tend to get weird.

The UI may look correct in each isolated state.

The bug happens because the transition left something behind.

A stale permission.

An old token.

A cached plan.

A lingering feature flag.

A half-updated record.

This guide on mapping user-journey test coverage to state transitions makes a good case for treating transition coverage as a first-class QA problem.

I think that idea applies whether QA is outsourced or entirely internal.

Happy paths hide structural gaps

Take a subscription flow.

You probably test:

Signup
    ↓
Choose plan
    ↓
Pay
    ↓
Dashboard

Great.

Now test:

Signup
    ↓
Choose plan
    ↓
Payment fails
    ↓
Retry
    ↓
Payment succeeds
    ↓
Cancel
    ↓
Reactivate
    ↓
Change card

Different system.

Same product.

The first scenario proves the happy path works.

The second starts probing whether your application handles state.

And this is where I've become skeptical of raw test counts.

Ten carefully chosen transition tests can sometimes tell you more than 100 page-oriented checks.

API testing has the same problem

The same mistake happens at the API layer.

A team may have lots of tests like:

POST /users → 201
GET /users/123 → 200
DELETE /users/123 → 204

Useful.

But that doesn't necessarily tell you whether the API still behaves correctly as a contract between systems.

There's a meaningful difference between:

Does this request return what I expect?

and:

Are the producer and consumer still honoring the same contract?

That's the distinction behind comparisons like Pact vs REST Assured.

They're not interchangeable ways of doing the same thing.

One pushes you toward contract-first compatibility between services.

The other is very good at request-level API checks.

Which one matters more depends on how your system is built.

But if your frontend, backend, mobile app, and third-party integrations are all evolving independently, testing isolated requests may not be enough.

Frontend teams are quietly becoming QA infrastructure teams

There's another shift happening.

Frontend teams increasingly own:

component tests
browser tests
visual checks
accessibility checks
CI workflows
test data setup
mock APIs
release validation

Which means "frontend testing" isn't really just frontend testing anymore.

The frontend team may now own a substantial part of the release confidence system.

That's a different job.

If a team owns production UI code and the automation around it, the tooling needs to fit their development workflow rather than behave like a separate QA island.

This guide on browser testing platforms when frontend teams own the automation too explores that operating model.

I think this matters more than a generic list of features.

A tool can be excellent in isolation and still be a bad fit if developers hate touching it.

Ownership matters more than people admit

Here's a question I rarely see in tool comparisons:

Who is expected to maintain the tests six months from now?

That answer changes everything.

Imagine two models.

Model A

Your internal team:

Creates tests
Maintains tests
Reviews failures
Updates selectors
Handles flaky runs
Owns CI

Model B

A service or platform handles much of:

Maintenance
Triage
Review
Repair

Those are not small differences.

They're completely different operating models.

This BlinqIO vs QA.tech comparison frames the decision around managed maintenance versus faster self-serve browser coverage.

That is a much better way to think about it than asking which product has more AI.

The important question is:

What work is my team still responsible for after we buy this?

AI testing tools are splitting into different philosophies

The same thing is happening across AI testing platforms.

Some products want to give you:

Fast self-service automation.

Others lean more toward:

Guided workflows, review, and assistance.

Those approaches can look very similar in a demo.

Type an instruction.

Watch the browser move.

Test created.

Done.

But they create different day-to-day workflows.

A comparison like Reflect vs Octomind is more useful when you look at it through that lens: self-serve browser coverage versus more guided review workflows.

Neither is automatically better.

A five-person startup may want to move as fast as possible with minimal process.

A large QA organization may care much more about reviewability, governance, and consistency.

Same category.

Different problem.

I think we're entering the second phase of AI test automation

The first phase was:

Can AI create a test?

That's basically settled.

Yes.

It can.

Sometimes surprisingly well.

The more interesting questions now are:

Can it maintain the test?

Can humans understand what it changed?

Can it model complicated state?

Can it handle multiple roles?

Can it deal with non-happy paths?

Can it produce evidence a developer trusts?

Can teams own the result long term?

Those are much harder problems.

There are broader comparisons of the current market, including this review of 10 AI-powered test automation tools in 2026, that look at practical scenarios rather than only feature checklists.

That's the direction I think tool evaluation needs to go.

"Has AI" isn't a meaningful differentiator anymore.

Almost everybody has AI.

The question is what happens after the first impressive demo.

Your test model might be wrong

This is the bigger issue.

A lot of automation suites are modeled like this:

Page
    ↓
Actions
    ↓
Assertions

But many real applications behave more like this:

State A
    ↓
Transition
    ↓
State B
    ↓
Transition
    ↓
State C

The second model forces you to ask different questions.

For example:

What states can this user be in?

What transitions are allowed?

Which transitions should be impossible?

What data changes during the transition?

What permissions change?

What happens if the transition is interrupted?

What happens if the user repeats it?

Can the transition be reversed?

That's where some of the nastiest bugs live.

Here's a simple exercise

Take one important workflow in your application.

Not the whole product.

Just one.

Maybe:

Subscription

Write down every meaningful state.

For example:

Trial
Active
Past due
Canceled
Expired
Reactivated

Now draw the transitions.

Trial → Active
Trial → Expired
Active → Past due
Past due → Active
Past due → Canceled
Canceled → Reactivated

Then compare that graph with your automated tests.

Which edges are actually covered?

You may discover something uncomfortable.

You have excellent coverage of each page.

But weak coverage of the transitions that connect them.

This changes how I think about "critical paths"

Teams often say:

We automate our critical paths.

Usually they mean:

Signup
Login
Checkout
Create project
Invite user

Those are important.

But they're not really paths.

They're destinations.

A true critical path includes how users get into and out of important states.

For example, inviting a user isn't just:

Admin sends invitation
User accepts

It's also:

Admin sends invitation
Invitation expires
Admin resends
User accepts old link
User accepts new link
Admin changes role
User logs in with stale session

That's where "works in QA" turns into "why does this one customer have admin access?"

Better coverage can mean fewer tests

This is the counterintuitive part.

If you model transitions properly, you may not need more tests.

You may need better-selected tests.

Instead of 30 slight variations of:

Open page
Click button
Check confirmation

you might want 10 scenarios deliberately chosen to cross the risky boundaries in your system.

This is why I don't think test-count growth is necessarily a sign of a better automation program.

Sometimes it means the strategy is becoming more thorough.

Sometimes it means you're accumulating redundant scripts.

The number itself tells you almost nothing.

The question I'd ask instead of "How much is automated?"

I'd stop asking:

What percentage of QA is automated?

That's too vague.

I'd ask:

Which important state transitions can still reach production without being exercised automatically?

That's a much harder question.

It's also actionable.

Because once you know the answer, you can decide whether each gap deserves:

a browser test
an API test
a contract test
a component test
a manual exploratory test
or no test at all

Now you're designing a testing strategy.

Not just collecting automation.

The goal isn't maximum automation

It's useful to remember what all of this is for.

The goal isn't:

10,000 tests

The goal is:

Fewer important surprises in production

Those are not the same thing.

You can have a massive automation suite that repeatedly confirms the obvious.

Or a smaller one that aggressively probes the transitions where your system is most likely to break.

If I had to choose, I'd take the second one.

So before you add another 200 tests this quarter, try something simpler.

Draw the important states in your product.

Draw the transitions between them.

Then put your current automated coverage on top.

You might discover that the gaps aren't where you thought they were.

And you might also discover that your impressive test count has been measuring the wrong thing.

Most Flaky Tests Aren’t Flaky. Your Test Environment Is Lying to You.

Antoine Dubois — Fri, 02 Oct 2026 17:59:50 +0000

We use the word flaky way too casually in test automation.

A test passes nine times.

Then it fails.

Someone reruns it.

It passes.

And the verdict is:

"Eh. Flaky test."

Ticket closed.

Or worse, somebody adds another two-second wait and commits it.

await page.waitForTimeout(2000);

Excellent.

Software engineering.

The uncomfortable reality is that a lot of supposedly flaky tests aren't random at all.

They're deterministic failures caused by conditions we forgot to control.

Browser state.

Cache state.

Session state.

Network timing.

Webhooks.

Timezone.

Locale.

Scroll position.

Service workers.

And increasingly, AI-generated changes to tests that nobody has really inspected.

The browser knows exactly why the test failed.

We just didn't collect enough information to figure it out.

State is probably guilty until proven innocent

Here's a test automation debugging rule I've developed over the years:

If something passes locally, fails in CI, passes when retried, and nobody can explain why...

Look at state.

Not selectors.

State.

Browser automation is basically one giant experiment where we keep trying to pretend every execution starts from the same universe.

It usually doesn't.

Consider something as ordinary as authentication.

A browser context may contain:

cookies
local storage
session storage
IndexedDB
cached assets
service worker data
authentication tokens

Change one of those and you can get a completely different application.

This gets especially interesting with Playwright because browser contexts make isolation convenient enough that it's easy to assume isolation is perfect.

It isn't always.

This guide on debugging Playwright tests that fail after browser context storage changes covers exactly this class of problem.

And it's worth understanding because the failure usually doesn't look like a storage failure.

It looks like:

Expected element to be visible.
Element was not found.

The selector wasn't necessarily wrong.

The user may simply have landed in a different application state.

Service workers make this even more fun

If you really want to spend an afternoon questioning your career choices, debug a cache-related failure involving a service worker.

You can have:

Browser
   ↓
Service Worker
   ↓
Cache
   ↓
Network

Which means what your automation sees isn't necessarily what your server just deployed.

Maybe the new JavaScript bundle exists.

Maybe the browser is still using the old one.

Maybe the service worker has updated but isn't controlling the current page.

Maybe the cache invalidation worked on Chrome 1 but not Chrome 2.

Maybe your "offline" test isn't actually offline because something was already cached.

These are all legitimate application behaviors.

They're also easy to mislabel as automation flakiness.

There's a useful walkthrough on testing service worker updates, cache invalidation, and offline fallbacks without carrying flaky browser state between runs.

This is exactly the kind of scenario I'd include when evaluating a browser automation platform.

Not:

"Can it click the Login button?"

Everything can click the Login button.

Until a sticky banner covers it.

Speaking of things covering buttons...

Modern browser UIs are surprisingly hostile to automation.

Sticky headers.

Floating chat widgets.

Cookie banners.

Animated navigation.

Lazy-loaded content.

Infinite scroll.

Responsive layouts.

A perfectly valid element can exist in the DOM and still be impossible to click.

You might have:

Target element:
x = 420
y = 87

Unfortunately, your sticky navigation bar occupies:

y = 0 → 96

Your automation framework sees the button.

The user visually sees the button.

The click lands somewhere else.

Then somebody "fixes" the problem with:

await page.locator('#save').click({ force: true });

Which is occasionally correct.

And occasionally just hides a real user-facing problem.

There's a practical guide to testing sticky headers, scroll jank, and element occlusion without creating flaky click assertions.

The important distinction is:

"The automation couldn't click it"

and

"A user couldn't reliably click it"

are not always the same bug.

Good testing infrastructure should help you distinguish between them.

Sleep is not synchronization

Another common source of "flakiness":

sleep(5000);

This means:

"I have absolutely no idea when the thing will happen, but five seconds feels emotionally safe."

Sometimes five seconds is enough.

Sometimes the event takes six.

Congratulations.

You now have a flaky test.

Webhooks are a perfect example.

Imagine:

Browser action
     ↓
Backend request
     ↓
Payment provider
     ↓
Webhook
     ↓
Worker
     ↓
Database update
     ↓
UI refresh

How long does that take?

The wrong answer is:

Probably three seconds.

The better answer is:

Wait for evidence that the expected event happened.

That could mean polling an API, waiting for a database condition through a test helper, checking an event endpoint, or verifying a known downstream effect.

There's a good breakdown of testing webhooks in QA without relying on fragile sleep-based assertions.

This sounds obvious when written out.

And yet I still regularly see mature automation suites containing:

Wait 5 seconds
Wait 10 seconds
Wait 3 seconds

At some point you're not testing the application anymore.

You're testing whether your timing guesses were lucky.

Time itself is state

Here's another fun one.

A test passes in Bucharest.

It fails on a cloud browser in Virginia.

The page says:

October 3

The assertion expects:

October 2

Both can be correct.

Welcome to timezones.

International applications introduce several hidden inputs:

browser timezone
server timezone
operating system locale
browser language
daylight saving rules
currency formatting
decimal separators
12-hour vs 24-hour time
date ordering

Even a simple price can become:

$1,234.56
€1.234,56
1 234,56 €

depending on context.

Hardcoding rendered strings is a wonderful way to create tests that pass everywhere except where your users live.

This guide on testing timezone, locale, and currency rendering without flaky browser assertions gets into these problems in more detail.

Locale testing is one of those things teams postpone until they acquire international customers.

Then suddenly it's a release blocker.

Role-based applications multiply the problem

Now take everything above and add permissions.

Imagine an admin platform with:

Super Admin
Admin
Manager
Editor
Viewer
External Collaborator

Each role may see a different navigation.

Different buttons.

Different API responses.

Different data.

Different redirects.

Different session rules.

And now you've multiplied your state space significantly.

Testing these systems isn't just about whether your automation tool supports Chrome.

It's about whether you can reliably create isolated sessions and understand failures when one role behaves differently from another.

This breakdown of browser testing platforms for role-based admin panels, session state, and debuggable failures highlights some useful criteria.

For complex business applications, I'd care a lot about this.

A demo recording a checkout flow tells you very little about how the platform behaves with six permission levels and stale sessions.

And after deployment, you're still testing

There's another mistake I see fairly often.

Teams think:

CI passed → deployment successful

Those are different statements.

CI verifies one environment.

Your users interact with another.

After deployment you can still encounter:

bad environment variables
broken CDN assets
DNS issues
expired credentials
missing migrations
third-party outages
routing failures
region-specific problems

Which is why synthetic monitoring is basically test automation that refuses to go home after deployment.

A useful synthetic check should do more than scream:

CHECKOUT FAILED

You want to know where it failed and whether rollback might actually help.

There's a useful overview of selecting synthetic monitoring for post-deploy smoke checks, rollback verification, and API failure triage.

For important flows, the lifecycle really looks more like:

Build
  ↓
Automated tests
  ↓
Deploy
  ↓
Synthetic checks
  ↓
Real users

Your release doesn't become correct just because GitHub Actions turned green.

AI test repair adds an interesting economic problem

AI is now starting to "solve" flaky automation too.

A locator changes.

The AI repairs it.

The test keeps running.

Sounds great.

Sometimes it is.

But there's a question I think gets overlooked:

How much autonomous repair should happen before a human should review what's changing?

Imagine an agent repairs 100 tests.

Maybe 97 repairs are perfect.

Maybe three subtly changed what the test means.

For example:

Original:
Click "Delete User"

Repair:
Click "Disable User"

Both controls might exist in roughly the same place.

The test could continue successfully.

And you've now created something much worse than a failed test:

A passing test that's testing the wrong thing.

That's why I like the framing in this benchmark plan comparing autonomous test repair cost against human review time.

The metric shouldn't simply be:

How many tests did the AI repair?

It should also include:

repair cost
+
review cost
+
incorrect repair risk
+
time saved

That's a much more useful business equation.

The test itself is becoming an asset

There's another question teams don't ask early enough:

What happens if we leave this testing platform?

This mattered less when nearly everything was code.

A Selenium test was Python, Java, C#, JavaScript, whatever.

You owned it.

Now we're seeing more proprietary:

visual workflows
AI-generated models
natural-language tests
hosted execution formats
platform-specific steps
proprietary repair metadata

Some of these abstractions are valuable.

But abstraction has a price.

If you create 4,000 tests over three years, switching platforms becomes an entirely different conversation.

That's why measuring test artifact portability across browser clouds, open-source frameworks, and AI testing platforms is a worthwhile benchmark.

I'd want to know:

Can I export my tests?
In what format?
Can another system execute them?
Can humans understand them?
What happens to assertions?
What happens to screenshots and historical results?
What happens to AI-generated metadata?

Nobody worries about vendor lock-in when they have 20 tests.

They start worrying when they have 20,000.

Managed QA versus self-service AI is becoming another dividing line

We're also seeing two very different approaches emerge in AI testing.

One says:

Give us responsibility for the outcome.

The other says:

Here's powerful automation. You operate it.

Those are fundamentally different products even if both put "AI testing" on their homepage.

A managed service can remove maintenance work.

A self-service platform can give your team more direct control and faster iteration.

Neither model universally wins.

This comparison of QA Wolf vs QA.tech frames the difference around managed maintenance versus self-service browser coverage.

And if you're trying to understand the broader market, this list of the 10 best AI-powered test automation tools also gives a useful overview of how different AI testing products are approaching the problem.

Personally, I think category labels like "AI testing platform" are becoming less useful.

The more important questions are:

Who creates the tests?
Who maintains them?
Who investigates failures?
Who owns the artifacts?
Who decides when an AI repair is correct?

Those answers tell you much more about a platform than whether it has a chatbot.

My favorite flaky-test debugging trick

When a test fails intermittently, don't immediately modify it.

First try to classify the hidden variable.

Ask:

What changed between the passing run
and the failing run?

Check:

Browser state
Application state
Network state
Test data
Browser version
Viewport
Locale and timezone
Timing
Third-party dependencies
Deployment version

Then compare evidence from the runs.

The difference is usually somewhere in that list.

And if you don't have enough evidence to compare two executions?

That's the actual problem you should fix first.

Because adding:

await page.waitForTimeout(5000);

might make your dashboard greener.

But it doesn't mean your test suite became more reliable.

It just means you taught the failure to hide better.

Flakiness is information

This is probably the biggest mental shift.

A flaky test isn't merely annoying noise.

It's information.

Sometimes it tells you your test is poorly synchronized.

Sometimes it tells you your application behaves unpredictably.

Sometimes it tells you environments aren't isolated.

Sometimes it tells you your infrastructure behaves differently under load.

Sometimes it tells you that you've accidentally discovered a real race condition.

So before deleting the test, increasing the timeout, or clicking Retry for the sixth time, ask:

What condition makes this failure possible?

Because computers aren't usually rolling dice behind your back.

Something changed.

The job of good test automation isn't just to detect the failure.

It's to leave enough evidence behind that you can figure out what changed.

Most Test Failures Aren’t Really About the Test

Antoine Dubois — Tue, 25 Aug 2026 07:53:51 +0000

One of the weird things about test automation is that a failure rarely means just one thing.

A red test can mean:

the product is broken
the test is broken
the environment is different
the timing changed
some stale state leaked in
the browser behaved differently
or the test uncovered something nobody had thought about yet

That last category is usually the interesting one.

For example, Playwright tests that fail only when microtasks, timers, and API mocks interleave differently in CI are a good reminder that “works locally” is not much of a debugging strategy.

Accessibility testing has the same problem. Finding an issue is only step one. Evaluating AI accessibility testing tools also means looking at severity, triage, dynamic UI changes, and whether the results can actually influence a release decision.

Then there’s infrastructure.

Sometimes you don’t need another framework. You just need the right place to reproduce the problem. This BitBar vs Browserling comparison is useful because different browser clouds optimize for very different workflows.

And testing shouldn’t stop when deployment finishes.

This market map of synthetic monitoring and post-deploy browser smoke platforms looks at the tools teams can use when they need a fast answer to a very simple question:

Did we just break production?

Even browser state deserves more attention than it gets. Testing IndexedDB, local storage, and session resets sounds mundane until you realize bad cleanup can make your tests less realistic.

The common thread here is simple:

Reliable testing is rarely about adding more assertions.

It’s about understanding everything surrounding the assertion — timing, state, environment, infrastructure, and what happens after deployment.

That’s where most of the interesting bugs are hiding.

The Best Test Automation Tool Is Usually the One That Matches Your Mess

Antoine Dubois — Wed, 19 Aug 2026 22:00:42 +0000

Most test automation buying decisions start with a spreadsheet.

Browser testing? Check.

Mobile? Check.

API testing? Check.

AI? Check.

CI integration? Check.

Then somebody adds 40 more rows and the team spends three weeks comparing products that all appear to do roughly the same thing.

This is usually where the evaluation goes wrong.

The important question isn't:

Which tool has the most capabilities?

It's:

Where does our testing complexity actually live?

Because every team has a different kind of mess.

One company has a browser-first SaaS product and wants to stop maintaining a large Playwright framework.

Another has SAP, Windows desktop applications, APIs, and a mobile app that has been around for eight years.

Another has a modern web product, but half of its important workflows happen inside Stripe, embedded support widgets, rich-text editors, and third-party iframes.

Those teams should probably not buy the same thing.

Start with the surface area you actually have

A surprisingly common mistake is buying for hypothetical complexity.

"We might need desktop testing someday."

"We could have a native mobile application next year."

"We should probably support every possible protocol."

Maybe.

But you're paying for those decisions now.

If 90% of your regression work is browser-based, it makes sense to evaluate tools primarily around browser workflows, maintenance, debugging, parallel execution, and how quickly the rest of the team can work with the tests.

That's why comparisons such as Endtest vs Keysight Eggplant for browser-first teams are more useful when you read them through the lens of your actual environment rather than trying to declare a universal winner.

A browser-first SaaS team and a large enterprise testing packaged desktop software have different problems.

And that distinction matters.

On the other end of the spectrum, there are organizations where breadth really is the requirement.

If a company needs to test desktop applications, APIs, mobile apps, and legacy enterprise workflows from one stack, something like OpenText UFT One may make sense in a way that a browser-focused platform doesn't.

The lesson isn't that broad tools are bad or focused tools are better.

It's that unused flexibility is still complexity.

You maintain it.

You train people on it.

You pay for it.

And sometimes you design your entire QA process around capabilities you barely use.

The hard parts are usually at the edges

The happy-path demo is almost never where a testing tool struggles.

Click button.

Fill form.

Assert text.

Every serious browser testing product can show you a nice version of that.

The better evaluation is to pick the ugly workflows.

The ones that caused three bugs last quarter.

The ones people quietly exclude from regression because automation became annoying.

For example: embedded content.

Modern web applications increasingly contain payment providers, chat widgets, video players, authentication components, analytics tools, and other functionality hosted outside the application's origin.

That means cross-origin boundaries, sandbox restrictions, iframe switching, timing issues, and behavior that your application doesn't fully control.

If that describes your product, you should deliberately test cross-origin iframes, embedded widgets, and third-party sandboxes during a proof of concept.

Don't ask the vendor whether they support iframes.

Give the product your worst iframe.

That's a much better test.

The same applies to rich-text editing.

A contenteditable field looks like a text box until you automate it.

Then users paste formatted HTML from Google Docs, the browser normalizes whitespace, the editor inserts spans, keyboard shortcuts behave differently, sanitization removes markup, and undo history depends on how the content was inserted.

A test that compares the entire resulting DOM can become brittle almost immediately.

A better approach is to understand how to test contenteditable editors, paste sanitization, and undo behavior and then see how naturally the tool supports those patterns.

Again, don't test the brochure.

Test your mess.

AI makes the demo problem worse

AI has made test automation demos much more impressive.

You type:

Test the checkout flow.

Twenty seconds later, something is clicking through the application.

That's legitimately useful.

But it can also hide the most important part of the evaluation.

The first run isn't where AI testing gets expensive.

The interesting part is run 47.

The UI changed.

A selector stopped working.

A modal appears only for some accounts.

The test gets redirected.

A button moved.

A network request takes four seconds instead of one.

Now what?

Does the system recover consistently?

Does it make the same decision twice?

Can you see what it changed?

Can a human override it?

Does recovery fix the actual problem, or merely find some path that gets the test to green?

These questions are why I like the idea of using an explicit benchmark plan for comparing AI browser test tools on repeatability, recovery, and maintenance cost.

A useful benchmark doesn't ask each tool to create five tests and then award points for speed.

It should run those tests repeatedly.

Then intentionally break things.

Change labels.

Move elements.

Introduce latency.

Modify test data.

Add an intermediate dialog.

Run the same recovery scenario multiple times.

What you want to learn is not whether the AI can solve a problem once.

It's whether the behavior is predictable enough to trust.

Measure maintenance, not magic

There is a simple metric that rarely appears in software evaluations:

How many human minutes did this test require over the last 90 days?

Not execution minutes.

Human minutes.

Time spent:

updating selectors,
diagnosing failures,
rerunning pipelines,
fixing test data,
maintaining infrastructure,
reviewing AI-generated changes,
explaining the framework to new team members.

That number tells you far more about the true cost of automation than the license price.

A "free" framework can be expensive.

An enterprise platform can also be expensive.

A no-code tool can become expensive if every unusual workflow requires vendor support.

An AI tool can become expensive if nobody trusts its recovery decisions and every healed test has to be manually reviewed.

This is why tool comparisons should include operational cost, not just feature coverage.

The cheapest tool is often the one that creates the least recurring work for your particular team.

Run a proof of concept that tries to make the tool fail

Most proofs of concept are too polite.

The vendor helps configure the environment.

The team selects five clean workflows.

Everybody watches them pass.

Then the contract gets signed.

Six months later, the real application shows up.

Instead, I'd build a small evaluation suite containing the nastiest representative workflows you have.

Maybe:

A straightforward login and checkout flow.
A cross-origin iframe.
A file upload or download.
A contenteditable component.
A workflow with changing test data.
A test with deliberate latency.
A scenario where a locator is intentionally broken.
A workflow that fails halfway through and needs useful debugging evidence.

Run them repeatedly.

Then ask your QA engineers and developers a much simpler question:

Which system would you rather debug at 4:45 PM on a Friday?

That's probably more useful than another 80-row feature matrix.

Buy for the next two years, not the next ten

You should absolutely think about growth.

But there's a difference between planning ahead and buying an aircraft carrier because one day you might need to cross a lake.

If you're primarily building a web application, optimize for excellent browser testing.

If you're running a heterogeneous enterprise stack with desktop, mobile, and APIs, breadth may be worth the additional complexity.

If your UI depends heavily on third-party components, evaluate those specifically.

If AI maintenance is a major selling point, benchmark the maintenance rather than the test generation.

The best test automation platform isn't the one with the longest feature page.

It's the one whose strengths overlap with the things that make your application difficult to test.

Everything else is inventory.

And in software, inventory has carrying costs.

Most “Flaky Tests” Are Really Architecture Tests

Antoine Dubois — Sat, 08 Aug 2026 21:26:37 +0000

There’s a point in almost every browser automation project where someone says:

“The tests are getting flaky.”

And then the team starts fixing the tests.

Longer waits. More retries. New selectors. Another helper function. Maybe a wrapper around the wrapper around click().

Sometimes that’s the right answer.

But increasingly, I think “flaky test” is becoming a catch-all phrase for something much broader:

Modern frontends have become surprisingly difficult environments to observe deterministically.

The test isn’t always the problem.

The DOM Isn’t Really the DOM Anymore

A decade ago, browser automation had a fairly straightforward mental model.

Load page. Find element. Click element.

Today you might be dealing with:

Shadow DOM
nested iframes
third-party widgets
microfrontends
virtualized lists
components that render differently based on their container size
UI updates arriving over WebSockets
client-side navigation without full page loads

That’s a different problem.

For example, testing Shadow DOM, nested iframes, and embedded widgets is less about writing the perfect selector and more about whether your automation system understands the boundaries between those different contexts.

A perfectly valid selector can still be useless if your test is searching in the wrong DOM context.

Responsive Testing Got More Complicated Too

We used to think about responsiveness in terms of viewport width.

Desktop. Tablet. Mobile.

CSS container queries changed that assumption.

A component can now change dramatically while the browser window stays exactly the same size.

That means testing only a handful of viewport resolutions can miss real regressions. There’s a useful breakdown of testing container queries, resize behavior, and breakpoint edge cases that illustrates how these failures tend to appear.

This is one of those changes that looks small from the CSS side and surprisingly large from the QA side.

Then There’s Time

A lot of UI state isn’t created by page load anymore.

It arrives later.

Consider an application consuming WebSockets or Server-Sent Events.

The test opens the page.

The page says “Connected.”

The test waits for the transaction.

Nothing happens.

Failure.

Run it again.

Pass.

This is where arbitrary sleeps become dangerous. You’re trying to fix a state synchronization problem with a stopwatch.

A better approach is understanding how WebSocket, SSE, and live-data browser tests fail and synchronizing against meaningful application state instead of elapsed time.

The same principle applies to SPAs.

Browser history, back-button behavior, restored scroll positions, and cached component state can produce bugs that don’t appear when every automated test starts from a clean URL.

That makes client-side routing and scroll restoration worth treating as first-class behavior rather than navigation trivia.

Selectors Still Matter. Just Differently.

I don’t think selectors are going away.

But I do think teams often spend too much time trying to make one clever selector survive every imaginable frontend change.

That becomes especially painful with microfrontends.

One team renames a component.

Another team introduces a wrapper.

A shared UI package changes its DOM structure.

Suddenly 87 tests fail.

A sane selector strategy for microfrontends and shared component libraries is mostly about establishing contracts between developers and tests.

That’s much more scalable than XPath archaeology.

Virtualized Interfaces Are Another Trap

Infinite scroll and virtualized tables are great examples.

An item can conceptually exist in the application while not existing in the DOM.

That distinction matters.

It’s why comparing approaches such as Endtest vs Playwright for heavily virtualized interfaces gets interesting. The question isn’t merely whether the tool can click an element.

The question is how much machinery you have to build around that click.

The Useful Question

When a browser test fails, I’d stop asking:

“How do we make this test pass?”

At least initially.

Ask:

“What assumption did this test make about the application?”

Maybe it assumed the element was already rendered.

Maybe it assumed the route caused a page load.

Maybe it assumed the item existed in the DOM.

Maybe it assumed the component responded to viewport width.

Maybe it assumed there was only one document context.

Once you find that assumption, the failure usually becomes much less mysterious.

And strangely enough, your tests tend to become much less flaky too.

Visual Regression Testing Has a Noise Problem

Antoine Dubois — Fri, 07 Aug 2026 22:07:21 +0000

Visual regression testing sounds almost unfairly simple.

Take screenshot A.

Take screenshot B.

Compare them.

If they’re different, something changed.

Unfortunately, browsers have spent the last decade making “something changed” a surprisingly complicated concept.

Fonts render differently.

Content shifts by a pixel.

Animations fire.

Responsive layouts reflow.

Container queries change components independently of the viewport.

Web Components hide half the page inside Shadow DOM.

Suddenly your visual testing system is producing hundreds of differences, and the person reviewing them develops the highly sophisticated QA technique known as:

clicking Accept repeatedly until the red badges disappear.

At that point, you technically have visual regression testing.

You just don’t have much visual regression detection.

Signal matters more than sensitivity

The instinct when evaluating screenshot tools is to ask:

How precisely can it detect pixel differences?

That’s not the question I’d start with.

A tool that detects every single pixel difference may be worse than one that ignores harmless variation.

The real question is:

How quickly can a human distinguish an actual regression from noise?

Those are very different product requirements.

A useful evaluation should include baseline management, masking, thresholds, rendering consistency, review workflows, and what happens when the expected UI legitimately changes.

This guide to evaluating browser testing tools for visual diff noise, baseline drift, and screenshot review gets into those details.

Baseline drift is especially dangerous.

If developers routinely approve new screenshots without understanding the differences, your baseline slowly stops representing “correct.”

It represents “whatever the application looked like the last time someone got tired of reviewing screenshots.”

Fonts are chaos disguised as typography

Fonts deserve their own section because they cause a ridiculous amount of visual noise.

A font loads late.

The fallback font has slightly different metrics.

A headline becomes one pixel wider.

A card becomes three pixels taller.

Everything below it moves.

Your visual comparison now tells you that half the page changed.

Technically, it’s right.

Operationally, it’s useless.

The same issue shows up with:

dark mode,
themes,
localization,
dynamic content,
personalization,
responsive typography.

There’s a good exploration of this in what to evaluate in a visual regression tool for dynamic fonts, theming, and layout shifts.

A strong visual test system needs to understand that not every difference has equal importance.

A missing checkout button and a one-pixel font shift should not create identical urgency.

Responsive testing isn’t testing three viewport widths

For years, responsive testing meant:

desktop,
tablet,
mobile.

That model is increasingly incomplete.

CSS Grid and Flexbox often fail at awkward widths that nobody specifically designed.

A navigation menu might look perfect at 1440px and 375px but wrap badly at 847px.

A product card might collapse because one translation is longer than the English version.

These are the bugs users actually see.

This walkthrough of testing CSS Grid, Flexbox, and responsive wrapping without missing layout breakpoints makes an important point: meaningful breakpoints often come from the content, not from your CSS framework.

And container queries push this even further.

A component’s layout may now depend on its parent container rather than the browser viewport.

You can have two copies of the same component on the same page behaving differently.

That makes old-school “set browser width, take screenshot” strategies increasingly inadequate.

If your frontend uses them heavily, testing CSS container queries without creating visual noise is worth a look.

Then Shadow DOM arrives

Shadow DOM solves legitimate frontend architecture problems.

It also introduces a fun testing problem:

The element you can clearly see in the browser may not exist where your automation expects it to exist in the DOM tree.

That can affect:

selectors,
interactions,
accessibility checks,
screenshots,
debugging,
reusable page objects.

If your application uses a large component library based on Web Components, this shouldn’t be an edge-case checkbox in your tool evaluation.

It should be part of the proof of concept.

There’s a useful checklist for evaluating test automation platforms for Shadow DOM and encapsulated UI libraries.

And don’t just test a single toy component.

A framework being technically capable of entering a shadow root doesn’t tell you much about how it performs against a real application containing hundreds of components.

I like the more empirical approach described in benchmarking browser test stability on Shadow DOM-heavy frontends.

Run enough representative tests to discover where the maintenance cost actually appears.

Screenshot count is a terrible KPI

One final trap: measuring visual-test maturity by the number of screenshots.

“We have 4,000 screenshot assertions” sounds impressive.

It might also mean four engineers are approving meaningless diffs every morning.

The useful metrics are closer to:

regressions caught before production,
false-positive rate,
review time,
baseline churn,
percentage of visual checks people still trust.

That last one is hard to put on a dashboard.

But it might be the most important.

Because visual regression testing is only useful as long as humans believe the red pixels mean something.

The moment they stop believing that, you have a very expensive screenshot generator.

Most CI Failures Are Environment Failures

Antoine Dubois — Wed, 05 Aug 2026 19:09:02 +0000

When a browser test passes locally and fails in CI, the first diagnosis is often “flaky test.”

That diagnosis is convenient because it does not require us to explain anything.

Rerun the job. Add a retry. Increase the timeout. Move on.

Sometimes that is enough to unblock the pipeline. But it also teaches the team to treat unexplained behavior as normal.

Most CI-only failures are not random. They are deterministic reactions to differences we have not made visible yet.

Parallelism changes the system

A test suite can behave perfectly when executed one test at a time and collapse under parallel execution.

That is not necessarily a browser problem.

Parallel jobs introduce competition for:

CPU
Memory
Network bandwidth
Shared test accounts
Database records
File names
Ports
Rate-limited APIs
Preview environments

This guide to debugging GitHub Actions browser jobs that fail under parallelism is a good starting point because it treats parallelism as a change in system behavior, not merely a faster way to run the same tests.

Two tests using the same account may invalidate each other’s sessions.

Two jobs may update the same order.

A service that responds in 300 milliseconds locally may take three seconds when eight CI workers hit it at once.

The failure may appear in the browser, but its cause can live anywhere in the environment.

Look for predictive signals

Teams often wait until the pipeline is red before collecting diagnostic information.

By then, the most useful transient evidence may already be gone.

This article on signals that predict browser-test failures before CI turns red suggests watching the conditions around the test, not only its final status.

Useful leading indicators include:

Increasing page-load time
More locator retries
Slower API responses
Rising console-error counts
Higher memory use
Longer queue times
Increased test-data conflicts
Frequent browser restarts

A test that still passes after six retries is not healthy.

It is an early warning.

Preserve enough evidence to reconstruct the run

A screenshot is helpful, but it is rarely sufficient.

The browser may show an empty state because the API failed, the session expired, the frontend crashed, or the test reached the wrong route.

A practical diagnostic package should include:

Screenshot
Current URL
Console logs
Network failures
Browser and operating-system version
Viewport
DOM or page-source snapshot
Failed locator
Timing information
Test-data identifiers

This guide on diagnosing browser failures with console logs, network timing, and DOM snapshots explains why these artifacts are more valuable together than independently.

A console error can explain the blank screenshot.

A network trace can explain the console error.

A DOM snapshot can reveal that the expected element existed but was hidden behind a loading state.

The goal is not to collect everything. It is to preserve the shortest path to an explanation.

Playwright is not immune to CI drift

Playwright does a lot to make browser automation more reliable, including automatic waiting and strong browser tooling.

But it still runs inside an environment.

This guide to debugging Playwright tests that fail only in CI covers the differences that still matter:

Headless versus headed behavior
Browser versions
Fonts
Locale and timezone
Missing secrets
Viewport dimensions
CPU throttling
Network conditions
Container permissions

The right question is not, “Why is Playwright flaky?”

It is, “Which assumption in this test is different in CI?”

That wording changes the investigation.

Real-time flows expose weak waits

WebSocket and real-time applications are particularly good at revealing timing assumptions.

There may be no navigation to await. The UI changes when an event arrives, and that event may depend on another service completing work.

This guide on testing WebSocket and real-time UI flows without phantom failures recommends waiting for meaningful state rather than arbitrary time.

Do not sleep for two seconds and hope the message arrives.

Wait until:

The status changes
The expected item appears
The notification contains the relevant event
The backend confirms the state
The UI stops showing an intermediate condition

A fixed delay is a guess disguised as synchronization.

Sessions and tabs are environment state too

Multi-tab and cross-domain flows create additional failure modes.

A test may open a payment provider, authenticate in a second tab, return to the original application, and expect the session to persist.

Locally, the browser may already contain cookies or cached permissions. In CI, it starts from a clean environment.

This guide to evaluating browser tools for multi-tab, session-persistence, and cross-domain workflows highlights why these workflows should be part of tool evaluation.

A platform may handle simple single-page tests well and still struggle when browser context becomes part of the scenario.

Build a failure narrative

At Endtest, we have been working toward results that show the failed step, the surrounding logs, screenshots, and the information needed to investigate what happened. AI-assisted failure analysis can help suggest a likely cause, but it should always point back to the evidence.

That is the standard I would use for any test system.

A failure result should tell a coherent story:

What did the test attempt?

What did the browser observe?

Which environment conditions mattered?

What changed from the last successful run?

Who should investigate next?

Retries can keep a pipeline moving.

Evidence makes the system better.

Your CI Failure Probably Is Not Random

Antoine Dubois — Tue, 04 Aug 2026 22:20:28 +0000

There is a particular kind of browser test failure that can consume an entire afternoon.

The test passes on your laptop.

It passes when you rerun it locally.

It may even pass when you SSH into the CI machine and execute it manually.

But inside the real pipeline, it fails.

The easiest explanation is that the test is flaky. Sometimes that is true. More often, “flaky” is just a word we use when we have not identified the environmental difference yet.

Start by assuming the environments are different

Your laptop and the CI runner are not the same environment.

Even when they use the same browser version, they may differ in:

CPU availability
Network latency
Screen dimensions
Font availability
Locale
Timezone
Cached assets
Test data
Browser permissions
Feature flags
Secrets and authentication state

This checklist for E2E tests that fail only in CI is a useful place to begin because it separates timing, data, and environment drift instead of treating every failure as another waiting problem.

Adding a larger timeout can make the test pass. It does not necessarily fix the cause.

Sometimes it simply turns a five-second mystery into a thirty-second mystery.

Time is test data

Date-sensitive applications are especially good at producing failures that look random.

A billing screen may behave differently near the end of the month. A calendar may calculate the first day of the week from the browser locale. A promotion may expire according to the server timezone rather than the browser timezone.

The problem becomes more confusing when the developer, CI service, and production environment are in three different regions.

This guide on testing locale, timezone, and calendar-dependent interfaces recommends treating time configuration as an explicit part of the test rather than an ambient property of the machine.

That means recording the timezone and locale in the test result. It also means using fixed dates when the workflow permits it.

A screenshot showing “March 31” is not enough evidence if you do not know which timezone produced it.

Real-time interfaces need state-based waits

WebSocket-driven interfaces create a different class of timing problem.

A user performs an action, the server processes an event, and the browser updates when a message arrives. There may be no navigation and no obvious network request for the automation framework to wait for.

Sleeping for two seconds works until the server needs three.

This article on testing WebSocket-driven UI flows without chasing race conditions makes the stronger approach clear: wait for the product state you actually care about.

Do not wait because “the UI probably finished.”

Wait until:

The expected notification appears
The status changes
The item enters the correct list
The relevant event has been reflected in the DOM
A specific backend state can be verified

A test should synchronize with meaning, not time.

Rendering problems often masquerade as assertion failures

A visual assertion can fail even when the underlying product behavior is correct.

Fonts may load later in CI. Images can be decoded at a different speed. Cookie banners may alter the viewport. A container may shift after a client-side component initializes.

The guide to debugging layout shift before it becomes visual test flakiness is relevant even for teams that do not run screenshot comparisons.

Layout shifts can cause ordinary click failures too. The test identifies the correct element, but another component moves it between locating and clicking.

Before blaming the automation framework, check whether the page is visually stable.

Performance budgets should be selective

Performance checks belong in CI, but running a complete performance audit on every change can make developers resent the pipeline.

That usually leads to one of two outcomes:

The checks are ignored.
The checks are removed.

A better approach is described in this article on enforcing frontend performance budgets without slowing every merge.

Use lightweight checks for every pull request and reserve expensive measurements for relevant changes, scheduled builds, or release candidates.

The objective is not to collect every possible metric. It is to catch meaningful regressions while developers can still act on them.

Pull requests are their own environment

One of the more confusing CI patterns is a test that fails on pull requests but succeeds after the same code reaches the main branch.

This is often caused by configuration rather than application behavior:

Forked pull requests may not receive secrets.
Preview environments may use different URLs.
Pull-request workflows may have different permissions.
The branch may be tested before dependent assets are deployed.
Cache keys may differ.
Conditional workflow steps may run only on the default branch.

This analysis of GitHub Actions browser tests that fail only on pull requests is a reminder to compare workflow execution paths, not only test code.

The same test command does not guarantee the same test environment.

Build a failure package, not just a failure message

“Element not found” is barely useful as a diagnostic result.

A useful CI failure should preserve enough evidence to reconstruct what the browser experienced:

Screenshot
Current URL
Browser and operating system
Viewport size
Console errors
Relevant network failures
Page source or DOM snapshot
Locale and timezone
Test data identifiers
The exact failed step

This is where modern automation platforms can save significant time. Whether you use a managed platform such as Endtest or your own Playwright or Selenium infrastructure, the run should produce an investigation package.

The goal is not merely to know that the test failed.

The goal is to reduce the distance between failure and explanation.

Most CI-only failures are not random. They are deterministic reactions to variables the team has not made visible yet.

The best debugging improvement is often not another retry.

It is better evidence.

Your CI Pipeline Does Not Need More Tests. It Needs Better Decisions

Antoine Dubois — Mon, 03 Aug 2026 21:25:34 +0000

It is easy to measure the size of a test suite.

You can count test cases, browsers, devices, assertions, executions, and minutes of runtime.

It is much harder to measure whether the suite helps the team make better decisions.

That is the real job of CI.

A pipeline should help answer a practical question:

Is this change safe enough to move forward?

When the pipeline cannot answer that question clearly, teams compensate by adding more tests, rerunning failures, and asking developers to inspect increasingly large logs.

The result is slower CI without greater confidence.

Stop running everything because it feels safer

Running every automated test for every code change sounds responsible.

It is also frequently wasteful.

A documentation update probably does not need the same validation as a change to authentication, billing, or shared application state. Yet many pipelines treat them identically because the organization has no reliable way to estimate impact.

This creates two problems.

First, feedback becomes slow. Developers wait for unrelated tests before learning whether their change is usable.

Second, important failures become harder to notice because they are mixed into a much larger volume of routine execution.

Test impact analysis attempts to solve this by mapping changes to the areas, dependencies, and user flows most likely to be affected.

A practical guide to building a test impact analysis workflow for faster CI/CD decisions explains how to approach this without pretending the mapping will be perfect from day one.

You do not need a flawless prediction engine.

You need a process that is better than “run absolutely everything and hope someone reads the result.”

A failure without context is barely useful

A browser test fails on step 17.

The screenshot shows a loading spinner.

What happened?

Perhaps the API was slow. Perhaps a JavaScript exception prevented rendering. Perhaps the previous click never registered. Perhaps the test reached the page with an expired session.

A single screenshot rarely answers the question.

This is where session replay becomes useful. A good replay can show the sequence of interactions, DOM changes, console errors, network requests, redirects, and timing leading up to a failure.

But simply recording video is not enough.

A useful debugging workflow should connect the replay to the failed step and make it easy to compare the expected path with the actual one.

This article on building a browser session replay debugging workflow for flaky UI tests offers a practical framework.

The important shift is to design replay around diagnosis rather than archiving.

Nobody benefits from terabytes of test recordings that are difficult to search and rarely opened.

Webhooks need first-class evidence

Webhook failures are a perfect example of CI ambiguity.

The application performs an action. A remote service is supposed to receive an event. The pipeline times out.

Now the investigation begins.

Was the webhook generated? Was it sent? Did DNS fail? Did the receiving endpoint reject the signature? Was the event delivered twice? Did the endpoint accept it but fail during processing?

Without structured evidence, each pipeline run becomes a small forensic project.

A reliable webhook test should preserve:

The event identifier
The payload
The signature headers
Delivery attempts
Response codes
Retry timing
The receiving system’s processing result
Correlation identifiers across services

The guide to testing webhooks in CI without turning every pipeline run into a mystery covers the mechanics in more detail.

The broader lesson applies beyond webhooks: distributed behavior needs distributed evidence.

A browser screenshot cannot explain what happened inside an asynchronous backend workflow.

Flakiness should be detected as a pattern

Most teams notice flaky tests informally.

Someone says, “That one fails sometimes.”

Another person says, “Just rerun it.”

Eventually the workaround becomes part of the culture.

The problem with this approach is that human memory is poor at recognizing gradual changes. A test that failed once every fifty runs may now fail once every twelve, but nobody notices because each failure is handled independently.

Anomaly detection can help by treating test behavior as a time series.

You can monitor:

Changes in failure frequency
Sudden increases in runtime
Browser-specific instability
Failures concentrated on certain workers
New retry dependence
Correlation with deployments or infrastructure changes

This does not require an elaborate machine-learning system. Even a rolling baseline can reveal meaningful changes.

The article on adding flaky-test anomaly detection before developers start ignoring failures provides a useful implementation path.

The earlier you identify deterioration, the cheaper it is to fix.

Once the team has normalized rerunning failures, the technical problem has become a cultural problem too.

Sometimes the test is fine and the grid is slow

Not every inconsistent browser test is caused by a weak locator or missing wait.

Distributed browser infrastructure introduces its own failure modes.

A Selenium Grid may experience latency between the test runner, hub, browser node, application server, and external dependencies. Each individual delay may be small. Combined across hundreds of commands, they can change the timing enough to expose failures.

Signs of grid-related latency include:

Commands becoming slow across unrelated tests
Failures clustering on specific nodes
Timeouts increasing during parallel execution
Large differences between local and remote execution
Browser startup delays
Long gaps between a command and its corresponding application activity

This guide to spotting Selenium Grid network bottlenecks before they become flaky tests explains what to measure.

The important point is diagnostic discipline.

Do not rewrite a stable test because the network is overloaded. Do not increase every timeout because one node is unhealthy. Fix the layer that is actually failing.

CI should reduce uncertainty

A fast pipeline is useful.

A comprehensive pipeline is useful.

But neither matters if the result does not guide action.

The best CI systems are designed around decisions:

Which tests are relevant to this change?
What exactly happened when one failed?
Is the failure new or part of an existing pattern?
Did the application fail, or did the test infrastructure fail?
Can the team proceed safely?

When CI answers those questions, testing becomes a development tool.

When it does not, the pipeline becomes a ceremony everyone waits for and nobody fully trusts.