Most teams calculate the cost of test automation by looking at how long it takes to create the first version of the test suite.
That is usually the least expensive part.
The real cost appears six months later, when the application has changed, the original test author has moved to another project, and nobody is entirely sure whether the failing test represents a product bug or a stale assumption.
I have seen teams spend far more time maintaining automation than they originally spent building it. The problem is rarely a single bad technology decision. It is usually a collection of small decisions that compound.
The wrong locator here. A test with no clear owner there. A self-healing system that quietly changes behavior. Eventually, the test suite becomes another product that the company must maintain.
Difficult interfaces reveal weak abstractions
A simple login form can make almost any automation tool look good.
The more useful evaluation happens when you point the tool at the awkward parts of your application:
- Nested iframes
- Shadow DOM components
- Dynamic tables
- Custom controls
- Reusable web components
- Elements rendered differently across browsers
A good starting point is this guide on evaluating automation tools for Shadow DOM, iframes, and other hard-to-test surfaces.
The goal is not to find a tool that claims to support every frontend technology. Nearly every vendor can create a convincing demo.
The question is what happens when the component changes.
Can someone on the QA team understand the generated locator? Can they replace it without rewriting the test? Does the failure report explain which DOM boundary caused the problem?
Testing modern components also requires a more deliberate approach to selectors. This guide to testing web components and slot-based interfaces without fragile selectors highlights an important principle: tests should target stable product meaning, not incidental implementation details.
That distinction sounds academic until a frontend refactor breaks 80 tests overnight.
Code generation does not eliminate maintenance
Playwright has made browser automation considerably more approachable. But approachable is not the same as maintenance-free.
The hidden maintenance cost of Playwright tests is rarely the initial setup. It is the ongoing engineering work:
- Updating fixtures
- Diagnosing timing problems
- Managing test data
- Reviewing generated code
- Maintaining CI infrastructure
- Teaching new team members how the framework works
None of these are arguments against Playwright. They are arguments against calling the software “free” simply because the license costs nothing.
The same principle applies to AI-generated tests. Generating code quickly is valuable, but speed at creation can hide fragility.
Small wording changes are a common example. This analysis of why AI-generated tests fail after minor UI copy changes explains what happens when an agent uses visible text as both the intent and the locator.
A human understands that “Continue,” “Next,” and “Proceed” may represent the same action. A brittle test sees three unrelated strings.
Self-healing can create silent failures
Self-healing sounds like the perfect solution to locator maintenance.
A button moves, the old selector fails, and the system finds the new button automatically.
But there is a dangerous edge case: the test can heal itself by selecting the wrong element.
That is worse than a normal failure.
A normal failure stops the run and asks someone to investigate. An incorrect repair can allow the test to pass while validating the wrong behavior.
This article on preventing self-healing agents from rewriting the wrong locator outlines the controls I would expect from any serious implementation:
- A visible explanation of what changed
- The old and new locator
- Confidence information
- Screenshots or DOM evidence
- Human approval for ambiguous replacements
- A way to roll back the change
Self-healing should reduce repetitive work. It should not remove accountability.
The owner matters more than the framework
The most reliable test suites I have seen have one thing in common: ownership is obvious.
Someone knows why each important test exists. Someone decides whether a failure should block a release. Someone removes redundant tests instead of letting the suite grow forever.
When ownership is vague, automation slowly turns into an archaeological project.
This guide on auditing test suite ownership before an automation program stalls provides a useful exercise. For each major area, identify:
- Who owns the product behavior?
- Who owns the automated coverage?
- Who investigates failures?
- Who approves major test changes?
- Who decides when a test should be deleted?
If the answer to all five questions is “the QA team,” you probably do not have ownership. You have a queue.
Optimize for understandable automation
This is one reason I prefer automation systems that keep tests readable after AI has helped create them.
For example, the Endtest AI Test Creation Agent can help translate a natural-language instruction into test steps, while the resulting test remains visible and editable. AI Assertions can evaluate behavior that is difficult to capture with a rigid text comparison, but the broader workflow still needs to be understandable by the team.
The AI is an accelerator, not the owner.
That is the standard I would use for any automation platform or framework:
Can another person understand the test, diagnose its failure, and safely change it without asking the original author?
If the answer is no, the team has not eliminated maintenance.
It has merely postponed it.
Top comments (0)