Developers love evaluating test automation at creation time.
How quickly can I write the first test?
How elegant is the syntax?
How clever is the AI generation?
Can I get this demo working before lunch?
Those are reasonable questions.
They’re also responsible for a lot of bad purchasing and architecture decisions.
Because test suites spend almost none of their lives being created.
They spend their lives being maintained.
That changes the economics completely.
Generation is cheap now
AI has made generating Playwright tests almost comically easy.
Describe a workflow.
Get some code.
Run it.
Fix a locator.
Done.
This is genuinely useful.
But code generation solves the first 5% of the test’s life.
The remaining 95% involves questions like:
- Who understands this test six months from now?
- Who fixes it when the UI changes?
- How much context must an AI agent read before modifying it?
- Are utilities duplicated across generated files?
- Can non-specialists review what the test actually does?
- Does anyone notice when the test stops representing the product?
There’s a good explanation of the maintenance side in why AI-generated Playwright tests are hard to maintain.
AI reduces the cost of producing code.
It doesn’t automatically reduce the cost of owning code.
Those are not the same thing.
The hidden metric is handoff cost
Imagine your automation engineer leaves.
How long before someone else can confidently change the suite?
One hour?
Two days?
Three weeks?
That number tells you quite a bit about the quality of your automation system.
Handoffs expose architecture problems faster than almost anything else.
A suite can look wonderfully engineered to the person who built it while being completely opaque to everyone else.
That’s why enterprise features such as SSO, permissions, audit history, understandable test structure, and team workflows aren’t merely procurement bureaucracy.
They affect whether testing becomes a team capability or a private kingdom maintained by one specialist.
This guide on choosing browser testing tools for teams that need SSO, audit logs, and clean handoffs covers a lot of those less-glamorous requirements.
They aren’t exciting.
Neither are backups.
You tend to appreciate both after something goes wrong.
Measure coverage of the product, not volume of automation code
Another thing I’d avoid is using lines of automation code as a KPI.
It incentivizes exactly the wrong behavior.
More abstractions.
More helper classes.
More framework.
More code.
A team can produce 40,000 lines of Selenium or Playwright while still failing to cover the three workflows that make the company money.
A better question is:
Are our tests keeping up with the product?
If the engineering team completes ten meaningful features in a sprint but automation only catches up with four, your coverage debt is increasing.
You can make this visible with a simple feature-to-test matrix.
You don’t need a sophisticated dashboard.
Something like:
| Feature | Automated | Critical paths | Browser coverage |
|---|---|---|---|
| Checkout | Yes | 5/5 | Chrome, Safari |
| Subscription upgrade | Yes | 3/4 | Chrome |
| Team invitations | No | 0/3 | — |
Suddenly “we have 2,000 automated tests” becomes much less impressive if the new product surface isn’t being covered.
The same applies to team adoption.
If one engineer creates every test and nobody else touches them, I wouldn’t call that a successful automation program yet.
I’d call it a dependency.
This broader article on software quality metrics CTOs should track in 2026 gets into coverage, adoption, automation velocity, and the actual economics of quality.
Free frameworks aren’t free
There’s another strange way automation gets evaluated.
Framework A costs $0.
Platform B costs money.
Therefore Framework A is cheaper.
That math would make sense if engineer time cost $0.
It does not.
Suppose maintaining an internal automation framework consumes even 15 hours per week across your team.
You have:
- dependency upgrades,
- flaky selector fixes,
- CI troubleshooting,
- browser compatibility,
- reporting,
- authentication helpers,
- retries,
- screenshots,
- test data utilities,
- parallelization,
- infrastructure.
None of those appear on the Playwright invoice.
They appear on payroll.
And because they’re distributed across engineers, many organizations never add them up.
The expensive part of software is rarely the npm package.
It’s everything humans have to do around it.
Outsourcing doesn’t remove ownership either
The same principle applies when outsourcing QA.
An external team can give you more testing capacity.
But you still need to know what evidence you're getting.
Accessibility is a good example.
A vendor saying “we test accessibility” might mean:
We run an automated scanner.
That’s useful.
It is not the same as testing keyboard navigation, focus behavior, screen-reader semantics, or complicated interactive components.
If you're evaluating outside QA help, this guide to assessing outsourcing partners for accessibility coverage, evidence, and release readiness gives you some practical questions to ask.
The principle is identical whether the testing is internal or external:
You need understandable evidence.
Not activity.
Optimize for boring Tuesday mornings
I’ve become increasingly convinced that technical tools should be evaluated by imagining an ordinary Tuesday six months after adoption.
Not the demo.
Not the proof of concept.
Tuesday.
Three developers have merged UI changes.
Two tests failed overnight.
The person who originally built the automation is on vacation.
A release is planned for 2 PM.
Can the team quickly answer:
- What broke?
- Is it a product bug?
- What changed?
- Which workflow is affected?
- Who can fix the test?
- Can someone safely update it?
- Do we trust the rest of the suite?
If yes, you have a good testing system.
If the answer is:
We need Kevin.
Then you have a Kevin-based testing framework.
And Kevin, presumably, would like to take vacations.
The best automation system isn’t necessarily the one with the most elegant API.
It’s the one that continues producing useful information after the novelty of setting it up has disappeared.
That’s when the real test begins.
Top comments (0)