Why I ran this experiment
I lead a QA team of eight on a high-traffic e-commerce product, and for the last year the same question kept landing in my inbox from leadership, from my own engineers, and honestly from my own head at 11pm: can AI just write our tests now?
Every vendor demo says yes. Every LinkedIn thread says either "automation engineers are finished" or "it's all hype." Neither is a real answer. So instead of arguing, I ran a controlled experiment: for 30 days I used AI tools — primarily Claude, Cursor, and Antigravity — as the first draft author for our end-to-end tests, and I tracked what actually happened.
This is the honest verdict. Not a sales pitch, not a takedown. What worked, what broke, and where we would actually spend the money.
The setup
To keep this fair rather than anecdotal, I fixed the rules up front:
Stack: our real Playwright + TypeScript suite, running against staging.
Scope: every new E2E test written during the 30 days had to start as an AI-generated draft. No writing from a blank file by hand.
Tools: Claude for reasoning about scenarios and reviewing diffs, Cursor for in-editor generation and refactors, Antigravity for agentic multi-step tasks.
Measured: time-to-first-passing-test, number of human edits per test, flakiness over the following two weeks, and how many AI-written tests survived code review unchanged.
The goal was not "did the AI write something," it was "did the AI write something I would let my team merge."
What the AI was genuinely good at
The boilerplate disappeared. Page objects, selector scaffolding, the repetitive shape of a DESCRIBE, IT block — the AI produced these faster and more consistently than a human typing them out. For well-understood flows like "add to cart" or "apply a coupon," a usable first draft arrived in under a minute.
It was also a surprisingly good rubber duck. When I described a flaky checkout scenario and pasted the failing logs, the suggestions about wait conditions and race conditions were frequently the same ones I would have reached eventually — just faster. As a reviewer of my own code, it caught a couple of genuine bugs in assertions I had written half-asleep.
And for pure translation tasks it was excellent: "convert these three WebdriverIO tests to Playwright," or "rewrite this callback-style helper as async/await." Mechanical, well-defined, verifiable — exactly the kind of work where AI shines.
Where it fell apart
The failures were consistent, and they clustered around one thing: the AI does not know what actually matters in your product.
It writes tests that pass, not tests that protect. It happily asserted that a button existed rather than that the checkout total was correct after a regional tax change — because it has no idea that the tax logic is the part that has bitten us in production. Domain risk is invisible to it.
Selectors were the second failure mode. The AI loves brittle selectors — long CSS chains, text matches that break the moment marketing changes a label. Left unedited, these are flakiness factories. Every generated test needed a human pass to swap in stable, intent-revealing locators.
The third, and most dangerous, was confident wrong-ness. A few generated tests passed for the wrong reason — a bad wait that masked a real failure, or an assertion loose enough to be meaningless. These are worse than no test at all, because they create false confidence. If I had merged on green without reading, we would have shipped gaps disguised as coverage.
The numbers (for my suite, not a benchmark)
I want to be careful here: this is one team, one codebase, 30 days. Treat it as a data point, not a law.
Roughly 60% of the drafting time vanished for standard flows. But zero tests merged without human edits — the median was a handful of edits per test, mostly selectors and assertions. And the tests that needed the most rework were exactly the high-value, domain-heavy ones. So the time I saved on easy tests, I partly reinvested in the hard ones — which, honestly, is where I want my senior people spending time anyway.
What this means for QA roles
The "AI replaces automation engineers" framing has it backwards. What AI removed from my team's week was the low-judgement typing. What it made more valuable was the thing juniors do not have yet: knowing what to test, what a meaningful assertion looks like, and smelling a test that is green for the wrong reason.
If your value as an SDET is producing test code, that value is genuinely compressing. If your value is deciding what deserves a test and why, it just went up. The role is shifting from author to editor-in-chief — and editors need more judgement, not less.
How I would actually adopt AI in a QA team
After 30 days, here is the workflow I would defend to my leadership. Use AI for the first draft of anything mechanical — page objects, boilerplate steps, migrations, refactors. Never merge an AI-written test on green alone; require a human to verify why it passes. Keep a human in charge of selector strategy and of deciding which flows are worth testing at all. And spend the reclaimed time on the tests that are hard precisely because they encode real business risk.
The honest verdict
AI did not write my tests. It wrote my first drafts, fast, and then handed me the interesting 40% of the work. For a team that already knows what good testing looks like, that is a real multiplier. For a team that does not, it is a very efficient way to generate a false sense of safety.
The tools are good enough to change how we work and nowhere near good enough to change whether we need to think. That gap — between passing and protecting — is the whole job. It is not going anywhere.
I am a QA Lead with 12 years in software quality, currently building AI-assisted quality and reliability systems in e-commerce. If you are navigating AI adoption in your own QA team and want to compare notes, I am always up for that conversation.
Top comments (0)