Last month a teammate pasted a Playwright test into our PR channel and wrote "AI generated this in 4 seconds, why are we still writing tests by hand."
The test passed. It also asserted nothing. It clicked a button, waited 3 seconds, and checked that the page still existed. Green tick, zero value.
I have been doing test automation for a bit over three years now, mostly Playwright on web and Flutter on mobile. I have been using AI heavily in that workflow for about six months. Some of it genuinely changed how I work. A lot of it is noise that people are too excited to admit is noise.
Here is the honest split.
Where AI actually earns its place
1. Turning a bug report into a test case.
This is the single biggest win and almost nobody talks about it. Our QA team writes bug reports in plain language. "Cart total does not update when you remove the last item while a coupon is applied." I paste that into the model along with our page object file, and I get a reasonable failing test in under a minute.
Not a perfect test. A reasonable one. I still rewrite the assertions. But the boring scaffolding, the imports, the fixture setup, the navigation steps, all of that is done. That is maybe 60 percent of the typing gone.
2. Explaining a flaky test you did not write.
You know the feeling. A test fails once every 20 runs. It was written 14 months ago by someone who left. You open it and there are four nested waits and a hardcoded timeout of 8000ms.
Paste the test plus the trace, ask what the race condition probably is. The model is right maybe half the time, but even when it is wrong it gives you a hypothesis to disprove, which is faster than staring at the file. I have written more about the patterns behind flaky end to end tests if you want the non AI side of that problem.
3. Locator suggestions for messy DOMs.
Give it the HTML chunk, ask for the most stable locator. It will usually push you toward getByRole and getByLabel instead of the CSS selector nightmare you were about to write. It is basically a linter with opinions.
Where it falls apart
It does not know what matters.
AI writes tests for the happy path because the happy path is what is in the code. It will never ask "what happens if the payment webhook arrives twice." That question comes from having been burned by a duplicate webhook at 2am. That is domain knowledge, not pattern matching.
Roughly 80 percent of the real bugs I have caught came from tests nobody would think to generate.
Mobile web is where it really struggles.
This is the part that surprised me. AI models are trained on a mountain of desktop web test code. Ask them about viewport specific behaviour, touch targets, or the way a sticky header eats your click on a 390px screen and the quality drops hard. It will confidently give you a desktop solution and call it mobile.
I ended up writing my own reference for Playwright mobile web testing because I kept getting the same wrong suggestions. Device emulation, real touch events, and orientation handling still need a human who has actually seen the bug.
Flutter is worse.
If your app is Flutter, brace yourself. The training data is thin. Ask for a widget test and you get something plausible looking that uses an API that changed two versions ago. Ask for a smoke suite and it gives you web patterns wearing a Flutter costume.
I built out our smoke and regression testing setup for Flutter almost entirely by hand for this reason. AI helped with the boilerplate inside each test. It helped with nothing about the structure.
Self healing locators are mostly marketing.
Every AI testing tool sells this. In practice, a locator that silently repairs itself is a locator that stops telling you the UI changed. Sometimes the UI changing IS the bug. I would rather my test break loudly.
What my actual workflow looks like now
- I write the test plan myself. Plain English, in the ticket. What should break, and why I care.
- AI generates the skeleton from that plan plus my existing page objects.
- I rewrite every assertion. Every single one.
- I run it 20 times locally before it goes near CI.
- If it is flaky at step 4, I delete it and start over rather than adding waits. Step 3 is the one people skip and it is the one that matters. A generated assertion checks that something exists. A written assertion checks that something is correct. Those are completely different jobs.
The rest of my test automation notes go deeper on the CI side if you are setting this up fresh.
The uncomfortable part
AI made me faster at writing tests. It did not make me better at knowing which tests to write. And the second skill is the entire job.
I think there is a real risk for people entering QA right now. If you learn to prompt before you learn to reason about failure modes, you will produce a very large, very green test suite that catches nothing. I have reviewed a few of those already. They are worse than having no tests, because they create confidence.
Now tell me I am wrong
Three things I want to hear from you in the comments, and I will reply to every single one.
One. What is the dumbest test AI has ever generated for you? I want to collect these. Mine was a test that asserted expect(true).toBe(true) after a login flow.
Two. Has anyone here actually had self healing locators work in a real production suite? I am genuinely open to being proven wrong on this. If you have a case where it saved you, I want the details.
Three. If you test Flutter or React Native, has AI been useful to you at all, or is your experience as rough as mine?
I am also curious whether anyone has moved to an MCP based setup where the model drives the browser directly instead of generating code. I have tried it twice and both times it was slower than just writing the test, but I suspect I was holding it wrong.
Drop your take below. Especially if you disagree.
Top comments (0)