DEV Community: Umair Iftikhar

What I Learned Wiring Playwright Into MCP: The Good, the Flaky, and the Surprising

Umair Iftikhar — Fri, 26 Jun 2026 19:05:07 +0000

The first time it worked, I just sat there.

I'd typed one sentence into a chat window. Something like "go to the staging site, log in with the test account, and add two items to the cart." A browser opened on its own. The cursor moved. Fields filled in. A product landed in the cart. I hadn't written a single locator, a click, or one line of a spec file.

My first reaction wasn't "this is the future." It was "what exactly just happened, and can I trust any of it?"

That question is pretty much what this whole post is about. I've spent the last few years writing Playwright suites the normal way. Selectors, fixtures, page objects, all of it. Wiring Playwright into MCP felt like watching a tool I knew well start speaking a different language. Some of it was genuinely brilliant. Some of it drove me up the wall. And a few things caught me completely off guard.

What this actually is, in plain terms
If you haven't touched it yet, here's the short version.

MCP, the Model Context Protocol, is a way for an AI model to call external tools through one shared interface. The Playwright MCP server is one of those tools. Instead of you writing the browser automation, it hands the model a set of browser actions. Navigate, click, type, read the page, take a snapshot. The model decides what to do and Playwright carries it out.

The thing that surprised me earliest was how it sees the page. I assumed screenshots, with some vision model squinting at pixels. It isn't, at least not by default. It works off the accessibility tree, the structured, labelled version of the page that screen readers rely on. That one decision explains a lot of what comes next, both the good and the bad.

So you're not really recording a test anymore. You're describing what you want and letting something else turn that into actions. After years of being deliberate about every selector, that's a strange thing to sit with.

The good

I'll start with what genuinely impressed me, because there was plenty.

Exploring an app got fast. Normally when I land on something unfamiliar I poke around by hand, work out the flows, then start scripting. With MCP in the loop I could just ask it to walk through a feature while I watched. It found steps and edge cases I'd have reached eventually, only quicker. As a way to understand a new system before writing a single real test, it paid for itself straight away.

The accessibility tree approach held up better than I expected. Because it reads structured page data instead of pixels, a layout shifting a few points or a colour changing doesn't throw it. It reads the page the way the page describes itself. When it found an element, it usually found the right one, by role and label, which is honestly how we're all told we should be writing locators anyway.

It opened the door for people who don't code. I showed it to a manual tester on our team who has never been comfortable writing automation. Within an hour they were driving real flows in plain English. That's not a small thing. So much testing knowledge sits with people who can't translate it into a framework, and this closes part of that gap.

Drafting the boring scaffolding got quicker. Even when I went back to writing proper Playwright code, letting the model map a flow first gave me a decent structure to harden by hand. Less staring at an empty file.

For a few days I was sold. Then I pushed it on real work and the cracks showed.

The flaky

Here's where I have to be straight with you, because the title promised flaky and the title wasn't bluffing.

It isn't deterministic, and tests are meant to be. Run the same instruction twice and you don't always get the same path. Once it logged in through the main form. Another time it spotted a "continue with email" route I'd forgotten was even there and went that way instead. Both reached the goal. But a test that takes a different road on each run is a test you can't fully trust, because you no longer know exactly what it checked. In automation, "it got there somehow" is not the same as "it did the thing I meant."

It resolves ambiguity confidently, and sometimes wrongly. When my instruction was vague, it didn't stop to ask. It picked a reading and ran with it, totally sure of itself. "Submit the form" once meant a Save Draft button rather than the real submit, and the run looked perfectly fine while testing the wrong thing. A green result that quietly checked the wrong path is the most dangerous green there is.

It stalls on the genuinely odd parts. Custom date pickers, drag and drop, canvas elements, anything that isn't a clean, clearly labelled control. These tripped it up far more than they trip up a locator I've tuned by hand for that exact widget. The accessibility tree strength turns into a weakness the moment the interface stops describing itself properly. And plenty of real apps describe themselves badly.

Cost and speed pile up. Every step means a model deciding what to do next. For a quick bit of exploring, fine. For a full regression run, the latency and the token cost are real, and a plain Playwright suite still wins comfortably on both.

None of this means it's broken. It means it behaves like a sharp assistant improvising, not a deterministic test runner. Different tools, different jobs.

The surprising

A few things I didn't see coming at all.

The bottleneck moved to how well I describe things. I thought the hard part would be setup. It wasn't. The hard part was writing instructions precise enough to get the same behaviour twice. The clearer and tighter my wording, the better it ran. Which, once I sat with it, is just a fuzzier version of the discipline good test design already asks for. The skill didn't vanish. It changed shape.

Watching it work became its own debugging tool. Seeing the model talk through its choices, "the login link wasn't visible, so I opened the menu first," surfaced assumptions about my own app I'd long stopped noticing. A couple of times it exposed genuine clunkiness in the UX, not test problems. I went in to judge a testing tool and came out with notes for the product team.

It made me value my boring old suite more, not less. I'd half expected to walk away feeling replaceable. Instead I came away clearer on what handwritten, deterministic tests are actually for. The predictability I used to take for granted turns out to be the entire point in the places that matter.

So where would I actually use it

I keep landing on a simple split.

For exploring, prototyping, getting to know a new app, and helping teammates who don't write code contribute, Playwright through MCP is a real addition to the toolbox. I'll keep using it there. It's fast, it's approachable, and it lowers a barrier that's been stubbornly high for years.

For the regression suite that gates a release, though, especially anything I'd have to defend to someone who can stop that release, I still want deterministic tests I wrote and can explain. It's the same conclusion I keep reaching with AI in testing in general. Lean on it where being wrong costs little, and keep a firm hand where it doesn't.

That's not a dig at the tooling. It's just knowing which job you're doing.

Where I've landed, for now

Wiring Playwright into MCP was one of the more interesting things I've done in a while, mostly because it refused to resolve neatly into "great" or "useless." It's both, depending on what you ask of it.

The good is genuinely good. The flaky is genuinely flaky. And the most surprising part was realising the tool didn't reduce the need for testing judgement. It just moved it, from writing the steps to deciding which steps to trust.

I'm still working out exactly where it fits in my week. These are early notes, not a verdict.

If you've wired Playwright into MCP, or any model into a real browser, I'd love to hear where it held up for you and where it fell apart. I get the feeling we're all still mapping the edges of this one.

What I've Learned Using AI for Testing in Industries Where Mistakes Actually Cost Something

Umair Iftikhar — Thu, 25 Jun 2026 12:57:18 +0000

A while back I sat through a demo where someone generated an entire regression suite from a single prompt. It was genuinely impressive. The tests appeared in seconds, the locators looked clean, and the presenter barely touched the keyboard.

My first thought wasn't "this is going to replace me." It was "I would never let that run on a claims platform without a human reading every line first."

That gap, between what AI testing tools can do and what you can actually trust them to do in a regulated environment, is something I think about constantly. I've spent the last five years building automation frameworks for clients in fintech, insurance, and healthcare. In those worlds the conversation about AI sounds very different from the optimism I usually see online.

It isn't that AI doesn't belong in testing. It clearly does, and I use it most days. It's that the cost of being wrong changes everything about how you use it.

When a passing test is the dangerous outcome

Most AI testing tools are built, whether the makers say so or not, for environments where a false positive is irritating rather than serious.

If a test wrongly passes on an ecommerce checkout, a bad order slips through. You refund the customer and move on. If a test wrongly passes on a claims workflow, or a dosage calculation in a health system, or a transaction check in a payments engine, you are in completely different territory. Audit exposure. Regulatory questions. Real harm to real people.

So nothing here is an argument against AI. It's an argument for understanding the room you're standing in before you decide how much to trust the machine.

The three things that quietly break

Three areas come up again and again.

The first is test data. In a normal product, fake data is trivial. Invent a name, invent an email, done. In healthcare that data has to respect HIPAA. In fintech you're handling account numbers and transaction histories. In insurance, policy data is both legally and personally sensitive. Tools that generate test data on demand will happily hand you something that looks perfectly realistic and quietly breaks your data handling rules the moment it lands in a test environment. I've seen suites seeded with data exactly like that. It looked fine. It would have failed an audit on sight. The tool wasn't being careless. It simply had no idea what world it was operating in.

The second is the audit trail. Regulated work gets audited, sometimes by people with the power to stop a release. When an auditor asks why a test passed, "the model wrote it that way" is not an answer anyone will accept. Every test in that kind of pipeline needs a reason a human can read and defend. AI can absolutely help draft the test. What it can't do is own the intent. That part stays with an engineer who can explain the logic out loud.

The third is change. In consumer software you ship, test, and fix at speed. In an insurance or healthcare platform, a single release can pass through compliance gates, change boards, and approvals before it ever reaches production. This is where the feature that lets tests repair their own locators gets interesting. Tests that quietly rewrite themselves when the interface shifts sound wonderful, until you remember that in a validated suite an undocumented automatic change is still a change, and under frameworks like GAMP 5 that can drag you back into validating the whole suite again. A model that helpfully fixed your tests overnight can hand you a compliance problem that takes weeks to untangle.

Where it genuinely earns its place

I don't want this to read as a list of reasons to avoid AI, because I lean on it. The trick is putting it where the downside is small.

Drafting is the obvious one. Handing a model a requirements document or a set of user stories and asking for a first cut of test cases saves real time. In regulated work that draft is never the finished article. A human reviews it, adds the compliance edge cases the model missed, and writes down the reasoning. But starting from something solid beats starting from a blank file, and the model often surfaces a scenario a tired reviewer would skip.

Coverage analysis is similar. Feeding a spec to a model and asking what scenarios it implies is useful less as an answer and more as a second opinion. When a missed scenario can mean a failed audit, another set of eyes, even synthetic ones, has value.

Then there's the plumbing. Page objects, data factories, pipeline configuration, all the mechanical scaffolding nobody enjoys writing. Tools like GitHub Copilot have made that part of my week noticeably faster, and the risk stays low because I'm reading and integrating the output, not shipping it untouched.

The skill that actually matters now

Here's the part I think a lot of people are dancing around.

AI is not going to delete QA jobs in regulated industries. But it is going to change what makes a QA engineer worth hiring.

Knowing how to prompt a model for a test case is not really a skill anymore. It's a baseline, like knowing how to use version control. The people who become hard to replace are the ones who know when to overrule the model. The engineer who reads a generated test and instantly spots that it ignores a data residency rule in the third scenario. Who notices that a locator was silently rewritten in a way that changes what the test actually checks. Who can sit across from an auditor and explain, in plain language, what every test is verifying and why it exists at all.

That kind of judgement rests on domain knowledge. Understanding how a payment really settles, how a claim moves through a system, how health data is allowed to be handled. A model can't stand in for that, and in regulated industries it's exactly what makes an engineer valuable.

The future I see isn't AI replacing testers. It's testers who understand both the rules and the tooling, leaning on the tooling to move faster and on the rules to know where to push back.

A rough test I apply

Before I let anything a model produced into a regulated suite, I ask myself three questions.

Can I explain this to an auditor in plain language? If I can't describe the test, its inputs, and its expected results to someone outside the technical team, it needs more documentation before it goes anywhere near the suite.

Does it touch sensitive data? If it does, anything the model generated or suggested gets a compliance review before it runs anywhere that isn't fully isolated.

Is this suite under formal validation? If the project falls under something like GAMP 5 or 21 CFR Part 11, any automated change, including one the model made on its own, goes through change control. No shortcuts.

None of that is red tape for its own sake. It's the line between automation that helps a regulated product ship safely and automation that turns into a liability you discover at the worst possible moment.

The bigger picture

AI is going to reshape testing. I believe that, and I'm actively learning the new tools as they arrive.

But in regulated industries the change will look different. Slower, more deliberate, with human judgement carrying real weight for longer. That isn't a weakness. When the software you're testing handles a medical record, a mortgage, or someone's insurance claim, a testing mistake reaches a lot further than a bad afternoon for one user.

The engineers who can hold both ideas at once, moving faster with AI while keeping the rigour these industries demand, are the ones who will set the standard for what good looks like next.

I'm still working mine out. These are just the questions I've found worth asking.

If you've used AI tools inside a regulated environment, I'd genuinely like to hear what has held up for you and what hasn't.