Testing Out Three New Playwright AI Tools
The other day, I heard about three new Playwright AI tools that came out in October 2025.
From the Playwright Test Agents docs:
"Playwright comes with three Playwright Test Agents out of the box: 🎠planner, 🎠generator and 🎠healer.
These agents can be used independently, sequentially, or as chained calls in the agentic loop. Using them sequentially will produce test coverage for your product.
- 🎠"planner — explores the app and produces a Markdown test plan
- 🎠"generator — transforms the Markdown plan into Playwright Test files
- 🎠"healer — executes the test suite and automatically repairs failing tests"
... So of course, I had to check it out! How well did they do?
I found it fascinating that Playwright-test-planner could write a detailed test plan, one you could review and edit. And it was amazing that Playwright-test-generator could explore a website and find locators, and organize tests all on its own.
The problem is that it did make incorrect assumptions when finding locators for common elements such as a "heading". When it logged into a login page, it thought the build in Playwright "heading" would have been enough to find the "Secure Area". Except the heading type was in an "h2" tag, so I had to tweak it a bit.
All in all, it did well, getting me 80% there. If you want your automated test framework to store login credentials, or string messages in a separate data object, or store locators in private variables in page objects, you need to explicitly have prompts stating such.
Out of the box, you aren't going to get the best solution. You still need a Senior SDET to guide the AI in building the automation framework.
After all my experimentation with various coding AIs these past several months, I don't think they can replace SDETs, but they will save on a lot of typing!
Sidenote: Thank you, Butch Mayhew!
If you ever really enjoy a course on LinkedIn Learning, reach out to the author. Butch Mayhew, Playwright Ambassador, referred me to two tools that came out in October 2025: playwright-test-planner and playwright-test-generator — Playwright test agents you can plug into GitHub Copilot.
- View Code: github.com/tjmaher/playwright-generate-plan
Need expert Playwright services? Butch has set up a shingle with Howdy QA: "Save time and reduce testing costs with our expert Playwright automation services. From migration to training, we help you build a quality testing culture. From Zero to Automated with Playwright, Test Automation Health Audit, API Test Automation and Training & Education".
Other Playwright resources from Butch:
- Book: Hands-On Automated Testing with Playwright: Create fast, reliable, and scalable tests for modern web apps with Microsoft's automation framework
- LinkedIn Learning Courses: Learning Playwright and Playwright Essential Training: Abstractions, Fixtures, and Complex Scenarios
- Blog: https://playwrightsolutions.com/
Setting Up Windows 11:
First, I created a new directory on my Windows 11 desktop at home. I tend to store source code of my projects in D:\src. With VS Code, I opened up a new Power Shell terminal, changed to the D:\src directory and made a new directory:
- cd D:\src
- mkdir playwright-generate-plan
... then in VS Code I did File -> Open Folder and opened up the playwright-generate-plan directory I just created in D:\spec\playwright-generate-plan
Installing Playwright
According to Playwright Docs / Intro all I had to do in the terminal was to:
- npm init playwright@latest
- I selected the defaults of using TypeScript, place the end-to-end tests in "tests", and "Y" to create a GitHub Actions workflow, so I could create an eventual CI/CD pipeline.
- I chose it to install Chromium, Firefox, and Webkit (emulating Mac Safari) as the internal browsers.
After I installed it all, it printed out some helpful commands:
npx playwright test
Runs the end-to-end tests.
npx playwright test --ui
Starts the interactive UI mode.
npx playwright test --project=chromium
Runs the tests only on Desktop Chrome.
npx playwright test example
Runs the tests in a specific file.
npx playwright test --debug
Runs the tests in debug mode.
npx playwright codegen
Auto generate tests with Codegen.
We suggest that you begin by typing:
npx playwright test
And check out the following files:
- .\tests\example.spec.ts - Example end-to-end test
- .\playwright.config.ts - Playwright Test configuration
Visit https://playwright.dev/docs/intro for more information.
Open GitHub Copilot and plan away!
Opening up a new GitHub Copilot session, I selected from the dropdowns:
Local
playwright-test-planner
GPT-4.1 (the free version that would not use up any tokens)
... then I gave it the prompt...
Using Playwright and playwright-test-planner, outline positive
and negative test cases for
https://the-internet.herokuapp.com/login
* Go to https://the-internet.herokuapp.com/login and check
that the title is "Login Page".
* Login as tomsmith / SuperSecretPassword!
* Once on Secure Area, select the Logout button and make sure
that when back on the Login Page, that "You logged out of the
secure area!" appears.
Then, playwright-test (MCP server) went to work!
Instead of having to approve every few seconds, I set it to "Always Allow" without needing to review.
After a minute or two it came up with "login.plan.md". Not bad!
The Testplan is Auto-generated!
Login Page Test Plan
Application Overview
Test plan for the login functionality at https://the-internet.herokuapp.com/login, covering positive and negative scenarios including login, logout, and error handling.
Test Scenarios
1. Login and Logout Functionality
Seed: tests/seed.spec.ts
1.1. Positive Login and Logout Flow
File: tests/login-positive.spec.ts
Steps:
- Navigate to https://the-internet.herokuapp.com/login
- expect: The page loads successfully.
- Verify the visible heading is 'Login Page'
- expect: The heading 'Login Page' is present.
- Enter 'tomsmith' in the Username field
- expect: Username field contains 'tomsmith'.
- Enter 'SuperSecretPassword!' in the Password field
- expect: Password field contains 'SuperSecretPassword!'.
- Click the Login button
- expect: Redirected to the Secure Area page.
- expect: A message 'You logged into a secure area!' is displayed.
- Click the Logout button
- expect: Redirected back to the Login Page.
- expect: A message 'You logged out of the secure area!' is displayed.
1.2. Negative Login Scenarios
File: tests/login-negative.spec.ts
Steps:
- Navigate to https://the-internet.herokuapp.com/login
- expect: The page loads successfully.
- Enter invalid username 'wronguser' and invalid password 'wrongpass'
- expect: An error message 'Your username is invalid!' is displayed.
- Leave both Username and Password fields blank and click Login
- expect: An error message 'Your username is invalid!' is displayed.
Generate the Tests
Next, I switched the dropdown to "playwright-test-generator", and prompted it with:
"Using Playwright + TypeScript and playwright-test-generator, generate the tests for specs/login.plan.md. Refactor locators into page objects, and common elements into a base page".
It then came back with:
"To implement your request, I will generate Playwright tests
for both positive and negative login scenarios, refactoring
locators into page objects and extracting common elements
into a base page. No page object files currently exist, so I
will create:
* BasePage (for shared elements/methods)
* LoginPage (for login form and actions)
* SecureAreaPage (for secure area and logout)
I will then generate the tests using these page objects.
It automatically generated page objects for base.page.ts,login.page.ts,and secure-area.page.ts in the tests/pages folder.
Minor Error Locating Secure Area Heading
When I went to run the tests, though, the negative tests passed, but the "login-positive.spec.ts" tests failed, but since there were no screenshots, it was difficult to debug what was happening, so I asked GitHub Copilot to have playwright.config.ts turn video and screenshots to ON.
It did have one small problem: It couldn't find the label "Secure Area"..
- this.heading = page.getByRole('heading', { name: 'Secure Area' });
Examining the page, it showed that the heading was actually an "h2" tag. Once I update it to
- this.heading = page.locator('h2')
... everything worked out fine!
What a quick way to stand up a simple project. Thank you, Butch for referring me to the tools!
Happy Testing!
-T.J. Maher
Software Engineer in Test
Top comments (0)