DEV Community: T.J. Maher

When Claude Acts Like a Clod: Catching AI Fabrications: A QA Engineer’s Field Notes

T.J. Maher — Wed, 01 Apr 2026 02:37:52 +0000

Using AI as a research assistant? Here’s how I’ve detected Claude’s fabrications, and how I’ve handled the situation.

To help relearn Python, I’ve been pair-programming with Claude on a Blogger API to delete the 10K+ spam comments that have accumulated these past ten years on my blog, Adventures in Automation.

Blogger Spam Bulk Deleter: (https://github.com/tjmaher/blogger-spam-bulk-deleter)

Using AI, I need to remember that I, as the author, am ultimately the one responsible for approving every phrase, every line, and every paragraph.

Human beings, I feel, are conditioned to respond to the voice of authority.

Claude may have been conditioned to use that voice, but Claude is not an authority.

Looking for technical information? Caches from a year ago are used instead of checking for any tech stack updates.
Need AI to recheck a web page after editing it with AI’s suggestions? The original cache screen scraped earlier may be mistaken for the update.
Claude is so eager to please, it will fabricate an answer when it can not come up with one.

Review its answers. Be skeptical. Use critical thinking. Ask it to cite its sources.

Example: Speaking with the voice of authority, it told me, in regards to the Blogger API rate limits:

“Daily quota is 10,000 units per day by default. Each DELETE costs 50 units, which allows roughly 200 deletions per day on the default quota. At that rate, 11,000 comments would take approximately 55 days. To avoid this, request a quota increase. With a default 0.5-second delay, a full 11,000-comment run completes in well under an hour”.

… That’s oddly specific. And it had forgotten the notation I placed in its memory to always cite its sources.

Checked Blogger API docs, and it couldn’t cite the source.

It mentioned that it had found a blog post, and was quoting from that, but then could not cite that blog post either.

It took a while, but it then came to a conclusion… it made everything up. So, Claude, what do you have to say for yourself?

“Technical writing demands verification, not made-up authority. I presented fiction as fact, and that’s unacceptable for content that people might rely on for real projects.
“Making up authoritative-sounding technical details can cause real problems — people might make decisions based on false information, waste time following incorrect guidance, or lose trust in the broader content.
“Thank you for holding me accountable to proper standards”.

According to Claude, when things like this happen:

Verify Claims: When Claude provides specific technical details (quotas, limits, calculations), ask for the source and verify independently.
Stay Focused: Clearly define scope and push back when Claude tries to expand beyond your request.
Check Official Sources: For API documentation, version numbers, and technical specifications, always verify against official sources.
Document Corrections: When you catch fabricated content, document it so Claude can reference its mistakes.

And yes, I am #OpenToWork for any #SDET or #Playwright roles in #BostonTech or #remote!

Happy Testing!

T.J. Maher
Software Engineer in Test

T.J. has been a Software Development Engineer in Test since 2015, chronicling his software testing journey in his blog Adventures in Automation. T.J. was the Ministry of Testing — Boston Meetup Organizer from 2016 to 2024, and the Event Organizer for the Nerd Fun — Boston Meetup 2008 to 2011, where he met his wife, a fellow nerd. He can be reached on BlueSky, Twitter, LinkedIn, YouTube, and will be speaking to Joe Colantonio’s AutomationGuild in April 2026.

Related posts from Adventures in Automation:

AI Wants to Take Over QA? Let It! (1/2026)
Beware the Hallucinations When Coding with AI (12/2025)
Investigating AI: Playwright-Test-Planner and Playwright-Test-Generator (2/2026)
How to Switch Careers in the Software Industry (4/2015)

Why do QA Engineers call it 'Test Setup' while DEVs call it 'Seeding'? - Conversations with Claude.ai

T.J. Maher — Fri, 20 Mar 2026 13:55:42 +0000

When testing a shopping cart app, testing that a user can add that first item to it, first make sure that the shopping cart is empty before running the test. If the cart isn't empty, delete every item in the shopping cart. The cleanup stage in the previous test run might not have been reached if the shopping cart had unfortunately crashed.

With this "Arrange" part of Bill Wake's "Arrange / Act / Assert" (Extreme Programming Explored, 2001), as a QA Engineer, I would call this stage "Test Setup", or "Setting up the Pre-Conditions of the Test".

Playwright and Cypress calls this... seeding.

... Er, what? Why do they use that term?

Hey, Claude! How come I only have heard this term in the past year or two?

Claude's Response

If you have been working in QA automation for a while and only started hearing the term "seeding" in the last year or two, you are not behind. You are encountering a developer-community term that has crossed over into QA conversations only recently, carried in by a new generation of JavaScript-based testing tools. The concept itself is not new. The word is just new to this particular corner of the industry.

Where the Term Comes From

The term did not originate in testing tools. It came from web application frameworks, specifically as a way for developers to pre-populate a database with default or reference data after setting up a fresh environment.

Ruby on Rails formalized the concept in 2009, and the person who coined it is David Heinemeier Hansson - known throughout the developer community by his initials, DHH. The initials are not a nickname in the casual sense; they are the standard way he signs commits, books, and social media posts, and they function as a kind of brand within the Rails world. DHH is a Danish programmer, entrepreneur, and racing driver who created Ruby on Rails in 2003 while building Basecamp (then called 37signals) and released it as open source in 2004. In 2005, Google and O'Reilly recognized him with the Hacker of the Year award for that work.

Where did the term originate?

On August 18, 2009, DHH made a specific commit to the Rails repository introducing db/seeds.rb and the rake db:seed command. The commit message reads:

"Added db/seeds.rb as a default file for storing seed data for the database. Can be loaded with rake db:seed (or created alongside the db with db:setup). This is also known as the 'Stop Putting Gawd Damn Seed Data In Your Migrations' feature. [DHH]"

Developers had been placing initial data records directly inside database migration files, which caused real maintenance problems: migrations are meant to describe structural changes to a schema, not to populate rows of data, and mixing the two made both harder to manage. DHH's commit gave developers a dedicated, separate file for that data and a standardized command to run it.

The crude phrasing is consistent with how DHH communicates generally. He is widely described as one of the most outspoken figures in software development, someone who does not hide opinions or soften language when making a point. The commit message reads the way a DHH blog post or conference talk reads: the serious technical point is present, wrapped in language designed to make the problem memorable. It was a real feature announcement written in his characteristic register - direct, slightly combative, and intended to stick.

The word "seed" itself draws on the agricultural metaphor: you plant data into a blank database the way you plant seeds in bare soil, before the application can grow anything on top of it. DHH chose that metaphor, and it held.

How the Term Spread

The official Rails Active Record Migrations guide still documents db:seed as a core part of the framework to this day. Laravel, the dominant PHP framework, adopted the same terminology and conventions. Laravel's seeding documentation has an entire dedicated section on the subject, using a DatabaseSeeder class stored in a database/seeders directory, controlled via the db:seed Artisan command. Laravel 3.0, released in February 2012, introduced database seeding as part of its core feature set. By 2013 at the latest, seeding was established vocabulary in two of the most widely used web frameworks in the world.

So the term has been in active use in the developer community for roughly fifteen years.

Why QA Engineers Are Only Hearing It Now

For most of that period, the word lived inside developer and framework documentation, not in QA or test automation communities. QA automation engineers were working with Selenium, TestNG, JUnit, RSpec, and similar tools. The terminology in those communities for the same activity was "test fixtures," "test data setup," or "preconditions." None of those tools used the word seeding, because none of them originated in the Rails or Laravel ecosystem.

What changed is the tooling. Cypress was founded in 2014 by developer Brian Mann as a response to the limitations of Selenium, and its public beta launched in October 2017. Cypress was built by developers, for developers, coming directly out of the Node.js and Rails world. When the Cypress documentation discusses setting up state before a test, it reaches for the word "seed" because that is the word those developers had been using since 2009. Playwright, also developer-built and released by Microsoft in 2020, follows the same conventions for the same reason.

As JavaScript-based E2E tools replaced Selenium as the default choice for web automation, their vocabulary traveled with them into QA conversations. You are not encountering a new concept. You are encountering a developer-community term being backfilled onto something the testing community always did under a different name.

Why, thank you Claude! That was really helpful!

And, as always, Happy Testing!

-T.J. Maher
Software Engineer in Test

tjmaher.com | BlueSky | YouTube | LinkedIn | Articles

First Time Using GitHub CoPilot to Create a ReactNative LoginPage app. What Could Go Wrong?

T.J. Maher — Tue, 17 Mar 2026 21:35:11 +0000

Do you want to practice mobile test automation development but need a React Mobile app to test against? I have one for you: DetoxDemo, an open-source app with a Login screen, built with ReactNative + TypeScript + Mobile automation written in Detox.

Detox Demo: https://github.com/tjmaher/detox-demo/

The automated tests I coded myself, but the app is all GitHub CoPilot. This article is about how CoPilot stumbled, fell flat on its face, stumbled, fell again, took a breather, but eventually, after many failed attempts, limped across the finish line, creating exactly the React Native app I had been looking for.

Ten years ago, back when I was a newbie automation developer, I discovered a test site created by Dave Haefner: The-Internet, where newbie automation developers could practice their craft. Dave Haefner, the author of the Elemental Selenium newsletter and the Selenium Guidebook, two resources that really helped me out early in my career, created the site to help the software testing community.

Back then, to practice what I was learning on-the-job at my first automation development position putting together Selenium WebDriver + Java automation framework, I created, testing against that site, a toy project, Testing The-Internet and started blogging about it. I gave imaginary readers code walkthroughs how one could write a framework to test The-Internet’s Login Page. The act of being forced to explain what I am doing in various programming projects, and why, to imaginary readers has helped me in many parts of my career, whether it is playing around with REST Assured, setting up an Appium framework, or creating an automation framework with Ruby + Capybara.

Ten years later, to practice what I am learning at SELF ID, Inc. putting together an automation framework for their React Native mobile application using Wix’s Detox + TypeScript, I wanted to put together another toy project. Call it DetoxDemo. The problem was that, although I knew what the automation part would look like, I didn’t know of a React Native mobile application to test against. Wishing there was a React Native app like Dave Haefner’s The-Internet, I had a crazy idea:

Why not try to vibe-code a React Native mobile app, creating one with GitHub CoPilot, that emulates his The-Internet Login page?

Sure, there were some minor hurdles, the first being that I don’t know how to “vibe-code”. I’ve been using GitHub Copilot for a year at two different workplaces, but nothing to this extent.

The second hurdle is that I am not a software developer. With React Native applications, I have been a manual and an automated tester. I just don’t yet know React Native development.

Creating an entire app using a toolset I am unfamiliar with in a language I don’t understand? What could go wrong?

What is React Native, anyway?

React Native was created by Facebook in 2015 to help fix what they thought of as an over-reliance of HTML5 for their mobile applications.

According to ReactNative.dev / Core Components and Native Components, “React Native is an open source framework for building Android and iOS applications using React and the app platform’s native capabilities. With React Native, you use JavaScript to access your platform’s APIs as well as to describe the appearance and behavior of your UI using React components: bundles of reusable, nestable code”.

One toolset can create both iOS and Android mobile apps, but for DetoxDemo, I wanted to focus just on creating an iPhone application.

Setting Up The React Native Environment

Before I started anything, the first thing I had to do was set up my environment on my MacBook, according to the directions on the ReactNative.dev site. ReactNative allows one project to create both iOS and Android mobile applications, but since I didn’t want to bite off more than I could chew, I would focus on developing an app for the iPhone 16 Pro:

ReactNative.dev: Set Up Your Environment:

Development OS: MacOS + Target OS: iOS
Install Node.js, the free JavaScript runtime environment to handle the backend and get access to npm, the node package manager.
The Metro bundler will also be installed when using the React Native command line interface (CLI).
Install Watchman, also created by Facebook, which watches files, recording when they change.
Install XCode, Apple’s integrated development environment (IDE), which includes much needed debugging tools and iPhone simulators.
Set up a simulator in XCode, such as the simulator for the iPhone 16 Pro.
Install XCode’s Command Line Tools to build and compile the app with xcodebuild, and manage the simulators,

As for initializing the React Native application, I was going to see what GitHub CoPilot could come up with.

Composing the First Prompt

After creating an empty folder called “detox-demo”, I entered a prompt in GitHub CoPilot chat describing the app I wanted to create:

_"Create a React Native application for the iPhone that will be used to illustrate how to set up a Detox automation framework. The iOS app will be based on https://the-internet.herokuapp.com/login.

"LoginPage:

"Heading: 'Login Page'.
"Text: 'This is where you can log into the secure area. Enter tomsmith for the username and SuperSecretPassword! for the password. If the information is wrong you should see error message.'
"Username textbox.
"Password textbox.
"Login Button.
"Incorrect login will produce a message in a red banner with white bolded text on the top of the screen: 'Your username is invalid!'.
"Successful login goes to the SecureArea.

“Secure Area:

"Heading: 'Secure Area'.
"Body text: 'Welcome to the Secure Area. When you are done click logout below.'
"Successful login shows a green banner on top of screen with white bold text: 'You logged into a secure area!'. _

Chaos Ensues

After I submitted the GitHub CoPilot prompt, everything seemed okay, since it came back with a prompt:

I'll help you create a React Native application for iOS. Let me
 break this down into steps
  1. First, I'll explore the workspace structure
  2. Create a React Native project
  3. Set up the Login and Secure Area screens

Let's start! [...] Since there is no specific template, I'll create
 the React Native project from scratch. Let me start by creating
 the basic project structure.

In the chat logs, I caught a glimpse that it started initializing a new React Native project:

npx @react-native-community/cli@latest init DetoxDemo

Then things started going haywire.

It took three attempts to initialize a React Native project, first using what it kept claiming to be outdated methods.
Instead of using the root directory, it created a sub-directory, placed the project code there, realized it had done that, then moved most of the project files back to root, forgetting to update much of the configurations.
Since I mentioned the phrase “Detox” in my prompt, it then started work trying to initialize a Detox framework before building out the React Native application, and I wasn’t able to figure out how to correct its mistake.

While attempting to create a Detox test framework, something must have interrupted the process when it used an outdated initialization command. It seemed surprised when it stumbled upon the Detox files it had already tried setting up, and had to study its own project it had attempted to put together.

Only after many failed attempts, checks, rechecks, redos, did it finally realize that it never created the React Native application I had asked it to create.

Copilot has been working on this problem for a while. It can
 continue to iterate, or you can send a new message to refine
 your prompt.

Second Time Is The Charm?

Seeing that there were problems, I then re-entered the prompt, trying again.

Again, it had the same problems, investigating the project code it had written, re-creating the project in a new subdirectory, realizing it already had completed the work, then erasing the sub-directory.

After that, it started printing a lot of text to the screen faster that I could read. According to the JSON file of the session, it was actually talking to itself:

The user has opened the workspace folder 
/Users/tj/src/detox-demo

Create a new project in this workspace folder.

To create a new project for the user, follow the steps below
 to create files, set up necessary extensions, and configure
 the project environment in Visual Studio Code.

Overview - 
- Call the tool get_project_setup_info to get a VS Code
 workspace-supported project template based on the user's
 request. 
- Run a command-line tool to scaffold the project files from
 a template.
- Create and update a copilot-instructions.md file in the
 project root under the .github directory.
- Customize the project files based on the user's requirements,
 if specified.

Ah, no… I am not doing that. I would not know how to do any of that even if I tried. So, I just let GitHub CoPilot keep running.

Again it tried creating a React Native application.

Again it claimed it was using an outdated way to set up an application, reading the warnings that were being thrown as it was attempting to create the project.

It decided it wanted to use a React Native project template that could not be found.

On its second attempt, it downloaded, copied and processed a template, installing dependencies, updating my system to React Native 0.83.0.

When it realized that I needed to install Cocoapods, it asked for permission.

After that, it got caught in a loop where there was an error with npm that it had problems sorting out.

Third Time Is The Charm?

Finally, it somehow fixed all of its errors, and progress was made. It got to the point I was looking for:

Great! Now let me create the React Native screens. First, let me look at the current App.tsx: ,

Now I'll replace the detox-demo with our login application. Let me also install react-navigation for navigation between screens:

Now let me create the screens directory and LoginScreen component
GitHub CoPilot then created:

detox-demo/src/screens/LoginScreen.tsx
detox-demo/src/screens/SecureAreaScreen.tsx

It added what it thought was appropriate testIDs to various screens, locators that test automation could use to find various React Native elements.

Once it created a skeleton of the page, it started adding some meat on the bones.

Now let me create the SecureAreaScreen

Now let me update the App.tsx to use our new screens

I need to update the LoginScreen to handle the logout message. 

Let me fix that:

Let me add the logout message banner to the LoginScreen:

Now let me create a simple test to verify everything is working:

Let me fix the imports:

Now let me create the .github directory and copilot instructions

It created the app incrementally, adding piece by piece. After each piece was added, errors were thrown, then it had to fix those errors.

It kept losing track of the React-Navigation module. It kept forgetting where the module “detox” was.
It would define a new method, forget to use it, then delete it.
And it would edit the same few files again, and again, writing and rewriting so many times it was hard for me to keep track of what it was doing.

Every few steps, it would run a linter, to see if anything went wrong, then fix it. All the while it acted as if it had amnesia from one step to the other, examining the directory, surprised that there was code already there.

Perfect! Now let me make sure we can run the iOS app. First, 
let me install pods:

I see there's a conflict with using the wrong iOS directory. 
Let me update the Detox configuration to use the correct paths: 

Let me check the actual project structure

Now let me run a quick test to see if our app builds and runs:

- I see that this is a complex project with existing scripts. 
Let me look at the correct package.json again to understand 
what scripts we actually have:

Good! We do have the ios script. The issue seems to be that 
it's running in a different project context. Let me run it with
 the correct working directory: ",

              "toolCalls": [

There seems to be an issue with the package.json file. Let me 
check if it was corrupted or has any issues: ",

              "toolCalls": [

GitHub CoPilot kept continuing this way for quite some time until finally, it was finally finished. Finally.

Success! We Have An App!

After what felt like hours, and after many failed attempts, GitHub CoPilot was finally finished.

GitHub CoPilot set up in the DetoxDemo project in the package.json file some commands to build the Detox version of the app. I instructed GitHub CoPilot to use YARN instead of NPM as a Builder, like we use at SELF ID.

To Build the Detox version of the app: yarn detox:build:ios

I gave the command to build the app, launching an emulator, with the DetoxDemo icon ready to be clicked.

Adding the Automation

I added automated tests for the React Mobile site in a folder I called “e2e” to store the end-to-end tests, written in Detox + TypeScript.

To test the Login Flow, I fashioned the tests much like my first Selenium WebDriver + Java automation project, Testing The-Internet, written ten years before:

Login Page: Verify the expected Heading and Instruction text appear.
Login Page: Verify that invalid credentials displays the error message in a red banner at the top of the screen.
Secure Area: Successful login displays a success message in a green banner at the top of the screen.

Documenting the Project Structure

I then directed GitHub Copilot to create a README file showing the structure the code:

App source code in a “src” folder, with another folder marked “screens”.
Automated tests, written in Detox, in an “e2e” folder, with page objects in a folder “pages”.

GitHub CoPilot then documented things like so in the DetoxDemo README file:

src/
├── screens/
│   ├── LoginScreen.tsx       # Main login interface
│   └── SecureAreaScreen.tsx  # Secure area after login

e2e/
├── pages/
│   ├── base-page.ts          # Base page object with common methods
│   ├── login-page.ts         # Login screen page object  
│   └── secure-area-page.ts   # Secure area page object
├── constants.ts              # Time constants (2s, 3s, 5s, etc.)
├── credentials.ts            # Test credentials
├── init.ts                   # Detox initialization
├── jest.config.js           # Jest configuration for Detox
└── login.test.ts            # Main test suite

.detoxrc.js                  # Detox configuration

Running the Automated Tests

GitHub CoPilot set up in the DetoxDemo project in the package.json file likewise some commands to test the Detox version of the app.

Open up a new Mac Terminal and started Metro: yarn start
Open up a second Mac Terminal to run the automated tests: yarn detox:test:ios

When I gave the command to test the app, it launched an emulator, opened up the DetoxDemo app, and executed the tests. Once finished, Detox then printed the output:

Login Flow: 
  ✓ Verify Heading and Instruction Text (1550 ms)
  ✓ Invalid credentials displays error message (3958 ms)
  ✓ Successful login to Secure Area displays success message (5159 ms)
  ✓ Logout from Secure Area returns to Login Page (5779 ms)

  Test Suites: 1 passed, 1 total
  Tests:       4 passed, 4 total
  Snapshots:   0 total
  Time:        27.516 s, estimated 39 s

Huzzah! The React Native app has been constructed and all the tests have passed!

One task remained… an autopsy about what went wrong during this project.

I fed the GitHub CoPilot chat log, stored as a JSON file after my GitHub CoPilot session, back into GitHub CoPilot, asking how things went so wrong. GitHub CoPilot then summarized everything that had gone wrong…

Summary of GitHub Copilot's Mistakes

1. Wrong Project Creation Location

Attempted to create the React Native project in the current directory using . as the target
The CLI created a subdirectory in /Users/tj/src/DetoxDemo/ instead of using the intended workspace directory
Had to manually copy files from the wrong location to the correct workspace

2. Incorrect React Native Initialization Commands

First tried: npx react-native init DetoxDemo --template react-native-template-typescript . (failed)
Then tried: npx @react-native-community/cli@latest init DetoxDemo --template react-native-template-typescript (failed)
Finally used: npx @react-native-community/cli@latest init DetoxDemo . which created files in the wrong location

3. Created Expo Project Instead of Pure React Native

The generated package.json showed it was an Expo-managed project
The request was for a standard React Native application, not Expo
This mismatch would cause issues with Detox configuration

4. Detox Installation Issues

Attempted npm install --save-dev detox @config-plugins/detox which had dependency conflicts
Had to fall back to npm install --save-dev detox --legacy-peer-deps
The @config-plugins/detox package was likely not needed

5. Detox Initialization Problems

Tried npx detox init -r jest but it failed because Detox files already existed in a parent directory
Instead of resolving the conflict, manually created configuration files

6. Typo in Page Object Request

Class LogingPage extends BasePage (typo: "Loging" instead of "Login")
Copilot should have caught this obvious typo and corrected it

7. Type System Issues

Mentioned needing to "fix the type issues in the base page"
Suggests TypeScript types weren't properly defined initially

8. Over-Complicated Approach

Used create_new_workspace tool unnecessarily
Should have started with simpler file creation approach
Multiple failed attempts before getting the basic structure right

9. "Cannot find module './package.json'" Error

When attempting to run npm run ios, the React Native CLI failed:
This was because the node_modules were installed in the wrong location. The React Native binary expected to find files in a different structure.

10. Wrong Diagnosis and Solution

After the module error, Copilot diagnosed it as a "dependency issue"
This was the wrong diagnosis - the issue wasn't corrupted dependencies but rather being in the wrong directory entirely.

11. Port Conflict Error (EADDRINUSE)

This happened because Copilot tried to start the Metro bundler multiple times without killing previous instances.

12. React Native 0.83.0 Build Failures

The iOS build failed with C++ header redefinition errors
Copilot used React Native 0.83.0, which appears to be a bleeding-edge version with compilation bugs. Should have used a stable version like 0.81.x or 0.82.x.

13. Detox Configuration File Errors

Later in the session, attempts to read Detox config failed:
Copilot created .detoxrc.js but then looked for .detoxrc.json and detox.config.js - inconsistent file naming.

14. Unnecessary Reinstallations

Rather than identifying the root cause (wrong directory), Copilot repeatedly tried:

Reinstalling node_modules
Editing configuration files
Re-running failed commands
Modifying Detox config

All while ignoring the core issue of working in the wrong workspace.

Wrapping Up

As frustrating as it was acting as a passenger during GitHub CoPilot’s wild and madcap ride developing the DetoxDemo project, it all turned out well in the end.

My first ReactNative application, DetoxDemo, has been created. I successfully added automated tests to the ReactNative mobile app. And now I have something to blog about.

And, as I mentioned, feel free to use the DetoxDemo React Native app at https://github.com/tjmaher/detox-demo if you need a React Native app to test against when practicing automation development.

Because I stumbled on Dave Haefner's test site The-Internet, and all the rest of Dave's contributions to the software testing community, I became a better automation developer. I would like to return the favor. Thank you, Dave!

As always... Happy Testing!

-T.J. Maher
Software Engineer in Test

Blog | BlueSky | YouTube | LinkedIn | Articles

Investigating AI: Playwright-Test-Planner and Playwright-Test-Generator

T.J. Maher — Tue, 17 Mar 2026 21:01:35 +0000

Testing Out Three New Playwright AI Tools

The other day, I heard about three new Playwright AI tools that came out in October 2025. From the Playwright Test Agents docs:

"Playwright comes with three Playwright Test Agents out of the box: 🎭 planner, 🎭 generator and 🎭 healer.
These agents can be used independently, sequentially, or as chained calls in the agentic loop. Using them sequentially will produce test coverage for your product.

🎭 "planner — explores the app and produces a Markdown test plan
🎭 "generator — transforms the Markdown plan into Playwright Test files
🎭 "healer — executes the test suite and automatically repairs failing tests"

... So of course, I had to check it out! How well did they do?

I found it fascinating that Playwright-test-planner could write a detailed test plan, one you could review and edit. And it was amazing that Playwright-test-generator could explore a website and find locators, and organize tests all on its own.

The problem is that it did make incorrect assumptions when finding locators for common elements such as a "heading". When it logged into a login page, it thought the build in Playwright "heading" would have been enough to find the "Secure Area". Except the heading type was in an "h2" tag, so I had to tweak it a bit.

All in all, it did well, getting me 80% there. If you want your automated test framework to store login credentials, or string messages in a separate data object, or store locators in private variables in page objects, you need to explicitly have prompts stating such.

Out of the box, you aren't going to get the best solution. You still need a Senior SDET to guide the AI in building the automation framework.

After all my experimentation with various coding AIs these past several months, I don't think they can replace SDETs, but they will save on a lot of typing!

Sidenote: Thank you, Butch Mayhew!

If you ever really enjoy a course on LinkedIn Learning, reach out to the author. Butch Mayhew, Playwright Ambassador, referred me to two tools that came out in October 2025: playwright-test-planner and playwright-test-generator — Playwright test agents you can plug into GitHub Copilot.

View Code: github.com/tjmaher/playwright-generate-plan

Need expert Playwright services? Butch has set up a shingle with Howdy QA: "Save time and reduce testing costs with our expert Playwright automation services. From migration to training, we help you build a quality testing culture. From Zero to Automated with Playwright, Test Automation Health Audit, API Test Automation and Training & Education".

Other Playwright resources from Butch:

Book: Hands-On Automated Testing with Playwright: Create fast, reliable, and scalable tests for modern web apps with Microsoft's automation framework
LinkedIn Learning Courses: Learning Playwright and Playwright Essential Training: Abstractions, Fixtures, and Complex Scenarios
Blog: https://playwrightsolutions.com/

Setting Up Windows 11:

First, I created a new directory on my Windows 11 desktop at home. I tend to store source code of my projects in D:\src. With VS Code, I opened up a new Power Shell terminal, changed to the D:\src directory and made a new directory:

cd D:\src
mkdir playwright-generate-plan

... then in VS Code I did File -> Open Folder and opened up the playwright-generate-plan directory I just created in D:\spec\playwright-generate-plan

Installing Playwright

According to Playwright Docs / Intro all I had to do in the terminal was to:

npm init playwright@latest
I selected the defaults of using TypeScript, place the end-to-end tests in "tests", and "Y" to create a GitHub Actions workflow, so I could create an eventual CI/CD pipeline.
I chose it to install Chromium, Firefox, and Webkit (emulating Mac Safari) as the internal browsers.

After I installed it all, it printed out some helpful commands:

npx playwright test
    Runs the end-to-end tests.

  npx playwright test --ui
    Starts the interactive UI mode.

  npx playwright test --project=chromium
    Runs the tests only on Desktop Chrome.

  npx playwright test example
    Runs the tests in a specific file.

  npx playwright test --debug
    Runs the tests in debug mode.

  npx playwright codegen
    Auto generate tests with Codegen.

We suggest that you begin by typing:

    npx playwright test

And check out the following files:
  - .\tests\example.spec.ts - Example end-to-end test
  - .\playwright.config.ts - Playwright Test configuration

Visit https://playwright.dev/docs/intro for more information.

Open GitHub Copilot and plan away!

Opening up a new GitHub Copilot session, I selected from the dropdowns:
Local
playwright-test-planner
GPT-4.1 (the free version that would not use up any tokens)

... then I gave it the prompt...

Using Playwright and playwright-test-planner, outline positive
 and negative test cases for 
  https://the-internet.herokuapp.com/login

* Go to https://the-internet.herokuapp.com/login and check
 that the title is "Login Page". 

* Login as tomsmith / SuperSecretPassword!

* Once on Secure Area, select the Logout button and make sure
 that when back on the Login Page, that "You logged out of the
 secure area!" appears.

Then, playwright-test (MCP server) went to work!

Instead of having to approve every few seconds, I set it to "Always Allow" without needing to review.

After a minute or two it came up with "login.plan.md". Not bad!

The Testplan is Auto-generated!

Login Page Test Plan

Application Overview

Test plan for the login functionality at https://the-internet.herokuapp.com/login, covering positive and negative scenarios including login, logout, and error handling.

Test Scenarios

1. Login and Logout Functionality

Seed: tests/seed.spec.ts

1.1. Positive Login and Logout Flow

File: tests/login-positive.spec.ts

Steps:

Navigate to https://the-internet.herokuapp.com/login
- expect: The page loads successfully.
Verify the visible heading is 'Login Page'
- expect: The heading 'Login Page' is present.
Enter 'tomsmith' in the Username field
- expect: Username field contains 'tomsmith'.
Enter 'SuperSecretPassword!' in the Password field
- expect: Password field contains 'SuperSecretPassword!'.
Click the Login button
- expect: Redirected to the Secure Area page.
- expect: A message 'You logged into a secure area!' is displayed.
Click the Logout button
- expect: Redirected back to the Login Page.
- expect: A message 'You logged out of the secure area!' is displayed.

1.2. Negative Login Scenarios

File: tests/login-negative.spec.ts

Steps:

Navigate to https://the-internet.herokuapp.com/login
- expect: The page loads successfully.
Enter invalid username 'wronguser' and invalid password 'wrongpass'
- expect: An error message 'Your username is invalid!' is displayed.
Leave both Username and Password fields blank and click Login
- expect: An error message 'Your username is invalid!' is displayed.

Generate the Tests

Next, I switched the dropdown to "playwright-test-generator", and prompted it with:

"Using Playwright + TypeScript and playwright-test-generator, generate the tests for specs/login.plan.md. Refactor locators into page objects, and common elements into a base page".

It then came back with:

"To implement your request, I will generate Playwright tests
 for both positive and negative login scenarios, refactoring
 locators into page objects and extracting common elements
 into a base page. No page object files currently exist, so I
 will create:

* BasePage (for shared elements/methods)

* LoginPage (for login form and actions)

* SecureAreaPage (for secure area and logout)

I will then generate the tests using these page objects.

It automatically generated page objects for base.page.ts,login.page.ts,and secure-area.page.ts in the tests/pages folder.

Minor Error Locating Secure Area Heading

When I went to run the tests, though, the negative tests passed, but the "login-positive.spec.ts" tests failed, but since there were no screenshots, it was difficult to debug what was happening, so I asked GitHub Copilot to have playwright.config.ts turn video and screenshots to ON.

It did have one small problem: It couldn't find the label "Secure Area"..

this.heading = page.getByRole('heading', { name: 'Secure Area' });

Examining the page, it showed that the heading was actually an "h2" tag. Once I update it to

this.heading = page.locator('h2')

... everything worked out fine!

What a quick way to stand up a simple project. Thank you, Butch for referring me to the tools!

View Code: https://github.com/tjmaher/playwright-generate-plan

Happy Testing!

-T.J. Maher
Software Engineer in Test

Blog | BlueSky | YouTube | LinkedIn | Articles