DEV Community: Anton Gulin

How to Migrate from Selenium to Playwright in 2026: A Complete Guide

Anton Gulin — Tue, 22 Sep 2026 15:01:34 +0000

Why Migrate from Selenium to Playwright?

I've been using Selenium since 2015. It served me well at Apple and plenty of other companies. But after migrating a large suite to Playwright, I won't go back.

Here's how they compare:

Setup time:

Selenium: 30+ minutes
Playwright: 5 minutes
Auto-waiting:
Selenium: Manual (causes flaky tests)
Playwright: Built-in
Cross-browser:
Selenium: Needs separate drivers
Playwright: Bundled
Parallel testing:
Selenium: Requires Grid setup
Playwright: Built-in
Test execution:
Selenium: Slower
Playwright: 40-60% faster
Debugging:
Selenium: Stack traces only
Playwright: Trace viewer with recordings
If your Selenium tests are slow, flaky, or a pain to maintain, migration is worth it.

The Migration Strategy

Don't rewrite everything at once. Follow this incremental approach:

Phase 1: New tests in Playwright

Stop writing new Selenium tests. All new tests go in Playwright.

Phase 2: Migrate critical paths first

Identify your highest-value tests (login, checkout, core features) and migrate those.

Phase 3: Gradual Selenium deprecation

As you touch Selenium tests for maintenance, migrate them. Eventually, the old suite goes to zero.

This approach minimizes risk and lets your team learn Playwright gradually.

Step 1: Set Up Playwright Alongside Selenium

You don't need to remove Selenium to start. Create a parallel Playwright project:

npm init playwright@latest -- --quiet

Your project structure becomes:

tests/
├── selenium/  # Existing Selenium tests
└── playwright/  # New Playwright tests

Both can run in CI simultaneously during transition.

Step 2: Translate Common Patterns

Here's a side-by-side comparison of common operations:

Navigation:

Selenium (Java):

driver.get("https://example.com");

Playwright (TypeScript):

await page.goto('https://example.com');

Finding elements:

Selenium:

driver.findElement(By.id("username"));

driver.findElement(By.cssSelector(".submit-btn"));

driver.findElement(By.xpath("//button[text()='Submit']"));

Playwright:

page.locator('#username');

page.locator('.submit-btn');

page.getByRole('button', { name: 'Submit' });

Clicking:

Selenium:

driver.findElement(By.id("submit")).click();

Playwright:

await page.locator('#submit').click();

Or better:

await page.getByRole('button', { name: 'Submit' }).click();

Typing:

Selenium:

driver.findElement(By.id("email")).sendKeys("test@example.com");

Playwright:

await page.locator('#email').fill('test@example.com');

Or:

await page.getByLabel('Email').fill('test@example.com');

Waiting:

Selenium (the source of most flakiness):

WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));

wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("result")));

Playwright (auto-waits by default):

Just interact. Playwright waits automatically

await page.locator('#result').click();

Or explicitly assert visibility:

await expect(page.locator('#result')).toBeVisible();

Assertions:

Selenium (with JUnit):

String title = driver.getTitle();

assertEquals("Expected Title", title);

Playwright:

await expect(page).toHaveTitle('Expected Title');

Screenshots:

Selenium:

File screenshot = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);

FileUtils.copyFile(screenshot, new File("screenshot.png"));

Playwright:

await page.screenshot({ path: 'screenshot.png' });

Step 3: Migrate Page Objects

If you're using Page Object Model in Selenium, the pattern translates directly.

Selenium (Java):

public class LoginPage {
    private WebDriver driver;
    public LoginPage(WebDriver driver) {
        this.driver = driver;
    }
    public void login(String email, String password) {
        driver.findElement(By.id("email")).sendKeys(email);
        driver.findElement(By.id("password")).sendKeys(password);
        driver.findElement(By.id("submit")).click();
    }
}

Playwright (TypeScript):

export class LoginPage {
  constructor(private page: Page) {}
  async login(email: string, password: string) {
    await this.page.getByLabel('Email').fill(email);
    await this.page.getByLabel('Password').fill(password);
    await this.page.getByRole('button', { name: 'Sign in' }).click();
  }
}

The structure is almost identical. You're just swapping APIs.

Step 4: Replace WebDriver Waits with Auto-Wait

This is where you get the biggest reliability win.

Remove all explicit waits:

In Selenium, you probably have code like:

WebDriverWait wait = new WebDriverWait(driver, 10);

wait.until(ExpectedConditions.elementToBeClickable(By.id("button")));

driver.findElement(By.id("button")).click();

In Playwright, just:

await page.locator('#button').click();

Playwright automatically waits for:

Element to be attached to DOM
Element to be visible
Element to be enabled
Element to be stable (not animating) This single change eliminates most flaky tests.

Step 5: Set Up CI/CD

Playwright's CI setup is simpler than Selenium Grid.

GitHub Actions example:

name: Playwright Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/

No Selenium Grid. No browser driver management. Just install and run.

Common Migration Pitfalls

1. Trying to rewrite everything at once

This is the number one cause of failed migrations. Use the incremental approach.

2. Copying Selenium's explicit waits

Don't translate WebDriverWait to waitForSelector. Trust Playwright's auto-waiting.

3. Keeping fragile locators

Migration is a chance to improve. Replace XPath and positional CSS with role-based locators.

4. Not training the team

Playwright has different patterns. Invest in a training session or pair programming during the first weeks.

Migration Checklist

Before declaring migration complete:

All new tests written in Playwright
Critical user journeys migrated
CI/CD running Playwright tests
Team trained on Playwright patterns
Old Selenium tests deleted (don't leave dead code)
Trace viewer configured for failure debugging

How Long Does Migration Take?

Based on my experience:

Small suite (under 50 tests): 1-2 weeks

Medium suite (50-200 tests): 1-2 months

Large suite (200+ tests): 3-6 months

The biggest factor isn't the number of tests. It's how tangled your Selenium architecture is. Clean Page Objects migrate fast. Spaghetti code requires refactoring.

The Results Are Worth It

On that project, after completing migration:

Test execution time: 40% faster
Flaky test rate: Reduced by 80%
CI pipeline reliability: 98%+ pass rate
Developer experience: "I actually like writing tests now" That last point matters more than any metric. When tests are pleasant to write, people write more tests.

Need Help with Your Migration?

If you're planning a Selenium to Playwright migration and want guidance from someone who's done it multiple times, I can help. I offer consulting packages that include:

Migration strategy and roadmap
Team training on Playwright patterns
Architecture review of existing tests
Hands-on migration for critical paths Let's talk

Anton Gulin is the AI QA Architect, the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.

Let's save the clues from a failed Playwright test

Anton Gulin — Sat, 19 Sep 2026 15:00:56 +0000

Originally published on anton.qa.

Let's run a checkout test that fails once, then passes.
We'll use Playwright, a browser testing tool.
You'll compare two settings and open the recordings they keep.

A trace is a detailed test recording.
You can inspect actions, saved page views, and requests from that attempt.
A retry is another attempt after a failure.

The useful difference in this example:

on-first-retry keeps the successful retry's recording.
retain-on-failure keeps the original failure's recording.

The test creates both responses so you can repeat this comparison.
You can see the supplied error in the original failure's saved response.
It runs on your computer and doesn't contact a payment provider.

The complete runnable project includes all files.
An agent verified a fresh public download while preparing the article.

Set up the practice project

Use Node.js 24, which runs JavaScript programs.
Its included npm tool installs project packages.
You'll also need Git, which downloads and tracks source code.

Run these commands in your terminal:

git clone https://github.com/antongulin/anton-qa-resources.git
cd anton-qa-resources/posts/save-failed-playwright-test
npm ci
npx playwright install chromium

The project fixes Playwright at version 1.63.0 for this example.
The last command installs Chromium, the browser used for testing.

On Linux, missing browser libraries may require this command:

npx playwright install --with-deps chromium

You don't need to start the practice website yourself.
The tests start and stop their own local server at 127.0.0.1:4191.
If you've started npm run serve yourself, stop it with Ctrl+C first.

Run the example and check its recordings

npm run verify

This command runs the tests and checks the saved recordings.
It replaces the project's generated results from any previous run.

Expect the checkout test to fail first, then pass on retry.
Playwright labels that result flaky: it needed another attempt to pass.
Here, we created that behavior so you can repeat the comparison.

The verifier checks both attempt numbers and the recorded responses.
It also confirms that the failed recording contains the original failed check.
You should see successful verification despite the deliberately failed first attempts.

The project includes three additional checks for related browser-testing examples.
Those checks should pass too.

See how the test creates its failure

Open tests/core-trace.spec.ts in your editor.
The test uses testInfo.retry to read the attempt number.
Zero means the original attempt. One means the first retry.

It fills an order note and presses Save checkout.
A route handler, code that intercepts requests, supplies the checkout response.
On attempt zero, it supplies an error. On retry one, it supplies success.

A fixture means a controlled test setup here.
The example uses that word in its exact response messages:

Attempt	Response code	Message supplied by the test
Original, 0	503, service unavailable	Injected first-attempt save failure
Retry, 1	200, success	Saved on retry by synthetic fixture

Both attempts run the same assertion, a check of the expected result.
This exact excerpt comes from the test:

await expect(page.getByRole('status')).toHaveText('Saved on retry by synthetic fixture');

The first attempt displays the error message, so this check fails.
The retry displays the expected success message, so it passes.
No timing guess or random failure decides the outcome.

Compare the two recording settings

The project runs that test under two sets of settings.
Playwright calls these projects, named groups of test settings.

These two project entries come from playwright.config.ts:

[
  {
    name: 'on-first-retry',
    testMatch: /core-trace\.spec\.ts/,
    retries: 1,
    use: { trace: 'on-first-retry' },
  },
  {
    name: 'retain-on-failure',
    testMatch: /core-trace\.spec\.ts/,
    retries: 1,
    use: { trace: 'retain-on-failure' },
  },
]

These entries belong inside the existing configuration, followed by the companion project.
Keep the downloaded file intact. You don't need to edit it for this walkthrough.

on-first-retry records the first retry and keeps that recording.
In our example, you get the successful second attempt.

retain-on-failure records each attempt and keeps recordings of failed attempts.
In our example, you get the original failed attempt.

Playwright's recording settings describe additional modes and combinations.
The two here let you compare a successful retry with its original failure.

Open the successful retry

Run this command after verification:

npx playwright show-trace test-results/*on-first-retry-retry1/trace.zip

Trace Viewer opens the recording for inspection.
Select the Save checkout action, then look at the saved page view.
Open Network, the list of recorded requests.
Find the checkout request ending in /api/checkout.

The request asks the app to save the order note.
Its response carries the result.
You should find response code 200 and the supplied success message.

This recording lets you inspect what happened during the retry.
To inspect the first failure, open its own recording next.

Open the original failed attempt

npx playwright show-trace test-results/*retain-on-failure/trace.zip

Find the failed text check in the action list.
Compare the expected message with the received message.
The page received Injected first-attempt save failure.
The test expected Saved on retry by synthetic fixture.

Next, find /api/checkout under Network.
Inspect its response code and response body, the returned data.
The recording contains code 503 and the injected error message.

You can now connect the received response with the page's message and failed check.
In this example, the test supplied the error on purpose.
This demonstrates saved evidence, rather than diagnosing a real service outage.

The Trace Viewer guide explains its inspection panels.

Keep a record of what you found

Open the complete test report with:

npm run show-report

For the verifier's findings, open artifacts/verification-report.json in your editor.
JSON is a structured data format.
This file lists attempt numbers, saved response bodies, and recording file locations.

The verifier reads those recordings to confirm the comparison.
A filename alone doesn't establish which response the browser received.
The tested project revision preserves the version used for this article.

For your own investigation, record a few concrete details:

The test name and attempt number.
The expected result and what appeared instead.
The relevant response and a link to the saved recording.

That gives the next person somewhere useful to start.

Choose recording settings for your project

If you need the original failure, choose a mode that keeps that attempt.
If you're investigating the retry, record the retry.

Recording more attempts uses processing time and storage.
Saved pages and responses can also contain private data.
Decide who can open those files and when to delete them.

Try this comparison on the practice project before changing your team's settings.
You'll know which recording to look for when a real test fails.

How to Fix Flaky Tests in Playwright: 10 Battle-Tested Strategies

Anton Gulin — Tue, 15 Sep 2026 15:01:17 +0000

The Cost of Flaky Tests

A flaky test is one that sometimes passes and sometimes fails without any code changes. They seem harmless at first. Just re-run the pipeline, right?

Wrong. Flaky tests are silent killers:

Eroded trust: Teams stop believing test results
Wasted time: Engineers debug tests instead of building features
Slower releases: "Just re-run it" becomes the norm
Hidden bugs: Real failures get dismissed as "flaky" At one company, I inherited a test suite with a 68% pass rate. Not because the application was broken, but because the tests were. Here's how I fixed it.

Strategy 1: Use Role-Based Locators

The problem: CSS selectors and XPath break when developers change class names or restructure HTML.

The fix: Use Playwright's role-based locators:

Bad:

page.locator('#submit-btn')

page.locator('.form-container > button:nth-child(2)')

Good:

page.getByRole('button', { name: 'Submit' })

page.getByLabel('Email address')

page.getByText('Welcome back')

Role-based locators are more resilient because they target what users see, not implementation details.

Strategy 2: Never Use Hard-Coded Waits

*The problem: *waitForTimeout() is the number one cause of flaky tests.

await page.waitForTimeout(5000); // ❌ Please don't do this

Why it fails: 5 seconds might be enough on your machine but not in CI. Or it might be way too long, slowing tests unnecessarily.

The fix: Wait for specific conditions:

await page.waitForLoadState('networkidle');

await expect(page.getByRole('button')).toBeEnabled();

await expect(page.getByText('Success')).toBeVisible();

Playwright's auto-waiting handles most cases automatically. Trust it.

Strategy 3: Isolate Test State

The problem: Tests depend on state from previous tests.

test('login', ...); // Creates session

test('add to cart', ...); // Expects logged-in state

If the login test fails, the cart test also fails, but not because of a cart bug.

The fix: Each test should set up its own state:

test.beforeEach(async ({ page }) => {
  await loginAsUser(page, 'testuser');
});

Or use Playwright's storage state to share authentication without dependencies:

await page.context().storageState({ path: 'auth.json' });

Strategy 4: Handle Loading States Explicitly

The problem: Clicking a button that's still loading, or reading text before it's rendered.

The fix: Wait for loading indicators to disappear:

// Wait for spinner to go away
await expect(page.getByTestId('loading-spinner')).toBeHidden();
// Then interact with the element
await page.getByRole('button', { name: 'Submit' }).click();

Or wait for the element to be in a specific state:

await expect(page.getByRole('button')).toBeEnabled();
await page.getByRole('button').click();

Strategy 5: Use data-testid for Dynamic Content

The problem: Elements generated dynamically have unpredictable locators.

The fix: Add data-testid attributes for testing:

In your application code:

<button data-testid="checkout-button">Checkout</button>

In your test:

await page.getByTestId('checkout-button').click();

This creates a contract between frontend and tests that survives refactoring.

Strategy 6: Retry Failed Assertions (Not Whole Tests)

The problem: A test fails once and you re-run the entire suite.

The fix: Use Playwright's built-in expect retries:

// playwright.config.ts
export default defineConfig({
  expect: {
    timeout: 10000, // Wait up to 10 seconds for assertions
  },
});

Assertions like toBeVisible() and toHaveText() will automatically retry until timeout. No manual retries needed.

Strategy 7: Handle Network Variability

The problem: API calls take longer in CI than locally.

The fix: Wait for network responses explicitly:

// Wait for specific API call to complete
await page.waitForResponse(resp =>
  resp.url().includes('/api/products') && resp.status() === 200
);

Or use networkidle for simpler cases:

await page.goto('/dashboard', { waitUntil: 'networkidle' });

Strategy 8: Run Tests in Parallel Correctly

The problem: Tests interfere with each other when running in parallel.

Test A creates user "testuser@example.com"

Test B also creates user "testuser@example.com"

One fails due to duplicate email.

The fix: Use unique data per test:

import { faker } from '@faker-js/faker';
const email = faker.internet.email();
await page.getByLabel('Email').fill(email);

Or isolate tests in separate browser contexts (Playwright does this by default).

Strategy 9: Use Trace Viewer for Debugging

The problem: You can't see what happened when a test failed in CI.

The fix: Turn traces on for failures:

// playwright.config.ts
export default defineConfig({
  use: {
    trace: 'on-first-retry',
  },
});

Now, when a test fails and retries, Playwright captures:

Screenshots at every step
DOM snapshots
Network requests
Console logs Open traces with:

npx playwright show-trace trace.zip

This is the single best debugging tool for flaky tests.

Strategy 10: Set Realistic Timeouts

The problem: Default timeouts are too short for slow environments.

The fix: Configure appropriate timeouts based on your CI environment:

// playwright.config.ts
export default defineConfig({
  timeout: 60000,        // Test timeout: 60 seconds
  expect: {
    timeout: 10000,      // Assertion timeout: 10 seconds
  },
  use: {
    actionTimeout: 15000, // Click/fill timeout: 15 seconds
    navigationTimeout: 30000, // Page load timeout: 30 seconds
  },
});

Don't make them too long. Slow failures are frustrating. Find the right balance for your infrastructure.

Bonus: The Flaky Test Triage Process

When you encounter a flaky test, follow this process:

1. Reproduce locally

Run the test 10 times:

npx playwright test tests/checkout.spec.ts --repeat-each=10

If it passes every time locally but fails in CI, it's likely a timing or environment issue.

2. Check the trace

Open the trace file from CI and look for:

Slow network requests
Elements not visible when clicked
Unexpected modal or popup 3. Identify the root cause

Common causes:

Hard-coded wait times
Race conditions
Shared test state
Unstable locators 4. Fix or delete

If you can't fix it after 30 minutes, delete it. A flaky test is worse than no test. You can always rewrite it properly later.

My Flaky Test Scorecard

After applying these strategies on one project, we went from a 68% pass rate to 98%+ in three months.

Pass rate:

Before: 68%
After: 98.5%
Average test time:
Before: 4.2 minutes
After: 1.8 minutes
"Re-run pipeline" requests:
Before: Daily
After: Rare
Team trust in automation:
Before: Low
After: High
The biggest win wasn't technical. It was cultural. When tests are reliable, developers actually care about failures.

Need Help with Your Flaky Tests?

If your test suite is unreliable and slowing down your team, I can help. I've fixed test suites at Fortune 500 companies and can usually identify the main issues in just a few hours.

Book a consultation

Let's turn a browser task into a Playwright test

Anton Gulin — Sat, 12 Sep 2026 15:01:06 +0000

Originally published on anton.qa.

This walkthrough teaches one thing: turn recorded actions into a test that checks the result.

Our example is a small practice website with a display-name field.
The name starts as Alex. We change it to Sam and press Save.
We are editing this website, not a Chrome profile or Google account.

Playwright is a browser testing tool. It can record those actions as code.
We add a check: reload the page and confirm the name is still Sam.
The finished test repeats the actions and checks the result automatically.

Here is the problem that check can catch.
Our demo includes a deliberately broken Save button.
It says Saved, but reloading brings back Alex.
The test expects Sam, finds Alex, and fails.
With saving fixed, the same test passes.

The lesson: record the steps, then check that the intended change actually happened.

The sections below show how to build and run that example.
Playwright's command-line tool accepts typed terminal commands.
Version 0.1.19 added commands for recording browser actions.
The recorder provides the actions. We choose and add the result checks.

An agent ran this example while preparing the article.
It uses invented names and browser storage, not a customer account.

1. Create a small practice app

Use Node.js 24, which runs JavaScript programs.
The files use TypeScript, JavaScript with types.

Create a folder, then run these commands inside it:

npm init -y
npm pkg set type=module
npm install --save-dev @playwright/test@1.63.0 @playwright/cli@0.1.19
npx playwright install chromium
mkdir tests

Create server.ts with this app:

import { createServer } from 'node:http';

createServer((request, response) => {
  const url = new URL(request.url ?? '/', 'http://127.0.0.1');
  const broken = url.searchParams.get('broken') === '1';
  response.setHeader('Content-Type', 'text/html; charset=utf-8');
  response.end(`<!doctype html>
<html lang="en"><meta charset="utf-8"><title>Profile practice</title>
<body><main><h1>Profile settings</h1>
<form><label for="name">Display name</label>
<input id="name"><button>Save</button></form>
<p role="status"></p>
<p>Current name: <strong data-testid="current-name"></strong></p>
<script>
const field = document.querySelector('#name');
const current = document.querySelector('[data-testid="current-name"]');
field.value = localStorage.getItem('displayName') || 'Alex';
current.textContent = field.value;
document.querySelector('form').addEventListener('submit', event => {
  event.preventDefault();
  if (!${broken}) localStorage.setItem('displayName', field.value);
  current.textContent = localStorage.getItem('displayName') || 'Alex';
  document.querySelector('[role="status"]').textContent = 'Saved';
});
</script></main></body></html>`);
}).listen(4187, '127.0.0.1');

Browser storage keeps the name between page reloads.
This small app has a deliberate broken mode.
Adding ?broken=1 makes Save display success without storing the name.
That gives us a known problem to test.

Start the app in one terminal:

node server.ts

2. Record one task

In a second terminal, open a visible browser:

npx playwright-cli -s=profile open http://127.0.0.1:4187 --headed
npx playwright-cli -s=profile recording-start

Change Display name from Alex to Sam. Click Save.
Then stop recording:

npx playwright-cli -s=profile recording-stop
npx playwright-cli -s=profile close

The checked example returned these actions:

await page.getByRole('textbox', { name: 'Display name' }).click();
await page.getByRole('textbox', { name: 'Display name' }).press('ControlOrMeta+A');
await page.getByRole('textbox', { name: 'Display name' }).fill('Sam');
await page.getByRole('button', { name: 'Save' }).click();

Your output can differ with your clicks, browser, and operating system.
The local proof used simulated pointer and keyboard input.
These actions came from the recorder. The saved-result checks below were added separately.

We can simplify these steps. fill replaces the field's current text.
The first two actions are unnecessary for this particular test.

The recording describes what happened in the browser. It does not decide whether saving worked.

A field label also makes the action easier to read. Here, the test finds the textbox named Display name. It finds the Save button. Those descriptions connect the code to controls the reader can see.

For another page, review the recorded descriptions before keeping them. A changing number or unrelated click may not belong in the finished test.

3. Check the result after reload

Stop the app in the first terminal with Control-C.
The test runner will start its own copy.

Create playwright.config.ts:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  retries: 0,
  use: { baseURL: 'http://127.0.0.1:4187' },
  webServer: {
    command: 'node server.ts',
    url: 'http://127.0.0.1:4187',
    reuseExistingServer: false,
  },
});

Create tests/save.spec.ts:

import { test, expect } from '@playwright/test';

test('Save keeps the edited name after reload', async ({ page }) => {
  const path = process.env.APP_BROKEN === '1' ? '/?broken=1' : '/';
  await page.goto(path);

  await page.evaluate(() => localStorage.removeItem('displayName'));
  await page.reload();
  await expect(page.getByLabel('Display name')).toHaveValue('Alex');

  await page.getByRole('textbox', { name: 'Display name' }).fill('Sam');
  await page.getByRole('button', { name: 'Save', exact: true }).click();
  await expect(page.getByRole('status')).toHaveText('Saved');

  await page.reload();
  await expect(page.getByLabel('Display name')).toHaveValue('Sam');
  await expect(page.getByTestId('current-name')).toHaveText('Sam');
});

expect checks that the app matches an expected result.
Here, Sam must remain in both places after reloading.

The status message is useful feedback. It cannot prove that the name was saved.
Our final checks cover that missing part.

We also restore Alex before editing. Each run starts from a known value.
Playwright gives each test a fresh browser context, an isolated browser session.
That does not reset a real shared server or database.
This example uses only browser storage.

The reset and the final check answer different questions. The reset establishes where the test begins. The final check establishes what the action must preserve. Keeping both visible makes the example easier to change later.

4. Confirm the test catches the problem

Run the broken app mode first:

# macOS or Linux
APP_BROKEN=1 npx playwright test tests/save.spec.ts

PowerShell users can set the same value separately:

$env:APP_BROKEN='1'
npx playwright test tests/save.spec.ts

The test should fail after reloading. It expects Sam and finds Alex.
The Save message still said Saved, which explains why clicking alone was insufficient.

Now run the working mode:

# macOS or Linux
APP_BROKEN=0 npx playwright test tests/save.spec.ts

$env:APP_BROKEN='0'
npx playwright test tests/save.spec.ts

The same test should pass.

If your result differs, read the failed check before changing the test. Does it show Alex where Sam was expected? That is the intended failure in broken mode. A browser launch error or an occupied port is a setup problem instead.

First confirm the example opened the page. Then compare the expected name with the actual name. Keeping that distinction helps you fix the right part. A failed setup does not demonstrate that the saving check caught a defect.

During preparation, the broken run failed and the corrected run passed. Both used the same test, with no retries.
This proves the check caught our deliberately introduced problem.
It does not prove coverage for every possible saving problem.

5. Choose the next small flow

Try this with one task you already understand.

Write the expected result before adding more recorded steps.
For a saved setting, reload and check the value.
For a submitted form, check the resulting record where your test can inspect it.
Use test accounts and practice data when recording.

Keep unrelated clicks out of the finished test.
Give each test its own data where possible.
Keep checks that explain the user's expected outcome.

The useful question is simple: what should still be true after this action?

The recorder helps with the steps. Your understanding supplies the reason to check them.

Sources

Playwright CLI 0.1.19 release: recording commands and their scope.
Playwright best practices: test isolation and user-visible checks.
Playwright assertions: checks that wait for expected results.

About the author

Playwright Tutorial for Beginners: Your First Test in 10 Minutes

Anton Gulin — Tue, 08 Sep 2026 15:01:15 +0000

What You'll Learn

By the end of this tutorial, you'll have:

Playwright installed on your machine
Written and run your first automated test
Understood the basic Playwright concepts
A foundation to build real-world test suites Total time: about 10 minutes. Let's go.

Prerequisites

You need Node.js 18+ installed. Check your version:

node --version

If you need to install Node.js, download it from nodejs.org.

That's it. No other dependencies required.

Step 1: Create Your Project

Open your terminal and create a new folder:

mkdir playwright-tutorial

cd playwright-tutorial

Step 2: Install Playwright

Run the Playwright installer:

npm init playwright@latest

You'll see some prompts. Accept the defaults:

TypeScript or JavaScript? → TypeScript (recommended)
Where to put tests? → tests
Add GitHub Actions? → Yes (optional, but useful later)
Install browsers? → Yes This installs Playwright, downloads browser binaries, and creates a basic project structure.

Step 3: Explore the Project Structure

After installation, you'll see:

tests/: Your test files go here
playwright.config.ts: Configuration file
tests/example.spec.ts: A sample test (we'll replace this)

Step 4: Write Your First Test

Delete the example test and create a new file called tests/first.spec.ts:

import { test, expect } from '@playwright/test';
test('homepage has correct title', async ({ page }) => {
  await page.goto('https://playwright.dev');
  await expect(page).toHaveTitle(/Playwright/);
});

What this does:

test(): Defines a test with a name
page.goto(): Navigates to a URL
expect().toHaveTitle(): Asserts the page title contains "Playwright"

Step 5: Run Your Test

In your terminal:

npx playwright test

You should see:

Running 1 test using 1 worker
✓ first.spec.ts:3:1 › homepage has correct title (1.2s)
1 passed (2.1s)

Congratulations! You just ran your first Playwright test.

Step 6: See the Test Report

Playwright generates beautiful HTML reports. View it:

npx playwright show-report

This opens a browser with your test results, including timing, screenshots, and traces.

Step 7: Run Tests in UI Mode (Game Changer)

Playwright's UI mode lets you watch tests run in real-time:

npx playwright test --ui

This opens a visual interface where you can:

Watch tests execute step-by-step
Inspect DOM at any point
Debug failures visually
Re-run individual tests This is my favorite Playwright feature for development.

Step 8: Add More Assertions

Let's expand our test to click a link and verify navigation:

import { test, expect } from '@playwright/test';
test('can navigate to getting started page', async ({ page }) => {
  await page.goto('https://playwright.dev');
  // Click the Getting Started link
  await page.getByRole('link', { name: 'Get started' }).click();
  // Verify we're on the right page
  await expect(page).toHaveURL(/.*intro/);
  // Check the heading exists
  await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
});

New concepts:

getByRole(): Finds elements by their accessibility role (best practice)
click(): Clicks an element
toHaveURL(): Asserts the current URL matches a pattern
toBeVisible(): Asserts an element is visible on page

Step 9: Use the Code Generator

Playwright can generate test code by recording your actions:

npx playwright codegen playwright.dev

This opens a browser. Click around, and Playwright writes code for you in real-time. Copy the generated code into your test file.

This is extremely useful for learning locators and speeding up test creation.

Understanding Key Concepts

Locators:

Playwright finds elements using locators. Best practices:

getByRole(): By accessibility role (button, link, heading)
getByText(): By visible text
getByLabel(): By form label
getByTestId(): By data-testid attribute Avoid CSS selectors and XPath when possible. Role-based locators are more resilient.

Auto-waiting:

Playwright automatically waits for elements to be ready before interacting. No more:

await page.waitForSelector('.button'); // Not needed!

Just write:

await page.click('.button'); // Playwright handles waiting

This dramatically reduces flaky tests.

Next Steps

Now that you have the basics:

Add more tests: Start testing your own application
Learn Page Object Model: Organize tests for maintainability
Set up CI/CD: Run tests on every pull request
Explore API testing: Playwright also tests REST APIs

Common Beginner Mistakes to Avoid

1. Using sleep/delay:

Bad:

await page.waitForTimeout(5000);

Good: Let Playwright's auto-waiting handle it, or wait for specific conditions.

2. Fragile locators:

Bad:

page.locator('#app > div:nth-child(3) > button')

Good:

page.getByRole('button', { name: 'Submit' })

3. Not using test isolation:

Each test should be independent. Don't rely on state from previous tests.

Resources

Official docs: playwright.dev/docs/intro
Test generator:
npx playwright codegen [url]
UI mode:
npx playwright test --ui
Report viewer:
npx playwright show-report

Need Hands-On Help?

If you're building a test automation framework for your team and want guidance from someone who's done it at Apple and Fortune 500 companies, I offer consulting and training.

Learn more about working with me

How to tell whether an AI feature actually works

Anton Gulin — Sat, 05 Sep 2026 20:23:16 +0000

Your AI feature worked when you demoed it. That is the only evidence most teams collect before shipping.

In August 2026 Microsoft published a benchmark that measures the thing that demo misses. They ran 507 real business workflows through AI agents, across retail, hospitality, auto insurance, banking IT and consulting support. Every workflow was attempted 20 times, and each attempt was graded on what actually changed in the backend database, not on what the agent said it did.

The strongest model completed 65.36% of workflows on the first try. It completed all twenty attempts on only 25.25% of them.

Their paper is titled "One Success Isn't Reliability."

Testers have a shorter word for the gap between those two numbers. We call it a flaky test: one that passes and fails on the same code, with nothing changed but time.

This post is the method for measuring it in your own product.

Why one good run tells you almost nothing

A traditional test is deterministic. Given the same input and the same code, it returns the same answer every time. When it fails, something is broken.

An AI feature is not. The same prompt against the same model can return a different answer on Tuesday than it did on Monday. Temperature, model updates, retrieval order, and load all move the output.

So "it worked" is not a property of the feature. It is a property of one attempt. The question worth answering is: how often, and how far does it drift when it misses?

The ten-run check

Here is the smallest honest version. Drop it in a spec file, replace score() with whatever "good" means for your feature, and run it against the AI path you trust least.

import { test, expect } from '@playwright/test'

const RUNS = 10
const MIN_PASS_RATE = 0.9   // 9 of 10 runs must be good enough
const MAX_SPREAD = 0.34     // best run minus worst run

test('the summary feature holds up over ten runs', async ({ page }) => {
  const scores: number[] = []

  for (let i = 0; i < RUNS; i++) {
    await page.goto('/summarize')
    await page.getByRole('button', { name: 'Summarize' }).click()
    const text = await page.getByTestId('summary').innerText()
    scores.push(score(text))
  }

  const passRate = scores.filter(s => s >= 0.67).length / RUNS
  const spread = Math.max(...scores) - Math.min(...scores)
  console.log(`pass rate ${passRate}  spread ${spread.toFixed(2)}  [${scores}]`)

  expect(passRate).toBeGreaterThanOrEqual(MIN_PASS_RATE)
  expect(spread).toBeLessThanOrEqual(MAX_SPREAD)
})

// Yours to write. Start with the cheapest checks that are not "did it crash".
function score(text: string) {
  let s = 0
  if (text.length > 40) s += 0.34                  // it said something
  if (/\d{4}-\d{2}-\d{2}/.test(text)) s += 0.33    // it kept the date
  if (!/i cannot|as an ai/i.test(text)) s += 0.33  // it did not refuse
  return s
}

Two things make this useful instead of noisy.

Log all ten scores every run, and keep them. One bad day means nothing. The same feature drifting down across a week is the real signal.

Fail on the spread, not only on the average. A feature scoring 1.0, 1.0, 0.33, 1.0 has a worse problem than one scoring 0.8 every time. The average hides it. The spread does not.

The flag you already have

If you would rather not write a loop, Playwright ships this:

npx playwright test --repeat-each=10

It runs the same test ten times, and every run has to pass.

That tells you whether something failed. It does not tell you how far the answers moved, which is the part that warns you early. Use it as the free first step, then graduate to scoring when the feature matters.

The setting that hides all of this

Most projects carry this in playwright.config.ts:

export default defineConfig({
  retries: 2,
})

A failed test gets two more chances. If any of them passes, the run reports the test as flaky and still exits 0. The build goes green and nobody opens it.

Retries exist for a good reason: they keep a shared pipeline usable. But they are the opposite tool from the one above. Retries let one pass overwrite two failures. Repetition refuses to let that happen.

You need both, and most teams only have the first.

What to do this week

Pick the three AI-backed paths you trust least.
Run each ten times and write down the pass rate and the spread.
Decide, before you see the numbers, what pass rate would stop a release.
Put that number in the test, not in a document.

Step 3 is the one teams skip, and it is the one that makes the rest mean anything. A threshold chosen after seeing the result is not a threshold.

The part worth keeping

Microsoft needed 507 workflows, twenty runs each, and a graded backend state to say something testers have said for twenty years: one success is not reliability.

The measurement is not new. What is new is that the software now behaves differently on Tuesday than it did on Monday, so everyone needs the measurement, not just the people with "quality" in their title.

Page Object Model in Playwright with TypeScript: The Complete Guide

Anton Gulin — Tue, 01 Sep 2026 15:00:46 +0000

What is the Page Object Model?

The Page Object Model (POM) is a design pattern that creates an abstraction layer between your tests and your web pages. Instead of writing locators directly in tests, you encapsulate page elements and actions inside reusable classes.

Without POM (hard to maintain):

test('user can login', async ({ page }) => {
  await page.goto('/login');
  await page.locator('#email').fill('user@example.com');
  await page.locator('#password').fill('password123');
  await page.locator('button[type="submit"]').click();
  await expect(page.locator('.dashboard-welcome')).toBeVisible();
});

With POM (scalable):

test('user can login', async ({ page }) => {
  const loginPage = new LoginPage(page);
  await loginPage.goto();
  await loginPage.login('user@example.com', 'password123');
  await expect(loginPage.welcomeMessage).toBeVisible();
});

When the login form changes, you update one file, not 50 tests.

Why Use Page Objects?

1. Maintainability

Locators live in one place. When UI changes, update once.

2. Readability

Tests read like user stories: loginPage.login(), dashboard.navigateToSettings().

3. Reusability

4. Separation of concerns

Tests focus on behavior. Page objects handle implementation.

5. Team scalability

New team members understand tests faster.

Project Structure

Here's the structure I use for all my Playwright projects:

tests/
├── pages/
│   ├── BasePage.ts
│   ├── LoginPage.ts
│   ├── DashboardPage.ts
│   └── CheckoutPage.ts
├── components/
│   ├── Header.ts
│   ├── Modal.ts
│   └── Sidebar.ts
├── fixtures/
│   └── fixtures.ts
├── specs/
│   ├── login.spec.ts
│   ├── dashboard.spec.ts
│   └── checkout.spec.ts
└── playwright.config.ts

Creating a Base Page Class

Start with a base class that all pages extend:

// tests/pages/BasePage.ts
import { Page, Locator } from '@playwright/test';
export abstract class BasePage {
  protected page: Page;
  constructor(page: Page) {
    this.page = page;
  }
  async goto(): Promise<void> {
    await this.page.goto(this.url);
  }
  abstract get url(): string;
  // Common elements across all pages
  get header(): Locator {
    return this.page.getByRole('banner');
  }
  get footer(): Locator {
    return this.page.getByRole('contentinfo');
  }
  // Common actions
  async waitForPageLoad(): Promise<void> {
    await this.page.waitForLoadState('networkidle');
  }
}

Creating a Login Page Object

Now create a specific page class:

// tests/pages/LoginPage.ts
import { Page, Locator, expect } from '@playwright/test';
import { BasePage } from './BasePage';
export class LoginPage extends BasePage {
  constructor(page: Page) {
    super(page);
  }
  get url(): string {
    return '/login';
  }
  // Locators (use getters for lazy evaluation)
  get emailInput(): Locator {
    return this.page.getByLabel('Email address');
  }
  get passwordInput(): Locator {
    return this.page.getByLabel('Password');
  }
  get submitButton(): Locator {
    return this.page.getByRole('button', { name: 'Sign in' });
  }
  get errorMessage(): Locator {
    return this.page.getByRole('alert');
  }
  get forgotPasswordLink(): Locator {
    return this.page.getByRole('link', { name: 'Forgot password?' });
  }
  // Actions
  async login(email: string, password: string): Promise<void> {
    await this.emailInput.fill(email);
    await this.passwordInput.fill(password);
    await this.submitButton.click();
  }
  async loginAndExpectDashboard(email: string, password: string): Promise<void> {
    await this.login(email, password);
    await expect(this.page).toHaveURL(/.*dashboard/);
  }
  async loginWithInvalidCredentials(email: string, password: string): Promise<void> {
    await this.login(email, password);
    await expect(this.errorMessage).toBeVisible();
  }
}

Creating a Dashboard Page Object

// tests/pages/DashboardPage.ts
import { Page, Locator } from '@playwright/test';
import { BasePage } from './BasePage';
export class DashboardPage extends BasePage {
  constructor(page: Page) {
    super(page);
  }
  get url(): string {
    return '/dashboard';
  }
  // Locators
  get welcomeMessage(): Locator {
    return this.page.getByRole('heading', { name: /Welcome/ });
  }
  get userMenu(): Locator {
    return this.page.getByRole('button', { name: 'User menu' });
  }
  get logoutButton(): Locator {
    return this.page.getByRole('menuitem', { name: 'Logout' });
  }
  get settingsLink(): Locator {
    return this.page.getByRole('link', { name: 'Settings' });
  }
  // Actions
  async logout(): Promise<void> {
    await this.userMenu.click();
    await this.logoutButton.click();
  }
  async navigateToSettings(): Promise<void> {
    await this.settingsLink.click();
  }
}

Using Page Objects in Tests

Now your tests are clean and readable:

// tests/specs/login.spec.ts
import { test, expect } from '@playwright/test';
import { LoginPage } from '../pages/LoginPage';
import { DashboardPage } from '../pages/DashboardPage';
test.describe('Login functionality', () => {
  test('user can login with valid credentials', async ({ page }) => {
    const loginPage = new LoginPage(page);
    const dashboardPage = new DashboardPage(page);
    await loginPage.goto();
    await loginPage.login('valid@example.com', 'password123');
    await expect(dashboardPage.welcomeMessage).toBeVisible();
  });
  test('user sees error with invalid credentials', async ({ page }) => {
    const loginPage = new LoginPage(page);
    await loginPage.goto();
    await loginPage.loginWithInvalidCredentials('invalid@example.com', 'wrong');
    await expect(loginPage.errorMessage).toContainText('Invalid credentials');
  });
  test('user can navigate to forgot password', async ({ page }) => {
    const loginPage = new LoginPage(page);
    await loginPage.goto();
    await loginPage.forgotPasswordLink.click();
    await expect(page).toHaveURL(/.*forgot-password/);
  });
});

Advanced: Component Objects

For reusable UI components (modals, headers, sidebars), create component classes:

// tests/components/Modal.ts
import { Page, Locator } from '@playwright/test';
export class Modal {
  private page: Page;
  private container: Locator;
  constructor(page: Page) {
    this.page = page;
    this.container = page.getByRole('dialog');
  }
  get title(): Locator {
    return this.container.getByRole('heading');
  }
  get closeButton(): Locator {
    return this.container.getByRole('button', { name: 'Close' });
  }
  get confirmButton(): Locator {
    return this.container.getByRole('button', { name: 'Confirm' });
  }
  get cancelButton(): Locator {
    return this.container.getByRole('button', { name: 'Cancel' });
  }
  async close(): Promise<void> {
    await this.closeButton.click();
  }
  async confirm(): Promise<void> {
    await this.confirmButton.click();
  }
}

Use it in page objects:

// In DashboardPage.ts
async openDeleteModal(): Promise<Modal> {
  await this.deleteButton.click();
  return new Modal(this.page);
}

Advanced: Using Playwright Fixtures

For even cleaner tests, integrate page objects with Playwright fixtures:

// tests/fixtures/fixtures.ts
import { test as base } from '@playwright/test';
import { LoginPage } from '../pages/LoginPage';
import { DashboardPage } from '../pages/DashboardPage';
type MyFixtures = {
  loginPage: LoginPage;
  dashboardPage: DashboardPage;
};
export const test = base.extend<MyFixtures>({
  loginPage: async ({ page }, use) => {
    await use(new LoginPage(page));
  },
  dashboardPage: async ({ page }, use) => {
    await use(new DashboardPage(page));
  },
});
export { expect } from '@playwright/test';

Now tests are even cleaner:

// tests/specs/login.spec.ts
import { test, expect } from '../fixtures/fixtures';
test('user can login', async ({ loginPage, dashboardPage }) => {
  await loginPage.goto();
  await loginPage.login('user@example.com', 'password123');
  await expect(dashboardPage.welcomeMessage).toBeVisible();
});

Best Practices

1. Use getters for locators

Getters are lazily evaluated. The locator is resolved when accessed, not when the class is instantiated.

// Good
get submitButton(): Locator {
  return this.page.getByRole('button', { name: 'Submit' });
}
// Avoid (evaluated too early)
submitButton = this.page.getByRole('button', { name: 'Submit' });

2. Keep actions atomic

Each method should do one logical thing:

// Good
async login(email: string, password: string) { ... }
async fillEmail(email: string) { ... }
// Avoid combining unrelated actions
async loginAndCheckSettings() { ... }

3. Don't add assertions in page objects (usually)

Assertions belong in tests. Page objects encapsulate interactions.

Exception: Verification methods like loginAndExpectDashboard() can include assertions for convenience.

4. Use role-based locators

More resilient than CSS selectors:

// Good
page.getByRole('button', { name: 'Submit' })
// Fragile
page.locator('#submit-btn-primary')

Common Mistakes to Avoid

1. God page objects

If your page object has 50+ methods, break it into smaller components.

2. Exposing raw locators everywhere

Encapsulate interactions: loginPage.login() instead of loginPage.submitButton.click().

3. Not waiting for navigation

After clicks that navigate, wait for the URL or element:

await this.submitButton.click();

await this.page.waitForURL(/.*dashboard/);

4. Sharing state between tests

Each test should be independent. Don't rely on previous test state.

Folder Structure for Large Projects

For 100+ tests, organize by feature:

tests/
├── pages/
├── components/
├── fixtures/
├── auth/
│   ├── login.spec.ts
│   ├── register.spec.ts
│   └── password-reset.spec.ts
├── checkout/
│   ├── cart.spec.ts
│   ├── payment.spec.ts
│   └── confirmation.spec.ts
└── admin/
    ├── users.spec.ts
    └── settings.spec.ts

Need Architecture Help?

If you're building a test automation framework and want it done right from the start, I can help design the architecture, set up page objects, and train your team on best practices.

Get in touch

You Were Handed Testing and No One Trained You: A Developer's Field Guide

Anton Gulin — Sat, 29 Aug 2026 15:00:41 +0000

Your team moved testing to the left. That is the industry's word for it: shift-left. Testing that used to happen at the end, done by a separate QA team, now happens earlier, done by the people writing the code.

On my most recent project, that meant the developers. Most had never opened Playwright. None of them thought like a tester. And they still had to ship tested code every sprint.

The same five questions came in every week. This is the field guide I wish I could have handed each of them on day one. If testing just landed on your desk, start here.

1. "How do I wait for the page to be ready?"

The instinct is to sleep for a few seconds and hope. Do not. A sleep is a guess. Guess too short and the test is flaky. Guess too long and your suite crawls.

Playwright already waits for you. When you act on an element, it waits until that element is ready to use. Visible, stable, clickable. Then it acts.

// Fragile: a blind guess
await page.waitForTimeout(3000);
await page.locator('#save').click();

// Correct: Playwright waits for the button to be ready, then clicks
await page.getByRole('button', { name: 'Save' }).click();

Wait for the thing, never for the clock.

2. "Why does my test pass, then fail with no change?"

That is a flaky test. It is not bad luck. It means two things raced and your test did not say which one had to finish first.

The fix is not to add a sleep. The fix is to wait for the real signal, the row appears, the spinner is gone, the network call returned.

// Flaky: asserts before the row has rendered
await page.getByRole('button', { name: 'Add' }).click();
await expect(page.getByRole('row')).toHaveCount(1);

// Stable: web-first assertion retries until the state is true
await page.getByRole('button', { name: 'Add' }).click();
await expect(page.getByRole('row', { name: 'New item' })).toBeVisible();

A flaky test is not a small annoyance. It teaches the team to ignore red, and an ignored red is where real bugs ship.

3. "What am I even supposed to check?"

Check what the user gets. Not what the console logs, not that the page loaded, the actual outcome on the screen.

// Weak: proves the page rendered, nothing more
await expect(page).toHaveURL(/checkout/);

// Strong: proves the user sees the right total
await expect(page.getByTestId('order-total')).toHaveText('$41.00');

If your assertion would still pass when the feature is broken, it is decoration, not proof.

4. "My tests are so slow."

Most slow suites log in through the UI before every single test. If you have 200 tests, that is 200 logins. Do it once.

// One-time setup: save the signed-in state
await page.context().storageState({ path: 'auth.json' });

// Every test reuses it, no login step
test.use({ storageState: 'auth.json' });

Testing the login belongs in one test. The other 199 should assume it works.

5. "It works on my machine."

Then it is not tested. It is remembered. A test that only passes with your leftover data and your logged-in browser is not proof anyone else can trust.

Run it clean and isolated, every time: fresh data, no shared login, no state left behind from the last run. If it passes on a machine that knows nothing about you, it is real.

The one habit under all five

A test is a claim that the software works. Your job is to make that claim honest. Wait for real signals, check what the user sees, keep each test independent, and when a test fails, read the evidence before you touch it. Playwright's trace viewer records exactly what the browser did, so you can tell a real bug from a broken test in about two minutes.

Shift-left handed you testing. Nobody handed you the manual. Now you have a start.

Your Regression Suite Is a Museum: 5 Questions That Decide Delete vs Keep

Anton Gulin — Sat, 22 Aug 2026 15:00:46 +0000

Every sprint adds tests. Almost no sprint deletes one.

That one-way flow has a predictable end. The suite gets slower. Flaky tests
(tests that fail randomly) pile up. Engineers stop trusting red builds.
Eventually someone suggests rewriting everything, and the cycle restarts.

There is a cheaper fix. Audit the suite with five questions.
On the last suite I inherited, they retired a third of the tests.
Nothing those tests "guarded" ever broke.

Question 1: When did this test last fail for a real reason?

Your CI history (the record of server test runs) already knows.

A test that has been green for a year has two possible explanations.
Either the code it guards never changed, or the test cannot detect change.
Both are worth knowing. Only one deserves compute on every commit.

Practical rule: pull the last 90 days of runs. Tag every test that failed
only for environment reasons, or never failed at all. Those are audit candidates.

Question 2: What user risk does it guard?

Write the risk in one sentence, naming a user.

"It tests the profile page" is not a risk.
"A user loses saved work when the session expires" is a risk.

If nobody on the team can produce that sentence, the test is guarding
an implementation detail, not a user. Details change on purpose all the time.
Tests guarding them fail on purpose all the time. That is where flakiness lives.

Question 3: Would anyone notice if it vanished?

Run the experiment instead of debating it:

# in a branch: skip the suspect test, run everything else
npx playwright test --grep-invert "@museum-candidate"

If coverage of the named risk survives through other tests, and no gap
appears in the risk list from Question 2, the test was furniture.

Delete it in the branch. Keep the branch open a week. Merge with confidence.

Question 4: Does it check results or steps?

A test that clicks through checkout and asserts the button was clickable
is a tour, not a test.

// a tour: asserts the step happened
await page.getByRole('button', { name: 'Pay' }).click();
await expect(page).toHaveURL(/confirmation/);

// a test: asserts the outcome is real
await expect(page.getByTestId('invoice-total')).toHaveText('$34.20');
await expect(page.getByTestId('invoice-number')).not.toBeEmpty();

The URL can change while the invoice ships blank. Assert on the thing
the user came for.

Question 5: Can it fail at all?

Flip the condition and run it.

// original
await expect(status).toBe('paid');
// flipped: this MUST fail. If it passes, the test is dead.
await expect(status).not.toBe('paid');

A test that passes both ways asserts nothing. AI-written tests fail this
check more than any other kind, because generators optimize for green.

Sixty seconds per suspicious test. The flip never lies.

What you get back

Five questions, one afternoon, on the oldest third of your suite.

The suite you keep is faster, and every test in it can answer
"what breaks if I go red?" That is what makes a red build mean something.

And when an AI agent starts writing tests into your suite, this audit
is the contract you hold its work against. A generated test enters
only if it survives the same five questions.

How to Review AI-Generated Tests: Seven Checks Before You Keep Them

Anton Gulin — Sat, 15 Aug 2026 15:00:43 +0000

Review AI-generated tests with seven checks. Name the user risk. Break the product. Check the final result. Change the data. Read the failure. Repeat the run. Then decide whether the test could stop a bad release.

AI can write a clean test in seconds. The file may look finished. The names may sound correct. The test may even pass.

None of those facts prove value. A useful test catches a failure that matters to users. Your review must find that proof before the test joins your suite.

I use seven checks for that decision.

1. Name the user risk

Start with the person who could get hurt. Write the risk in one sentence.

For example: "A customer sees the wrong total and pays too much."

Avoid risks like "checkout may fail." That sentence does not name the damage. It also gives the test no clear target.

Ask two questions:

Who loses something?
What do they lose?

Money, time, access, and trust are clear answers. "The feature breaks" is not.

2. Break the product

A test should fail when its protected behavior breaks. Prove that before keeping it.

AI often writes a test that follows the correct steps. The test may never check the important result.

test('customer can pay', async ({ page }) => {
  await page.goto('/checkout')
  await page.getByRole('button', { name: 'Pay' }).click()
  await expect(page.getByText('Success')).toBeVisible()
})

This test checks a message. It does not check the charged amount.

Remove the payment action or return the wrong total. The test must fail. A passing result means the test protects nothing useful.

3. Check the final result

Clicks are steps. Results are proof.

The test above clicks the right button. A stronger test checks the amount and payment record.

await expect(page.getByTestId('order-total')).toHaveText('$120.00')
await expect(page.getByTestId('payment-status')).toHaveText('Paid')

An assertion means a result check. Playwright provides assertions that wait for results. The tool can wait. You still choose the right result.

Review every assertion. Ask what user outcome it proves. Rewrite assertions that only confirm page activity.

4. Change the data

AI tends to generate a clean example. Real users bring missing, wrong, and extreme values.

Test more than one amount. Include zero, a large value, and invalid text.

for (const amount of ['0', '999999', 'wrong']) {
  await page.getByLabel('Amount').fill(amount)
  await page.getByRole('button', { name: 'Pay' }).click()
  await expect(page.getByRole('alert')).toBeVisible()
}

The exact values depend on your product. The review question stays simple. Can one neat example hide a serious failure?

5. Read the failure

Run the test against a broken result. Then read its message.

Another engineer should understand the problem without opening the whole file. Compare these messages:

Expected: "$20.00"
Received: "$120.00"

Timeout after 30000ms

The first message points to the product error. The second sends someone searching through logs.

Use clear result checks and useful test names. A failure should shorten the investigation.

6. Run it twice

Use the same input twice. You should get the same result.

Repeated runs catch shared data and timing problems. They also expose tests that depend on another test.

Do not accept a passing second run as proof. Compare both runs. Investigate any difference before keeping the test.

7. Make the release decision

Finish with one question: Would this test stop a bad release?

Name the release it could stop. For example: "This test blocks checkout when totals are wrong."

Keep the test when the answer is clear. Rewrite or delete it when the answer stays vague.

This step protects the review queue. Teams do not need every generated test. They need the small set that proves important behavior.

A five-minute review example

Imagine that AI writes a checkout test. The test adds one product and completes payment. It checks the success message.

First, name the risk. A customer could pay the wrong total.

Next, change the price calculation. The test still passes because the message appears. You found the missing proof.

Add a check for the order total. Then try an expired coupon and an empty cart. The test should explain each failed result.

Run the test twice with the same data. Both runs should match.

Finish with the release question. This test should stop checkout when the charged total is wrong. The answer now names one clear release failure.

That review improved one test without adding more code than needed. The team gained useful proof instead of another passing file.

Use the card during review

You can complete this review in a few minutes:

Name the user risk.
Break the protected behavior.
Check the final result.
Change the input data.
Read the failure message.
Repeat the same run.
Name the bad release it stops.

AI can handle test volume. You still decide what deserves trust. Keep tests that fail correctly and explain why the release should stop.

Anton Gulin builds AI systems and measurement tools. He tests software for a living and applies that discipline to AI. He runs independent LLM benchmarks and builds production test systems. Find him at anton.qa or on LinkedIn.

Test Retries Hide Real Bugs: When a Rerun Helps and When It Lies

Anton Gulin — Sat, 08 Aug 2026 15:00:40 +0000

Your build is green. One of your tests failed five minutes ago. Both of those things are true, because the test passed on a retry. Here is the short answer. A retry never fixes anything. Keep retries on in CI, but treat a retried pass as a bug report. It opens a ticket. It never closes one.

Last Saturday I posted about rerunning failed tests in isolation. Two readers pushed back, and both were more right than my post.

The first said flakiness is a signal. Turning the alarm off does not put out the fire.

The second was more specific. A team reruns the failed test alone. It passes. Everyone moves on. And the real cause, a race condition, ships to production. A race condition means two things run at the same time and collide.

This post is the concession, and the setup I now recommend.

What the rerun actually changed

Look at what happens when a test fails in CI and someone reruns it alone.

The code did not change. The data did not change. One thing changed: the test ran without the other tests around it.

So the rerun did not prove the test is fine. It proved the test is fine when nothing else is running. That is a different sentence. Your users do not visit your app one at a time.

The race that a rerun hides

Here is the simplest version of the trap:

test('admin can rename a user', async () => {
  await renameUser('user-42', 'New Name');
  await expect(profileName).toHaveText('New Name');
});

test('report shows user names', async () => {
  const report = await openReport();
  await expect(report.row('user-42')).toContainText('New Name');
});

Both tests touch the same record. Run together, they sometimes collide, and one fails. Run alone, each passes every time.

The failing run was the only honest one. It was telling you the app has a timing bug. The rerun in isolation deleted the evidence.

Playwright already tells you the truth

Here is the part most teams never look at. When a Playwright test fails and then passes on a retry, the report does not say "passed."

It says flaky.

  3 passed
  1 flaky

The tool is honest. The habit is not. Most dashboards only show green or red, so a flaky pass reads as a pass, and the count nobody reads keeps growing.

The setup that keeps retries useful

Retries have one honest job: telling infrastructure noise apart from real signal. A container that started slowly is noise. A race condition is signal. You cannot tell them apart without evidence, so collect it:

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: { trace: 'on-first-retry' },
});

Two lines, two jobs. The retry detects. The trace records everything about the failing run, so you can diagnose it later instead of shrugging.

Locally, retries stay at zero. On your own machine you want the failure loud and immediate.

The rule

A retried pass opens a ticket. It never closes one.

Once a week, read the flaky list. Every entry is one of two things. Infrastructure noise you should fix in the pipeline. Or a real timing bug in the app that your tests found first. Both are work. Neither is "passed."

My Saturday post said rerunning a flaky test in isolation makes things worse. These two readers explained the mechanism better than I did. The rerun does not just waste time. It manufactures false confidence and deletes the only evidence you had.

The green build is not the goal. The true build is.

Reuse One Page Object Method for Success and Failure Cases

Anton Gulin — Sat, 01 Aug 2026 15:00:51 +0000

Your login helper exists twice. One version expects the dashboard. One version expects the error. Ninety percent of the two bodies is the same code. Here is the short answer. Keep one method per user action. Let an options object say what it waits for. The test still decides what the result means.

Last week I wrote about where test assertions belong. A reader came back with the harder version of the question. If business checks live in the test, how does one method serve both cases? Good question. This post is the answer.

The duplicate everyone has

Open any page object that is more than a year old. You will find this:

async login(user: string, pass: string) {
  await this.username.fill(user);
  await this.password.fill(pass);
  await this.submit.click();
  await this.page.waitForURL('**/dashboard');
}

async loginExpectingError(user: string, pass: string) {
  await this.username.fill(user);
  await this.password.fill(pass);
  await this.submit.click();
  await this.errorBanner.waitFor();
}

Two methods. Three identical lines each. One real difference: what the method waits for at the end.

Now the login form changes. Someone adds a "remember me" checkbox. You fix the first method. You forget the second. The negative test starts failing for a reason unrelated to the negative case.

That is the cost of the duplicate. Not ugliness. Drift.

Why the boolean flag makes it worse

The first fix everyone tries is a flag:

async login(user: string, pass: string, success: boolean) { ... }

Then the test reads like this:

await loginPage.login('anton', 'wrong-password', false);

False what? You have to open the page object to find out. And flags multiply. Six months later the same method takes three of them, and nobody can read the call site at all:

await loginPage.login('anton', 'pw', false, true, false);

A flag saves a method and costs a reader. Bad trade.

Copy the shape Playwright already uses

Playwright's own API solves this everywhere. Every action takes an optional options object with named keys:

await page.getByRole('button').click({ timeout: 5000, force: true });

You never pass a bare true to Playwright. You pass a name and a value. Do the same in your page objects:

type LoginOptions = { waitFor?: 'dashboard' | 'error' };

async login(user: string, pass: string, options: LoginOptions = {}) {
  const { waitFor = 'dashboard' } = options;
  await this.username.fill(user);
  await this.password.fill(pass);
  await this.submit.click();
  if (waitFor === 'dashboard') {
    await this.page.waitForURL('**/dashboard');
  } else {
    await this.errorBanner.waitFor();
  }
}

Two things make this work. The default keeps the common call short. The named key makes the rare call readable:

// happy path, unchanged
await loginPage.login('anton', 'correct-password');

// negative path, and you can read it without opening the class
await loginPage.login('anton', 'wrong-password', { waitFor: 'error' });

One method. One place to fix when the form changes.

Notice what the option does not say

The option is called waitFor, not expectSuccess. That is deliberate.

expectSuccess: false puts a judgment inside the page object. The class starts deciding what a correct login looks like. That breaks the rule from last week: business checks live in the test.

waitFor: 'error' only says which element the method should wait for before it returns. It is a timing instruction, not a verdict. The verdict stays where a reader expects it:

test('rejects a wrong password', async ({ loginPage }) => {
  await loginPage.login('anton', 'wrong-password', { waitFor: 'error' });
  await expect(loginPage.errorBanner).toHaveText('Wrong password');
});

Read that test aloud. It says what it does and what should be true. You never open the page object.

When two methods really are right

Sometimes the duplicate is not a duplicate. Keep separate methods when the steps themselves differ:

async loginWithPassword(user: string, pass: string) { ... }
async loginWithSso(user: string) { ... }

Different fields, different clicks, different waits. These are two user actions that happen to end in the same place.

The test: if the bodies differ by more than the final wait, keep them apart. If they differ only by the final wait, merge them.

And name methods after the action, never after the outcome. loginExpectingError names a result. login names what the user does. Results belong in the test name.

The rule, in three lines

One method per user action.
An options object says what it waits for, with a default for the common case.
The test says what the result means.

Apply it once to your login helper. Then look at your checkout helper, your search helper, and your upload helper. The same pair is usually hiding in all of them.