DEV Community: Anton Gulin

How to Implement AI in QA (2026): A Practical Framework

Anton Gulin — Sat, 04 Jul 2026 15:40:25 +0000

Key takeaways

To implement AI in QA, let AI do the work and let a fixed check decide the result. Use AI to generate tests, repair broken selectors, and explore your app. Never let AI grade its own output. Add an independent check the AI cannot change, test for repeatable results, and test that the agent did only what you asked.

What "AI in QA" actually means

AI in QA means using an AI model to help test software. That is the whole idea. The model can write test cases, fix tests that broke, click through your app like a user, or read a failure and guess the cause.

It does not mean the AI replaces testing. It means the AI does some of the work a tester used to do by hand. The judgment stays with you.

I test software for a living. The teams that win with AI in QA all draw the same line. AI does the work. A human, and a fixed check, decide if the work is good.

The one rule that keeps it safe

Here is the rule the rest of this guide hangs on.

Let AI do the work. Never let AI judge its own work.

Picture an AI agent that tests your login. It clicks around. It reports green. Everyone relaxes. But the agent decided what "pass" means. If it is too kind, it passes a broken page. Now you shipped a bug with a green check on top.

An AI that grades its own work is not a test. It is an opinion.

So you give the AI room to explore, and you keep one thing it cannot touch. A fixed check. A known-good answer. A hard assert (a check that fails loudly) on the real outcome. The agent finds the path. The fixed check says pass or fail.

In testing this fixed answer has a name: an oracle. The oracle is the part the system being tested is not allowed to influence. Keep your oracle out of the AI's reach and most AI-in-QA risk goes away.

Where AI helps in QA (the 4 good jobs)

These four jobs are where AI pays off today.

Writing tests. Point the model at a page or a user story. It drafts test cases, including edge cases a tired human skips. You review and keep the good ones.
Fixing broken tests. A button moved and the test broke. AI can find the new selector (how a test finds a button) and propose the fix. This is the biggest time-saver for most teams.
Exploring the app. An AI agent can wander your app like a curious user and report what feels broken. Great for finding the bug nobody wrote a test for.
Reading failures. When a test fails, AI can read the log and the trace and suggest the likely cause. It turns a wall of red into a short list to check.

In all four, the AI proposes. You and your fixed checks dispose.

Where AI lies in QA (the 4 traps)

This is the part most guides skip. AI in QA fails in four ways. Plan for each.

It passes a broken thing. A too-kind agent calls a broken page "fine." Fix: an independent oracle the agent cannot move.
It is not repeatable. The same input passes now and fails in ten minutes. Fix: run the same input twice and compare the shape of the answer.
It does too much. You asked for one thing. The agent also changed a setting or sent a message. Fix: a scope check on what it touched.
It depends on a model that can change. The model under your tool updates, or even goes offline, and your tests shift with it. Fix: pin the model version and keep a fallback.

That last one is not theoretical. In June 2026 a widely used model was pulled offline overnight. Teams that pinned their model switched to a fallback in one line. Teams that did not found out when their build broke.

A working example: an AI agent with an oracle it cannot move

Here is the pattern in code. An AI agent books a meeting room. Then three fixed checks decide if it really worked. The agent never grades itself.

import { test, expect } from '@playwright/test';
import { Stagehand } from '@browserbasehq/stagehand';

test('AI books a room — and only that', async ({ page }) => {
  // Pin the model. Do not let it auto-upgrade under your tests.
  const stage = new Stagehand({ env: 'LOCAL', model: 'anthropic/claude-opus-4-8' });
  await stage.init();

  // 1) Let the AI do the work. It decides HOW to book the room.
  await stage.act('Book room B for 2pm tomorrow, for 30 minutes');

  // 2) The fixed check the AI cannot move (the oracle).
  //    These helpers read your real database, not the agent's report.
  const booking = await getBookingFromDb({ room: 'B', time: '14:00' });
  expect(booking).toBeTruthy();          // it did the task
  expect(booking.durationMin).toBe(30);  // exactly what we asked for

  // 3) Scope check. Did it touch anything it should not have?
  const otherChanges = await getChangesExcept(booking.id);
  expect(otherChanges).toHaveLength(0);  // no surprise side effects
});

Read the three checks again. The agent's own "I booked it" is never trusted. The database is the oracle. The duration check catches a sloppy booking. The scope check catches the agent doing extra. (getBookingFromDb and getChangesExcept are your own helpers — they read real state, not the agent's words.)

To catch the repeatable-result trap, run the same prompt twice and compare:

test('same request, same result', async () => {
  const a = await runBooking('Book room B for 2pm tomorrow, 30 minutes');
  const b = await runBooking('Book room B for 2pm tomorrow, 30 minutes');
  // The wording of the agent's reply may differ. The outcome may not.
  expect(a.room).toBe(b.room);
  expect(a.durationMin).toBe(b.durationMin);
});

The model is allowed to phrase its answer differently each time. It is not allowed to book a different room.

The 3 tests every AI feature needs

If you ship an AI feature to users, these three tests catch the failures that page you at 2am. Most teams only write the first easy one ("does it give a good answer?").

The wrong-input test. Feed it junk, empty fields, another language, a user trying to break it. A good feature fails safely. A bad one is confidently wrong.
The same-input-twice test. Run the exact input twice. Same kind of answer? Different wording is fine. Pass-then-fail is not.
The scope test. Did it do only what you asked? Or did it also change a setting, send a message, or touch a file? Extra is not helpful. Extra is a future incident.

Where to start on Monday

You do not need a platform or a budget. Start small.

Pick one flaky test. Let an AI tool propose the fix. You keep final say.
Add one oracle. Take your most important flow and add a hard check on the real outcome, not the agent's report.
Pin your model. Lock the version your AI tools use. Add a fallback.
Add the scope check to one agent run. See what it touches when you are not looking.

Do those four and you have AI in QA that you can trust. The AI does more work. You keep the judgment. The fixed checks keep everyone honest.

That is the whole job: the gap between "the AI says it passed" and "it passed, for the right reason."

Anton Gulin is the AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.

How to Test Passkey (WebAuthn) Login in Playwright (2026)

Anton Gulin — Sat, 27 Jun 2026 15:43:03 +0000

Key takeaways

You can now test passkey login in Playwright with no hardware key. Playwright 1.61 added a virtual authenticator (a fake security key). Your test seeds a passkey, turns it on, and the page signs in as if a real key answered. It works in every browser and runs in CI. The API is browserContext.credentials, with three methods: create(), install(), and get().

What is a passkey, in plain words

A passkey is a login with no password. You sign in with your face, your fingerprint, or a device PIN. The hard part lives in your device. The website only sees a signed reply.

The browser standard behind this is called WebAuthn. It means "web authentication". When you log in, the browser runs a small back-and-forth with the site. The site asks. Your device answers and signs. This is the part that used to need real hardware.

Apple, Google, and most banks ship passkeys now. If your app has a "sign in with a passkey" button, you have this flow in production today.

Why nobody tested this flow

Here is the part nobody talks about. Almost nobody tests the passkey login.

For years it was hard. To test a passkey you needed a real security key plugged into the machine. You cannot plug a USB key into a CI server. CI is a remote build machine with no hands and no ports.

So the most important login flow shipped untested. The one thing a user does first. The one thing that locks them out if it breaks.

I test software for a living. An untested login is the scariest gap on the list. If sign-in breaks, nothing else matters. The cart, the dashboard, the settings page, all of it sits behind the door. On one project I watched a broken auth path block every other test for two days. The fix took ten minutes. Finding it took the two days.

Playwright 1.61 closed this gap.

What Playwright 1.61 added

Playwright 1.61 shipped on June 15, 2026. It added a virtual authenticator (a fake security key).

A virtual authenticator is software that pretends to be a hardware key. Your test creates one. It seeds a passkey into it. From then on, when the page calls the browser to sign in, Playwright answers for the key. No real device. No USB port. The page cannot tell the difference.

You reach it through a new class called Credentials, on the browser context: browserContext.credentials. It has three methods you will use most:

create() seeds a test passkey for a site.
install() turns the virtual key on for the page.
get() reads back any passkey the page registered, so you can save it and reuse it later.

This works in all three browser engines Playwright drives. So one test covers Chromium, Firefox, and WebKit.

How to test passkey login — a working example

Here is a complete test. It seeds a passkey, turns on the virtual key, then signs in. Read the comments for what each step does.

// passkey-login.spec.ts
// Tested against Playwright 1.61. Run with: npx playwright test
import { test, expect } from '@playwright/test';

test('user signs in with a passkey', async ({ browser }) => {
  // A fresh, clean browser session for this test.
  const context = await browser.newContext();

  // STEP 1 — Seed a passkey for our site.
  // 'example.com' is the site domain (the "relying party id").
  // With only the domain, Playwright makes a fresh key for us.
  await context.credentials.create('example.com');

  // STEP 2 — Turn the virtual key on.
  // From now on, the page's sign-in calls are answered by our key,
  // not by real hardware. Call this before the page loads.
  await context.credentials.install();

  // STEP 3 — Let the page use it.
  const page = await context.newPage();
  await page.goto('https://example.com/login');

  // The page calls the browser to sign in. Our key answers.
  await page.getByRole('button', { name: 'Sign in with a passkey' }).click();

  // Check the user is in.
  await expect(page.getByText('Welcome back')).toBeVisible();
});

The three steps map to the three method calls. First you create() a passkey for your site. Then you install() the virtual key, which makes the page's sign-in calls run through it. Then the page does its normal login, and your key answers in place of hardware.

One note on order. Call install() before the page touches sign-in. The virtual key only answers calls that happen after you turn it on.

Re-using a passkey across tests

Often you want to register a passkey once, then reuse it in many tests. A passkey holds a private key (a secret only your device knows). You can read that secret back with get() and seed it into a later test.

// In a setup test: register once, then read the passkey back.
const created = await context.credentials.get({ rpId: 'example.com' });
// `created` holds the passkey fields, including its keys.
// Save them, then seed an identical passkey in a later test:
await otherContext.credentials.create('example.com', {
  id: created[0].id,
  userHandle: created[0].userHandle,
  privateKey: created[0].privateKey,
  publicKey: created[0].publicKey,
});

This is how you keep one stable test user across a whole suite. You do not re-register on every test. You seed the same passkey each run. See the Credentials docs for the full field list.

The honest part: what this does not test

A virtual key is not a real key. So this approach tests your login flow, not the physical hardware. It will not catch a bug in a specific phone's secure chip or a real fingerprint reader. It tests the part you own: the page, the back-and-forth, the server check.

For most teams that is the right line. The browser and the operating system test the hardware path for you. Your job is to test that your app asks the right question and trusts the right answer. That is exactly what the virtual key lets you do, in CI, on every push.

Why this matters for your CI pipeline

Before 1.61, your passkey login had two test options. Skip it, or test it by hand. Both are bad. A skipped test means a silent break. A by-hand test runs once a release, not once a push.

Now it runs like any other test. It sits in your suite. It runs on every pull request. If someone changes the login and breaks the passkey path, the build goes red before the change ships. That is the whole point of a test. Catch the break in seconds, not from an angry user.

If your app supports passkeys, this is the test you write this week. The excuse is gone. The login everyone ships and nobody verifies is now testable.

Related reads on anton.qa:

Official sources:

Internal References

(Reviewer-only. The dev.to publisher strips this section — it lives after the bio and is not part of the published article. Same source receipts as the canonical blog.)

Canonical: https://anton.qa/blog/posts/test-passkey-login-playwright (live Wed 2026-06-24; dev.to publishes +72h once canonical is live).
All API claims (browserContext.credentials, create() / install() / get(), positional rpId, options fields, "works in all browsers", v1.61.0 shipped 2026-06-15) verified in the source blog against the Playwright v1.61.0 release notes + Credentials docs (2026-06-22). See 2026-06-24-blog-test-passkey-login-playwright.md § Internal References.
Cover is the 1000×420 dev.to-native export of the blog d-variant hero (same visual asset, dev.to size — AGENTS.md Rule 9 research-reuse clause; no fresh Subscribr needed). Source HTML: assets/devto-covers/var-test-passkey-login-playwright.html.
dev.to publishes via scripts/devto_publish.py --drain (hourly GitHub Action) at publish-at; status: scheduled + publish-at are the trigger. Tags ≤4, canonical globally unique.

Playwright Best Practices: 10 Rules AI Agents Get Wrong (2026)

Anton Gulin — Sat, 20 Jun 2026 15:51:36 +0000

Playwright best practices are the rules that keep browser tests stable and easy to read. Use role-based locators (find by what users see), web-first assertions that auto-wait, and isolated tests. Seed data through the API (direct requests), not the UI. Avoid hard waits, conditional logic, and tests tied to your HTML. Turn on traces and run in parallel.

An AI agent can write 50 Playwright tests in a minute. That feels fast.

Then those tests fail at random, and nobody knows why. The agent copied old patterns from its training data. It does not know the run failed last night.

This guide lists the 10 best practices that keep tests stable. For each one, I show a small correct example. I also show what AI code agents get wrong. AI tools like Copilot, Cursor, and even Playwright codegen (the test recorder) lean on stale habits. Someone has to fix that.

1. Find elements the way a user sees them

A locator (a pointer to an element) should match what a person sees on screen. Use getByRole, getByLabel, or getByText. These read like the page. They also survive a redesign of your HTML.

import { test, expect } from '@playwright/test';

test('user can sign in', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill('ada@example.com');
  await page.getByRole('button', { name: 'Sign in' }).click();
});

What AI agents get wrong: they reach for CSS or XPath (brittle path selectors) like page.locator('div.btn-primary > span'). Change one class name and the test breaks.

2. Use web-first assertions that wait for you

A web-first assertion (a check that auto-waits) retries until the page is ready. expect(locator).toBeVisible() waits on its own. You never add a fixed sleep.

import { test, expect } from '@playwright/test';

test('welcome message appears', async ({ page }) => {
  await page.goto('/dashboard');
  await expect(page.getByText('Welcome back')).toBeVisible();
});

What AI agents get wrong: they add await page.waitForTimeout(3000) (a hard pause). Hard waits are the top cause of flaky tests (tests that fail at random). Too short, the test fails. Too long, the suite crawls.

3. Keep every test isolated

Isolated means each test starts clean. No shared login. No leftover data from the test before. Playwright gives each test a fresh browser context (a clean session). Set up state in a hook, not across tests.

import { test, expect } from '@playwright/test';

test.beforeEach(async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill('ada@example.com');
  await page.getByRole('button', { name: 'Sign in' }).click();
});

test('shows the account name', async ({ page }) => {
  await expect(page.getByRole('heading', { name: 'Ada Lovelace' })).toBeVisible();
});

What AI agents get wrong: they chain tests, where test 2 needs test 1 to run first. One failure then breaks the whole file.

4. Seed state through the API, not the UI

To test a page, you often need data first. A user, an order, a draft. Do not click through ten screens to make it. Send the data straight to your backend with the request fixture (a built-in HTTP client). It is faster and steadier.

import { test, expect } from '@playwright/test';

test('opens an existing project', async ({ page, request }) => {
  const res = await request.post('/api/projects', {
    data: { name: 'Apollo' },
  });
  expect(res.ok()).toBeTruthy();

  await page.goto('/projects');
  await expect(page.getByText('Apollo')).toBeVisible();
});

What AI agents get wrong: they build the data through the UI every time. The test gets long and slow, and a setup step fails for reasons that have nothing to do with the real check.

5. Do not lean on test IDs by default

A test ID (a tag added just for tests, like data-testid) works as a fallback. But reach for getByRole and getByLabel first. Those test what a real user can do. A test ID only proves an attribute exists.

import { test, expect } from '@playwright/test';

test('cart shows one item', async ({ page }) => {
  await page.goto('/cart');
  // Prefer a real role over a test id.
  await expect(page.getByRole('listitem')).toHaveCount(1);
});

What AI agents get wrong: they paste data-testid on everything. The tests pass even when the button has no label and a screen reader (assistive software) cannot find it. The test misses a real bug.

6. Turn on traces for the first retry

A trace (a full recording of the run) shows every step, the DOM, and the network. Set it to record only on the first retry of a failed test. You get the evidence for failures, and clean runs stay fast.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 1,
  use: {
    trace: 'on-first-retry',
  },
});

What AI agents get wrong: they leave tracing off, or set trace: 'on' for every run. Off means no evidence when a test fails. Always-on slows the suite and fills your disk.

7. Run tests in parallel and shard them

Parallel means many tests run at once. Playwright does this by default. For one big file of independent tests, set parallel mode. To split a slow suite across machines, use sharding (run a slice per machine).

import { test, expect } from '@playwright/test';

test.describe.configure({ mode: 'parallel' });

test('loads home', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveTitle(/Home/);
});

Split across three machines on CI (your build server):

npx playwright test --shard=1/3

What AI agents get wrong: they write tests that share a database row or a single user. Run those in parallel and they fight each other, so you get flaky tests.

8. Keep `if` and `try` out of your tests

A test should walk one clear path. No branching. If a test asks "is the button there? if so click it," it hides a bug. The button should always be there. Assert it.

import { test, expect } from '@playwright/test';

test('checkout button works', async ({ page }) => {
  await page.goto('/cart');
  // Assert the state. Do not guess it with an if.
  const checkout = page.getByRole('button', { name: 'Checkout' });
  await expect(checkout).toBeEnabled();
  await checkout.click();
});

What AI agents get wrong: they wrap clicks in if (await locator.isVisible()) to stop errors. That hides the real failure. A test that skips its own check still goes green.

9. Test what users see, not how it is built

Test the behavior, not the internals. Check the visible result. Do not check a CSS class, a state variable, or a function name. Those change when you refactor (rewrite the code), even though the app still works.

import { test, expect } from '@playwright/test';

test('shows a success message after submit', async ({ page }) => {
  await page.goto('/contact');
  await page.getByLabel('Message').fill('Hello');
  await page.getByRole('button', { name: 'Send' }).click();
  // Check the user-facing result, not an internal class.
  await expect(page.getByText('Thanks, we got your message')).toBeVisible();
});

What AI agents get wrong: they assert on class="is-active" or an exact HTML shape. The test breaks on every redesign, even when nothing real changed.

10. Define projects in your config

A project (a named test setup) in playwright.config.ts runs the same tests under different settings. Use projects to cover Chromium, Firefox, and WebKit (the three main browser engines). One config, full coverage.

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  retries: 1,
  use: { trace: 'on-first-retry' },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
    { name: 'firefox', use: { ...devices['Desktop Firefox'] } },
    { name: 'webkit', use: { ...devices['Desktop Safari'] } },
  ],
});

What AI agents get wrong: they hard-code one browser, or copy a config with no projects array. The suite then tests Chrome only, and a Safari-only bug ships to users.

The human still owns the standard

AI writes the first draft fast. That part is real, and it is useful. But the first draft copies patterns from old code on the internet. It adds hard waits. It clicks through the UI to seed data. It wraps fragile steps in if blocks so the run stays green.

A green suite that proves nothing is worse than no suite. It buys false trust.

So the workflow is simple. Let the agent write the draft. Then a human reads it against these 10 rules and fixes what the agent got wrong. The agent moves fast. The human keeps the tests honest. That is the job of an AI QA Architect.

Build the tests with AI. Then make them stable yourself.

I measure how fast 42 LLMs actually answer. Here's the honest method.

Anton Gulin — Tue, 16 Jun 2026 05:06:22 +0000

I test software for a living. So when a vendor calls an AI model "fast," I don't trust the word. I measure it.

Most leaderboards rank how smart a model is. Almost none rank how fast it answers. You pick a model because it scored well, ship it, and then your users sit and wait.

Speed is two different numbers. People mix them up constantly.

The two numbers

Time to first token (TTFT). The wait before the first word appears. You feel this every time a chatbot "thinks" before replying.

Tokens per second (TPS). How fast the model writes once it starts. A token is a chunk of a word.

A model can be great at one and terrible at the other. You need both.

How I measure it

I run an independent tracker called ollamatps.com. It benchmarks 42 Ollama Cloud models. Here is the exact method, because a benchmark you cannot inspect is just a claim.

One fixed prompt. Every run asks the model to write a 400-word explanation of HTTP routing. Same prompt, every model, every time.
Fixed output cap. max_tokens is capped at 300. It never changes between runs.
Every ~10 minutes. Each model is re-tested continuously, not once.
TTFT is measured from the moment the request is sent to the first non-empty content chunk. It includes network round-trip and prompt-processing time. That is honest, because it is what you actually wait.
TPS is generation throughput only: eval_count / (total_duration - time_to_first_token). The startup wait is removed, so TPS measures pure writing speed.

Same prompt, same cap, same schedule. That is what makes two models comparable.

Building this was a testing job, not a coding job. Retries on failure. A reliability score per model. A circuit breaker for models that keep failing. If you cannot trust the measurement, the number is noise. That part is the same work I do on any test system.

What surprised me

Bigger is not faster.

The fastest model on the board is one of the smallest: a 30B model at over 200 tokens per second. The model literally named "ultra" is dead last, under 8 tokens per second.

And the wait varies wildly. TTFT ranges from about 0.3 seconds to 23 seconds across the 42 models. Same cloud. Roughly 80x difference in how long you wait for the first word.

If you picked your model on a benchmark score alone, you have no idea which of these you are getting.

Why I rebuilt it

The first version tracked fewer models and was less robust. I rebuilt the engine this month (v2) to be multi-provider and to test continuously. The live board updates every 10 minutes.

Watch it run: ollamatps.com

I'm Anton. I build AI tools and measure how fast they really run. Former Apple engineer. I make ollamatps.com and write about building and measuring AI at anton.qa.

How to Score Your AI Test Agents: Offline Evaluation with Trajectories (2026)

Anton Gulin — Sat, 13 Jun 2026 20:37:36 +0000

AI test agent evaluation is the practice of scoring the tests an AI agent writes, instead of trusting that they pass. You record the agent's run as a trajectory (a saved log of every step), replay it offline, and grade each step for correctness and relevance. Offline scoring needs no live API calls, so you can check agent quality on every pull request.

An AI agent can write 200 tests before lunch. That feels like progress.

Then a real bug ships, and not one of those tests caught it. The agent was confident, and it was wrong.

This guide shows how to stop guessing and start scoring. Stagehand 3.5.0 made the method first-class on June 3, 2026, but the pattern works for any agent.

1. "It passed" is not a score

A green test suite tells you the tests ran. It does not tell you the tests were right.

An AI agent makes three mistakes a human reviewer would catch:

It checks the wrong thing. The test passes, but it never asserts the real behavior.
It writes flaky tests (tests that fail at random). They go green often enough to look fine.
It tests a happy path and skips the edge case that actually breaks.

You cannot fix what you cannot measure. So the first job is a number, not a vibe.

2. Record the run as a trajectory

A trajectory is a saved recording of an agent's run. It captures each step: what the agent saw, what it decided, and what code it produced.

You capture it once, during the agent's normal run.

// Illustrative pattern — confirm the exact Stagehand 3.5 API before use.
const trajectory = await agent.run(task, { record: true });
await saveTrajectory(trajectory, "runs/checkout-flow.json");

The recording is the receipt. Now you can study the run after it finishes, as many times as you want.

3. Replay it offline

Offline means you grade the saved run without calling the live model again. No new API cost. No flaky network. Same input every time.

This matters for two reasons. It makes scoring cheap, so you can run it on every pull request. It makes scoring repeatable, so two engineers get the same result.

// Replay the saved run and score it, with no live API calls.
const run = await loadTrajectory("runs/checkout-flow.json");
const score = await evaluate(run, rubric);

4. Score each step with evaluation types

A single pass/fail hides too much. Grade the run on a few clear axes instead.

Correctness: did the test assert the behavior the task asked for?
Relevance: does each step move toward the goal, or wander?
Stability: would this test pass on a clean re-run, or is it flaky?
Coverage: did the agent test the edge case, or only the happy path?

Stagehand 3.5.0 added evaluation types for exactly this kind of offline scoring. You define the rubric once and apply it to every saved run.

const rubric = {
  correctness: (run) => run.asserts.some(a => a.target === task.goal),
  relevance:   (run) => run.steps.every(s => s.onTask),
  stability:   (run) => run.reruns.every(r => r.passed),
};

A run that scores correctness 7/10, relevance pass, flaky tests 0 is a run you can talk about. "It passed" is not.

5. Wire the score into CI

A score you read once and forget changes nothing. Turn it into a gate.

# CI step: fail the build if the agent's tests score too low.
- run: npx evaluate runs/ --min-correctness 0.8 --max-flaky 0

Now the agent earns trust the same way a junior engineer does. It ships work, the work gets graded, and only graded work reaches production.

6. Where this sits: the Evidence Layer

I design AI test systems on a 3-Layer System:

Orchestration: decides what to test.
Execution: runs the tests, where the agent writes code.
Evidence: proves the work is right.

Most teams build the first two layers and stop. They let the agent write tests and assume the green check means quality.

Offline evaluation is the Evidence Layer. It is the difference between an agent you hope works and an agent you can prove works.

The 5-line checklist

Record every agent run as a trajectory.
Replay it offline, with no live API calls.
Score it on correctness, relevance, stability, and coverage.
Gate your build on the score.
Keep the trajectory, so you can re-grade when the rubric improves.

Build the agent. Then prove it works.

Playwright Codegen: The Complete Guide (2026)

Anton Gulin — Mon, 08 Jun 2026 05:38:38 +0000

Playwright Codegen is a native CLI (command-line interface) tool that generates test scripts automatically as you interact with a browser. It records your actions—like clicks, form inputs, and page navigation—and translates them into clean TypeScript or JavaScript test code.

For most developers, writing test locators (how tests find buttons) takes up 60% of test writing time.

Codegen reduces that time to zero.

Here is how to use it, and how to scale it from a simple draft tool to a full production architecture.

1. How to Launch Playwright Codegen

To start the generator, run this command in your terminal:

npx playwright codegen demo.playwright.dev/todomvc

This launch command opens two windows:

A browser window: This is where you click, type, and record your test steps.
The Playwright Inspector: This is a tool window that displays the generated code in real time.

As you click on the page, the tool writes the test code automatically.

2. Capturing Assertions

A test without assertions (checks to verify behavior) is just a script.

Codegen allows you to record checks directly from the UI.

In the browser window, hover over any element and click one of the check buttons in the toolbar:

Assert Visibility: Verifies if an element is visible on the screen.
Assert Text: Verifies if an element contains specific text.
Assert Value: Verifies the input value of a form field.

This generates standard assertions like await expect(locator).toBeVisible() instantly.

3. Playwright Codegen Best Practices

Generated code is a draft.

To make it production-ready, apply these three rules:

Avoid Hardcoded Wait Times

Codegen does not generate sleep statements.

Playwright uses auto-waiting (waiting for elements to be ready).

Keep it that way.

Do not add manual timeouts.

Use Semantic Locators

Playwright prefers locators that represent user actions.

Codegen generates these by default:

// Good: accessible locator
await page.getByRole('button', { name: 'Submit' }).click();

// Bad: fragile CSS selector
await page.locator('#submit-btn-2').click();

Keep the accessible versions.

They prevent flaky tests (tests that fail randomly).

Isolate Your Auth State

Do not record login steps in every single test.

Use Codegen to save your authentication state once.

Run Codegen with this save option:

npx playwright codegen --save-storage=auth.json

Then, configure your tests to load auth.json before running.

This saves hours of run time in CI (continuous integration servers).

4. The Architectural View: From Draft to System

As an AI QA Architect, I view Codegen as a helper.

It is the entry point of the Execution Layer in the 3-Layer System:

Orchestration: Decides when to run tests.
Execution: The code that runs (where Codegen helps).
Evidence: Gathers logs and traces.

Codegen writes the initial code.

But it cannot design the framework.

It cannot handle API mocks (fake servers).

It cannot govern agentic testing systems (where AI agents write and heal tests).

Use Codegen to build the first block.

Then build the architecture around it.

Playwright vs Cypress vs Selenium in 2026

Anton Gulin — Sun, 24 May 2026 06:59:50 +0000

Playwright is the best default for new browser test automation in 2026. It gives cross-browser runs, parallel CI, API checks, and AI-agent evidence in one tool. Cypress still fits JavaScript-heavy teams that want fast local feedback. Selenium still fits legacy grids and strict browser labs.

That is the short answer.

The better answer depends on your system.

If AI agents will read your failures, the question changes.
You are no longer picking only a test runner.
You are picking the evidence layer.

What Changed In 2026

Most comparison posts still ask old questions.

They ask which tool has cleaner syntax.
They ask which tool is easier to learn.
They ask which tool starts faster.

Those questions still matter.
They are no longer enough.

AI agents need proof they can inspect.
Proof means screenshots, traces, browser state, and readable failures.

The human reviewer still owns the decision.
The agent only helps when the evidence is clear.

That is why Playwright now has the default seat.

Pick Playwright When Evidence Matters

Pick Playwright for new end-to-end test systems.
End-to-end means browser checks.

Playwright gives you one model across Chromium, Firefox, and WebKit.
Those are browser engines.
They are how pages run.

That matters for real product risk.

It also matters for AI-agent workflows.
AI agents means tools that act.

Playwright now documents Test Agents.
Those agents plan, generate, and repair tests.

The tool also has strong receipts.
Traces show what happened.
Screenshots show where it happened.
Reports help humans review the failure.

Use Playwright when you need:

browser coverage across engines
parallel CI at scale
trace-based debugging
API and UI checks together
AI-agent review paths
long-term framework ownership

CI means server test runs.

Pick Cypress When Local Feedback Matters Most

Cypress is still useful.

That sentence matters.
Tool debates get lazy when one side becomes a villain.

Cypress can be a strong fit for frontend teams.
It works well when developers want quick local feedback.
It also fits teams already built around Cypress Cloud.

Cypress documents cross-browser testing.
It also documents parallel runs through Cypress Cloud.

That can be enough for many product teams.

Use Cypress when:

your app is JavaScript-first
developers own most browser checks
fast local debugging is the main goal
Cypress Cloud is already approved
browser coverage needs are narrow
the suite is not agent-driven yet

The risk appears later.

As the suite grows, evidence gets more important.
That is where Playwright usually wins.

Keep Selenium When Migration Risk Is Higher

Selenium is not dead.

It is still the right answer for some teams.

Keep Selenium when a grid already exists.
Keep it when policy requires it.
Keep it when migration risk is higher than tool value.

But do not choose Selenium by default for new AI QA work.

You will spend too much time rebuilding the evidence layer.
You will also carry older suite habits forward.

Selenium can be stable.
The question is whether it helps the next system.

The Decision Table

Need	Best default	Why
New AI-agent test system	Playwright	Best evidence path
Broad browser engine coverage	Playwright	One model across major engines
Fast frontend feedback	Cypress	Strong local developer loop
Existing Cypress investment	Cypress	Migration may not pay yet
Legacy grid policy	Selenium	Use what the organization can run
Greenfield QA architecture	Playwright	Better long-term receipts

My Four-Question Test

I use four questions before I choose.

Who reads the failure first?
What proof do they need?
Where will the suite run?
What happens when the UI changes?

If the answer includes AI agents, I lean Playwright.

If the answer is one frontend team, Cypress can fit.

If the answer is legacy policy, keep Selenium.

The Practical Recommendation

Start new projects with Playwright.

Keep Cypress when it already serves the team.

Keep Selenium when migration would create more risk.

Then build the same rule across all three:

Every failed test needs a receipt.

That receipt should show:

what action ran
what page state existed
what assertion failed
what changed before failure
what a human must decide

The tool only matters because the evidence matters.

In 2026, that is the real comparison.

Author Bio

Playwright v1.60 Turns Test Failures Into Evidence

Anton Gulin — Mon, 18 May 2026 04:54:00 +0000

Playwright v1.60 makes failure evidence easier to capture during the run.

The main change is scoped HAR recording.

HAR means network request file.

It shows what the browser sent and received.

The release also adds file drops, ARIA boxes, and hard test aborts.

ARIA means accessibility map.

Together, these changes help CI failures explain themselves.

CI means automated build server.

The practical update

Use context.tracing.startHar() when network failures waste review time.

It records a HAR file inside Playwright tracing.

Tracing means run evidence capture.

Use locator.drop() when upload tests use custom events.

Drop API means file drop simulation.

Use page.ariaSnapshot({ boxes: true }) when AI tools inspect pages.

Boxes mean element positions.

Use test.abort() when shared setup finds unsafe state.

Fixtures mean shared test setup.

Code example

import { test, expect } from '@playwright/test';

test('upload records network evidence', async ({ context }) => {
  await using har = await context.tracing.startHar('upload.har', {
    content: 'embed',
    mode: 'minimal',
    urlFilter: /\/api\/upload/,
  });

  const page = await context.newPage();
  await page.goto('/upload');

  await page.locator('#dropzone').drop({
    files: {
      name: 'note.txt',
      mimeType: 'text/plain',
      buffer: Buffer.from('hello'),
    },
  });

  await expect(page.getByText('Upload complete')).toBeVisible();
});

The HAR starts before the page opens.

The drop step sends an in-memory file.

When the test scope ends, Playwright finalizes the HAR.

The rule

Do not treat this release as a feature list.

Treat it as an evidence upgrade.

Better tests do not just pass or fail.

They explain what happened.

Read the canonical version:

https://www.anton.qa/blog/posts/playwright-v1-60-evidence-first-testing

AI Test Automation Architecture: The 3-Layer System

Anton Gulin — Sun, 17 May 2026 23:53:47 +0000

AI test automation architecture is the system that tells AI what to test.

It also defines how to run tests and prove the result.

I split it into three layers: orchestration, execution, and evidence.

Without all three, AI testing becomes prompt output with no production gate.

Why tool lists fail

Most AI testing content starts with tools.

That is backwards.

AI means software that predicts.

Predictions can help QA teams move faster.

But predictions do not prove quality.

The 3-layer model

Layer	Plain meaning	Main question
Orchestration	test control plan	What risk should this cover?
Execution	actual test run	Did it run in the real pipeline?
Evidence	proof from runs	Can a human review it?

The practical gate

Use this before AI-generated tests ship:

Gate	Pass condition
Scope	The test maps to one named risk
Data	Test data setup is explicit
State	Browser state is controlled
Run	The test passes in CI
Evidence	Trace or equivalent proof exists
Review	A human can explain the failure mode

CI means automated build server.

MCP means tool connection standard.

Playwright is a browser test tool.

Together, they can help AI agents run useful tests.

But the architecture must prove each run.

The rule

Never ask AI to expand test coverage first.

Build the proof system before that.

Generation is cheap.

Evidence is the architecture.

Read the canonical version:

https://www.anton.qa/blog/posts/ai-test-automation-architecture-3-layer-system

How to Test MCP Servers Before They Break Your CI

Anton Gulin — Mon, 11 May 2026 18:15:50 +0000

Most teams install an MCP server and hope it works.

That is how you get 3 AM pages.

An MCP server is a bridge between AI agents and your tools. It can crash, leak data, or silently return garbage. If your AI agent relies on it, your whole pipeline breaks.

MCP means Model Context Protocol (standard tool link).

Do not only test startup. Test behavior and permissions too.

This post is the checklist I run on every MCP server before it touches production.

The three-layer test stack

Layer	What it catches	Tool
Discovery	Missing tools, broken metadata	MCP Inspector
Behavior	Silent failures, wrong output	pytest smoke tests
Security	Over-permissions, data leaks	Permission audit

Layer 1: Discovery with MCP Inspector

MCP Inspector is the official debugging tool. Start it with:

npx @anthropic-ai/mcp-inspector node dist/server.js

Check three things:

Does the server start without errors?
Does it list the tools it promises?
Does a sample request return the right shape?

Layer 2: Behavior with pytest

Here is a minimal smoke test. It checks that initialization returns valid JSON-RPC:

import subprocess, json

def test_mcp_server_responds():
    proc = subprocess.Popen(
        ["npx", "-y", "@anthropic-ai/mcp-server-filesystem", "/tmp"],
        stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True
    )
    proc.stdin.write(json.dumps({
        "jsonrpc":"2.0","id":1,
        "method":"initialize",
        "params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}
    }) + "\n")
    proc.stdin.flush()
    response = json.loads(proc.stdout.readline())
    assert response["id"] == 1
    proc.terminate()

Layer 3: Security with a permission audit

Check three things:

Does it need file system access (disk read/write)? Which paths?
Does it make network calls (external requests)? To which hosts?
Does it run shell commands (terminal execution)? Under which user?

If the answers are "all files, any host, root user," block it.

Where to find servers worth testing

Official MCP Registry — https://registry.modelcontextprotocol.io
GitHub — Search modelcontextprotocol topics
npm / pip — Search @anthropic-ai/mcp-server-*

Red flags: no commits in 6+ months, no tests, no README, permission requests that are too broad.

Verdict

Testing MCP servers is not optional. An untested server is a bug waiting to become an incident.

The three-layer stack catches common failure modes. MCP Inspector for manual checks. pytest for CI gates. Permission audit for last defense.

Anton Gulin is an AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET, now Lead Software Engineer in Test. anton.qa

Playwright MCP v0.0.73: How to Configure Browser Paths via Environment Variables

Anton Gulin — Mon, 04 May 2026 21:37:17 +0000

This post was originally published on anton.qa. The canonical version lives there.

Playwright MCP v0.0.73 fixes a critical gap where extension channels and executable paths could not be resolved from CI/CD environment variables.

If you run Playwright MCP in Docker, Kubernetes, or ephemeral CI workers, this release removes a class of environment-specific debugging that typically consumes 15–30 minutes per incident.

What changed

Two interconnected bug fixes:

Extension channel and executablePath now resolve from CLI flags and environment variables (#40572)
--browser channel flags now propagate on --extension paths (#40567)

Combined, these changes mean your Playwright MCP setup can now be fully environment-driven.

The pattern

export PLAYWRIGHT_BROWSERS_CHANNEL=chromium
export PLAYWRIGHT_EXTENSION_PATH=/path/to/browser-extension
npx playwright test

The resolution hierarchy is now:

CLI flags (highest priority)
Environment variables
Config file defaults
Built-in channel defaults

MCP Registry listing

Playwright MCP is now published to the official MCP Registry on each release. This simplifies enterprise procurement and governance for teams evaluating AI-assisted testing infrastructure.

The gotcha

Environment variables set in your shell may not propagate to the MCP process spawned by your AI tool. Test this before deploying to production.

For the full breakdown — including CI/CD examples and the subprocess propagation fix — read the canonical post at anton.qa.

Anton Gulin is an AI QA Architect. Former Apple SDET, now Lead Software Engineer in Test. anton.qa

Native Drag-and-Drop Automation Arrives in Playwright MCP

Anton Gulin — Tue, 28 Apr 2026 18:43:12 +0000

TL;DR

Playwright MCP v0.0.71 ships browser_drop. It gives you native drag-and-drop from any MCP client. No more evaluate scripts. No more mouse.move chains. Grid reordering, file drop zones, text editor drags — all work the same way a real user does.

Why This Release Matters

QA teams either abandon drag-and-drop testing or hack around it. But sortable grids, file uploads, and rich text editors are everywhere. And they have been painful to test forever.

I ran into this firsthand on one project. Solid Playwright coverage for clicks, typing, and navigation. But drag-and-drop? We used evaluate scripts. Or we tested it by hand. Both paths broke across browsers. Both were impossible to keep working.

Playwright MCP v0.0.71 fixes this with browser_drop. It uses Playwright's own Locator.drop — the same API your tests already use. Now any MCP client can call it.

How to Use browser_drop

Here's a complete example combining browser_drop with the new response body capture from browser_network_requests and the simplified expression support in browser_evaluate. This pipeline automates a file upload scenario, validates the server response, and confirms the UI state update:

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";

const server = new McpServer({
  name: "file-upload-automation",
  version: "1.0.0",
});

// Drop zone and file item selectors for a document management UI
const dropZoneSelector = '[data-testid="upload-zone"]';
const fileItemSelector = '[data-testid="file-item"]';

const uploadedStatusSelector = '[data-testid="upload-status"]';

// Tool: Simulate file drag-and-drop onto upload zone
server.addTool("upload_document_flow", {
  description: "Upload a document via drag-and-drop and validate response",
  inputSchema: {
    type: "object",
    properties: {
      fileName: { type: "string", description: "Name of file to upload" },
      fileId: { type: "string", description: "Unique file identifier" },
    },
  },
  async execute(fileName, fileId) {
    // Navigate to upload interface
    await server.tools.call("browser_navigate", { url: "https://internal-docs.example.com/upload" });

    // Locate drag source and drop target
    const dragSource = `text=${fileName}`;
    const dropTarget = dropZoneSelector;

    // Execute native drag-and-drop operation
    // browser_drop wraps Locator.drop - no evaluate or mouse.move workarounds
    const dropResult = await server.tools.call("browser_drop", {
      source: dragSource,
      target: dropTarget,
    });

    if (!dropResult.success) {
      return { error: `Drop operation failed: ${dropResult.error}` };
    }

    // Inspect server response body with mime-type detection
    const networkCapture = await server.tools.call("browser_network_requests", {
      urlPattern: "**/api/upload**",
      responseBody: true,
      responseHeaders: true,
    });

    // Extract upload confirmation
    const uploadResponse = networkCapture.requests?.[0];
    if (!uploadResponse?.responseBody) {
      return { error: "No upload response captured" };
    }

    // Validate response using plain expression (no function wrapper needed)
    const validationResult = await server.tools.call("browser_evaluate", {
      expression: `JSON.parse(arguments[0]).status === "success"`,
      args: [uploadResponse.responseBody],
    });

    // Confirm UI reflects successful upload
    const statusText = await server.tools.call("browser_evaluate", {
      expression: `document.querySelector("${uploadedStatusSelector}")?.textContent`,
    });

    return {
      uploaded: true,
      serverResponse: uploadResponse.responseBody,
      uiStatus: statusText.result,
      validationPassed: validationResult.result === true,
    };
  },
});

Three new tools working together: browser_drop handles the drag. browser_network_requests captures the server response (full body, not just status codes). browser_evaluate runs plain JavaScript — no function wrapper needed.

The Gotcha Nobody Is Talking About

browser_drop needs both elements to be on screen. That's correct Playwright behavior. But here's the catch: if you navigate to a page and the drag target sits below the fold, the drop fails.

The fix: Call browser_evaluate to scroll the target into view before calling browser_drop, or use the scroll option if your Playwright version supports it. This catches teams off guard in CI where viewport sizes are smaller than local development.

// Before browser_drop: ensure target is in viewport
await server.tools.call("browser_evaluate", {
  expression: `document.querySelector("${dropTarget}").scrollIntoView()`,
});

This is not a bug. It's how Playwright works. But it catches teams when they test on a big screen and deploy to CI. CI viewports are smaller. The element you tested locally is off screen in the pipeline.

What This Changes in Your CI Pipeline

With browser_drop, you can test drag-and-drop flows through MCP. Not by hand. Not with broken scripts.

On one project, Selenium to Playwright gave us 40% faster tests. But drag-and-drop still broke in headless mode. We wrote evaluate scripts. They stopped working every sprint. browser_drop puts native drag-and-drop into MCP. No scripts. No workarounds.

What this actually means:

Fewer flaky tests. Native drag-and-drop is tested across browsers. evaluate + mouse.move sequences are not.
Simpler AI test generation. AI tools call browser_drop directly. No fragile mouse chains.
Faster CI. Native operations run faster than JavaScript-injected drag scripts.

Verdict

Playwright MCP v0.0.71 is worth upgrading for browser_drop alone. The response body capture and plain expression support make it better. But drag-and-drop was the missing piece. Now it's there.

The catch is real but small. Scroll your target into view before you drop. One line. Add it to your tool definitions and move on.

If you run MCP-based test infrastructure, this kills the last reason to fall back to evaluate for drag-and-drop. Upgrade. Add the scroll guard. Ship.

Reference: Playwright Locator.drop API documentation

DEV Community: Anton Gulin

How to Implement AI in QA (2026): A Practical Framework

Key takeaways

What "AI in QA" actually means

The one rule that keeps it safe

Where AI helps in QA (the 4 good jobs)

Where AI lies in QA (the 4 traps)

A working example: an AI agent with an oracle it cannot move

The 3 tests every AI feature needs

Where to start on Monday

How to Test Passkey (WebAuthn) Login in Playwright (2026)

Key takeaways

What is a passkey, in plain words

Why nobody tested this flow

What Playwright 1.61 added

How to test passkey login — a working example

Re-using a passkey across tests

The honest part: what this does not test

Why this matters for your CI pipeline

Internal References

Playwright Best Practices: 10 Rules AI Agents Get Wrong (2026)

1. Find elements the way a user sees them

2. Use web-first assertions that wait for you

3. Keep every test isolated

4. Seed state through the API, not the UI

5. Do not lean on test IDs by default

6. Turn on traces for the first retry

7. Run tests in parallel and shard them

8. Keep if and try out of your tests

9. Test what users see, not how it is built

10. Define projects in your config

The human still owns the standard

I measure how fast 42 LLMs actually answer. Here's the honest method.

The two numbers

How I measure it

What surprised me

Why I rebuilt it

How to Score Your AI Test Agents: Offline Evaluation with Trajectories (2026)

1. "It passed" is not a score

2. Record the run as a trajectory

3. Replay it offline

4. Score each step with evaluation types

5. Wire the score into CI

6. Where this sits: the Evidence Layer

The 5-line checklist

Playwright Codegen: The Complete Guide (2026)

1. How to Launch Playwright Codegen

2. Capturing Assertions

3. Playwright Codegen Best Practices

Avoid Hardcoded Wait Times

Use Semantic Locators

Isolate Your Auth State

4. The Architectural View: From Draft to System

Playwright vs Cypress vs Selenium in 2026

What Changed In 2026

Pick Playwright When Evidence Matters

Pick Cypress When Local Feedback Matters Most

Keep Selenium When Migration Risk Is Higher

The Decision Table

My Four-Question Test

The Practical Recommendation

Author Bio

Playwright v1.60 Turns Test Failures Into Evidence

The practical update

Code example

The rule

AI Test Automation Architecture: The 3-Layer System

Why tool lists fail

The 3-layer model

The practical gate

The rule

How to Test MCP Servers Before They Break Your CI

The three-layer test stack

Layer 1: Discovery with MCP Inspector

Layer 2: Behavior with pytest

Layer 3: Security with a permission audit

Where to find servers worth testing

Verdict

Playwright MCP v0.0.73: How to Configure Browser Paths via Environment Variables

What changed

8. Keep `if` and `try` out of your tests