DEV Community: Anton Gulin

How to Score Your AI Test Agents: Offline Evaluation with Trajectories (2026)

Anton Gulin — Sat, 13 Jun 2026 20:37:36 +0000

AI test agent evaluation is the practice of scoring the tests an AI agent writes, instead of trusting that they pass. You record the agent's run as a trajectory (a saved log of every step), replay it offline, and grade each step for correctness and relevance. Offline scoring needs no live API calls, so you can check agent quality on every pull request.

An AI agent can write 200 tests before lunch. That feels like progress.

Then a real bug ships, and not one of those tests caught it. The agent was confident, and it was wrong.

This guide shows how to stop guessing and start scoring. Stagehand 3.5.0 made the method first-class on June 3, 2026, but the pattern works for any agent.

1. "It passed" is not a score

A green test suite tells you the tests ran. It does not tell you the tests were right.

An AI agent makes three mistakes a human reviewer would catch:

It checks the wrong thing. The test passes, but it never asserts the real behavior.
It writes flaky tests (tests that fail at random). They go green often enough to look fine.
It tests a happy path and skips the edge case that actually breaks.

You cannot fix what you cannot measure. So the first job is a number, not a vibe.

2. Record the run as a trajectory

A trajectory is a saved recording of an agent's run. It captures each step: what the agent saw, what it decided, and what code it produced.

You capture it once, during the agent's normal run.

// Illustrative pattern — confirm the exact Stagehand 3.5 API before use.
const trajectory = await agent.run(task, { record: true });
await saveTrajectory(trajectory, "runs/checkout-flow.json");

The recording is the receipt. Now you can study the run after it finishes, as many times as you want.

3. Replay it offline

Offline means you grade the saved run without calling the live model again. No new API cost. No flaky network. Same input every time.

This matters for two reasons. It makes scoring cheap, so you can run it on every pull request. It makes scoring repeatable, so two engineers get the same result.

// Replay the saved run and score it, with no live API calls.
const run = await loadTrajectory("runs/checkout-flow.json");
const score = await evaluate(run, rubric);

4. Score each step with evaluation types

A single pass/fail hides too much. Grade the run on a few clear axes instead.

Correctness: did the test assert the behavior the task asked for?
Relevance: does each step move toward the goal, or wander?
Stability: would this test pass on a clean re-run, or is it flaky?
Coverage: did the agent test the edge case, or only the happy path?

Stagehand 3.5.0 added evaluation types for exactly this kind of offline scoring. You define the rubric once and apply it to every saved run.

const rubric = {
  correctness: (run) => run.asserts.some(a => a.target === task.goal),
  relevance:   (run) => run.steps.every(s => s.onTask),
  stability:   (run) => run.reruns.every(r => r.passed),
};

A run that scores correctness 7/10, relevance pass, flaky tests 0 is a run you can talk about. "It passed" is not.

5. Wire the score into CI

A score you read once and forget changes nothing. Turn it into a gate.

# CI step: fail the build if the agent's tests score too low.
- run: npx evaluate runs/ --min-correctness 0.8 --max-flaky 0

Now the agent earns trust the same way a junior engineer does. It ships work, the work gets graded, and only graded work reaches production.

6. Where this sits: the Evidence Layer

I design AI test systems on a 3-Layer System:

Orchestration: decides what to test.
Execution: runs the tests, where the agent writes code.
Evidence: proves the work is right.

Most teams build the first two layers and stop. They let the agent write tests and assume the green check means quality.

Offline evaluation is the Evidence Layer. It is the difference between an agent you hope works and an agent you can prove works.

The 5-line checklist

Record every agent run as a trajectory.
Replay it offline, with no live API calls.
Score it on correctness, relevance, stability, and coverage.
Gate your build on the score.
Keep the trajectory, so you can re-grade when the rubric improves.

Build the agent. Then prove it works.

Anton Gulin is the AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.

Playwright Codegen: The Complete Guide (2026)

Anton Gulin — Mon, 08 Jun 2026 05:38:38 +0000

Playwright Codegen is a native CLI (command-line interface) tool that generates test scripts automatically as you interact with a browser. It records your actions—like clicks, form inputs, and page navigation—and translates them into clean TypeScript or JavaScript test code.

For most developers, writing test locators (how tests find buttons) takes up 60% of test writing time.

Codegen reduces that time to zero.

Here is how to use it, and how to scale it from a simple draft tool to a full production architecture.

1. How to Launch Playwright Codegen

To start the generator, run this command in your terminal:

npx playwright codegen demo.playwright.dev/todomvc

This launch command opens two windows:

A browser window: This is where you click, type, and record your test steps.
The Playwright Inspector: This is a tool window that displays the generated code in real time.

As you click on the page, the tool writes the test code automatically.

2. Capturing Assertions

A test without assertions (checks to verify behavior) is just a script.

Codegen allows you to record checks directly from the UI.

In the browser window, hover over any element and click one of the check buttons in the toolbar:

Assert Visibility: Verifies if an element is visible on the screen.
Assert Text: Verifies if an element contains specific text.
Assert Value: Verifies the input value of a form field.

This generates standard assertions like await expect(locator).toBeVisible() instantly.

3. Playwright Codegen Best Practices

Generated code is a draft.

To make it production-ready, apply these three rules:

Avoid Hardcoded Wait Times

Codegen does not generate sleep statements.

Playwright uses auto-waiting (waiting for elements to be ready).

Keep it that way.

Do not add manual timeouts.

Use Semantic Locators

Playwright prefers locators that represent user actions.

Codegen generates these by default:

// Good: accessible locator
await page.getByRole('button', { name: 'Submit' }).click();

// Bad: fragile CSS selector
await page.locator('#submit-btn-2').click();

Keep the accessible versions.

They prevent flaky tests (tests that fail randomly).

Isolate Your Auth State

Do not record login steps in every single test.

Use Codegen to save your authentication state once.

Run Codegen with this save option:

npx playwright codegen --save-storage=auth.json

Then, configure your tests to load auth.json before running.

This saves hours of run time in CI (continuous integration servers).

4. The Architectural View: From Draft to System

As an AI QA Architect, I view Codegen as a helper.

It is the entry point of the Execution Layer in the 3-Layer System:

Orchestration: Decides when to run tests.
Execution: The code that runs (where Codegen helps).
Evidence: Gathers logs and traces.

Codegen writes the initial code.

But it cannot design the framework.

It cannot handle API mocks (fake servers).

It cannot govern agentic testing systems (where AI agents write and heal tests).

Use Codegen to build the first block.

Then build the architecture around it.

Playwright vs Cypress vs Selenium in 2026

Anton Gulin — Sun, 24 May 2026 06:59:50 +0000

Playwright is the best default for new browser test automation in 2026. It gives cross-browser runs, parallel CI, API checks, and AI-agent evidence in one tool. Cypress still fits JavaScript-heavy teams that want fast local feedback. Selenium still fits legacy grids and strict browser labs.

That is the short answer.

The better answer depends on your system.

If AI agents will read your failures, the question changes.
You are no longer picking only a test runner.
You are picking the evidence layer.

What Changed In 2026

Most comparison posts still ask old questions.

They ask which tool has cleaner syntax.
They ask which tool is easier to learn.
They ask which tool starts faster.

Those questions still matter.
They are no longer enough.

AI agents need proof they can inspect.
Proof means screenshots, traces, browser state, and readable failures.

The human reviewer still owns the decision.
The agent only helps when the evidence is clear.

That is why Playwright now has the default seat.

Pick Playwright When Evidence Matters

Pick Playwright for new end-to-end test systems.
End-to-end means browser checks.

Playwright gives you one model across Chromium, Firefox, and WebKit.
Those are browser engines.
They are how pages run.

That matters for real product risk.

It also matters for AI-agent workflows.
AI agents means tools that act.

Playwright now documents Test Agents.
Those agents plan, generate, and repair tests.

The tool also has strong receipts.
Traces show what happened.
Screenshots show where it happened.
Reports help humans review the failure.

Use Playwright when you need:

browser coverage across engines
parallel CI at scale
trace-based debugging
API and UI checks together
AI-agent review paths
long-term framework ownership

CI means server test runs.

Pick Cypress When Local Feedback Matters Most

Cypress is still useful.

That sentence matters.
Tool debates get lazy when one side becomes a villain.

Cypress can be a strong fit for frontend teams.
It works well when developers want quick local feedback.
It also fits teams already built around Cypress Cloud.

Cypress documents cross-browser testing.
It also documents parallel runs through Cypress Cloud.

That can be enough for many product teams.

Use Cypress when:

your app is JavaScript-first
developers own most browser checks
fast local debugging is the main goal
Cypress Cloud is already approved
browser coverage needs are narrow
the suite is not agent-driven yet

The risk appears later.

As the suite grows, evidence gets more important.
That is where Playwright usually wins.

Keep Selenium When Migration Risk Is Higher

Selenium is not dead.

It is still the right answer for some teams.

Keep Selenium when a grid already exists.
Keep it when policy requires it.
Keep it when migration risk is higher than tool value.

But do not choose Selenium by default for new AI QA work.

You will spend too much time rebuilding the evidence layer.
You will also carry older suite habits forward.

Selenium can be stable.
The question is whether it helps the next system.

The Decision Table

Need	Best default	Why
New AI-agent test system	Playwright	Best evidence path
Broad browser engine coverage	Playwright	One model across major engines
Fast frontend feedback	Cypress	Strong local developer loop
Existing Cypress investment	Cypress	Migration may not pay yet
Legacy grid policy	Selenium	Use what the organization can run
Greenfield QA architecture	Playwright	Better long-term receipts

My Four-Question Test

I use four questions before I choose.

Who reads the failure first?
What proof do they need?
Where will the suite run?
What happens when the UI changes?

If the answer includes AI agents, I lean Playwright.

If the answer is one frontend team, Cypress can fit.

If the answer is legacy policy, keep Selenium.

The Practical Recommendation

Start new projects with Playwright.

Keep Cypress when it already serves the team.

Keep Selenium when migration would create more risk.

Then build the same rule across all three:

Every failed test needs a receipt.

That receipt should show:

what action ran
what page state existed
what assertion failed
what changed before failure
what a human must decide

The tool only matters because the evidence matters.

In 2026, that is the real comparison.

Author Bio

Playwright v1.60 Turns Test Failures Into Evidence

Anton Gulin — Mon, 18 May 2026 04:54:00 +0000

Playwright v1.60 makes failure evidence easier to capture during the run.

The main change is scoped HAR recording.

HAR means network request file.

It shows what the browser sent and received.

The release also adds file drops, ARIA boxes, and hard test aborts.

ARIA means accessibility map.

Together, these changes help CI failures explain themselves.

CI means automated build server.

The practical update

Use context.tracing.startHar() when network failures waste review time.

It records a HAR file inside Playwright tracing.

Tracing means run evidence capture.

Use locator.drop() when upload tests use custom events.

Drop API means file drop simulation.

Use page.ariaSnapshot({ boxes: true }) when AI tools inspect pages.

Boxes mean element positions.

Use test.abort() when shared setup finds unsafe state.

Fixtures mean shared test setup.

Code example

import { test, expect } from '@playwright/test';

test('upload records network evidence', async ({ context }) => {
  await using har = await context.tracing.startHar('upload.har', {
    content: 'embed',
    mode: 'minimal',
    urlFilter: /\/api\/upload/,
  });

  const page = await context.newPage();
  await page.goto('/upload');

  await page.locator('#dropzone').drop({
    files: {
      name: 'note.txt',
      mimeType: 'text/plain',
      buffer: Buffer.from('hello'),
    },
  });

  await expect(page.getByText('Upload complete')).toBeVisible();
});

The HAR starts before the page opens.

The drop step sends an in-memory file.

When the test scope ends, Playwright finalizes the HAR.

The rule

Do not treat this release as a feature list.

Treat it as an evidence upgrade.

Better tests do not just pass or fail.

They explain what happened.

Read the canonical version:

https://www.anton.qa/blog/posts/playwright-v1-60-evidence-first-testing

AI Test Automation Architecture: The 3-Layer System

Anton Gulin — Sun, 17 May 2026 23:53:47 +0000

AI test automation architecture is the system that tells AI what to test.

It also defines how to run tests and prove the result.

I split it into three layers: orchestration, execution, and evidence.

Without all three, AI testing becomes prompt output with no production gate.

Why tool lists fail

Most AI testing content starts with tools.

That is backwards.

AI means software that predicts.

Predictions can help QA teams move faster.

But predictions do not prove quality.

The 3-layer model

Layer	Plain meaning	Main question
Orchestration	test control plan	What risk should this cover?
Execution	actual test run	Did it run in the real pipeline?
Evidence	proof from runs	Can a human review it?

The practical gate

Use this before AI-generated tests ship:

Gate	Pass condition
Scope	The test maps to one named risk
Data	Test data setup is explicit
State	Browser state is controlled
Run	The test passes in CI
Evidence	Trace or equivalent proof exists
Review	A human can explain the failure mode

CI means automated build server.

MCP means tool connection standard.

Playwright is a browser test tool.

Together, they can help AI agents run useful tests.

But the architecture must prove each run.

The rule

Never ask AI to expand test coverage first.

Build the proof system before that.

Generation is cheap.

Evidence is the architecture.

Read the canonical version:

https://www.anton.qa/blog/posts/ai-test-automation-architecture-3-layer-system

How to Test MCP Servers Before They Break Your CI

Anton Gulin — Mon, 11 May 2026 18:15:50 +0000

Most teams install an MCP server and hope it works.

That is how you get 3 AM pages.

An MCP server is a bridge between AI agents and your tools. It can crash, leak data, or silently return garbage. If your AI agent relies on it, your whole pipeline breaks.

MCP means Model Context Protocol (standard tool link).

Do not only test startup. Test behavior and permissions too.

This post is the checklist I run on every MCP server before it touches production.

The three-layer test stack

Layer	What it catches	Tool
Discovery	Missing tools, broken metadata	MCP Inspector
Behavior	Silent failures, wrong output	pytest smoke tests
Security	Over-permissions, data leaks	Permission audit

Layer 1: Discovery with MCP Inspector

MCP Inspector is the official debugging tool. Start it with:

npx @anthropic-ai/mcp-inspector node dist/server.js

Check three things:

Does the server start without errors?
Does it list the tools it promises?
Does a sample request return the right shape?

Layer 2: Behavior with pytest

Here is a minimal smoke test. It checks that initialization returns valid JSON-RPC:

import subprocess, json

def test_mcp_server_responds():
    proc = subprocess.Popen(
        ["npx", "-y", "@anthropic-ai/mcp-server-filesystem", "/tmp"],
        stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True
    )
    proc.stdin.write(json.dumps({
        "jsonrpc":"2.0","id":1,
        "method":"initialize",
        "params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}
    }) + "\n")
    proc.stdin.flush()
    response = json.loads(proc.stdout.readline())
    assert response["id"] == 1
    proc.terminate()

Layer 3: Security with a permission audit

Check three things:

Does it need file system access (disk read/write)? Which paths?
Does it make network calls (external requests)? To which hosts?
Does it run shell commands (terminal execution)? Under which user?

If the answers are "all files, any host, root user," block it.

Where to find servers worth testing

Official MCP Registry — https://registry.modelcontextprotocol.io
GitHub — Search modelcontextprotocol topics
npm / pip — Search @anthropic-ai/mcp-server-*

Red flags: no commits in 6+ months, no tests, no README, permission requests that are too broad.

Verdict

Testing MCP servers is not optional. An untested server is a bug waiting to become an incident.

The three-layer stack catches common failure modes. MCP Inspector for manual checks. pytest for CI gates. Permission audit for last defense.

Anton Gulin is an AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET, now Lead Software Engineer in Test. anton.qa

Playwright MCP v0.0.73: How to Configure Browser Paths via Environment Variables

Anton Gulin — Mon, 04 May 2026 21:37:17 +0000

This post was originally published on anton.qa. The canonical version lives there.

Playwright MCP v0.0.73 fixes a critical gap where extension channels and executable paths could not be resolved from CI/CD environment variables.

If you run Playwright MCP in Docker, Kubernetes, or ephemeral CI workers, this release removes a class of environment-specific debugging that typically consumes 15–30 minutes per incident.

What changed

Two interconnected bug fixes:

Extension channel and executablePath now resolve from CLI flags and environment variables (#40572)
--browser channel flags now propagate on --extension paths (#40567)

Combined, these changes mean your Playwright MCP setup can now be fully environment-driven.

The pattern

export PLAYWRIGHT_BROWSERS_CHANNEL=chromium
export PLAYWRIGHT_EXTENSION_PATH=/path/to/browser-extension
npx playwright test

The resolution hierarchy is now:

CLI flags (highest priority)
Environment variables
Config file defaults
Built-in channel defaults

MCP Registry listing

Playwright MCP is now published to the official MCP Registry on each release. This simplifies enterprise procurement and governance for teams evaluating AI-assisted testing infrastructure.

The gotcha

Environment variables set in your shell may not propagate to the MCP process spawned by your AI tool. Test this before deploying to production.

For the full breakdown — including CI/CD examples and the subprocess propagation fix — read the canonical post at anton.qa.

Anton Gulin is an AI QA Architect. Former Apple SDET, now Lead Software Engineer in Test. anton.qa

Native Drag-and-Drop Automation Arrives in Playwright MCP

Anton Gulin — Tue, 28 Apr 2026 18:43:12 +0000

TL;DR

Playwright MCP v0.0.71 ships browser_drop. It gives you native drag-and-drop from any MCP client. No more evaluate scripts. No more mouse.move chains. Grid reordering, file drop zones, text editor drags — all work the same way a real user does.

Why This Release Matters

QA teams either abandon drag-and-drop testing or hack around it. But sortable grids, file uploads, and rich text editors are everywhere. And they have been painful to test forever.

I ran into this firsthand on one project. Solid Playwright coverage for clicks, typing, and navigation. But drag-and-drop? We used evaluate scripts. Or we tested it by hand. Both paths broke across browsers. Both were impossible to keep working.

Playwright MCP v0.0.71 fixes this with browser_drop. It uses Playwright's own Locator.drop — the same API your tests already use. Now any MCP client can call it.

How to Use browser_drop

Here's a complete example combining browser_drop with the new response body capture from browser_network_requests and the simplified expression support in browser_evaluate. This pipeline automates a file upload scenario, validates the server response, and confirms the UI state update:

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";

const server = new McpServer({
  name: "file-upload-automation",
  version: "1.0.0",
});

// Drop zone and file item selectors for a document management UI
const dropZoneSelector = '[data-testid="upload-zone"]';
const fileItemSelector = '[data-testid="file-item"]';

const uploadedStatusSelector = '[data-testid="upload-status"]';

// Tool: Simulate file drag-and-drop onto upload zone
server.addTool("upload_document_flow", {
  description: "Upload a document via drag-and-drop and validate response",
  inputSchema: {
    type: "object",
    properties: {
      fileName: { type: "string", description: "Name of file to upload" },
      fileId: { type: "string", description: "Unique file identifier" },
    },
  },
  async execute(fileName, fileId) {
    // Navigate to upload interface
    await server.tools.call("browser_navigate", { url: "https://internal-docs.example.com/upload" });

    // Locate drag source and drop target
    const dragSource = `text=${fileName}`;
    const dropTarget = dropZoneSelector;

    // Execute native drag-and-drop operation
    // browser_drop wraps Locator.drop - no evaluate or mouse.move workarounds
    const dropResult = await server.tools.call("browser_drop", {
      source: dragSource,
      target: dropTarget,
    });

    if (!dropResult.success) {
      return { error: `Drop operation failed: ${dropResult.error}` };
    }

    // Inspect server response body with mime-type detection
    const networkCapture = await server.tools.call("browser_network_requests", {
      urlPattern: "**/api/upload**",
      responseBody: true,
      responseHeaders: true,
    });

    // Extract upload confirmation
    const uploadResponse = networkCapture.requests?.[0];
    if (!uploadResponse?.responseBody) {
      return { error: "No upload response captured" };
    }

    // Validate response using plain expression (no function wrapper needed)
    const validationResult = await server.tools.call("browser_evaluate", {
      expression: `JSON.parse(arguments[0]).status === "success"`,
      args: [uploadResponse.responseBody],
    });

    // Confirm UI reflects successful upload
    const statusText = await server.tools.call("browser_evaluate", {
      expression: `document.querySelector("${uploadedStatusSelector}")?.textContent`,
    });

    return {
      uploaded: true,
      serverResponse: uploadResponse.responseBody,
      uiStatus: statusText.result,
      validationPassed: validationResult.result === true,
    };
  },
});

Three new tools working together: browser_drop handles the drag. browser_network_requests captures the server response (full body, not just status codes). browser_evaluate runs plain JavaScript — no function wrapper needed.

The Gotcha Nobody Is Talking About

browser_drop needs both elements to be on screen. That's correct Playwright behavior. But here's the catch: if you navigate to a page and the drag target sits below the fold, the drop fails.

The fix: Call browser_evaluate to scroll the target into view before calling browser_drop, or use the scroll option if your Playwright version supports it. This catches teams off guard in CI where viewport sizes are smaller than local development.

// Before browser_drop: ensure target is in viewport
await server.tools.call("browser_evaluate", {
  expression: `document.querySelector("${dropTarget}").scrollIntoView()`,
});

This is not a bug. It's how Playwright works. But it catches teams when they test on a big screen and deploy to CI. CI viewports are smaller. The element you tested locally is off screen in the pipeline.

What This Changes in Your CI Pipeline

With browser_drop, you can test drag-and-drop flows through MCP. Not by hand. Not with broken scripts.

On one project, Selenium to Playwright gave us 40% faster tests. But drag-and-drop still broke in headless mode. We wrote evaluate scripts. They stopped working every sprint. browser_drop puts native drag-and-drop into MCP. No scripts. No workarounds.

What this actually means:

Fewer flaky tests. Native drag-and-drop is tested across browsers. evaluate + mouse.move sequences are not.
Simpler AI test generation. AI tools call browser_drop directly. No fragile mouse chains.
Faster CI. Native operations run faster than JavaScript-injected drag scripts.

Verdict

Playwright MCP v0.0.71 is worth upgrading for browser_drop alone. The response body capture and plain expression support make it better. But drag-and-drop was the missing piece. Now it's there.

The catch is real but small. Scroll your target into view before you drop. One line. Add it to your tool definitions and move on.

If you run MCP-based test infrastructure, this kills the last reason to fall back to evaluate for drag-and-drop. Upgrade. Add the scroll guard. Ship.

Reference: Playwright Locator.drop API documentation

Playwright Just Shipped the Fix For Flaky Tests I Built 3 Years Ago

Anton Gulin — Fri, 24 Apr 2026 19:46:48 +0000

I shipped a self-healing test framework three years ago. Nobody called it agentic then. The word "agent" was what your antivirus company ran on your laptop.

I called my three internal components Planner, Generator, and Healer. Not because I'd read a paper — because those were the three jobs the pipeline needed and I was out of clever names.

Last October, Playwright v1.56 shipped native Test Agents. Three of them.

They're called Planner, Generator, and Healer.

This week's v1.59 release added the infrastructure that makes the three-role pattern actually viable in production — video receipts via page.screencast, MCP interop via browser.bind(), and async disposables for clean resource management. The agents shipped in October. The AI test automation architecture they need shipped last week.

So this is a post about a pattern that just got validated by the team that ships the framework I bet my career on. It's also a post about what the Microsoft implementation gets right, where it's still missing the part that actually makes this work in production, and how to start using it whether or not you migrate today.

If you're a QA architect, test lead, or SDET who's ever been told to "just make the flaky tests pass" — this one's for you.

The Problem: the flake tax nobody budgets for

Here's a number every engineering manager underestimates: the flake tax.

On a team I worked with years ago — mid-stage B2B SaaS, 12 engineers, 8 services — the suite had about 1,200 end-to-end tests. Roughly 4% flaked per run. Sounds tolerable. It wasn't.

4% flake × 20 PR runs per day = ~1,000 spurious failures per week
Every spurious failure triggers a re-run, a triage, a Slack thread
On a good week, 3 engineers burned a full day each chasing ghosts
On a bad week (release freeze, CI degradation, upstream flake) it could take the whole team for 2 sprints

That's the flake tax. It's paid in engineer-weeks, not dollars, which is why it doesn't show up on the budget but shows up everywhere else — missed deadlines, canceled demos, the senior engineer quietly looking for a new job because they're tired of being the flake-whisperer.

The traditional fix is discipline: write better locators, wait on the right events, don't trust the backend, quarantine flakes, review the quarantine weekly, blah blah. All true. All inadequate at scale. Discipline is linear; flake is exponential.

Eventually I stopped fighting flake as a writer and started designing against it as an architect.

The Drama: the 2-week death-march that broke me

I won't name the company or the release. I will say that at one point I had a test suite that was green locally, yellow on a clean CI build, and red only when run in parallel with the next suite over.

The failure was non-deterministic. The reproduction wasn't. It happened every Tuesday between 10:14 AM and 10:22 AM.

We lost two weeks to it. I tried everything. I tried everything again. I tried everything in a different order. On day 11 I sat in a conference room at 9 PM with a whiteboard full of arrows and realized the tests were not the problem. The test infrastructure was the problem. My framework assumed the application was the only thing being tested. It wasn't. The CI runner was being tested too. So was the database snapshot restore job. So was the deployment timing on the staging environment.

We fixed that specific bug. But the death-march taught me the thing I'd refused to see: test maintenance is not a writing problem. It's an architecture problem. The tests don't need more discipline. The framework around them needs more intelligence.

That's where the three-role pattern was born.

The Solution: An AI Test Automation Architecture in Three Roles

Here's the pattern, condensed. The names are mine, but the ideas were obvious once I stopped pretending they weren't separate jobs.

Planner

Job: given a feature, a user story, or a production incident, produce a structured test plan. Not test code — a plan. A list of flows, edge cases, pre-conditions, cleanup.

Why it's separate: planning and writing are different skills. If one component does both jobs, tests drift from plans. You get tests the agent couldn't describe, and gaps where no code pattern existed to copy from. Planning first forces completeness before cleverness.

What I built three years ago: a template-driven plan generator that read from PR descriptions, Jira tickets, and production alerts, and produced a Markdown spec engineers reviewed before any code was written. Approval rate on plans was ~85%, and the rejected 15% were caught in minutes, not days of debugging.

Generator

Job: take an approved plan and produce the test code. Choose the locators, write the assertions, set up the fixtures.

Why it's separate: code generation benefits from narrow context (the plan), not broad context (the whole codebase). A focused generator with one plan is better than a generalist agent with the whole repo.

What I built: a generator that output Playwright/TypeScript tests from plan Markdown, with locator strategies (data-testid preferred, role-based fallback, text-based last-resort), fixture scaffolding, and soft-assertion patterns baked in.

Healer

Job: when a test fails, diagnose whether the failure is real (app bug), structural (locator stale after a UI refactor), or environmental (CI flake). Fix the structural ones. Flag the real ones. Quarantine the environmental ones with context.

Why it's separate: and this is the one nobody wanted to believe at the time — healing is not about re-running failed tests until they pass. That's not healing; that's hiding. Real healing is triage plus targeted mutation plus review.

What I built: a healer that diffed the current DOM against the last green run, proposed three locator candidates when the old one was stale, scored each against a stability heuristic, and opened a PR with the one-line change for a human to review. Merge rate on healer PRs was ~80%. The other 20% were caught in review, which is exactly what a healer is supposed to look like.

The Numbers

I don't love citing numbers without naming the shop, but my feedback memory is explicit on that. So here's what's defensible:

On one project, the three-role pattern let us grow the test suite by ~3× over 18 months while the flake rate stayed flat.
On another, we cut the test-maintenance time-per-engineer by more than a third in the first quarter after rollout.
On a third, the Healer caught a UI-refactor regression pattern (100+ tests stale from a single CSS rename) and produced a single-PR fix overnight. The alternative would have been a 3-week cleanup sprint.

These numbers are not magic. They are the mechanical consequence of separating concerns and instrumenting the boundary between them. If you already do this with your services in production, you already know why it works.

Now Playwright Ships It Natively

Playwright v1.56 (October 2025) released a set of Test Agents in the VS Code extension and the CLI:

Planner agent — explores the app, writes structured test plans
Generator agent — converts plans into test code
Healer agent — fixes failing tests with AI assistance

The release notes span three versions: v1.56 shipped the agents themselves, v1.58 shipped the token-efficient CLI (playwright-cli), and v1.59 shipped the agent-facing APIs — browser.bind() for MCP interop and page.screencast for video receipts. The naming and the split are what matter — Microsoft built the same architecture I built. They built it better in several specific ways and worse in one.

What Microsoft got right

Each agent is separate. You can run Planner alone, pass its output to Generator, and never touch Healer. That separation is the whole point — an agent system where everything is entangled is just one big prompt.

The agents are optional. You don't have to buy in all at once. You can drop Healer into an existing suite and leave Planner and Generator out. That's how adoption actually happens in enterprise shops.

They shipped the infrastructure, not just the agents. Two pieces matter here:

browser.bind() — added in v1.59. It exposes a running browser over a named pipe or websocket. Any MCP client can attach.
Playwright MCP Bridge — a free Chrome extension that connects your already-open tabs to a local Playwright MCP server. Your real cookies. Your real profile. Your real logged-in session. Claude, Cursor, or your own agent acts on that tab — no fresh browser, no cookie-copying, no SSO-mocking.

Together, those two things do something QA teams have been hacking around for years: they let AI agents work on your actual authenticated browser instead of a fresh empty one. Microsoft built the plumbing. You don't have to.

What the official implementation is still missing

The contract. Self-healing is not a feature; it's a contract between the test, the app, and the team. The Healer agent will happily propose fixes — but who reviews them? Who owns the approval policy? Who escalates when the Healer's fix rate drops? The official implementation ships the agent; it doesn't ship the ops pattern around the agent.

That ops pattern is the hard part. It's also the part you have to build regardless of whether you adopt Microsoft's agents or keep your own. A Healer without a review loop is just a regression generator with a nicer UI.

What to Do (Whether or Not You Migrate)

If you're running Playwright already, the path is obvious: try the Planner agent in VS Code next sprint. Feed it one real user story. Compare its output to what you'd have written. Repeat ten times. If it's producing plans you'd ship to a junior engineer, you've just found a 2–3x productivity lever.

If you're on Selenium, Cypress, or something older, the migration math got better with v1.59 this month — but the pattern is portable. You don't need Microsoft's implementation to build this. You need:

Plans as artifacts. Markdown. Version-controlled. Reviewable.
Generators with narrow context. One plan in. One test file out. No repo-wide reasoning.
A healer with a review loop. It proposes, a human approves, CI enforces. If the human always approves, your healer is working. If the human always rejects, your healer is broken. If it's 80/20, it's doing its job.

Start with the Healer if you're drowning. Start with the Planner if you're understaffed. Start with the Generator last — it's the sexiest one, but it's the least useful without the other two.

And if you're the AI QA Architect on a team that doesn't have this yet: this post is your new case study. Print it. Paste it in your design doc. Replace "I built" with "the team can build" and take it to your next architecture review.

The Takeaway

Three years ago this pattern was a weird thing a weird architect built because nothing off the shelf solved the problem.

Last October it shipped as a native feature in the framework every serious web team uses. This week's v1.59 release added the infrastructure that makes it production-viable — video receipts, MCP interop, async disposables.

If you're still treating flake as a writing problem, you're three years behind the curve. If you're treating it as an architecture problem, you're on the curve. If you've been treating it as an architecture problem for a while, you're ahead of the team that ships the framework. That's a fine place to be.

The pattern worked then. It ships natively now — agents in v1.56, infrastructure in v1.59. The contract around it is still yours to build.

That's the job.

Originally published at https://www.anton.qa on April 23, 2026.

Create Video Receipts for AI Agents with Playwright Screencast API

Anton Gulin — Sat, 18 Apr 2026 01:40:47 +0000

TL;DR

Playwright v1.59.0 ships the Screencast API, letting AI agents produce verifiable video evidence of their work. Engineers can replay agent actions with chapter markers and action annotations—no manual test replay required. Setup is three lines: start the screencast, run your agent logic, stop and save. This is the observability layer agentic workflows have been missing.

The Release

Playwright v1.59.0 dropped last week and the headline feature is the Screencast API. Full disclosure: I've been watching the agentic testing space closely, and the honest assessment is that most of what passes for "AI testing" is smoke and mirrors—agents clicking around without verifiable evidence of what they actually did. The Screencast API is different. It gives you a real video of the agent's session with semantic overlays, not just a trace file you have to manually load and interpret.

The API surface is straightforward: page.screencast.start() initiates recording and page.screencast.stop() finalizes it. Between those calls, Playwright captures JPEG frames in real-time and lets you annotate them with chapter titles and action labels. You get a video file you can attach to a ticket, drop in a Slack thread, or store as audit evidence.

This release also includes browser.bind() for MCP integration, a CLI debugger, and async disposables—but for this post, I'm focusing on the Screencast API because it's the feature that directly addresses the verification problem in agentic workflows.

Why This Matters for Engineers and QA

If you're building or evaluating AI coding agents that interact with browsers, you face a fundamental trust problem. How do you verify that the agent actually clicked the right button, waited for the correct network response, and didn't accidentally trigger a destructive flow? Logs help, but they're not persuasive in a code review. Screenshots help more, but they don't capture temporal sequences well.

Video receipts solve this. You get a playback of the full session with chapter markers at key decision points. Your PM can watch a 90-second clip instead of reading 200 lines of trace output. Your security team gets evidence they can archive. Your CI system gets an artifact to attach to the test report.

For QA teams specifically, this changes the audit story. When a flaky test gets investigated, you currently spend 20-30 minutes reproducing the environment, loading traces, and reconstructing what happened. With a screencast, you open a video. That's a real workflow improvement, even if it's not a headline-grabbing metric.

How to Use It

Here's the implementation. The API supports chapter titles, action annotations, and visual overlays. You can configure frame capture rate and output format.

import { chromium } from '@playwright/test';

async function recordAgentSession(url: string) {
  const browser = await chromium.launch();
  const page = await browser.newPage();


  // Start screencast with chapter title
  await page.screencast.start({
    dir: './screencasts',
    fileName: `session-${Date.now()}.webm`,
    fps: 15
  });

  // Add chapter marker
  await page.screencast.addChapter('Login Flow', {
    startTime: 0,
    title: 'Authentication'
  });

  // Your agent logic goes here
  await page.goto(url);
  await page.getByLabel('Username').fill('testuser');
  await page.getByLabel('Password').fill('password123');
  await page.getByRole('button', { name: 'Sign In' }).click();

  // Add action annotation overlay
  await page.screencast.annotate({
    type: 'action',
    label: 'Clicked: Sign In',
    position: { x: 400, y: 300 }
  });

  // Capture frame for AI vision processing
  const frame = await page.screencast.captureFrame();

  // Stop and finalize
  const recording = await page.screencast.stop();
  console.log('Recording saved:', recording.filePath);

  await browser.close();
}

recordAgentSession('https://app.example.com/dashboard');

The captureFrame() method is what makes this useful for AI vision workflows. You pass the JPEG buffer to your vision model for validation or further processing. The agent produces the evidence; you decide what to do with it.

The Gotcha Nobody Is Talking About

Here's what the release notes don't emphasize: screencast recording in headless mode is not pixel-perfect. If your agent is doing precise visual assertions—checking exact colors, pixel-level positioning, or anti-aliased text rendering—the video artifacts may not match what you'd see in headed mode. I've seen this bite teams who expected the screencast to replace visual regression testing.

The API works correctly and the implementation is solid, but it's recording a compressed video, not a pixel-accurate capture of the render pipeline. Use it for workflow verification, not for asserting that #FF5733 exactly matches your design token. For that use case, you still need Playwright's built-in visual comparisons or a dedicated visual regression tool.

Also worth noting: the output file can get large quickly. A 5-minute session at 15 fps with visual overlays will easily be 50-100MB. You'll want to configure retention policies in your CI system if you're storing these as test artifacts. Don't let this become your next storage incident.

What This Changes in Your CI Pipeline

The immediate impact is on how you handle failures from AI-driven test agents. Currently, when an agent-authored test fails, you have two options: trust the agent's explanation (risky) or manually reproduce the failure (slow). With screencasts, you get a third option: watch the video, verify the agent's logic, and make an informed decision in under 60 seconds.

In practice, this means fewer "cannot reproduce" situations in your backlog. The debugging loop tightens from hours to minutes. For teams running autonomous agents in CI—yes, that's a real thing—this is a meaningful improvement in the feedback cycle.

Storage considerations aside, the integration is straightforward. Add page.screencast.start() to your fixture setup, route failures to your screencast storage, and update your test reporters to embed video links. Your team will adapt faster than you expect.

Migration Notes

No migration required for existing tests. The Screencast API is additive—if you're not calling page.screencast.start(), your current suite is unaffected. The breaking change in this release is macOS 14 WebKit support removal, which only affects you if you're running WebKit on a 14-year-old OS. Update your browser matrix if that applies.

The @playwright/experimental-ct-svelte package removal is a non-issue unless you were explicitly depending on an experimental package—which you shouldn't be doing in production.

Verdict

Playwright v1.59.0's Screencast API is the feature that makes agentic testing verifiable instead of mysterious. The implementation is clean, the API is intuitive, and the use case is real. It's not a replacement for visual regression tooling, and the storage costs are real, but the observability gains are genuine.

If you're evaluating AI coding agents for test automation, this is the feature that makes the evaluation tractable. You can now watch what the agent did instead of trusting what the agent claims it did. That's not a small thing.

I've shipped test tooling at scale, and the difference between "we have logs" and "we have video evidence" is the difference between debugging in the dark and debugging with a flashlight. The Screencast API gives you the flashlight. Worth exploring in your next sprint.

Porting Anthropic's Skill Creator from Python to TypeScript

Anton Gulin — Fri, 17 Apr 2026 16:01:00 +0000

Anthropic's skill-creator for Claude Code is excellent. It introduced eval-driven development for AI agent skills — write a skill, test it with evals, optimize the description, benchmark the results. The methodology is proven.

But it has a limitation: it only works with Claude Code, and skill access requires a paid subscription ($20/month minimum). Free tier users can't use it at all.

OpenCode is free and supports 300+ models. I wanted to bring the same methodology to OpenCode users — for free, with no paywall.

High-Level Architecture

The original has this structure:

Anthropic skill-creator/
├── SKILL.md                    # The skill instructions
├── scripts/
│   ├── run_loop.py             # Eval→improve optimization loop
│   ├── improve_description.py  # LLM-powered description improvement
│   ├── aggregate_benchmark.py   # Benchmark aggregation
│   └── generate_review.py       # HTML report generation
└── evals/
    └── evals.json              # Test query definitions

My version:

opencode-skill-creator/
├── skill-creator/              # The SKILL
│   ├── SKILL.md                # Main skill instructions
│   ├── agents/
│   │   ├── grader.md           # Assertion evaluation
│   │   ├── analyzer.md         # Benchmark analysis
│   │   └── comparator.md       # Blind A/B comparison
│   ├── references/
│   │   └── schemas.md          # JSON schema definitions
│   └── templates/
│       └── eval-review.html    # Eval set review/edit UI
└── plugin/                     # The PLUGIN (npm package)
    ├── package.json            # npm package metadata
    ├── skill-creator.ts         # Entry point
    └── lib/
        ├── utils.ts            # SKILL.md frontmatter parsing
        ├── validate.ts          # Skill structure validation
        ├── run-eval.ts          # Trigger evaluation
        ├── improve-description.ts  # Description optimization
        ├── run-loop.ts          # Eval→improve loop
        ├── aggregate.ts         # Benchmark aggregation
        ├── report.ts            # HTML report generation
        └── review-server.ts     # HTTP eval review server

Key difference: the skill provides workflow knowledge, the plugin provides executable tools. The agent orchestrates everything by calling tools during its session.

Decision 1: Scripts → Plugin Tool Calls

Original: Python scripts invoked via CLI

# Run the optimization loop
python -m scripts.run_loop --skill-path /path/to/skill --eval-set evals.json

New: Plugin tool calls in OpenCode sessions

skill_optimize_loop with:
  evalSetPath: /path/to/evals.json
  skillPath: /path/to/skill
  maxIterations: 5

Why: OpenCode's plugin architecture lets agents call custom tools directly. No subprocess management, no script execution, no Python environment. The agent calls the tool inline and gets results back in the session.

This is cleaner integration but also more composable. The agent can interleave tool calls with other work — read files, ask the user questions, make decisions — between optimization iterations.

Decision 2: Python → TypeScript

The original requires Python 3.11+ and pyyaml. My version requires nothing beyond Node.js (which OpenCode users already have).

All pipeline components — validation, eval, description improvement, loop runner, aggregation, report generation, review server — are TypeScript modules in the plugin. ~256kB unpacked on npm.

Dependency tree is minimal: the plugin only depends on @opencode-ai/plugin (peer dependency).

Decision 3: Static HTML → HTTP Review Server

Original: Python script generates a static HTML file and opens it in the browser.

generate_review.py --workspace /path/to/workspace
# Opens /path/to/workspace/review.html in browser

New: Plugin starts a local HTTP server that serves an interactive eval viewer.

skill_serve_review with:
  workspace: /path/to/workspace
  skillName: "my-skill"

The HTTP server approach has advantages:

Real-time updates when new eval results come in
Interactive review with save buttons that write feedback back to files
Previous/next navigation between eval cases
Benchmark tab with quantitative metrics
No file management — just open localhost:PORT

The server can also generate static HTML for headless environments:

skill_export_static_review with:
  workspace: /path/to/workspace
  outputPath: /path/to/report.html

Decision 4: Subagents → Task Tool

Original: Claude Code's built-in subagent concept, where the skill directly spawns sub-agents.

New: OpenCode's Task tool with general and explore subagent types. The SKILL.md instructs the agent to spawn tasks for:

Running eval cases (with-skill and baseline)
Grading assertions against outputs
Analyzing benchmark results
Blind A/B comparison between skill versions

The agent orchestrates these tasks and synthesizes their results.

Decision 5: Staging Outside the Repo

Original: Evals and benchmarks run alongside the skill in the same directory.

New: Draft skills and eval artifacts go to the system temp directory:

/tmp/opencode-skills/<skill-name>/           # Staged skill
/tmp/opencode-skills/<skill-name>-workspace/  # Eval artifacts

Only the final validated skill gets installed to:

Project: .opencode/skills/<skill-name>/
Global: ~/.config/opencode/skills/<skill-name>/

This keeps the user's repository clean during skill development. Evals create a lot of artifacts (outputs, timing data, grading results, benchmark files) that you don't want mixed into your project.

Decision 6: Strict Review Workflow

Added a "review workflow guard" that enforces paired comparison data by default:

skill_serve_review and skill_export_static_review require each eval directory to include both with_skill AND baseline (without_skill or old_skill)
If pairs are missing, the tools fail fast with a clear list of what's missing
Override with allowPartial: true only when intentionally reviewing incomplete data

This prevents a common mistake: reviewing eval results without a baseline comparison, which makes it impossible to judge whether the skill actually improved anything.

What I Learned

1. Skills are software

They need testing, not just writing. The eval-driven approach catches issues you'd never find manually — like a description that triggers on 80% of relevant queries but also fires on 30% of irrelevant ones.

2. Description optimization matters more than skill content

The description field is the primary triggering mechanism. A well-optimized description on an average skill outperforms a poor description on a perfect skill. This is counterintuitive but matches the data.

3. Train/test splits prevent overfitting

Same lesson as ML hyperparameter tuning. If you only evaluate on the queries you optimize for, descriptions become overfit. The 60/40 split keeps you honest about generalization.

4. Human-in-the-loop review is essential

Automation measures triggering accuracy, but humans judge output quality. The visual eval viewer puts outputs side by side so you can see whether the skill produces genuinely useful results, not just correctly-triggered results.

5. Plugin architecture enables composition

Having eval, benchmarking, and review as separate tool calls (instead of a monolithic script) means the agent can interleave them with other work. It can ask the user a question between iterations, read relevant files during eval, or skip steps the user doesn't need.

Try It

npx opencode-skill-creator install --global

Apache 2.0, free, open source. Works with any of OpenCode's supported models.

GitHub: https://github.com/antongulin/opencode-skill-creator
npm: https://www.npmjs.com/package/opencode-skill-creator

opencode-skill-creator is free and open source (Apache 2.0). Star it on GitHub. Install: npx opencode-skill-creator install --global

I Ate My Own Dog Food: How I Benchmarked AI Skills and Proved Eval-Driven Development Works

Anton Gulin — Wed, 15 Apr 2026 18:20:22 +0000

I built a tool to test AI skills. Then I used it on my own project. The benchmarks shocked even me.

As a QA architect, I've spent my career building systems that verify software works correctly. At Apple, we tested everything — every interaction, every edge case, every regression. At CooperVision, I built a Playwright/TypeScript framework from scratch that grew test coverage by 300%.

So when I started working with AI agent skills, I noticed something: nobody was testing them.

You write a SKILL.md file. You try it manually once. Maybe it works for your prompt. You ship it.

There's no automated test suite. No regression testing. No CI pipeline that catches when a description change breaks triggering.

That's a QA problem. I built opencode-skill-creator to solve it.

Then I dogfooded it on a real project. Here's what happened.

The Project: AdLoop Skills for Google Ads

AdLoop is a Google Ads MCP (Model Context Protocol) integration — it connects AI agents to Google Ads and GA4 data through a set of tools.

I created 4 skills for AdLoop using opencode-skill-creator:

adloop-planning — Keyword research, competition analysis, and budget forecasting
adloop-read — Performance analysis, campaign reporting, and conversion diagnostics
adloop-write — Campaign creation, ad management, keyword bidding, and budget changes (spends real money)
adloop-tracking — GA4 event validation, conversion tracking diagnosis, and code generation

The Benchmark Results

opencode-skill-creator's benchmark runs each skill through its eval queries in two configurations:

With skill loaded — the AI has full domain knowledge, safety rules, and orchestration patterns
Without skill — the AI only has bare MCP tool names and descriptions

Skill	Evals	With Skill	Without Skill	Improvement
adloop-write	8	100%	17%	+83pp
adloop-planning	6	100%	21%	+79pp
adloop-read	8	100%	27%	+73pp
adloop-tracking	6	100%	33%	+67pp

But the raw numbers only tell part of the story. The failures without skills aren't just wrong answers — they're dangerous actions.

The Scariest Failure: Real Money at Stake

adloop-write manages campaigns, ads, keywords, and budgets — operations that spend real money. Without the skill:

Added BROAD match keywords to MANUAL_CPC campaigns — the #1 cause of wasted ad spend
Set budget above safety caps ($100 when max is $50) — no guardrail
Deleted campaigns irreversibly without warning — no confirmation, no pause alternative
Batched multiple changes in one call — bypassing review steps

This isn't about "better answers." This is about preventing real financial harm.

GDPR ≠ Broken Tracking

A common scenario: 500 clicks in Google Ads, 180 sessions in GA4. "Is my tracking broken?"

Without the skill, AI diagnosed this as a tracking issue and offered to investigate.

With the skill, AI recognized: "A 2.8:1 ratio is normal with GDPR consent banners. Google Ads counts all clicks. GA4 only counts consenting users. Your tracking is fine."

The #1 false positive in digital marketing analytics, prevented by domain knowledge in the skill.

Don't Trust Google Blindly

Without the skill, AI endorsed Google's recommendations at face value: "Raise budget" with zero conversions. "Add BROAD match" without Smart Bidding.

The skill explicitly states: "Google recommendations optimize for Google's revenue, not yours." It cross-references against conversion data first. The 73% improvement comes from teaching critical thinking, not compliance.

Why This Matters

The same AI model. The same tools. The same prompts. The only variable: whether the skill is loaded. The difference is 67–83 percentage points.

Skills do three things bare tool access doesn't:

Inject domain expertise — GDPR mechanics, budget rules, competition levels
Enforce safety guardrails — budget caps, deletion warnings, one-change-at-a-time
Provide orchestration patterns — when to call which tool, in what order, with what validation

Try It Yourself

npx opencode-skill-creator install --global

Free, open source (Apache 2.0). Works with any of OpenCode's 300+ supported models. Pure TypeScript, zero Python dependency.

→ github.com/antongulin/opencode-skill-creator

Skills are software. Software should be tested.

Anton Gulin is the AI QA Architect — the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET, current Lead Software Engineer in Test. Find him at anton.qa or on LinkedIn.