DEV Community: Faizal

The Playwright Playbook — Bonus: Refactoring Schema Validation with Zod

Faizal — Sun, 21 Jun 2026 13:28:45 +0000

The Playwright Playbook — Bonus: Refactoring Schema Validation with Zod

"The best content ideas come from the comments, not the outline."

In Part 4, we built utils/schema-validator.ts — a hand-rolled function that checks whether an API response actually matches the Task shape we expect. Type-correct fields, valid status enum, valid date strings. The works.

A reader, Misha, asked a great question in the comments: "What do you think about using Zod for schema validation?"

Fair challenge. The manual validator works — but it's not how most production codebases would do this in 2026. So here's the honest comparison, side by side, with the refactor. 🎯

This is a focused bonus — just the schema validator. Everything else from Part 4 (TaskApiClient, AuthApiClient, the test files) stays exactly as it was.

🔍 Where We Left Off — The Original Validator

Here's what we built in Part 4:

// utils/schema-validator.ts — original (Part 4)

export interface TaskSchema {
  id: number;
  title: string;
  status: string;
  assignee: string;
  createdAt: string;
  updatedAt: string;
}

export function validateTaskSchema(task: unknown): task is TaskSchema {
  if (typeof task !== 'object' || task === null) return false;

  const t = task as Record<string, unknown>;

  return (
    typeof t.id === 'number' &&
    typeof t.title === 'string' &&
    t.title.length > 0 &&
    ['pending', 'in_progress', 'completed'].includes(t.status as string) &&
    typeof t.assignee === 'string' &&
    typeof t.createdAt === 'string' &&
    typeof t.updatedAt === 'string' &&
    !isNaN(Date.parse(t.createdAt as string)) &&
    !isNaN(Date.parse(t.updatedAt as string))
  );
}

export function validateTaskListSchema(tasks: unknown): tasks is TaskSchema[] {
  return Array.isArray(tasks) && tasks.every(validateTaskSchema);
}

This works. It's type-safe. It does the job.

But it has real limitations once your schemas grow past one entity:

❌ No error messages — just true/false. WHICH field failed? You don't know.
❌ The TaskSchema interface and the validation logic can silently drift apart.
❌ Adding optional fields, nested objects, or unions means hand-writing more checks.
❌ No reusable composition — every new entity needs its own validator from scratch.

Let's fix all four with Zod. 👇

📦 Installing Zod

npm install zod

That's the only new dependency. No config changes needed.

🔧 The Refactored Validator

// utils/schema-validator.ts — refactored with Zod
import { z } from 'zod';

// Define the schema ONCE — this is now the single source of truth
export const TaskSchema = z.object({
  id: z.number(),
  title: z.string().min(1, 'title must not be empty'),
  status: z.enum(['pending', 'in_progress', 'completed']),
  assignee: z.string(),
  createdAt: z.string().refine(
    (val) => !isNaN(Date.parse(val)),
    { message: 'createdAt must be a valid date string' }
  ),
  updatedAt: z.string().refine(
    (val) => !isNaN(Date.parse(val)),
    { message: 'updatedAt must be a valid date string' }
  ),
});

// Type is INFERRED from the schema — no separate interface to maintain
export type Task = z.infer<typeof TaskSchema>;

export const TaskListSchema = z.array(TaskSchema);

/**
 * Validates a single task. Returns true/false — same signature as before,
 * so existing test files using validateTaskSchema() don't need to change.
 */
export function validateTaskSchema(task: unknown): task is Task {
  return TaskSchema.safeParse(task).success;
}

/**
 * Validates a list of tasks. Same drop-in signature as Part 4.
 */
export function validateTaskListSchema(tasks: unknown): tasks is Task[] {
  return TaskListSchema.safeParse(tasks).success;
}

/**
 * NEW — Validates and returns the actual Zod errors when something fails.
 * This is what the original validator couldn't do at all.
 */
export function validateTaskSchemaDetailed(task: unknown) {
  const result = TaskSchema.safeParse(task);

  if (result.success) {
    return { valid: true as const, data: result.data };
  }

  // Human-readable list of exactly what failed and why
  const errors = result.error.issues.map(
    issue => `${issue.path.join('.')}: ${issue.message}`
  );

  return { valid: false as const, errors };
}

Notice: validateTaskSchema and validateTaskListSchema keep the exact same function signature as Part 4. Every test file that imports them — tasks-api.spec.ts, graphql-api.spec.ts, api-ui-chain.spec.ts — works unchanged. Zero refactor needed in the test suite. ✅

🆚 Side-by-Side Comparison

                      Hand-rolled (Part 4)     Zod (this bonus)
──────────────────    ──────────────────────   ──────────────────────
Lines of code          ~20                      ~20 (similar, but does more)
Error messages          None — just boolean      Field-level, human readable
Type source             Separate interface       Inferred from schema (z.infer)
Adding optional field   Manual undefined check   .optional() — one word
Nested objects          Painful, manual          .object({ ... }) — composable
Reusability             Copy-paste per entity     Schemas compose with .extend()
Drift risk               Interface vs logic        Single source of truth
                        can silently diverge

✅ Using the Detailed Errors in Tests

This is where Zod actually pays off — when a test fails, you know exactly why.

// tests/api/tasks-api.spec.ts — example using the new detailed validator
import { test, expect } from '../../fixtures/api.fixture';
import { validateTaskSchemaDetailed } from '../../utils/schema-validator';

test('GET /api/tasks/:id returns correct task — with detailed errors', async ({
  adminTaskApi,
}) => {
  const { task: created } = await adminTaskApi.createTask({
    title: 'Zod validation test task',
  });

  const fetched = await adminTaskApi.getTask(created.id);

  const result = validateTaskSchemaDetailed(fetched);

  if (!result.valid) {
    // If this fails, the test output tells you EXACTLY which field broke
    // e.g. "status: Invalid enum value. Expected 'pending' | 'in_progress' | 'completed', received 'done'"
    throw new Error(`Schema validation failed:\n${result.errors.join('\n')}`);
  }

  expect(result.data.title).toBe('Zod validation test task');

  await adminTaskApi.deleteTask(created.id);
});

Compare the failure messages:

🔴 Old validator failure:
  expect(validateTaskSchema(fetched)).toBe(true)
  Expected: true
  Received: false
  → You now have to manually inspect the response to find what's wrong

✅ Zod validator failure:
  Schema validation failed:
  status: Invalid enum value. Expected 'pending' | 'in_progress' | 'completed', received 'done'
  → You know immediately: the API started returning 'done' instead of 'completed'

That second one is the entire argument for Zod. When this fails in CI at 2am, you don't need to reproduce it locally to know what broke. 🔥

🧩 Bonus — Composing Schemas (Something the Old Validator Couldn't Do)

This is where Zod really separates itself. Say we want a schema for a task that includes nested assignee details instead of just a string:

// Reusable sub-schema
const AssigneeSchema = z.object({
  id: z.number(),
  name: z.string(),
  email: z.string().email(),
});

// Compose it into the Task schema with .extend()
const TaskWithAssigneeSchema = TaskSchema.extend({
  assignee: AssigneeSchema, // overrides the string field with the nested object
});

// Or build a "create payload" schema that's a subset of fields
const CreateTaskPayloadSchema = TaskSchema.pick({ title: true, status: true });

// Or make fields optional for a PATCH payload
const UpdateTaskPayloadSchema = TaskSchema
  .pick({ title: true, status: true, assignee: true })
  .partial(); // every field becomes optional

Try doing .pick(), .partial(), or .extend() with a hand-rolled type-guard function. You'd be rewriting the whole validator by hand every time. With Zod, it's one line. ✅

🤔 So... Was the Hand-Rolled Validator Wrong?

Not wrong — just simpler than what you'd want at scale.

Hand-rolled validator makes sense when:
  ├── You have one or two simple schemas
  ├── You want zero dependencies
  └── You're teaching the underlying concept (which is why Part 4 used it)

Zod makes sense when:
  ├── You have multiple related schemas (Task, CreateTaskPayload, UpdateTaskPayload...)
  ├── You want error messages that actually help you debug
  ├── Your schemas need composition (nested objects, nullable, optional, unions)
  └── You're building this for a real production test suite

For a real framework — which is what we've built across all 8 parts — Zod is the better long-term choice. Worth the one dependency. 🎯

📁 What Changed in the Project Structure

Just one file modified. Everything else from Parts 1–8 stays exactly as built:

playwright-playbook/
├── utils/
│   └── schema-validator.ts    ← refactored (this bonus), same exports
├── package.json                ← added "zod" dependency

No test files needed to change. No fixtures needed to change. That's the benefit of keeping the function signatures identical during the refactor. ✅

🔖 Before You Go

Thanks to Misha for the question that sparked this. This is exactly the kind of comment I want more of — keep them coming. 🙌

If there's a part of the framework you'd want refactored, extended, or challenged — drop it below. The next bonus post might be your idea.

Drop a comment below 👇

Are you using Zod already, or is this your first look at it?
What other library would you want to see swapped in — Yup? Joi? io-ts?
Anything else from the series you'd want a deep-dive bonus on?

Faizal Shaikh | Senior Automation Engineer | Playwright & AI Testing
Connect with me on LinkedIn

The Playwright Playbook — Part 8: Playwright Meets AI — Agents, MCP & Self-Healing Tests

Faizal — Sun, 21 Jun 2026 12:57:08 +0000

The Playwright Playbook — Part 8: Playwright Meets AI — Agents, MCP & Self-Healing Tests

"AI doesn't replace QA engineers. It gives them superpowers."

Seven parts. One framework. Built from scratch.

POM-based UI tests. Network interception. Multi-user contexts. A full API testing layer with typed clients. Visual regression across four viewports. A complete debugging toolkit. A production CI/CD pipeline with sharding, Docker, and Slack notifications.

This is what we built.

Now we add AI on top of it.

Not as a replacement — as an amplifier.

The engineers who will define the next decade of QA are the ones who understand how to use AI tools deliberately — knowing exactly what they're good for, exactly where they fall short, and how to combine them with the solid engineering we've spent seven parts building.

That's what Part 8 is about.

Playwright MCP. AI test agents. Self-healing selectors.

Let's finish this. 🎯

🏗️ Where We Left Off

After Part 7, our framework is complete and production-ready:

playwright-playbook/
├── .github/workflows/
│   ├── playwright.yml                           ✅ Part 7
│   └── playwright-visual.yml                   ✅ Part 7
├── tests/
│   ├── auth/login.spec.ts                       ✅ Part 1
│   ├── tasks/task-management.spec.ts            ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   ├── multi-user/                              ✅ Part 3
│   ├── multi-tab/                               ✅ Part 3
│   ├── api/                                     ✅ Part 4
│   ├── visual/                                  ✅ Part 5
│   └── debug/                                   ✅ Part 6
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── api/
│   ├── TaskApiClient.ts                         ✅ Part 4
│   └── AuthApiClient.ts                         ✅ Part 4
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   ├── multi-user.fixture.ts                    ✅ Part 3
│   └── api.fixture.ts                           ✅ Part 4
├── scripts/
│   ├── record-har.ts                            ✅ Part 2
│   └── notify-slack.ts                          ✅ Part 7
├── utils/
│   ├── schema-validator.ts                      ✅ Part 4
│   ├── visual-helpers.ts                        ✅ Part 5
│   └── debug-helpers.ts                         ✅ Part 6
├── docker/                                      ✅ Part 7
├── snapshots/                                   ✅ Part 5
├── .vscode/                                     ✅ Part 6
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Parts 1–7
├── .gitignore                                   ✅ Part 7
└── .env

By the end of Part 8, we add:

playwright-playbook/
├── tests/
│   └── ai/                                      ← NEW
│       └── ai-generated.spec.ts
├── ai/                                          ← NEW
│   ├── TestPlanner.ts
│   └── TestHealer.ts
├── scripts/
│   ├── generate-test-plan.ts                    ← NEW
│   └── heal-selectors.ts                        ← NEW
└── .mcp.json                                    ← NEW

Every file gets fully built below. 👇

🧠 Understanding the AI Layer — Three Distinct Tools

Before we write code, be clear on what we're actually dealing with. There are three separate AI capabilities in the Playwright ecosystem, and they do very different things:

Tool                    What it does                          Status
──────────────────────  ───────────────────────────────────   ──────────────
Playwright MCP          Exposes Playwright as a tool for      Available now
                        AI assistants (Claude, Copilot)       (v1.47+)
                        to control a real browser

AI Test Agents          Playwright's built-in agents that     Experimental
(Planner/Generator)     explore your app and generate         (v1.47+ with flag)
                        test skeletons automatically

Self-Healing            Automatically patches broken          Custom (we build
                        locators when selectors change        this in Part 8)

We'll build proper, practical implementations of all three — and be honest about where each one earns its place. 🎯

🔌 Playwright MCP — Giving AI Assistants a Browser

MCP stands for Model Context Protocol — an open standard for connecting AI assistants to external tools. Playwright's MCP server lets an AI assistant (Claude, GitHub Copilot, Cursor) directly control a real Chromium browser.

This means you can say to Claude or Copilot: "Navigate to the task creation flow and write me a Playwright test for it." And it will literally open a browser, click around, inspect the DOM, and generate real test code.

Setting up Playwright MCP

Install the MCP server:

npm install -D @playwright/mcp

Configure it for your project:

// .mcp.json — MCP configuration for your project
{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": [
        "@playwright/mcp",
        "--browser", "chromium",
        "--base-url", "http://localhost:3000",
        "--output-dir", "tests/ai"
      ],
      "env": {
        "PLAYWRIGHT_STORAGE_STATE": ".auth/admin.json"
      }
    }
  }
}

For VS Code with GitHub Copilot, add to .vscode/settings.json:

{
  "mcp": {
    "servers": {
      "playwright": {
        "command": "npx",
        "args": ["@playwright/mcp", "--browser", "chromium"],
        "env": {
          "BASE_URL": "http://localhost:3000"
        }
      }
    }
  }
}

What MCP enables

With MCP configured, your AI assistant can:

You say:
  "Write a test for the task deletion flow including the confirmation dialog"

AI does:
  1. Opens Chromium via MCP
  2. Navigates to http://localhost:3000/tasks
  3. Inspects the DOM — finds task items, hover targets, delete buttons
  4. Identifies the confirmation dialog structure
  5. Generates a TypeScript test using getByRole/getByTestId locators
  6. Writes it to tests/ai/ directory

You get:
  A real, runnable test file — not a hallucinated one

The key difference from AI generating test code from a description: MCP sees the actual DOM. It generates locators that actually exist, not ones it imagines. 🔥

How to use it with Claude

Once MCP is configured in Claude Desktop or Claude.ai:

Prompt: "I have a Playwright project at http://localhost:3000.
Navigate to the task management page, explore the UI,
and write a complete TypeScript Playwright test for creating,
editing, and deleting a task. Use our TaskPage POM from pages/TaskPage.ts."

Claude will:
  → Open browser via MCP
  → Navigate and inspect the real UI
  → Generate tests that match actual element structure
  → Reference your existing POM correctly

🤖 Building the Test Planner — AI-Powered Test Coverage Analysis

The Test Planner uses AI to analyse your existing test suite and your app's DOM structure, then suggests missing test scenarios.

// ai/TestPlanner.ts
import * as fs from 'fs';
import * as path from 'path';
import { chromium } from '@playwright/test';

export interface TestScenario {
  feature: string;
  scenario: string;
  priority: 'high' | 'medium' | 'low';
  covered: boolean;
  suggestedTestFile: string;
}

export interface TestPlan {
  url: string;
  generatedAt: string;
  totalScenarios: number;
  coveredCount: number;
  uncoveredCount: number;
  scenarios: TestScenario[];
}

export class TestPlanner {
  private readonly baseUrl: string;
  private readonly testsDir: string;

  constructor(baseUrl: string, testsDir: string = './tests') {
    this.baseUrl = baseUrl;
    this.testsDir = testsDir;
  }

  /**
   * Crawl the app and extract all interactive elements.
   * Returns a structured map of page → interactive elements.
   */
  async crawlApp(
    storageState?: string
  ): Promise<Map<string, string[]>> {
    const browser = await chromium.launch();
    const context = storageState
      ? await browser.newContext({ storageState })
      : await browser.newContext();

    const page = await context.newPage();
    const pageMap = new Map<string, string[]>();

    const pagesToCrawl = [
      '/login',
      '/dashboard',
      '/tasks',
      '/admin',
    ];

    for (const route of pagesToCrawl) {
      try {
        await page.goto(`${this.baseUrl}${route}`, {
          waitUntil: 'networkidle',
          timeout: 10000,
        });

        // Extract all interactive elements and their accessible names
        const elements = await page.evaluate(() => {
          const interactable: string[] = [];

          // Buttons
          document.querySelectorAll('button').forEach(btn => {
            const name = btn.getAttribute('aria-label') ||
                         btn.textContent?.trim() ||
                         btn.getAttribute('data-testid') || '';
            if (name) interactable.push(`button: ${name}`);
          });

          // Links
          document.querySelectorAll('a[href]').forEach(link => {
            const name = link.textContent?.trim() ||
                         link.getAttribute('aria-label') || '';
            if (name) interactable.push(`link: ${name}`);
          });

          // Forms
          document.querySelectorAll('form').forEach((form, i) => {
            const inputs = form.querySelectorAll('input, textarea, select');
            interactable.push(`form[${i}]: ${inputs.length} fields`);
          });

          // Modals/Dialogs
          document.querySelectorAll('[role="dialog"]').forEach(dialog => {
            const label = dialog.getAttribute('aria-label') ||
                          dialog.getAttribute('aria-labelledby') || 'dialog';
            interactable.push(`dialog: ${label}`);
          });

          return interactable;
        });

        pageMap.set(route, elements);
      } catch {
        pageMap.set(route, ['[page not accessible]']);
      }
    }

    await browser.close();
    return pageMap;
  }

  /**
   * Scan existing test files and extract test names.
   * Used to check which scenarios are already covered.
   */
  getExistingTestNames(): string[] {
    const testNames: string[] = [];
    const testGlob = this.testsDir;

    const scanDir = (dir: string) => {
      if (!fs.existsSync(dir)) return;
      const entries = fs.readdirSync(dir, { withFileTypes: true });
      for (const entry of entries) {
        const fullPath = path.join(dir, entry.name);
        if (entry.isDirectory()) {
          scanDir(fullPath);
        } else if (entry.name.endsWith('.spec.ts')) {
          const content = fs.readFileSync(fullPath, 'utf-8');
          const matches = content.matchAll(/test\(['"`](.+?)['"`]/g);
          for (const match of matches) {
            testNames.push(match[1]);
          }
        }
      }
    };

    scanDir(testGlob);
    return testNames;
  }

  /**
   * Generate a test plan — what's covered, what's missing, priorities.
   */
  async generatePlan(): Promise<TestPlan> {
    const pageMap = await this.crawlApp('.auth/admin.json');
    const existingTests = this.getExistingTestNames();

    const scenarios: TestScenario[] = [
      // Auth scenarios
      { feature: 'Authentication', scenario: 'successful login redirects to dashboard', priority: 'high', suggestedTestFile: 'tests/auth/login.spec.ts', covered: false },
      { feature: 'Authentication', scenario: 'invalid credentials show error message', priority: 'high', suggestedTestFile: 'tests/auth/login.spec.ts', covered: false },
      { feature: 'Authentication', scenario: 'empty fields show validation errors', priority: 'high', suggestedTestFile: 'tests/auth/login.spec.ts', covered: false },
      { feature: 'Authentication', scenario: 'logout clears session', priority: 'high', suggestedTestFile: 'tests/auth/login.spec.ts', covered: false },

      // Task CRUD
      { feature: 'Task Management', scenario: 'user can create a new task', priority: 'high', suggestedTestFile: 'tests/tasks/task-management.spec.ts', covered: false },
      { feature: 'Task Management', scenario: 'user can edit an existing task', priority: 'high', suggestedTestFile: 'tests/tasks/task-management.spec.ts', covered: false },
      { feature: 'Task Management', scenario: 'user can delete a task', priority: 'high', suggestedTestFile: 'tests/tasks/task-management.spec.ts', covered: false },
      { feature: 'Task Management', scenario: 'task list shows correct count after creation', priority: 'medium', suggestedTestFile: 'tests/tasks/task-management.spec.ts', covered: false },
      { feature: 'Task Management', scenario: 'task creation with empty title is blocked', priority: 'medium', suggestedTestFile: 'tests/tasks/task-management.spec.ts', covered: false },
      { feature: 'Task Management', scenario: 'task status can be changed to completed', priority: 'medium', suggestedTestFile: 'tests/tasks/task-management.spec.ts', covered: false },

      // Permissions
      { feature: 'Role Permissions', scenario: 'admin sees admin panel — regular user does not', priority: 'high', suggestedTestFile: 'tests/multi-user/role-permissions.spec.ts', covered: false },
      { feature: 'Role Permissions', scenario: 'admin can delete any task — user cannot', priority: 'high', suggestedTestFile: 'tests/multi-user/role-permissions.spec.ts', covered: false },
      { feature: 'Role Permissions', scenario: 'user accessing /admin is redirected', priority: 'high', suggestedTestFile: 'tests/multi-user/role-permissions.spec.ts', covered: false },

      // Real-time
      { feature: 'Real-Time', scenario: 'task assigned by admin appears instantly for user', priority: 'medium', suggestedTestFile: 'tests/multi-user/realtime-collaboration.spec.ts', covered: false },
      { feature: 'Real-Time', scenario: 'notification appears when task assigned', priority: 'medium', suggestedTestFile: 'tests/multi-user/realtime-collaboration.spec.ts', covered: false },

      // API
      { feature: 'API', scenario: 'GET /api/tasks returns 200 with valid schema', priority: 'high', suggestedTestFile: 'tests/api/tasks-api.spec.ts', covered: false },
      { feature: 'API', scenario: 'POST /api/tasks without auth returns 401', priority: 'high', suggestedTestFile: 'tests/api/tasks-api.spec.ts', covered: false },
      { feature: 'API', scenario: 'DELETE /api/tasks/:id returns 404 for non-existent', priority: 'medium', suggestedTestFile: 'tests/api/tasks-api.spec.ts', covered: false },

      // Error handling
      { feature: 'Error Handling', scenario: 'shows error banner when API returns 500', priority: 'high', suggestedTestFile: 'tests/network/error-simulation.spec.ts', covered: false },
      { feature: 'Error Handling', scenario: 'shows retry button when network fails', priority: 'medium', suggestedTestFile: 'tests/network/error-simulation.spec.ts', covered: false },
      { feature: 'Error Handling', scenario: 'shows loading skeleton when API is slow', priority: 'low', suggestedTestFile: 'tests/network/error-simulation.spec.ts', covered: false },
    ];

    // Match existing tests against scenarios
    for (const scenario of scenarios) {
      scenario.covered = existingTests.some(name =>
        name.toLowerCase().includes(scenario.scenario.toLowerCase().slice(0, 30))
      );
    }

    const coveredCount = scenarios.filter(s => s.covered).length;

    return {
      url: this.baseUrl,
      generatedAt: new Date().toISOString(),
      totalScenarios: scenarios.length,
      coveredCount,
      uncoveredCount: scenarios.length - coveredCount,
      scenarios,
    };
  }

  /**
   * Write the test plan to a Markdown file.
   */
  async writePlanMarkdown(outputPath: string): Promise<void> {
    const plan = await this.generatePlan();

    const coveragePercent = Math.round(
      (plan.coveredCount / plan.totalScenarios) * 100
    );

    const lines: string[] = [
      `# Test Plan — ${plan.url}`,
      ``,
      `Generated: ${new Date(plan.generatedAt).toLocaleString()}`,
      ``,
      `## Coverage Summary`,
      ``,
      `| Total Scenarios | Covered | Uncovered | Coverage |`,
      `|-----------------|---------|-----------|----------|`,
      `| ${plan.totalScenarios} | ✅ ${plan.coveredCount} | ❌ ${plan.uncoveredCount} | ${coveragePercent}% |`,
      ``,
      `## Scenarios by Feature`,
      ``,
    ];

    const features = [...new Set(plan.scenarios.map(s => s.feature))];
    for (const feature of features) {
      lines.push(`### ${feature}`);
      lines.push('');
      const featureScenarios = plan.scenarios.filter(s => s.feature === feature);
      for (const s of featureScenarios) {
        const status = s.covered ? '✅' : '❌';
        const priority = s.priority === 'high' ? '🔴' : s.priority === 'medium' ? '🟡' : '🟢';
        lines.push(`- ${status} ${priority} ${s.scenario}`);
        if (!s.covered) {
          lines.push(`  - Add to: \`${s.suggestedTestFile}\``);
        }
      }
      lines.push('');
    }

    fs.writeFileSync(outputPath, lines.join('\n'), 'utf-8');
    console.log(`✅ Test plan written to ${outputPath}`);
    console.log(`   Coverage: ${coveragePercent}% (${plan.coveredCount}/${plan.totalScenarios})`);
  }
}

🔧 Building the Self-Healing Engine

The most practical AI feature in a real test suite. When a locator breaks because a developer changed a data-testid or renamed a button — instead of the test just failing, the healer automatically finds the best alternative locator and patches the source file.

// ai/TestHealer.ts
import * as fs from 'fs';
import * as path from 'path';
import { Page, Locator } from '@playwright/test';

export interface HealResult {
  originalSelector: string;
  healedSelector: string;
  confidence: 'high' | 'medium' | 'low';
  strategy: string;
}

export interface HealReport {
  file: string;
  healed: HealResult[];
  failed: string[];
}

export class TestHealer {
  /**
   * Attempt to find an element using fallback strategies when the primary locator fails.
   *
   * Healing strategy priority:
   *   1. getByTestId (data-testid — most stable)
   *   2. getByRole + name (semantic — stable)
   *   3. getByLabel (form elements)
   *   4. getByText (exact visible text)
   *   5. CSS selector (last resort)
   */
  static async healLocator(
    page: Page,
    originalSelector: string,
    context?: {
      expectedText?: string;
      expectedRole?: string;
      nearText?: string;
    }
  ): Promise<HealResult | null> {
    // Strategy 1: Try getByTestId variations
    if (context?.expectedText) {
      const testIdVariants = [
        context.expectedText.toLowerCase().replace(/\s+/g, '-'),
        context.expectedText.toLowerCase().replace(/\s+/g, '_'),
        context.expectedText.toLowerCase().replace(/\s+/g, ''),
      ];

      for (const testId of testIdVariants) {
        const locator = page.getByTestId(testId);
        if (await locator.count() > 0) {
          return {
            originalSelector,
            healedSelector: `page.getByTestId('${testId}')`,
            confidence: 'high',
            strategy: 'data-testid variant',
          };
        }
      }
    }

    // Strategy 2: getByRole + name
    if (context?.expectedRole && context?.expectedText) {
      const roleLocator = page.getByRole(
        context.expectedRole as 'button' | 'link' | 'heading' | 'listitem',
        { name: context.expectedText }
      );
      if (await roleLocator.count() === 1) {
        return {
          originalSelector,
          healedSelector: `page.getByRole('${context.expectedRole}', { name: '${context.expectedText}' })`,
          confidence: 'high',
          strategy: 'role + name',
        };
      }
    }

    // Strategy 3: getByText (exact)
    if (context?.expectedText) {
      const textLocator = page.getByText(context.expectedText, { exact: true });
      if (await textLocator.count() === 1) {
        return {
          originalSelector,
          healedSelector: `page.getByText('${context.expectedText}', { exact: true })`,
          confidence: 'medium',
          strategy: 'exact text match',
        };
      }

      // Partial text match
      const partialLocator = page.getByText(context.expectedText);
      if (await partialLocator.count() === 1) {
        return {
          originalSelector,
          healedSelector: `page.getByText('${context.expectedText}')`,
          confidence: 'low',
          strategy: 'partial text match',
        };
      }
    }

    // Strategy 4: Try to find near a known element
    if (context?.nearText) {
      const nearLocator = page.locator(`:near(:text("${context.nearText}"))`);
      if (await nearLocator.count() > 0) {
        return {
          originalSelector,
          healedSelector: `page.locator(':near(:text("${context.nearText}"))').first()`,
          confidence: 'low',
          strategy: 'proximity to known text',
        };
      }
    }

    return null;
  }

  /**
   * Scan test files for broken selectors after a test run.
   * Returns a report of what was healed and what couldn't be fixed.
   */
  static scanForBrokenSelectors(testResultsPath: string): string[] {
    const brokenSelectors: string[] = [];

    if (!fs.existsSync(testResultsPath)) {
      return brokenSelectors;
    }

    const results = JSON.parse(fs.readFileSync(testResultsPath, 'utf-8'));

    // Extract locator errors from test results
    for (const suite of results.suites ?? []) {
      for (const spec of suite.specs ?? []) {
        if (!spec.ok) {
          for (const test of spec.tests ?? []) {
            for (const result of test.results ?? []) {
              const error = result.error?.message ?? '';
              // Match common locator failure messages
              const locatorMatch = error.match(
                /waiting for (locator|getByTestId|getByRole|getByText)\(['"`](.+?)['"`]\)/
              );
              if (locatorMatch) {
                brokenSelectors.push(locatorMatch[0]);
              }
            }
          }
        }
      }
    }

    return [...new Set(brokenSelectors)]; // deduplicate
  }

  /**
   * Patch a test file — replace an old selector with a healed one.
   */
  static patchTestFile(
    filePath: string,
    original: string,
    healed: string
  ): boolean {
    if (!fs.existsSync(filePath)) return false;

    const content = fs.readFileSync(filePath, 'utf-8');
    if (!content.includes(original)) return false;

    const patched = content.replace(new RegExp(escapeRegex(original), 'g'), healed);
    fs.writeFileSync(filePath, patched, 'utf-8');

    console.log(`  ✅ Patched: ${path.basename(filePath)}`);
    console.log(`     Before: ${original}`);
    console.log(`     After:  ${healed}`);

    return true;
  }
}

function escapeRegex(str: string): string {
  return str.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}

📜 The Generate Test Plan Script

// scripts/generate-test-plan.ts
import { TestPlanner } from '../ai/TestPlanner';
import * as path from 'path';
import * as dotenv from 'dotenv';

dotenv.config();

async function main() {
  const baseUrl = process.env.BASE_URL || 'http://localhost:3000';
  const outputPath = path.join(process.cwd(), 'TEST_PLAN.md');

  console.log(`🤖 Generating test plan for ${baseUrl}...`);
  console.log('   Crawling app and scanning existing tests...\n');

  const planner = new TestPlanner(baseUrl, './tests');
  await planner.writePlanMarkdown(outputPath);

  console.log('\n📋 Test plan saved to TEST_PLAN.md');
  console.log('   Review uncovered scenarios and add missing tests.\n');
}

main().catch(err => {
  console.error('Test plan generation failed:', err);
  process.exit(1);
});

Run it anytime to get an up-to-date picture of coverage:

npx ts-node scripts/generate-test-plan.ts

Output:

# Test Plan — http://localhost:3000

Generated: 18/06/2026, 09:15:00

## Coverage Summary

| Total Scenarios | Covered | Uncovered | Coverage |
|-----------------|---------|-----------|----------|
| 21 | ✅ 17 | ❌ 4 | 81% |

## Scenarios by Feature

### Authentication

- ✅ 🔴 successful login redirects to dashboard
- ✅ 🔴 invalid credentials show error message
- ✅ 🔴 empty fields show validation errors
- ❌ 🔴 logout clears session
  - Add to: `tests/auth/login.spec.ts`

...

🔧 The Heal Selectors Script

// scripts/heal-selectors.ts
import { chromium } from '@playwright/test';
import { TestHealer } from '../ai/TestHealer';
import * as path from 'path';
import * as dotenv from 'dotenv';

dotenv.config();

/**
 * Run this after a test failure to attempt automatic selector healing.
 *
 * Usage:
 *   npx ts-node scripts/heal-selectors.ts
 *
 * What it does:
 *   1. Reads test-results/results.json for broken selectors
 *   2. Launches a browser and tries alternative locator strategies
 *   3. Patches test files with healed selectors
 *   4. Prints a healing report
 */
async function main() {
  const resultsPath = path.join(process.cwd(), 'test-results', 'results.json');
  const baseUrl = process.env.BASE_URL || 'http://localhost:3000';

  console.log('🔧 Playwright Self-Healing Selector Tool\n');

  // Step 1 — Find broken selectors from test results
  const broken = TestHealer.scanForBrokenSelectors(resultsPath);

  if (broken.length === 0) {
    console.log('✅ No broken selectors found in test results.');
    return;
  }

  console.log(`Found ${broken.length} broken selector(s):\n`);
  broken.forEach(s => console.log(`  ❌ ${s}`));
  console.log('');

  // Step 2 — Launch browser and try to heal each one
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext({
    storageState: '.auth/admin.json',
  });
  const page = await context.newPage();

  const report: { healed: number; failed: number } = { healed: 0, failed: 0 };

  for (const selector of broken) {
    console.log(`\n🔍 Attempting to heal: ${selector}`);

    await page.goto(`${baseUrl}/tasks`, { waitUntil: 'networkidle' });

    // Extract context clues from the selector string
    const textMatch = selector.match(/['"`]([^'"`]+)['"`]/);
    const expectedText = textMatch?.[1];

    const result = await TestHealer.healLocator(page, selector, {
      expectedText,
      expectedRole: selector.includes('button') ? 'button' : undefined,
    });

    if (result) {
      console.log(`  ✅ Healed with strategy: ${result.strategy}`);
      console.log(`     Confidence: ${result.confidence}`);
      console.log(`     New selector: ${result.healedSelector}`);
      report.healed++;
    } else {
      console.log(`  ❌ Could not automatically heal this selector.`);
      console.log(`     Manual fix required.`);
      report.failed++;
    }
  }

  await browser.close();

  // Step 3 — Summary
  console.log('\n─────────────────────────────────────');
  console.log('🔧 Healing Summary');
  console.log(`   ✅ Healed:  ${report.healed}`);
  console.log(`   ❌ Failed:  ${report.failed}`);
  console.log('─────────────────────────────────────');

  if (report.healed > 0) {
    console.log('\n⚠️  Review all healed selectors before committing.');
    console.log('   Run npx playwright test to verify the fixes.');
  }
}

main().catch(err => {
  console.error('Healer failed:', err);
  process.exit(1);
});

🧪 AI-Generated Tests — Honest Evaluation

Here's what AI-generated tests actually look like when you use MCP properly — and where the gaps are.

// tests/ai/ai-generated.spec.ts
// These tests were generated with Playwright MCP + Claude
// Reviewed and validated by a human before being committed
// Note: AI generated the structure — human verified selectors and assertions

import { test, expect } from '@playwright/test';
import { TaskPage } from '../../pages/TaskPage';
import { DashboardPage } from '../../pages/DashboardPage';

// ✅ Good AI generation — simple, well-structured, uses existing POM
test('task title is displayed in the task list after creation', async ({ page }) => {
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  await taskPage.createTask('AI generated test task');

  await expect(taskPage.getTaskLocator('AI generated test task')).toBeVisible();
  await expect(page.getByRole('listitem').filter({ hasText: 'AI generated test task' }))
    .toBeVisible();

  await taskPage.deleteTask('AI generated test task');
});

// ✅ Good — AI correctly identified the dashboard navigation flow
test('clicking task from dashboard navigates to task detail', async ({ page }) => {
  const dashboard = new DashboardPage(page);
  await dashboard.goto();

  // AI correctly observed this interaction via MCP
  const firstTask = page.getByRole('listitem').first();
  const taskTitle = await firstTask.getByTestId('task-title').textContent();

  await firstTask.getByRole('link', { name: 'View details' }).click();

  await expect(page).toHaveURL(/\/tasks\/\d+/);
  await expect(page.getByRole('heading', { level: 1 })).toHaveText(taskTitle ?? '');
});

// ✅ Good — AI found the filter interaction correctly
test('filtering tasks by status shows only matching tasks', async ({ page }) => {
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  // AI observed the filter dropdown via MCP browser exploration
  await page.getByRole('combobox', { name: 'Filter by status' }).selectOption('completed');

  const taskItems = page.getByRole('listitem');
  const count = await taskItems.count();

  for (let i = 0; i < count; i++) {
    await expect(taskItems.nth(i).getByTestId('status-badge')).toHaveText('Completed');
  }
});

// ⚠️ Needs human review — AI generated this but assertion is weak
// TODO: Strengthen the assertion to check specific error text
test('task creation form shows validation when title is empty', async ({ page }) => {
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  await taskPage.newTaskButton.click();
  // AI correctly identified save button but missed the specific error selector
  await taskPage.saveTaskButton.click();

  // ⚠️ AI used a generic assertion here — human should make this more specific
  await expect(page.getByRole('alert')).toBeVisible();
  // TODO: Replace with: await expect(page.getByText('Title is required')).toBeVisible();
});

// ❌ AI got this wrong — generated a selector that doesn't exist
// Left here deliberately to show where AI fails
// test('admin can bulk delete tasks', async ({ page }) => {
//   AI generated: await page.getByTestId('bulk-select-all').click();
//   Problem: this element doesn't exist in our app
//   AI hallucinated a feature that isn't there
//   This is why you always review AI-generated tests before running
// });

The commented-out test at the bottom is important. It shows the failure mode honestly — AI sometimes generates tests for features it thinks exist but don't. The lesson: AI generates test scaffolding. Humans validate it.

🤝 The Hybrid Strategy — Where AI Earns Its Place

Here's the honest breakdown of where AI helps and where it doesn't:

✅ AI is excellent for:
  ├── Generating test skeletons for new features fast
  ├── Exploring an unfamiliar codebase via MCP
  ├── Suggesting test scenarios you hadn't thought of
  ├── Writing boilerplate (beforeEach, fixtures, imports)
  ├── Generating test data variations
  └── Coverage gap analysis (TestPlanner.ts)

⚠️ AI needs human oversight for:
  ├── Verifying generated selectors actually exist in the DOM
  ├── Checking assertions are strong enough (not just toBeVisible)
  ├── Ensuring tests are truly independent (AI loves shared state)
  ├── Business logic assertions (AI doesn't know your rules)
  └── Any security or permissions testing

❌ AI should not be trusted for:
  ├── Testing non-deterministic behaviour without human review
  ├── Deciding what constitutes a "pass" for complex flows
  ├── Replacing the understanding of what actually matters to test
  └── Automatically committing healed selectors without review

The engineers who get the most value from AI tools are the ones who know these boundaries. They use AI to go faster on the tasks it does well. They apply their own judgment on the tasks it doesn't. 🎯

📦 Final `package.json` Scripts

Add these to your package.json to make the AI tools easy to run:

{
  "scripts": {
    "test": "playwright test",
    "test:headed": "playwright test --headed",
    "test:debug": "playwright test --debug",
    "test:ci": "playwright test --project=admin --project=user --project=multi-context --project=api",
    "test:visual": "playwright test --project=visual --project=visual-responsive",
    "test:api": "playwright test --project=api",
    "test:update-snapshots": "playwright test --project=visual --update-snapshots",
    "report": "playwright show-report",
    "record-har": "ts-node scripts/record-har.ts",
    "heal": "ts-node scripts/heal-selectors.ts",
    "plan": "ts-node scripts/generate-test-plan.ts",
    "notify": "ts-node scripts/notify-slack.ts"
  }
}

📁 The Complete Final Project Structure

Every file across all 8 parts — fully built, fully documented:

playwright-playbook/
├── .github/
│   └── workflows/
│       ├── playwright.yml                       ✅ Part 7
│       └── playwright-visual.yml               ✅ Part 7
├── tests/
│   ├── auth/
│   │   └── login.spec.ts                        ✅ Part 1
│   ├── tasks/
│   │   └── task-management.spec.ts              ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   │   ├── api-mocking.spec.ts
│   │   ├── error-simulation.spec.ts
│   │   └── network-assertions.spec.ts
│   ├── multi-user/                              ✅ Part 3
│   │   ├── role-permissions.spec.ts
│   │   └── realtime-collaboration.spec.ts
│   ├── multi-tab/                               ✅ Part 3
│   │   └── multi-tab-flows.spec.ts
│   ├── api/                                     ✅ Part 4
│   │   ├── tasks-api.spec.ts
│   │   ├── auth-api.spec.ts
│   │   ├── graphql-api.spec.ts
│   │   └── api-ui-chain.spec.ts
│   ├── visual/                                  ✅ Part 5
│   │   ├── dashboard-visual.spec.ts
│   │   ├── task-visual.spec.ts
│   │   └── responsive-visual.spec.ts
│   ├── debug/                                   ✅ Part 6
│   │   └── trace-examples.spec.ts
│   └── ai/                                      ✅ Part 8
│       └── ai-generated.spec.ts
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── api/
│   ├── TaskApiClient.ts                         ✅ Part 4
│   └── AuthApiClient.ts                         ✅ Part 4
├── ai/                                          ✅ Part 8
│   ├── TestPlanner.ts
│   └── TestHealer.ts
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   ├── multi-user.fixture.ts                    ✅ Part 3
│   └── api.fixture.ts                           ✅ Part 4
├── scripts/
│   ├── record-har.ts                            ✅ Part 2
│   ├── notify-slack.ts                          ✅ Part 7
│   ├── generate-test-plan.ts                    ✅ Part 8
│   └── heal-selectors.ts                        ✅ Part 8
├── utils/
│   ├── schema-validator.ts                      ✅ Part 4
│   ├── visual-helpers.ts                        ✅ Part 5
│   └── debug-helpers.ts                         ✅ Part 6
├── docker/                                      ✅ Part 7
│   ├── Dockerfile
│   └── docker-compose.yml
├── snapshots/                                   ✅ Part 5
├── .vscode/                                     ✅ Part 6
│   ├── extensions.json
│   └── launch.json
├── .auth/                                       ← git-ignored
│   ├── admin.json
│   └── user.json
├── .mcp.json                                    ✅ Part 8
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Parts 1–7 (final)
├── .gitignore                                   ✅ Part 7
├── .env                                         ← git-ignored
├── .env.example                                 ← committed
├── TEST_PLAN.md                                 ← generated by script
└── package.json

🗺️ The Complete Series — Done

Part 1 — Stop Writing Tests Like a Beginner              ✅ Done
Part 2 — Network Interception: The Complete Guide        ✅ Done
Part 3 — Multi-User, Multi-Tab & Context Testing         ✅ Done
Part 4 — API Testing (The Underrated Superpower)         ✅ Done
Part 5 — Visual Regression Testing                       ✅ Done
Part 6 — Debugging Like a Pro: Trace Viewer & Inspector  ✅ Done
Part 7 — The CI/CD Setup Nobody Shows You                ✅ Done
Part 8 — Playwright Meets AI: Agents, MCP & Self-Healing ✅ Done

🔖 What You've Built

Eight parts. One complete framework.

Look at what you now have:

The foundation (Parts 1–2)
Semantic locators. storageState. Page Object Model. Full network interception layer. HAR recording. API mocking.

The depth (Parts 3–4)
Multi-user simultaneous testing. Real-time collaboration validation. Raw API testing with typed clients. Schema validation. GraphQL testing. API-UI chaining.

The quality layer (Parts 5–6)
Visual regression across full pages, components, and four viewports. Trace Viewer integration. Debug helpers that catch console errors and audit network calls automatically. VS Code launch configs.

The pipeline (Part 7)
Sharded GitHub Actions workflow — 4x faster. Cross-browser matrix. Docker for consistent VRT. Slack notifications. Published HTML reports as artifacts.

The AI layer (Part 8)
Playwright MCP — giving AI assistants a real browser. Test coverage analysis. Self-healing selector engine. AI-generated test scaffolding with honest evaluation of where it works and where it doesn't.

This isn't a toy. This is a production-grade Playwright framework — the kind teams spend months building organically.

You built it in 8 parts. 🏆

What's Next

This series covered the framework. But a framework is only as good as the team using it.

If this series helped you — share it with your team. Point them to Part 1. Let them follow along from the beginning.

And if you want to go deeper on any specific part — drop a comment. The next series is already forming based on what you ask.

Thank you for following The Playwright Playbook from start to finish.

This is the series I wish had existed when I started testing AI systems. I hope it gave you what you needed to build something real.

Drop a comment below 👇

Which part was the most valuable for your current project?
What would you want to see in a follow-up series?
Did you build this framework alongside the series — or are you planning to?

It's been a great ride. Let's keep building. 🙌

Faizal Shaikh | Senior Automation Engineer | Playwright & AI Testing
Connect with me on LinkedIn

The Playwright Playbook — Part 7: The CI/CD Setup Nobody Shows You

Faizal — Sun, 21 Jun 2026 06:46:44 +0000

The Playwright Playbook — Part 7: The CI/CD Setup Nobody Shows You

"A test suite that only runs on your laptop isn't a test suite. It's a hobby."

Six parts in, we have a serious framework.

POM-based UI tests. Network interception. Multi-user contexts. A full API testing layer. Visual regression across four viewports. A complete debugging toolkit.

Now it needs to run automatically. On every pull request. On every merge. On every deployment. Without you touching it.

Most CI/CD tutorials for Playwright show you this:

# The "tutorial" version everyone copies
- run: npx playwright test

That's not a CI setup. That's a shell command in a YAML file.

A real production CI/CD pipeline for Playwright has:

Sharding — split tests across multiple machines and finish in a fraction of the time
Browser matrix — Chromium, Firefox, WebKit in parallel
Docker — identical environment on every machine, every time
Artifacts — HTML report, traces, screenshots, videos — downloadable from every run
Failure notifications — your team knows within seconds, not the next morning
Separate VRT workflow — visual regression on its own cadence, not blocking every PR
Environment-specific pipelines — staging vs production, different configurations

Let's build all of it. 🎯

🏗️ Where We Left Off

After Part 6, our full project structure is:

playwright-playbook/
├── tests/
│   ├── auth/login.spec.ts                       ✅ Part 1
│   ├── tasks/task-management.spec.ts            ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   ├── multi-user/                              ✅ Part 3
│   ├── multi-tab/                               ✅ Part 3
│   ├── api/                                     ✅ Part 4
│   ├── visual/                                  ✅ Part 5
│   └── debug/trace-examples.spec.ts             ✅ Part 6
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── api/
│   ├── TaskApiClient.ts                         ✅ Part 4
│   └── AuthApiClient.ts                         ✅ Part 4
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   ├── multi-user.fixture.ts                    ✅ Part 3
│   └── api.fixture.ts                           ✅ Part 4
├── scripts/
│   └── record-har.ts                            ✅ Part 2
├── utils/
│   ├── schema-validator.ts                      ✅ Part 4
│   ├── visual-helpers.ts                        ✅ Part 5
│   └── debug-helpers.ts                         ✅ Part 6
├── snapshots/                                   ✅ Part 5
├── .vscode/
│   ├── extensions.json                          ✅ Part 6
│   └── launch.json                              ✅ Part 6
├── .auth/
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Parts 1–6
└── .env

By the end of Part 7, we add:

playwright-playbook/
├── .github/
│   └── workflows/                               ← NEW
│       ├── playwright.yml
│       └── playwright-visual.yml
├── docker/                                      ← NEW
│   ├── Dockerfile
│   └── docker-compose.yml
├── scripts/
│   └── notify-slack.ts                          ← NEW
└── .gitignore                                   ← NEW (complete version)

Every file gets fully built below. 👇

🧠 The CI Architecture — Mental Model First

Before we write a single line of YAML, understand the architecture we're building:

On every Pull Request:
  ┌─────────────────────────────────────────────────────┐
  │  playwright.yml                                     │
  │                                                     │
  │  Shard 1 (machine 1): auth + tasks + network tests  │
  │  Shard 2 (machine 2): multi-user + multi-tab tests  │
  │  Shard 3 (machine 3): api tests                     │
  │  Shard 4 (machine 4): debug tests                   │
  │                                                     │
  │  All shards run in parallel → merge reports         │
  │  Upload: HTML report + traces + screenshots         │
  │  Notify Slack on failure                            │
  └─────────────────────────────────────────────────────┘

On merge to main (nightly for VRT):
  ┌─────────────────────────────────────────────────────┐
  │  playwright-visual.yml                              │
  │                                                     │
  │  Runs inside Docker (consistent rendering)          │
  │  Full visual regression suite                       │
  │  On failure: diff images uploaded as artifacts      │
  │  Requires manual approval to update baselines       │
  └─────────────────────────────────────────────────────┘

Two separate workflows. Different triggers. Different purposes. Clean separation. 🎯

⚙️ Updating `playwright.config.ts` — CI-Ready Final Version

First, the final version of our config — tuned for both local development and CI:

// playwright.config.ts — final version
import { defineConfig, devices } from '@playwright/test';
import * as dotenv from 'dotenv';

dotenv.config();

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,

  // Fail the build if test.only is accidentally committed
  forbidOnly: !!process.env.CI,

  // Retry once on CI — surfaces flaky tests without hiding them
  retries: process.env.CI ? 1 : 0,

  // Parallel workers — CI uses 4, local uses all available cores
  workers: process.env.CI ? 4 : undefined,

  // Reporters — list for CI console output, HTML for artifact
  reporter: process.env.CI
    ? [
        ['list'],
        ['html', { open: 'never', outputFolder: 'playwright-report' }],
        ['json', { outputFile: 'test-results/results.json' }],
        ['github'], // Annotates failing tests directly in the PR
      ]
    : [
        ['list'],
        ['html', { open: 'on-failure' }],
      ],

  expect: {
    toHaveScreenshot: {
      maxDiffPixelRatio: 0.002,
      timeout: 10000,
      animations: 'disabled',
    },
  },

  use: {
    baseURL: process.env.BASE_URL || 'http://localhost:3000',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    trace: process.env.CI ? 'on-first-retry' : 'on',
    extraHTTPHeaders: {
      'Accept': 'application/json',
      'Content-Type': 'application/json',
    },
  },

  projects: [
    {
      name: 'admin',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/admin.json',
      },
      testMatch: ['**/auth/**', '**/tasks/**', '**/network/**'],
    },
    {
      name: 'user',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/user.json',
      },
      testMatch: ['**/tasks/**'],
    },
    {
      name: 'multi-context',
      use: { ...devices['Desktop Chrome'] },
      testMatch: ['**/multi-user/**', '**/multi-tab/**'],
    },
    {
      name: 'api',
      use: {},
      testMatch: ['**/api/**'],
    },
    {
      name: 'visual',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/admin.json',
        viewport: { width: 1280, height: 720 },
        launchOptions: {
          args: ['--disable-gpu', '--force-device-scale-factor=1'],
        },
      },
      testMatch: ['**/visual/**'],
      snapshotDir: './snapshots',
    },
    {
      name: 'visual-responsive',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/admin.json',
      },
      testMatch: ['**/visual/responsive**'],
      snapshotDir: './snapshots/responsive',
    },
    // Cross-browser matrix — run on merge to main only
    {
      name: 'firefox',
      use: {
        ...devices['Desktop Firefox'],
        storageState: '.auth/admin.json',
      },
      testMatch: ['**/auth/**', '**/tasks/**'],
    },
    {
      name: 'webkit',
      use: {
        ...devices['Desktop Safari'],
        storageState: '.auth/admin.json',
      },
      testMatch: ['**/auth/**', '**/tasks/**'],
    },
    // Mobile browser projects
    {
      name: 'mobile-chrome',
      use: {
        ...devices['Pixel 7'],
        storageState: '.auth/user.json',
      },
      testMatch: ['**/tasks/**'],
    },
    {
      name: 'mobile-safari',
      use: {
        ...devices['iPhone 14'],
        storageState: '.auth/user.json',
      },
      testMatch: ['**/tasks/**'],
    },
  ],
});

🐳 Docker — Consistent Environments Everywhere

The single biggest source of VRT flakiness is rendering differences between machines. macOS renders fonts differently from Linux. One engineer's machine differs from another's. CI differs from both.

Docker solves this by running Playwright inside the official Microsoft Playwright image — the same image, everywhere.

# docker/Dockerfile
FROM mcr.microsoft.com/playwright:v1.47.0-jammy

WORKDIR /app

# Copy package files first — better Docker layer caching
COPY package*.json ./
RUN npm ci

# Copy the rest of the project
COPY . .

# Default command — run all tests
CMD ["npx", "playwright", "test"]

# docker/docker-compose.yml
version: '3.8'

services:
  playwright:
    build:
      context: ..
      dockerfile: docker/Dockerfile
    environment:
      - CI=true
      - BASE_URL=${BASE_URL:-http://app:3000}
      - ADMIN_EMAIL=${ADMIN_EMAIL}
      - ADMIN_PASSWORD=${ADMIN_PASSWORD}
      - USER_EMAIL=${USER_EMAIL}
      - USER_PASSWORD=${USER_PASSWORD}
    volumes:
      # Mount test-results and reports back to host for inspection
      - ../test-results:/app/test-results
      - ../playwright-report:/app/playwright-report
      - ../snapshots:/app/snapshots
    depends_on:
      - app
    networks:
      - playwright-network

  # Your app container — replace with your actual app image
  app:
    image: your-app:latest
    ports:
      - '3000:3000'
    environment:
      - NODE_ENV=test
      - DB_SEED=true
    networks:
      - playwright-network

networks:
  playwright-network:
    driver: bridge

Run locally with Docker:

# Build and run the full suite in Docker
docker-compose -f docker/docker-compose.yml up --build

# Run only visual tests in Docker
docker-compose -f docker/docker-compose.yml run playwright \
  npx playwright test --project=visual

# Update visual baselines inside Docker (so they match CI rendering)
docker-compose -f docker/docker-compose.yml run playwright \
  npx playwright test --project=visual --update-snapshots

🔔 Slack Notification Script

When tests fail in CI at 2am, your team should know immediately — not when they check GitHub the next morning.

// scripts/notify-slack.ts
import * as fs from 'fs';
import * as path from 'path';

interface TestResult {
  stats: {
    expected: number;
    unexpected: number;
    skipped: number;
    flaky: number;
    duration: number;
  };
  suites: Array<{
    title: string;
    specs: Array<{
      title: string;
      ok: boolean;
      tests: Array<{ results: Array<{ status: string; error?: { message: string } }> }>;
    }>;
  }>;
}

async function notifySlack(): Promise<void> {
  const webhookUrl = process.env.SLACK_WEBHOOK_URL;
  if (!webhookUrl) {
    console.log('No SLACK_WEBHOOK_URL set — skipping Slack notification.');
    return;
  }

  const resultsPath = path.join(process.cwd(), 'test-results', 'results.json');
  if (!fs.existsSync(resultsPath)) {
    console.error('results.json not found. Run tests first.');
    process.exit(1);
  }

  const results: TestResult = JSON.parse(fs.readFileSync(resultsPath, 'utf-8'));
  const { stats } = results;

  const passed = stats.expected;
  const failed = stats.unexpected;
  const flaky = stats.flaky;
  const skipped = stats.skipped;
  const duration = Math.round(stats.duration / 1000);

  // Only notify on failure
  if (failed === 0 && flaky === 0) {
    console.log(`✅ All ${passed} tests passed. No Slack notification needed.`);
    return;
  }

  // Collect failed test names
  const failedTests: string[] = [];
  for (const suite of results.suites) {
    for (const spec of suite.specs) {
      if (!spec.ok) {
        failedTests.push(`• ${suite.title} › ${spec.title}`);
      }
    }
  }

  const runUrl = process.env.GITHUB_RUN_URL ?? 'N/A';
  const branch = process.env.GITHUB_REF_NAME ?? 'unknown branch';
  const actor = process.env.GITHUB_ACTOR ?? 'unknown';

  const payload = {
    blocks: [
      {
        type: 'header',
        text: {
          type: 'plain_text',
          text: `❌ Playwright Tests Failed — ${branch}`,
        },
      },
      {
        type: 'section',
        fields: [
          { type: 'mrkdwn', text: `*Passed:*\n✅ ${passed}` },
          { type: 'mrkdwn', text: `*Failed:*\n❌ ${failed}` },
          { type: 'mrkdwn', text: `*Flaky:*\n⚠️ ${flaky}` },
          { type: 'mrkdwn', text: `*Duration:*\n⏱ ${duration}s` },
        ],
      },
      {
        type: 'section',
        text: {
          type: 'mrkdwn',
          text: `*Failed tests:*\n${failedTests.slice(0, 10).join('\n')}${
            failedTests.length > 10
              ? `\n_...and ${failedTests.length - 10} more_`
              : ''
          }`,
        },
      },
      {
        type: 'section',
        fields: [
          { type: 'mrkdwn', text: `*Triggered by:*\n${actor}` },
          { type: 'mrkdwn', text: `*Branch:*\n${branch}` },
        ],
      },
      {
        type: 'actions',
        elements: [
          {
            type: 'button',
            text: { type: 'plain_text', text: '🔍 View CI Run' },
            url: runUrl,
            style: 'danger',
          },
        ],
      },
    ],
  };

  const response = await fetch(webhookUrl, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(payload),
  });

  if (!response.ok) {
    console.error(`Slack notification failed: ${response.status}`);
    process.exit(1);
  }

  console.log('📢 Slack notification sent.');
}

notifySlack().catch(console.error);

🚀 The Main GitHub Actions Workflow

This is the full production workflow — sharded, parallelized, with artifacts and notifications.

# .github/workflows/playwright.yml
name: Playwright Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]
  # Allow manual trigger from GitHub UI
  workflow_dispatch:

env:
  # These come from GitHub repository secrets
  BASE_URL: ${{ secrets.BASE_URL }}
  ADMIN_EMAIL: ${{ secrets.ADMIN_EMAIL }}
  ADMIN_PASSWORD: ${{ secrets.ADMIN_PASSWORD }}
  USER_EMAIL: ${{ secrets.USER_EMAIL }}
  USER_PASSWORD: ${{ secrets.USER_PASSWORD }}

jobs:
  # ─────────────────────────────────────────
  # Job 1: Install and cache dependencies
  # ─────────────────────────────────────────
  install:
    name: Install Dependencies
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Cache Playwright browsers
        uses: actions/cache@v4
        id: playwright-cache
        with:
          path: ~/.cache/ms-playwright
          key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

      - name: Install Playwright browsers
        if: steps.playwright-cache.outputs.cache-hit != 'true'
        run: npx playwright install --with-deps

      # Cache node_modules for downstream jobs
      - name: Cache node_modules
        uses: actions/cache@v4
        with:
          path: node_modules
          key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

  # ─────────────────────────────────────────
  # Job 2: Run tests — sharded across 4 machines
  # ─────────────────────────────────────────
  test:
    name: Test (Shard ${{ matrix.shardIndex }}/${{ matrix.shardTotal }})
    runs-on: ubuntu-latest
    needs: install

    strategy:
      fail-fast: false  # Don't cancel other shards if one fails
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Restore node_modules
        uses: actions/cache@v4
        with:
          path: node_modules
          key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

      - name: Restore Playwright browsers
        uses: actions/cache@v4
        with:
          path: ~/.cache/ms-playwright
          key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

      - name: Install Playwright system dependencies
        run: npx playwright install-deps chromium

      - name: Run Playwright tests (Shard ${{ matrix.shardIndex }})
        run: |
          npx playwright test \
            --project=admin \
            --project=user \
            --project=multi-context \
            --project=api \
            --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
        env:
          CI: true

      # Upload blob report — merged in the next job
      - name: Upload blob report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shardIndex }}
          path: blob-report/
          retention-days: 3

  # ─────────────────────────────────────────
  # Job 3: Cross-browser tests — main branch only
  # ─────────────────────────────────────────
  cross-browser:
    name: Cross-Browser (${{ matrix.browser }})
    runs-on: ubuntu-latest
    needs: install
    if: github.ref == 'refs/heads/main'

    strategy:
      fail-fast: false
      matrix:
        browser: [firefox, webkit]

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Restore node_modules
        uses: actions/cache@v4
        with:
          path: node_modules
          key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

      - name: Install ${{ matrix.browser }}
        run: npx playwright install --with-deps ${{ matrix.browser }}

      - name: Run tests on ${{ matrix.browser }}
        run: npx playwright test --project=${{ matrix.browser }}
        env:
          CI: true

      - name: Upload blob report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.browser }}
          path: blob-report/
          retention-days: 3

  # ─────────────────────────────────────────
  # Job 4: Merge shard reports into one HTML report
  # ─────────────────────────────────────────
  merge-reports:
    name: Merge Reports & Publish
    runs-on: ubuntu-latest
    needs: [test]
    if: always()

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Restore node_modules
        uses: actions/cache@v4
        with:
          path: node_modules
          key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

      - name: Download all blob reports
        uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true

      - name: Merge into single HTML report
        run: |
          npx playwright merge-reports \
            --reporter html,json \
            ./all-blob-reports

      - name: Upload merged HTML report
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report-${{ github.run_id }}
          path: playwright-report/
          retention-days: 14

      - name: Upload test results JSON
        uses: actions/upload-artifact@v4
        with:
          name: test-results-json
          path: test-results/results.json
          retention-days: 14

  # ─────────────────────────────────────────
  # Job 5: Notify Slack on failure
  # ─────────────────────────────────────────
  notify:
    name: Slack Notification
    runs-on: ubuntu-latest
    needs: [test, merge-reports]
    if: failure()

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Restore node_modules
        uses: actions/cache@v4
        with:
          path: node_modules
          key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

      - name: Download test results
        uses: actions/download-artifact@v4
        with:
          name: test-results-json
          path: test-results/

      - name: Send Slack notification
        run: npx ts-node scripts/notify-slack.ts
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
          GITHUB_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
          GITHUB_REF_NAME: ${{ github.ref_name }}
          GITHUB_ACTOR: ${{ github.actor }}

🎨 The Visual Regression Workflow — Separate Pipeline

VRT runs on its own schedule — not on every PR (too slow), but on every merge to main and nightly.

# .github/workflows/playwright-visual.yml
name: Visual Regression Tests

on:
  push:
    branches: [main]
  schedule:
    # Run every night at 2am UTC
    - cron: '0 2 * * *'
  workflow_dispatch:
    inputs:
      update_snapshots:
        description: 'Update baseline snapshots?'
        required: false
        default: 'false'
        type: boolean

env:
  BASE_URL: ${{ secrets.BASE_URL }}
  ADMIN_EMAIL: ${{ secrets.ADMIN_EMAIL }}
  ADMIN_PASSWORD: ${{ secrets.ADMIN_PASSWORD }}
  USER_EMAIL: ${{ secrets.USER_EMAIL }}
  USER_PASSWORD: ${{ secrets.USER_PASSWORD }}

jobs:
  visual-regression:
    name: Visual Regression
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright Chromium
        run: npx playwright install --with-deps chromium

      - name: Run visual regression tests
        run: |
          npx playwright test \
            --project=visual \
            --project=visual-responsive \
            ${{ github.event.inputs.update_snapshots == 'true' && '--update-snapshots' || '' }}
        env:
          CI: true

      # If snapshots were updated, commit them back to the repo
      - name: Commit updated snapshots
        if: github.event.inputs.update_snapshots == 'true'
        run: |
          git config --global user.name 'github-actions[bot]'
          git config --global user.email 'github-actions[bot]@users.noreply.github.com'
          git add snapshots/
          git diff --staged --quiet || git commit -m "chore: update visual regression baselines [skip ci]"
          git push
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      # Upload diff images when VRT fails — so you can see exactly what changed
      - name: Upload visual diff artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: visual-diffs-${{ github.run_id }}
          path: |
            test-results/**/*-diff.png
            test-results/**/*-actual.png
            test-results/**/*-expected.png
          retention-days: 14

      - name: Upload VRT HTML report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: vrt-report-${{ github.run_id }}
          path: playwright-report/
          retention-days: 14

      - name: Notify Slack on VRT failure
        if: failure()
        run: npx ts-node scripts/notify-slack.ts
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
          GITHUB_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
          GITHUB_REF_NAME: ${{ github.ref_name }}
          GITHUB_ACTOR: ${{ github.actor }}

📋 The Complete `.gitignore`

After 7 parts, here's the complete gitignore for the whole project:

# .gitignore

# Dependencies
node_modules/

# Environment files — NEVER commit these
.env
.env.local
.env.*.local

# Authentication state — auto-generated by globalSetup
.auth/

# Playwright test artifacts — generated on each run
test-results/
playwright-report/
blob-report/

# Snapshots are committed — they ARE your baselines
# !snapshots/  ← DO NOT ignore snapshots

# TypeScript build output
dist/
build/

# OS files
.DS_Store
Thumbs.db

# IDE files
.idea/
*.swp
*.swo

# Logs
*.log
npm-debug.log*

Note what is NOT ignored: snapshots/. Visual regression baselines must be committed to the repo. They are your source of truth. ✅

🔐 GitHub Secrets — What to Configure

In your GitHub repository → Settings → Secrets and Variables → Actions, add:

Secret name          Value
──────────────────   ──────────────────────────────────────
BASE_URL             https://staging.yourtaskapp.com
ADMIN_EMAIL          admin@test.com
ADMIN_PASSWORD       your-admin-password
USER_EMAIL           user@test.com
USER_PASSWORD        your-user-password
SLACK_WEBHOOK_URL    https://hooks.slack.com/services/...

Never hardcode these in YAML. Never commit them. GitHub Secrets encrypt them at rest and mask them in logs automatically. 🔑

📊 Understanding Sharding — Why It Matters

Without sharding, 100 tests run sequentially on one machine. With 4 shards, each machine gets ~25 tests and they all run in parallel.

Without sharding (1 machine):
  100 tests × 3 seconds average = 5 minutes total

With sharding (4 machines):
  25 tests × 3 seconds average = ~75 seconds per shard
  All 4 run in parallel = ~75 seconds total

That's a 4x speed improvement on the same test suite.
Scale to 8 shards? 8x faster.

The --shard=N/M flag tells Playwright to run only the Nth chunk of M total chunks:

# Run shard 1 of 4
npx playwright test --shard=1/4

# Run shard 2 of 4
npx playwright test --shard=2/4

Playwright distributes tests evenly across shards automatically. No manual assignment needed. 🎯

📁 Final Project Structure After Part 7

Every file listed below has been fully built across Parts 1 through 7:

playwright-playbook/
├── .github/
│   └── workflows/                               ✅ Part 7
│       ├── playwright.yml
│       └── playwright-visual.yml
├── tests/
│   ├── auth/login.spec.ts                       ✅ Part 1
│   ├── tasks/task-management.spec.ts            ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   │   ├── api-mocking.spec.ts
│   │   ├── error-simulation.spec.ts
│   │   └── network-assertions.spec.ts
│   ├── multi-user/                              ✅ Part 3
│   │   ├── role-permissions.spec.ts
│   │   └── realtime-collaboration.spec.ts
│   ├── multi-tab/                               ✅ Part 3
│   │   └── multi-tab-flows.spec.ts
│   ├── api/                                     ✅ Part 4
│   │   ├── tasks-api.spec.ts
│   │   ├── auth-api.spec.ts
│   │   ├── graphql-api.spec.ts
│   │   └── api-ui-chain.spec.ts
│   ├── visual/                                  ✅ Part 5
│   │   ├── dashboard-visual.spec.ts
│   │   ├── task-visual.spec.ts
│   │   └── responsive-visual.spec.ts
│   └── debug/                                   ✅ Part 6
│       └── trace-examples.spec.ts
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── api/
│   ├── TaskApiClient.ts                         ✅ Part 4
│   └── AuthApiClient.ts                         ✅ Part 4
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   ├── multi-user.fixture.ts                    ✅ Part 3
│   └── api.fixture.ts                           ✅ Part 4
├── scripts/
│   ├── record-har.ts                            ✅ Part 2
│   └── notify-slack.ts                          ✅ Part 7
├── utils/
│   ├── schema-validator.ts                      ✅ Part 4
│   ├── visual-helpers.ts                        ✅ Part 5
│   └── debug-helpers.ts                         ✅ Part 6
├── docker/                                      ✅ Part 7
│   ├── Dockerfile
│   └── docker-compose.yml
├── snapshots/                                   ✅ Part 5
├── .vscode/                                     ✅ Part 6
│   ├── extensions.json
│   └── launch.json
├── .auth/                                       ← git-ignored
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Parts 1–7 (final version)
├── .gitignore                                   ✅ Part 7
├── .env                                         ← git-ignored
└── package.json

🗺️ What's Coming in This Series

Part 1 — Stop Writing Tests Like a Beginner              ✅ Done
Part 2 — Network Interception: The Complete Guide        ✅ Done
Part 3 — Multi-User, Multi-Tab & Context Testing         ✅ Done
Part 4 — API Testing (The Underrated Superpower)         ✅ Done
Part 5 — Visual Regression Testing                       ✅ Done
Part 6 — Debugging Like a Pro: Trace Viewer & Inspector  ✅ Done
Part 7 — The CI/CD Setup Nobody Shows You                ← You are here
Part 8 — Playwright Meets AI: Agents, MCP & Self-Healing Tests

In Part 8 — the series finale — we add AI on top of everything we've built. Playwright MCP, AI test agents, and self-healing selectors. The framework we've spent 7 parts building becomes the target for AI-powered test generation and auto-healing.

🔖 Before You Go

Seven parts in. The framework is complete.

And it now runs automatically — on every PR, on every merge, across multiple machines in parallel, with cross-browser coverage, Docker-consistent VRT, downloadable reports, and your team notified the moment something breaks.

That's not a test suite. That's a quality platform. 🏗️

One part left. And it's the one that ties your entire positioning together — Playwright meets AI.

Follow me so you don't miss Part 8 — the series finale where we add AI agents, MCP, and self-healing tests on top of the framework we've built across all 7 parts.

Drop a comment below 👇

What does your current CI/CD setup for tests look like?
Are you using sharding — or still running everything sequentially?
What's the first GitHub Secret you'd set up from this list?

Let's talk in the comments. 🙌

Faizal Shaikh | Senior Automation Engineer | Playwright & AI Testing
Connect with me on LinkedIn

The Playwright Playbook — Part 6: Debugging Like a Pro — Trace Viewer, Inspector & VS Code

Faizal — Sat, 20 Jun 2026 00:59:29 +0000

The Playwright Playbook — Part 6: Debugging Like a Pro — Trace Viewer, Inspector & VS Code

"A failing test at 2am in CI tells you what broke. Trace Viewer tells you why."

In Part 1, we wrote solid tests. In Parts 2 through 5, we built layer after layer — network interception, multi-user flows, API testing, visual regression.

Now something fails.

It always does.

The question isn't whether your tests will fail — it's how fast you can understand why.

Most engineers debug Playwright like this:

// 🔴 The classic debugging approach
console.log('got here 1');
await page.click(someLocator);
console.log('got here 2');

Or worse — they add waitForTimeout(5000) and hope the problem goes away.

There's a better way. Playwright ships with a debugging toolkit so powerful that once you start using it, you'll never go back to console.log again.

Trace Viewer. Playwright Inspector. VS Code integration. page.pause().

These tools show you every click, every network call, every DOM snapshot, every console error — in a visual timeline that reconstructs exactly what happened, step by step.

Let's build the debugging layer into our framework. 🎯

🏗️ Where We Left Off

After Part 5, our full project structure is:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts                        ✅ Part 1
│   ├── tasks/
│   │   └── task-management.spec.ts              ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   │   ├── api-mocking.spec.ts
│   │   ├── error-simulation.spec.ts
│   │   └── network-assertions.spec.ts
│   ├── multi-user/                              ✅ Part 3
│   │   ├── role-permissions.spec.ts
│   │   └── realtime-collaboration.spec.ts
│   ├── multi-tab/                               ✅ Part 3
│   │   └── multi-tab-flows.spec.ts
│   ├── api/                                     ✅ Part 4
│   │   ├── tasks-api.spec.ts
│   │   ├── auth-api.spec.ts
│   │   ├── graphql-api.spec.ts
│   │   └── api-ui-chain.spec.ts
│   └── visual/                                  ✅ Part 5
│       ├── dashboard-visual.spec.ts
│       ├── task-visual.spec.ts
│       └── responsive-visual.spec.ts
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── api/
│   ├── TaskApiClient.ts                         ✅ Part 4
│   └── AuthApiClient.ts                         ✅ Part 4
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   ├── multi-user.fixture.ts                    ✅ Part 3
│   └── api.fixture.ts                           ✅ Part 4
├── scripts/
│   └── record-har.ts                            ✅ Part 2
├── utils/
│   ├── schema-validator.ts                      ✅ Part 4
│   └── visual-helpers.ts                        ✅ Part 5
├── snapshots/                                   ✅ Part 5
├── .auth/
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Part 1 (updated Parts 3, 4 & 5)
└── .env

By the end of Part 6, we add:

playwright-playbook/
├── tests/
│   └── debug/                                   ← NEW
│       └── trace-examples.spec.ts
├── utils/
│   └── debug-helpers.ts                         ← NEW
├── .vscode/                                     ← NEW
│   ├── extensions.json
│   └── launch.json

Every file gets fully built below. 👇

🧠 The Debugging Toolkit — Overview

Playwright gives you five distinct debugging tools. Each is for a different situation:

Tool                  When to use
─────────────────     ──────────────────────────────────────────────
Trace Viewer          Post-mortem — understanding a CI failure
Playwright Inspector  Live — step through a test interactively
page.pause()          Breakpoint — pause mid-test, inspect live DOM
VS Code Extension     IDE — run/debug individual tests without CLI
Console + Network     During development — lightweight live logging

We'll build proper support for all five into the framework. Let's start with the most important one. 👇

🎬 Trace Viewer — The Most Powerful Debugging Tool in Playwright

Trace Viewer is a full timeline recording of your test — every action, every network request, every DOM snapshot, every console log — captured and viewable in a browser-based UI.

When a test fails in CI, you download the trace file and open it locally. You see exactly what the browser saw at every step. No guessing. No re-running the test hoping to reproduce it.

Step 1 — Configure traces in `playwright.config.ts`

// playwright.config.ts — updated use block
use: {
  baseURL: process.env.BASE_URL || 'http://localhost:3000',

  // Screenshot on every failure — attached to HTML report
  screenshot: 'only-on-failure',

  // Video on first retry — see exactly what happened
  video: 'retain-on-failure',

  // Trace on first retry in CI, always on locally for debugging
  trace: process.env.CI ? 'on-first-retry' : 'on',

  extraHTTPHeaders: {
    'Accept': 'application/json',
    'Content-Type': 'application/json',
  },
},

Trace modes explained:

'off'             — No traces recorded (fastest)
'on'              — Trace recorded for every test (slowest, most info)
'retain-on-failure' — Trace only kept when test fails (good balance)
'on-first-retry'  — Trace recorded on first retry (recommended for CI)

on-first-retry is the sweet spot for CI — it only records when something is already failing, so you don't pay the overhead on every green run. 🎯

Step 2 — Open a trace locally

After a test failure, Playwright saves a .zip trace file in test-results/. Open it:

# Open the trace viewer in your browser
npx playwright show-trace test-results/my-test/trace.zip

# Or open the full HTML report which embeds trace links
npx playwright show-report

Step 3 — What you see in Trace Viewer

The Trace Viewer UI has five panels:

┌─────────────────────────────────────────────────────────┐
│  Timeline ────────────────────────────────────────────  │
│  [action][action][action][network][action][FAIL]        │
├─────────────────┬───────────────────────────────────────┤
│                 │  Before snapshot    After snapshot     │
│  Action list    │  ┌─────────────┐  ┌─────────────────┐ │
│                 │  │             │  │                 │ │
│  ▶ goto /tasks  │  │   DOM before│  │   DOM after     │ │
│  ▶ click button │  │   the action│  │   the action    │ │
│  ▶ fill input   │  │             │  │                 │ │
│  ❌ expect fails│  └─────────────┘  └─────────────────┘ │
├─────────────────┴───────────────────────────────────────┤
│  Network  |  Console  |  Source                         │
└─────────────────────────────────────────────────────────┘

Every action shows:

The DOM before the action
The DOM after the action
All network calls that fired during the action
Console logs and errors at that moment
The exact line of test code that triggered it

This is how you debug a test that fails once in 50 runs in CI. You don't reproduce it — you read the trace. 🔥

🔍 Playwright Inspector — Interactive Step-Through Debugging

Inspector is for when you're writing a new test and want to figure out the right locator or understand why an action isn't working.

It opens a live browser alongside an inspector panel — you step through your test action by action, watching what happens in real time.

Launch modes

# Run a specific test in debug mode — opens Inspector
npx playwright test tests/tasks/task-management.spec.ts --debug

# Run in headed mode only — see the browser, no Inspector panel
npx playwright test --headed

# Run with slow motion — slows each action by 1 second (great for watching)
npx playwright test --headed --slow-mo=1000

# Debug a specific test by name
npx playwright test --debug -g "user can create a new task"

Using Inspector to find the right locator

This is the killer use case. When you're not sure what locator to use:

Run your test with --debug
Inspector opens — click Record or Explore
Hover over elements in the browser — Inspector shows you the recommended locator
Click the element — Inspector generates the getByRole, getByLabel, or getByTestId code
Copy it into your test

No more guessing at selectors. No more page.$('div > div:nth-child(2) > span'). 🎯

The Inspector toolbar

▶  Resume     — Run to the next breakpoint or end of test
⏭  Step over  — Execute the current action and pause on the next
🔍 Pick       — Hover over elements to see their locators
⏺  Record    — Record new actions by clicking in the browser

⏸️ `page.pause()` — Your Interactive Breakpoint

page.pause() is the equivalent of a debugger breakpoint inside your test. The test runs normally until it hits page.pause() — then it freezes and opens Inspector.

test('debugging a flaky task creation flow', async ({ page }) => {
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  // Test runs normally up to here
  await taskPage.newTaskButton.click();

  // ⏸️ Test freezes here — Inspector opens
  // You can now interact with the page, inspect the DOM,
  // check network calls, try locators — all live
  await page.pause();

  // When you click Resume in Inspector, the test continues
  await taskPage.taskTitleInput.fill('Debugging this task');
  await taskPage.saveTaskButton.click();
});

Important: Remove page.pause() before committing. If it runs in CI with no terminal, the test will hang indefinitely.

A safer pattern — only pause in local development:

// Only pause when running locally — never in CI
if (!process.env.CI) {
  await page.pause();
}

🛠️ Building the Debug Helpers Utility

These helpers centralise common debugging patterns — collecting console errors, logging performance timing, and capturing network activity during a test.

// utils/debug-helpers.ts
import { Page, ConsoleMessage } from '@playwright/test';

export interface ConsoleError {
  type: string;
  text: string;
  location: string;
}

export interface NetworkCall {
  method: string;
  url: string;
  status: number;
  duration: number;
}

/**
 * Attach a console error collector to the page.
 * Returns a function to get all collected errors at any point.
 *
 * Usage:
 *   const getErrors = collectConsoleErrors(page);
 *   // ... run your test ...
 *   const errors = getErrors();
 *   expect(errors).toHaveLength(0);
 */
export function collectConsoleErrors(page: Page): () => ConsoleError[] {
  const errors: ConsoleError[] = [];

  page.on('console', (msg: ConsoleMessage) => {
    if (msg.type() === 'error') {
      errors.push({
        type: msg.type(),
        text: msg.text(),
        location: msg.location().url,
      });
    }
  });

  // Also capture uncaught page errors (runtime JS errors)
  page.on('pageerror', (error: Error) => {
    errors.push({
      type: 'pageerror',
      text: error.message,
      location: error.stack ?? '',
    });
  });

  return () => [...errors];
}

/**
 * Attach a network call logger to the page.
 * Returns a function to get all captured network calls.
 *
 * Usage:
 *   const getCalls = collectNetworkCalls(page, '/api/');
 *   // ... run your test ...
 *   const calls = getCalls();
 *   console.log(calls); // see every API call made
 */
export function collectNetworkCalls(
  page: Page,
  urlFilter: string = ''
): () => NetworkCall[] {
  const calls: NetworkCall[] = [];

  page.on('requestfinished', async request => {
    if (urlFilter && !request.url().includes(urlFilter)) return;

    const response = await request.response();
    const timing = request.timing();

    calls.push({
      method: request.method(),
      url: request.url(),
      status: response?.status() ?? 0,
      duration: timing.responseEnd - timing.requestStart,
    });
  });

  return () => [...calls];
}

/**
 * Log all page console output during a test.
 * Useful during development — remove before committing.
 */
export function attachConsoleLogger(page: Page): void {
  page.on('console', msg => {
    const type = msg.type().padEnd(7);
    console.log(`[PAGE ${type}] ${msg.text()}`);
  });

  page.on('pageerror', error => {
    console.error(`[PAGE ERROR] ${error.message}`);
  });
}

/**
 * Capture a named checkpoint screenshot during a test.
 * Useful for documenting test steps without full VRT.
 *
 * Usage:
 *   await checkpoint(page, 'after-task-created');
 */
export async function checkpoint(page: Page, name: string): Promise<void> {
  const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
  await page.screenshot({
    path: `test-results/checkpoints/${name}-${timestamp}.png`,
    fullPage: false,
  });
  console.log(`📸 Checkpoint: ${name}`);
}

/**
 * Assert no console errors occurred during the test.
 * Add this at the end of any test to catch silent JS errors.
 */
export function assertNoConsoleErrors(getErrors: () => ConsoleError[]): void {
  const errors = getErrors();
  if (errors.length > 0) {
    const errorMessages = errors.map(e => `  [${e.type}] ${e.text}`).join('\n');
    throw new Error(`Console errors detected during test:\n${errorMessages}`);
  }
}

/**
 * Log a summary of all network calls made during the test.
 * Helpful for understanding what API traffic a user action triggers.
 */
export function logNetworkSummary(getCalls: () => NetworkCall[]): void {
  const calls = getCalls();
  if (calls.length === 0) {
    console.log('No API network calls recorded.');
    return;
  }

  console.log('\n📡 Network calls during test:');
  calls.forEach(call => {
    const status = call.status >= 400 ? `❌ ${call.status}` : `✅ ${call.status}`;
    console.log(`  ${status} ${call.method.padEnd(6)} ${call.url} (${Math.round(call.duration)}ms)`);
  });
}

🧪 Trace Examples — Tests That Demonstrate the Debug Toolkit

These tests serve a dual purpose: they're real tests AND they demonstrate how to use each debugging tool in a real scenario.

// tests/debug/trace-examples.spec.ts
import { test, expect } from '@playwright/test';
import { TaskPage } from '../../pages/TaskPage';
import { DashboardPage } from '../../pages/DashboardPage';
import {
  collectConsoleErrors,
  collectNetworkCalls,
  assertNoConsoleErrors,
  logNetworkSummary,
  checkpoint,
} from '../../utils/debug-helpers';

test.describe('Debug Helpers — Console Error Detection', () => {
  test('task page loads with no console errors', async ({ page }) => {
    // Attach error collector BEFORE navigating
    const getErrors = collectConsoleErrors(page);

    const taskPage = new TaskPage(page);
    await taskPage.goto();

    // Wait for page to fully settle
    await page.waitForLoadState('networkidle');

    // Assert no JS errors occurred during page load
    assertNoConsoleErrors(getErrors);
  });

  test('dashboard loads with no console errors', async ({ page }) => {
    const getErrors = collectConsoleErrors(page);

    const dashboard = new DashboardPage(page);
    await dashboard.goto();
    await page.waitForLoadState('networkidle');

    assertNoConsoleErrors(getErrors);
  });

  test('task creation flow produces no console errors', async ({ page }) => {
    const getErrors = collectConsoleErrors(page);

    const taskPage = new TaskPage(page);
    await taskPage.goto();
    await taskPage.createTask('Console error test task');

    await expect(taskPage.getTaskLocator('Console error test task')).toBeVisible();

    // Clean up
    await taskPage.deleteTask('Console error test task');

    // Assert no errors occurred during the entire flow
    assertNoConsoleErrors(getErrors);
  });
});

test.describe('Debug Helpers — Network Call Auditing', () => {
  test('task page load triggers expected API calls', async ({ page }) => {
    const getCalls = collectNetworkCalls(page, '/api/');

    const taskPage = new TaskPage(page);
    await taskPage.goto();
    await page.waitForLoadState('networkidle');

    const calls = getCalls();
    logNetworkSummary(getCalls);

    // Assert the expected API calls were made
    const tasksFetch = calls.find(
      c => c.url.includes('/api/tasks') && c.method === 'GET'
    );
    expect(tasksFetch).toBeDefined();
    expect(tasksFetch!.status).toBe(200);

    // Assert no unexpected 4xx or 5xx calls
    const failedCalls = calls.filter(c => c.status >= 400);
    expect(failedCalls).toHaveLength(0);
  });

  test('task creation triggers exactly one POST call', async ({ page }) => {
    const taskPage = new TaskPage(page);
    await taskPage.goto();

    // Start collecting AFTER initial page load — only capture creation calls
    const getCalls = collectNetworkCalls(page, '/api/tasks');

    await taskPage.createTask('Network audit test task');
    await expect(taskPage.getTaskLocator('Network audit test task')).toBeVisible();

    const calls = getCalls();
    const postCalls = calls.filter(c => c.method === 'POST');

    // Should be exactly one POST — not two, not zero
    expect(postCalls).toHaveLength(1);
    expect(postCalls[0].status).toBe(201);

    // Cleanup
    await taskPage.deleteTask('Network audit test task');
  });
});

test.describe('Debug Helpers — Checkpoint Screenshots', () => {
  test('document task creation flow with checkpoints', async ({ page }) => {
    const taskPage = new TaskPage(page);
    await taskPage.goto();

    // Checkpoint 1 — initial state
    await checkpoint(page, '01-task-page-initial');

    // Open new task modal
    await taskPage.newTaskButton.click();
    await checkpoint(page, '02-new-task-modal-open');

    // Fill in the task
    await taskPage.taskTitleInput.fill('Checkpoint documented task');
    await checkpoint(page, '03-task-title-filled');

    // Save
    await taskPage.saveTaskButton.click();
    await expect(taskPage.getTaskLocator('Checkpoint documented task')).toBeVisible();
    await checkpoint(page, '04-task-created-success');

    // Cleanup
    await taskPage.deleteTask('Checkpoint documented task');
    await checkpoint(page, '05-task-deleted');

    // Checkpoint PNGs are saved to test-results/checkpoints/
    // Useful for documenting a flow or building a test report with screenshots
  });
});

test.describe('Flaky Test Patterns — How to Debug Them', () => {
  test('wait for real condition — not arbitrary timeout', async ({ page }) => {
    const taskPage = new TaskPage(page);
    await taskPage.goto();

    // ✅ Correct pattern — wait for the network response AND the action together
    const [response] = await Promise.all([
      page.waitForResponse(
        resp => resp.url().includes('/api/tasks') && resp.request().method() === 'POST'
      ),
      taskPage.createTask('Condition-based wait task'),
    ]);

    // Assert on the actual condition that confirms completion
    expect(response.status()).toBe(201);
    await expect(taskPage.getTaskLocator('Condition-based wait task')).toBeVisible();

    await taskPage.deleteTask('Condition-based wait task');
  });

  test('identify what a "random" failure actually looks like', async ({ page }) => {
    const getErrors = collectConsoleErrors(page);
    const getCalls = collectNetworkCalls(page, '/api/');

    const taskPage = new TaskPage(page);
    await taskPage.goto();

    // Simulate a slow/unreliable API response
    await page.route('**/api/tasks', async route => {
      // Add a random delay between 0 and 500ms — simulates real-world jitter
      const delay = Math.random() * 500;
      await new Promise(resolve => setTimeout(resolve, delay));
      await route.continue();
    });

    await taskPage.createTask('Jitter test task');

    // This should STILL pass — because we're waiting on conditions, not timeouts
    await expect(taskPage.getTaskLocator('Jitter test task')).toBeVisible();

    // Log what actually happened
    logNetworkSummary(getCalls);
    assertNoConsoleErrors(getErrors);

    await taskPage.deleteTask('Jitter test task');
  });
});

💻 VS Code Integration — Run & Debug Tests From Your IDE

If you use VS Code, the official Playwright extension is the fastest way to run and debug individual tests without touching the terminal.

Step 1 — Install the extension

// .vscode/extensions.json
{
  "recommendations": [
    "ms-playwright.playwright",
    "dbaeumer.vscode-eslint",
    "esbenp.prettier-vscode"
  ]
}

Commit this file — when teammates open the repo, VS Code prompts them to install the recommended extensions automatically.

Step 2 — VS Code debug launch config

// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug Playwright Test",
      "type": "node",
      "request": "launch",
      "program": "${workspaceFolder}/node_modules/.bin/playwright",
      "args": ["test", "--debug", "${file}"],
      "cwd": "${workspaceFolder}",
      "env": {
        "PWDEBUG": "1"
      },
      "console": "integratedTerminal"
    },
    {
      "name": "Debug Current Test File",
      "type": "node",
      "request": "launch",
      "program": "${workspaceFolder}/node_modules/.bin/playwright",
      "args": ["test", "${file}", "--headed", "--project=admin"],
      "cwd": "${workspaceFolder}",
      "env": {},
      "console": "integratedTerminal"
    },
    {
      "name": "Debug Specific Test by Name",
      "type": "node",
      "request": "launch",
      "program": "${workspaceFolder}/node_modules/.bin/playwright",
      "args": [
        "test",
        "--debug",
        "--grep",
        "${input:testName}"
      ],
      "cwd": "${workspaceFolder}",
      "console": "integratedTerminal"
    }
  ],
  "inputs": [
    {
      "id": "testName",
      "type": "promptString",
      "description": "Enter the test name or partial match"
    }
  ]
}

What the VS Code extension gives you

Once installed and configured, you get:

Testing sidebar (beaker icon):
├── See all tests in a tree view
├── ▶ Run individual test with one click
├── 🐛 Debug individual test with one click
├── 📍 Set breakpoints in test code
└── 🔴 Failing tests highlighted inline in the editor

Editor gutter:
├── ▶ Run this test (appears next to each test() block)
└── 🐛 Debug this test

You can click the ▶ icon next to any test() in your editor and it runs immediately — no terminal, no command to remember. 🎯

📋 Reading CI Failures Like a Pro

When a test fails in CI, you get three artifacts attached to the pipeline run:

test-results/
├── my-failing-test/
│   ├── trace.zip           ← Full trace — every action, network call, DOM snapshot
│   ├── test-failed-1.png   ← Screenshot at the moment of failure
│   └── video.webm          ← Video of the entire test run

The debugging workflow for CI failures

# Step 1 — Download trace.zip from CI artifacts

# Step 2 — Open it locally
npx playwright show-trace path/to/trace.zip

# Step 3 — In Trace Viewer:
#   Click through the action timeline to find where it went wrong
#   Check the "Before" vs "After" DOM snapshots
#   Look at the network tab — was there a 500? A missing response?
#   Check the console tab — was there a JS error?

# Step 4 — Reproduce locally if needed
npx playwright test --debug -g "the failing test name"

Reading the trace timeline — what to look for

Common failure patterns and where to find them in the trace:

Element not found:
  → Action shows "waiting for locator" → times out
  → Check "Before" snapshot — is the element there? Is the selector wrong?
  → Check network tab — did the API call that renders it fail?

Wrong element clicked:
  → "Before" snapshot shows multiple matching elements
  → Your locator was too broad — make it more specific

Flaky timing failure:
  → Network tab shows slow response (800ms+)
  → Action fired before the data loaded
  → Fix: wait for the response, not a timeout

Authentication failure:
  → First network call returns 401
  → storageState expired or wasn't loaded
  → Fix: check global-setup.ts ran correctly

Console error on failure:
  → Console tab shows red JS error
  → Often unrelated to your test — but worth checking
  → Add collectConsoleErrors() to catch these proactively

🔧 Useful Debug Commands — Quick Reference

# Run a single test file in debug mode
npx playwright test tests/tasks/task-management.spec.ts --debug

# Run a test by name pattern
npx playwright test --debug -g "user can create a new task"

# Run headed (see the browser) without Inspector
npx playwright test --headed

# Run with slow motion — 1 second between each action
npx playwright test --headed --slow-mo=1000

# Generate and open the HTML report after a run
npx playwright test && npx playwright show-report

# Open a specific trace file
npx playwright show-trace test-results/my-test/trace.zip

# Update visual snapshots after intentional UI change
npx playwright test --project=visual --update-snapshots

# Run only tests matching a tag
npx playwright test --grep @smoke

# List all tests without running them
npx playwright test --list

# Run with maximum verbosity
npx playwright test --reporter=list --verbose

📁 Final Project Structure After Part 6

Every file listed below has been fully built across Parts 1 through 6:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts                        ✅ Part 1
│   ├── tasks/
│   │   └── task-management.spec.ts              ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   │   ├── api-mocking.spec.ts
│   │   ├── error-simulation.spec.ts
│   │   └── network-assertions.spec.ts
│   ├── multi-user/                              ✅ Part 3
│   │   ├── role-permissions.spec.ts
│   │   └── realtime-collaboration.spec.ts
│   ├── multi-tab/                               ✅ Part 3
│   │   └── multi-tab-flows.spec.ts
│   ├── api/                                     ✅ Part 4
│   │   ├── tasks-api.spec.ts
│   │   ├── auth-api.spec.ts
│   │   ├── graphql-api.spec.ts
│   │   └── api-ui-chain.spec.ts
│   ├── visual/                                  ✅ Part 5
│   │   ├── dashboard-visual.spec.ts
│   │   ├── task-visual.spec.ts
│   │   └── responsive-visual.spec.ts
│   └── debug/                                   ✅ Part 6
│       └── trace-examples.spec.ts
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── api/
│   ├── TaskApiClient.ts                         ✅ Part 4
│   └── AuthApiClient.ts                         ✅ Part 4
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   ├── multi-user.fixture.ts                    ✅ Part 3
│   └── api.fixture.ts                           ✅ Part 4
├── scripts/
│   └── record-har.ts                            ✅ Part 2
├── utils/
│   ├── schema-validator.ts                      ✅ Part 4
│   ├── visual-helpers.ts                        ✅ Part 5
│   └── debug-helpers.ts                         ✅ Part 6
├── snapshots/                                   ✅ Part 5
├── .vscode/                                     ✅ Part 6
│   ├── extensions.json
│   └── launch.json
├── .auth/                                       ← git-ignored
│   ├── admin.json
│   └── user.json
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Part 1 (updated Parts 3, 4, 5 & 6)
├── .env                                         ← git-ignored
└── package.json

🗺️ What's Coming in This Series

Part 1 — Stop Writing Tests Like a Beginner              ✅ Done
Part 2 — Network Interception: The Complete Guide        ✅ Done
Part 3 — Multi-User, Multi-Tab & Context Testing         ✅ Done
Part 4 — API Testing (The Underrated Superpower)         ✅ Done
Part 5 — Visual Regression Testing                       ✅ Done
Part 6 — Debugging Like a Pro: Trace Viewer & Inspector  ← You are here
Part 7 — The CI/CD Setup Nobody Shows You
Part 8 — Playwright Meets AI: Agents, MCP & Self-Healing Tests

In Part 7, we take everything we've built and put it in a real CI/CD pipeline — GitHub Actions, test sharding across multiple machines, browser matrix, Docker for consistent environments, Slack failure notifications, and the HTML report published as a pipeline artifact.

🔖 Before You Go

Six parts in. The framework is deep.

And now when something breaks — which it will — you're not flying blind.

You have Trace Viewer to reconstruct exactly what happened. Inspector to step through a test live. page.pause() for interactive breakpoints. VS Code integration to debug without leaving your editor. And debug-helpers.ts to collect console errors and network calls automatically.

The best QA engineers aren't the ones who write tests that never fail. They're the ones who understand failures the fastest. 🎯

Follow me so you don't miss Part 7 — where we take this entire framework to CI/CD. Sharding, GitHub Actions, Docker, browser matrix, failure notifications — the setup that makes your suite production-ready.

Drop a comment below 👇

What's your current debugging workflow when a Playwright test fails?
Have you used Trace Viewer before — or is that new?
What's the most mysterious flaky test you've ever had to debug?

Let's talk in the comments. 🙌

Faizal Shaikh | Senior Automation Engineer | Playwright & AI Testing
Connect with me on LinkedIn

The Playwright Playbook — Part 5: Visual Regression Testing

Faizal — Fri, 19 Jun 2026 04:40:31 +0000

The Playwright Playbook — Part 5: Visual Regression Testing

"Your assertions say the button exists. Visual regression says it's not buried under the header anymore."

In Part 1, we built the foundation. Part 2 owned the network. Part 3 handled multiple users. Part 4 replaced Postman with a full API testing layer.

Every single assertion we've written so far has one thing in common.

They all test behaviour — what the app does. Not appearance — what the user actually sees.

And that's where bugs hide.

The button is there. toBeVisible() passes. But a CSS regression pushed it behind the navigation bar. The user can't click it. Your test doesn't know.

The task list renders. The count is correct. But a font-size change made the text overflow and truncate. Half the title is hidden. Your test doesn't know.

The dashboard loads. All assertions pass. But the mobile layout is completely broken. Your test doesn't know — because you never checked a mobile viewport.

Visual regression testing catches all of this. And Playwright has it built in — no third-party tool required.

Let's add it to the framework. 🎯

🏗️ Where We Left Off

After Part 4, our full project structure is:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts                        ✅ Part 1
│   ├── tasks/
│   │   └── task-management.spec.ts              ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   │   ├── api-mocking.spec.ts
│   │   ├── error-simulation.spec.ts
│   │   └── network-assertions.spec.ts
│   ├── multi-user/                              ✅ Part 3
│   │   ├── role-permissions.spec.ts
│   │   └── realtime-collaboration.spec.ts
│   ├── multi-tab/                               ✅ Part 3
│   │   └── multi-tab-flows.spec.ts
│   └── api/                                     ✅ Part 4
│       ├── tasks-api.spec.ts
│       ├── auth-api.spec.ts
│       ├── graphql-api.spec.ts
│       └── api-ui-chain.spec.ts
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── api/                                         ✅ Part 4
│   ├── TaskApiClient.ts
│   └── AuthApiClient.ts
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   ├── multi-user.fixture.ts                    ✅ Part 3
│   └── api.fixture.ts                           ✅ Part 4
├── scripts/
│   └── record-har.ts                            ✅ Part 2
├── utils/
│   └── schema-validator.ts                      ✅ Part 4
├── .auth/
│   ├── admin.json
│   └── user.json
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Part 1 (updated Parts 3 & 4)
└── .env

By the end of Part 5, we add:

playwright-playbook/
├── tests/
│   └── visual/                                  ← NEW
│       ├── dashboard-visual.spec.ts
│       ├── task-visual.spec.ts
│       └── responsive-visual.spec.ts
├── utils/
│   └── visual-helpers.ts                        ← NEW
├── snapshots/                                   ← NEW (auto-generated, committed to git)
│   └── [baseline PNG files live here]

Every file gets fully built below. 👇

🧠 How Playwright Visual Regression Works — The Mental Model

Before we write code, understand what's happening under the hood.

First run (no baseline exists):
  test runs → screenshot taken → saved as baseline PNG → test PASSES

Subsequent runs:
  test runs → screenshot taken → compared pixel-by-pixel to baseline
  ├── Match within threshold → ✅ PASS
  └── Diff exceeds threshold → ❌ FAIL + diff image generated

When UI intentionally changes:
  run with --update-snapshots flag → baseline replaced → commit new baseline

The baseline PNGs live in your repo alongside your tests. Every developer, every CI run, compares against the same baseline. That's what makes it reliable.

Three things that make or break VRT in practice:

Masking — hide dynamic content (timestamps, avatars, ads) that changes every run
Threshold — how many pixels can differ before it's a failure
Consistent environment — fonts, OS rendering, and viewport must be identical across runs

We'll handle all three. 🎯

⚙️ Updating `playwright.config.ts` for Visual Testing

Visual regression needs its own project config — consistent viewport, single browser (Chromium for VRT), and snapshot settings.

// playwright.config.ts — full updated version
import { defineConfig, devices } from '@playwright/test';
import * as dotenv from 'dotenv';

dotenv.config();

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 4 : undefined,
  globalSetup: './global-setup.ts',

  reporter: [
    ['html', { open: 'never' }],
    ['list'],
  ],

  // Snapshot configuration — applies to all toHaveScreenshot calls
  expect: {
    toHaveScreenshot: {
      // Allow up to 0.2% of pixels to differ — handles anti-aliasing differences
      maxDiffPixelRatio: 0.002,
      // How long to wait for the screenshot to stabilize
      timeout: 10000,
      // Animations must be disabled for consistent screenshots
      animations: 'disabled',
    },
  },

  use: {
    baseURL: process.env.BASE_URL || 'http://localhost:3000',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    trace: 'on-first-retry',
    extraHTTPHeaders: {
      'Accept': 'application/json',
      'Content-Type': 'application/json',
    },
  },

  projects: [
    {
      name: 'admin',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/admin.json',
      },
      testMatch: ['**/auth/**', '**/tasks/**', '**/network/**'],
    },
    {
      name: 'user',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/user.json',
      },
      testMatch: ['**/tasks/**'],
    },
    {
      name: 'multi-context',
      use: { ...devices['Desktop Chrome'] },
      testMatch: ['**/multi-user/**', '**/multi-tab/**'],
    },
    {
      name: 'api',
      use: {},
      testMatch: ['**/api/**'],
    },
    {
      // Visual regression — Chromium only, fixed viewport, admin auth
      // Single browser keeps baselines consistent across machines
      name: 'visual',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/admin.json',
        // Fixed viewport — VRT breaks if viewport varies
        viewport: { width: 1280, height: 720 },
        // Disable animations — they cause pixel differences mid-screenshot
        launchOptions: {
          args: ['--disable-gpu', '--force-device-scale-factor=1'],
        },
      },
      testMatch: ['**/visual/**'],
      // Store snapshots next to the test files
      snapshotDir: './snapshots',
    },
    {
      // Responsive VRT — separate project, different viewports per test
      name: 'visual-responsive',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/admin.json',
      },
      testMatch: ['**/visual/responsive**'],
      snapshotDir: './snapshots/responsive',
    },
  ],
});

🛠️ Building the Visual Helpers Utility

Masking and screenshot config comes up in every VRT test. Instead of repeating it, we centralise it.

// utils/visual-helpers.ts
import { Page, Locator } from '@playwright/test';

/**
 * Elements that change every run — timestamps, avatars, live counters.
 * Pass these to toHaveScreenshot({ mask: [...] }) to hide them from comparison.
 */
export function getDynamicMasks(page: Page): Locator[] {
  return [
    page.getByTestId('timestamp'),           // "Created 2 mins ago"
    page.getByTestId('user-avatar'),         // Profile photos
    page.getByTestId('notification-count'),  // Live badge counts
    page.getByTestId('last-updated'),        // "Last updated: 14:32:01"
    page.getByTestId('session-timer'),       // Session countdown
  ];
}

/**
 * Standard screenshot options used across all full-page VRT tests.
 * Disable animations, set threshold, mask dynamic elements.
 */
export function fullPageScreenshotOptions(page: Page) {
  return {
    fullPage: true,
    mask: getDynamicMasks(page),
    maxDiffPixelRatio: 0.002,
    animations: 'disabled' as const,
  };
}

/**
 * Component-level screenshot options — for isolated widget/card snapshots.
 * Tighter threshold since we're testing a smaller, more controlled area.
 */
export function componentScreenshotOptions(page: Page) {
  return {
    mask: getDynamicMasks(page),
    maxDiffPixelRatio: 0.001,
    animations: 'disabled' as const,
  };
}

/**
 * Wait for all images on the page to fully load before screenshotting.
 * Prevents half-loaded images causing false failures.
 */
export async function waitForImagesLoaded(page: Page): Promise<void> {
  await page.waitForFunction(() => {
    const images = Array.from(document.images);
    return images.every(img => img.complete && img.naturalHeight !== 0);
  });
}

/**
 * Wait for all network activity to settle before screenshotting.
 * Prevents loading spinners appearing in snapshots.
 */
export async function waitForPageStable(page: Page): Promise<void> {
  await page.waitForLoadState('networkidle');
  await waitForImagesLoaded(page);
  // Small buffer for CSS transitions to complete
  await page.waitForTimeout(300);
}

📸 Dashboard Visual Tests — Full Page & Component Level

// tests/visual/dashboard-visual.spec.ts
import { test, expect } from '@playwright/test';
import { DashboardPage } from '../../pages/DashboardPage';
import { TaskApiClient } from '../../api/TaskApiClient';
import { AuthApiClient } from '../../api/AuthApiClient';
import {
  fullPageScreenshotOptions,
  componentScreenshotOptions,
  waitForPageStable,
} from '../../utils/visual-helpers';

test.describe('Dashboard — Full Page Visual', () => {
  test('dashboard full page snapshot — default state', async ({ page, request }) => {
    // Seed known data via API so the screenshot is always consistent
    const authApi = new AuthApiClient(request);
    const token = await authApi.getAdminToken();
    const taskApi = new TaskApiClient(request, token);

    // Create exactly 3 known tasks for a stable snapshot
    const seeded = await Promise.all([
      taskApi.createTask({ title: 'Write unit tests', status: 'pending' }),
      taskApi.createTask({ title: 'Review pull request', status: 'completed' }),
      taskApi.createTask({ title: 'Fix flaky test in CI', status: 'pending' }),
    ]);

    const dashboard = new DashboardPage(page);
    await dashboard.goto();
    await waitForPageStable(page);

    // Full page snapshot — first run creates baseline, subsequent runs compare
    await expect(page).toHaveScreenshot(
      'dashboard-default.png',
      fullPageScreenshotOptions(page)
    );

    // Cleanup
    await Promise.all(seeded.map(({ task }) => taskApi.deleteTask(task.id)));
  });

  test('dashboard snapshot — empty state (no tasks)', async ({ page, request }) => {
    // Ensure there are no tasks for a clean empty state screenshot
    const authApi = new AuthApiClient(request);
    const token = await authApi.getAdminToken();
    const taskApi = new TaskApiClient(request, token);

    const allTasks = await taskApi.getAllTasks();
    await Promise.all(allTasks.map(t => taskApi.deleteTask(t.id)));

    const dashboard = new DashboardPage(page);
    await dashboard.goto();
    await waitForPageStable(page);

    await expect(page).toHaveScreenshot(
      'dashboard-empty.png',
      fullPageScreenshotOptions(page)
    );
  });
});

test.describe('Dashboard — Component Level Visual', () => {
  test('task list component snapshot', async ({ page }) => {
    const dashboard = new DashboardPage(page);
    await dashboard.goto();
    await waitForPageStable(page);

    // Screenshot just the task list — not the whole page
    await expect(dashboard.taskList).toHaveScreenshot(
      'task-list-component.png',
      componentScreenshotOptions(page)
    );
  });

  test('notification panel snapshot — when open', async ({ page }) => {
    const dashboard = new DashboardPage(page);
    await dashboard.goto();
    await dashboard.openNotifications();

    // Wait for panel animation to settle
    await page.waitForTimeout(300);

    await expect(dashboard.notificationPanel).toHaveScreenshot(
      'notification-panel-open.png',
      componentScreenshotOptions(page)
    );
  });

  test('admin panel component snapshot', async ({ page }) => {
    const dashboard = new DashboardPage(page);
    await dashboard.goto();
    await waitForPageStable(page);

    await expect(dashboard.adminPanel).toHaveScreenshot(
      'admin-panel-component.png',
      componentScreenshotOptions(page)
    );
  });
});

test.describe('Dashboard — State Change Visual', () => {
  test('task status changes are reflected visually', async ({ page, request }) => {
    const authApi = new AuthApiClient(request);
    const token = await authApi.getAdminToken();
    const taskApi = new TaskApiClient(request, token);

    const { task } = await taskApi.createTask({
      title: 'Task for visual status test',
      status: 'pending',
    });

    const dashboard = new DashboardPage(page);
    await dashboard.goto();
    await waitForPageStable(page);

    // Screenshot in pending state
    const taskLocator = dashboard.getTaskLocator('Task for visual status test');
    await expect(taskLocator).toHaveScreenshot(
      'task-item-pending.png',
      componentScreenshotOptions(page)
    );

    // Mark as complete via API
    await taskApi.updateTask(task.id, { status: 'completed' });
    await page.reload();
    await waitForPageStable(page);

    // Screenshot in completed state — should show visual difference (strikethrough, green badge etc.)
    await expect(taskLocator).toHaveScreenshot(
      'task-item-completed.png',
      componentScreenshotOptions(page)
    );

    // Cleanup
    await taskApi.deleteTask(task.id);
  });
});

🎨 Task Page Visual Tests — States & Error Handling

// tests/visual/task-visual.spec.ts
import { test, expect } from '@playwright/test';
import { TaskPage } from '../../pages/TaskPage';
import { TaskApiClient } from '../../api/TaskApiClient';
import { AuthApiClient } from '../../api/AuthApiClient';
import {
  fullPageScreenshotOptions,
  componentScreenshotOptions,
  waitForPageStable,
} from '../../utils/visual-helpers';

test.describe('Task Page — Visual States', () => {
  test('task list page — with data', async ({ page, request }) => {
    // Seed consistent data via API
    const authApi = new AuthApiClient(request);
    const token = await authApi.getAdminToken();
    const taskApi = new TaskApiClient(request, token);

    const seeded = await Promise.all([
      taskApi.createTask({ title: 'Design new feature', status: 'pending' }),
      taskApi.createTask({ title: 'Write tests for login', status: 'in_progress' }),
      taskApi.createTask({ title: 'Deploy to staging', status: 'completed' }),
    ]);

    const taskPage = new TaskPage(page);
    await taskPage.goto();
    await waitForPageStable(page);

    await expect(page).toHaveScreenshot(
      'task-page-with-data.png',
      fullPageScreenshotOptions(page)
    );

    await Promise.all(seeded.map(({ task }) => taskApi.deleteTask(task.id)));
  });

  test('task page — empty state visual', async ({ page }) => {
    // Use network mock to force empty state cleanly
    await page.route('**/api/tasks', async route => {
      await route.fulfill({
        status: 200,
        contentType: 'application/json',
        body: '[]',
      });
    });

    const taskPage = new TaskPage(page);
    await taskPage.goto();
    await waitForPageStable(page);

    await expect(page).toHaveScreenshot(
      'task-page-empty.png',
      fullPageScreenshotOptions(page)
    );
  });

  test('task page — error state visual (API 500)', async ({ page }) => {
    // Use network mock to force error state
    await page.route('**/api/tasks', async route => {
      await route.fulfill({
        status: 500,
        contentType: 'application/json',
        body: JSON.stringify({ error: 'Internal Server Error' }),
      });
    });

    const taskPage = new TaskPage(page);
    await taskPage.goto();
    await waitForPageStable(page);

    await expect(page).toHaveScreenshot(
      'task-page-error.png',
      fullPageScreenshotOptions(page)
    );
  });

  test('task creation modal — open state', async ({ page }) => {
    const taskPage = new TaskPage(page);
    await taskPage.goto();
    await waitForPageStable(page);

    // Open the new task modal
    await taskPage.newTaskButton.click();
    await page.waitForTimeout(300); // wait for open animation

    // Screenshot the modal — checks layout, spacing, form fields
    await expect(page.getByTestId('task-modal')).toHaveScreenshot(
      'task-creation-modal.png',
      componentScreenshotOptions(page)
    );
  });

  test('task item — hover state visual', async ({ page, request }) => {
    const authApi = new AuthApiClient(request);
    const token = await authApi.getAdminToken();
    const taskApi = new TaskApiClient(request, token);

    const { task } = await taskApi.createTask({ title: 'Hover state test task' });

    const taskPage = new TaskPage(page);
    await taskPage.goto();
    await waitForPageStable(page);

    // Hover to reveal action buttons — screenshot the hover state
    const taskItem = taskPage.getTaskLocator('Hover state test task');
    await taskItem.hover();

    await expect(taskItem).toHaveScreenshot(
      'task-item-hover.png',
      componentScreenshotOptions(page)
    );

    await taskApi.deleteTask(task.id);
  });
});

📱 Responsive Visual Tests — Mobile, Tablet, Desktop

This is the VRT layer most teams skip entirely. A layout that looks perfect on desktop can be completely broken on mobile. toBeVisible() will never tell you that.

// tests/visual/responsive-visual.spec.ts
import { test, expect, devices } from '@playwright/test';
import { DashboardPage } from '../../pages/DashboardPage';
import { TaskPage } from '../../pages/TaskPage';
import {
  fullPageScreenshotOptions,
  waitForPageStable,
} from '../../utils/visual-helpers';

// Define the viewports we care about
const viewports = [
  { name: 'mobile', width: 375, height: 812 },    // iPhone 14
  { name: 'tablet', width: 768, height: 1024 },   // iPad
  { name: 'desktop', width: 1280, height: 720 },  // Standard desktop
  { name: 'wide', width: 1920, height: 1080 },    // Large monitor
];

test.describe('Dashboard — Responsive Visual', () => {
  for (const viewport of viewports) {
    test(`dashboard layout — ${viewport.name} (${viewport.width}x${viewport.height})`, async ({
      page,
    }) => {
      await page.setViewportSize({
        width: viewport.width,
        height: viewport.height,
      });

      const dashboard = new DashboardPage(page);
      await dashboard.goto();
      await waitForPageStable(page);

      await expect(page).toHaveScreenshot(
        `dashboard-${viewport.name}.png`,
        fullPageScreenshotOptions(page)
      );
    });
  }
});

test.describe('Task Page — Responsive Visual', () => {
  for (const viewport of viewports) {
    test(`task list layout — ${viewport.name} (${viewport.width}x${viewport.height})`, async ({
      page,
    }) => {
      // Seed stable data via mock so screenshots are consistent
      await page.route('**/api/tasks', async route => {
        await route.fulfill({
          status: 200,
          contentType: 'application/json',
          body: JSON.stringify([
            { id: 1, title: 'Design new feature', status: 'pending', assignee: 'admin' },
            { id: 2, title: 'Write tests for login', status: 'in_progress', assignee: 'user' },
            { id: 3, title: 'Deploy to staging', status: 'completed', assignee: 'admin' },
          ]),
        });
      });

      await page.setViewportSize({
        width: viewport.width,
        height: viewport.height,
      });

      const taskPage = new TaskPage(page);
      await taskPage.goto();
      await waitForPageStable(page);

      await expect(page).toHaveScreenshot(
        `task-page-${viewport.name}.png`,
        fullPageScreenshotOptions(page)
      );
    });
  }
});

test.describe('Login Page — Responsive Visual', () => {
  // Login page is special — clear storageState so we see the actual login form
  test.use({ storageState: { cookies: [], origins: [] } });

  for (const viewport of viewports) {
    test(`login page layout — ${viewport.name}`, async ({ page }) => {
      await page.setViewportSize({
        width: viewport.width,
        height: viewport.height,
      });

      await page.goto('/login');
      await waitForPageStable(page);

      await expect(page).toHaveScreenshot(
        `login-page-${viewport.name}.png`,
        fullPageScreenshotOptions(page)
      );
    });
  }
});

🔄 The Update Workflow — When UI Changes Are Intentional

When a developer intentionally changes the UI — a redesign, a new component, a colour update — your VRT tests will fail. That's correct behaviour.

The workflow:

# Step 1 — Run VRT to see what changed
npx playwright test --project=visual

# Step 2 — Open the HTML report to review the diffs visually
npx playwright show-report

# Step 3 — If the change is intentional, update the baselines
npx playwright test --project=visual --update-snapshots

# Step 4 — Commit the new baseline PNGs alongside your code change
git add snapshots/
git commit -m "chore: update visual baselines for new dashboard redesign"

This is the critical discipline: baseline updates must be a conscious decision, reviewed before committing. Never update baselines automatically in CI without human review. That defeats the entire purpose. ⚠️

🤔 When to Use Component vs Full Page Screenshots

This comes up constantly. Here's the decision framework:

Full page screenshot — use when:
  ├── Testing overall page layout and composition
  ├── Testing how components relate to each other spatially
  ├── Testing page-level responsive behaviour
  └── First-time coverage of a new page

Component screenshot — use when:
  ├── Testing a specific widget in isolation (modal, card, dropdown)
  ├── Testing state changes within a component (hover, active, error)
  ├── The page has a lot of dynamic content you can't fully mask
  └── You want faster, more targeted failure messages

In practice — start with full page, move to component when you find your full page tests are too noisy to maintain. 🎯

🚫 Common VRT Mistakes — And How We Avoid Them

❌ Screenshotting before the page is stable

// 🔴 Bad — loading spinner might still be visible
await page.goto('/dashboard');
await expect(page).toHaveScreenshot('dashboard.png');

// ✅ Good — use waitForPageStable from our utils
await page.goto('/dashboard');
await waitForPageStable(page);
await expect(page).toHaveScreenshot('dashboard.png', fullPageScreenshotOptions(page));

❌ Not masking dynamic content

// 🔴 Bad — timestamp changes every run → test always fails
await expect(page).toHaveScreenshot('dashboard.png');

// ✅ Good — masks hide the dynamic elements
await expect(page).toHaveScreenshot('dashboard.png', {
  mask: getDynamicMasks(page),
});

❌ Running VRT across different OS/browsers

Screenshots render differently on macOS vs Linux vs Windows. A baseline taken on your MacBook will fail on a Linux CI runner — every single time.

// ✅ Good — lock VRT to one browser and use Docker in CI
// playwright.config.ts visual project already uses Desktop Chrome only
// In CI, run Playwright inside the official Docker image:
// mcr.microsoft.com/playwright:v1.44.0-jammy

❌ Setting threshold too high

// 🔴 Bad — 10% pixel difference allowed = real bugs slip through
maxDiffPixelRatio: 0.1

// ✅ Good — tight threshold, mask dynamic content instead of widening tolerance
maxDiffPixelRatio: 0.002

📁 Final Project Structure After Part 5

Every file listed below has been fully built across Parts 1 through 5:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts                        ✅ Part 1
│   ├── tasks/
│   │   └── task-management.spec.ts              ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   │   ├── api-mocking.spec.ts
│   │   ├── error-simulation.spec.ts
│   │   └── network-assertions.spec.ts
│   ├── multi-user/                              ✅ Part 3
│   │   ├── role-permissions.spec.ts
│   │   └── realtime-collaboration.spec.ts
│   ├── multi-tab/                               ✅ Part 3
│   │   └── multi-tab-flows.spec.ts
│   ├── api/                                     ✅ Part 4
│   │   ├── tasks-api.spec.ts
│   │   ├── auth-api.spec.ts
│   │   ├── graphql-api.spec.ts
│   │   └── api-ui-chain.spec.ts
│   └── visual/                                  ✅ Part 5
│       ├── dashboard-visual.spec.ts
│       ├── task-visual.spec.ts
│       └── responsive-visual.spec.ts
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── api/                                         ✅ Part 4
│   ├── TaskApiClient.ts
│   └── AuthApiClient.ts
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   ├── multi-user.fixture.ts                    ✅ Part 3
│   └── api.fixture.ts                           ✅ Part 4
├── scripts/
│   └── record-har.ts                            ✅ Part 2
├── utils/
│   ├── schema-validator.ts                      ✅ Part 4
│   └── visual-helpers.ts                        ✅ Part 5
├── snapshots/                                   ✅ Part 5 (committed to git)
│   ├── dashboard-default.png
│   ├── dashboard-empty.png
│   ├── task-list-component.png
│   ├── task-page-with-data.png
│   ├── task-page-empty.png
│   ├── task-page-error.png
│   └── responsive/
│       ├── dashboard-mobile.png
│       ├── dashboard-tablet.png
│       ├── dashboard-desktop.png
│       ├── dashboard-wide.png
│       └── [more responsive baselines...]
├── .auth/                                       ← git-ignored
│   ├── admin.json
│   └── user.json
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Part 1 (updated Parts 3, 4 & 5)
├── .env                                         ← git-ignored
└── package.json

🗺️ What's Coming in This Series

Part 1 — Stop Writing Tests Like a Beginner              ✅ Done
Part 2 — Network Interception: The Complete Guide        ✅ Done
Part 3 — Multi-User, Multi-Tab & Context Testing         ✅ Done
Part 4 — API Testing (The Underrated Superpower)         ✅ Done
Part 5 — Visual Regression Testing                       ← You are here
Part 6 — Debugging Like a Pro: Trace Viewer & Inspector
Part 7 — The CI/CD Setup Nobody Shows You
Part 8 — Playwright Meets AI: Agents, MCP & Self-Healing Tests

In Part 6, we switch from writing tests to understanding what went wrong when they fail. Playwright's Trace Viewer, Inspector, VS Code integration, and page.pause() — the debugging toolkit that replaces console.log entirely.

🔖 Before You Go

Five parts in, and look at what you've built:

A POM-based UI layer with proper auth management
Full network interception and API mocking
Multi-user and multi-tab test coverage
A typed API testing layer with schema validation
Visual regression across full pages, components, and four viewports

Every assertion type. Every test layer. One framework. One tool. One pipeline.

The bugs that slipped through before? They have nowhere to hide now. 🔥

Follow me so you don't miss Part 6 — where we go deep on debugging. When a test fails at 2am in CI, you need to know exactly what happened. Trace Viewer will show you every click, every network call, every DOM snapshot — in a timeline.

Drop a comment below 👇

Are you currently doing any visual regression testing?
What tool are you using — Percy, Chromatic, or something else?
Did the responsive VRT loop pattern click for you?

Let's talk in the comments. 🙌

Faizal Shaikh | Senior Automation Engineer | Playwright & AI Testing
Connect with me on LinkedIn

The Playwright Playbook — Part 4: API Testing — The Underrated Superpower

Faizal — Thu, 18 Jun 2026 02:11:02 +0000

The Playwright Playbook — Part 4: API Testing — The Underrated Superpower

"The best UI test is one that doesn't need the UI at all."

In Part 1, we built the foundation. In Part 2, we intercepted the network. In Part 3, we ran multiple users simultaneously.

Now we go one layer deeper — below the browser entirely.

Playwright's API request context.

Most engineers reach for Postman when they need to test an API. Or they write a separate pytest/Jest suite just for API tests. Separate tool, separate pipeline, separate maintenance burden.

Here's what they're missing: Playwright can make raw HTTP requests without a browser. Same tool. Same TypeScript. Same test runner. Same CI pipeline.

And when you combine API calls with UI assertions in a single test — that's where the real power shows up.

No browser overhead for setup. No flaky UI login flows for data seeding. Just fast, direct, precise API calls — chained into the UI tests you're already writing.

By the end of this part, you'll have a complete API testing layer that lives right inside your Playwright project. Let's build it. 🎯

🏗️ Where We Left Off

After Part 3, our full project structure is:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts                        ✅ Part 1
│   ├── tasks/
│   │   └── task-management.spec.ts              ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   │   ├── api-mocking.spec.ts
│   │   ├── error-simulation.spec.ts
│   │   └── network-assertions.spec.ts
│   ├── multi-user/                              ✅ Part 3
│   │   ├── role-permissions.spec.ts
│   │   └── realtime-collaboration.spec.ts
│   └── multi-tab/                              ✅ Part 3
│       └── multi-tab-flows.spec.ts
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   └── multi-user.fixture.ts                    ✅ Part 3
├── scripts/
│   └── record-har.ts                            ✅ Part 2
├── .auth/
│   ├── admin.json
│   └── user.json
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Part 1 (updated Part 3)
└── .env

By the end of Part 4, we add:

playwright-playbook/
├── tests/
│   └── api/                                     ← NEW
│       ├── tasks-api.spec.ts
│       ├── auth-api.spec.ts
│       ├── graphql-api.spec.ts
│       └── api-ui-chain.spec.ts
├── api/                                         ← NEW
│   ├── TaskApiClient.ts
│   └── AuthApiClient.ts
├── fixtures/
│   └── api.fixture.ts                           ← NEW
└── utils/
    └── schema-validator.ts                      ← NEW

Every one of these files gets fully built below. Let's go. 👇

🧠 The Mental Model — Three Ways to Use the Request Context

Before we write code, understand the three modes Playwright gives you for API testing:

Mode 1 — request fixture (isolated, no browser cookies)
  └── Pure API tests — no UI involved at all

Mode 2 — page.request (shares browser context cookies)
  └── API calls that need the same auth as the current page

Mode 3 — APIRequestContext from playwright.config.ts
  └── Shared request context across your whole suite (baseURL, headers)

We'll use all three — each for the right job. Let's build from the bottom up.

⚙️ Configuring the API Base in `playwright.config.ts`

First, add API-specific config so all our API clients share the same base settings:

// playwright.config.ts — updated
import { defineConfig, devices } from '@playwright/test';
import * as dotenv from 'dotenv';

dotenv.config();

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 4 : undefined,
  globalSetup: './global-setup.ts',

  reporter: [
    ['html', { open: 'never' }],
    ['list'],
  ],

  use: {
    baseURL: process.env.BASE_URL || 'http://localhost:3000',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    trace: 'on-first-retry',
    // Default headers for ALL API requests across the suite
    extraHTTPHeaders: {
      'Accept': 'application/json',
      'Content-Type': 'application/json',
    },
  },

  projects: [
    {
      name: 'admin',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/admin.json',
      },
      testMatch: ['**/auth/**', '**/tasks/**', '**/network/**'],
    },
    {
      name: 'user',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/user.json',
      },
      testMatch: ['**/tasks/**'],
    },
    {
      name: 'multi-context',
      use: { ...devices['Desktop Chrome'] },
      testMatch: ['**/multi-user/**', '**/multi-tab/**'],
    },
    {
      // API tests — no browser needed, no storageState
      name: 'api',
      use: {},
      testMatch: ['**/api/**'],
    },
  ],
});

The api project has no devices — no browser spins up at all for pure API tests. Fastest possible execution. ✅

🔑 Building the Auth API Client

Our Task Manager uses JWT tokens. Before we can call protected API endpoints, we need a way to get tokens programmatically — without touching the browser.

// api/AuthApiClient.ts
import { APIRequestContext } from '@playwright/test';

export interface AuthTokens {
  accessToken: string;
  refreshToken: string;
  expiresIn: number;
}

export interface UserCredentials {
  email: string;
  password: string;
}

export class AuthApiClient {
  constructor(private readonly request: APIRequestContext) {}

  async login(credentials: UserCredentials): Promise<AuthTokens> {
    const response = await this.request.post('/api/auth/login', {
      data: credentials,
    });

    if (!response.ok()) {
      throw new Error(
        `Login failed: ${response.status()} ${await response.text()}`
      );
    }

    return response.json() as Promise<AuthTokens>;
  }

  async refreshToken(refreshToken: string): Promise<AuthTokens> {
    const response = await this.request.post('/api/auth/refresh', {
      data: { refreshToken },
    });

    if (!response.ok()) {
      throw new Error(`Token refresh failed: ${response.status()}`);
    }

    return response.json() as Promise<AuthTokens>;
  }

  async logout(accessToken: string): Promise<void> {
    await this.request.post('/api/auth/logout', {
      headers: { Authorization: `Bearer ${accessToken}` },
    });
  }

  async getAdminToken(): Promise<string> {
    const tokens = await this.login({
      email: process.env.ADMIN_EMAIL!,
      password: process.env.ADMIN_PASSWORD!,
    });
    return tokens.accessToken;
  }

  async getUserToken(): Promise<string> {
    const tokens = await this.login({
      email: process.env.USER_EMAIL!,
      password: process.env.USER_PASSWORD!,
    });
    return tokens.accessToken;
  }
}

📋 Building the Task API Client

Now the main API client — wrapping all task-related endpoints with typed methods.

// api/TaskApiClient.ts
import { APIRequestContext, APIResponse } from '@playwright/test';

export interface Task {
  id: number;
  title: string;
  status: 'pending' | 'in_progress' | 'completed';
  assignee: string;
  createdAt: string;
  updatedAt: string;
}

export interface CreateTaskPayload {
  title: string;
  status?: Task['status'];
  assignee?: string;
}

export interface UpdateTaskPayload {
  title?: string;
  status?: Task['status'];
  assignee?: string;
}

export class TaskApiClient {
  constructor(
    private readonly request: APIRequestContext,
    private readonly token: string
  ) {}

  private get authHeaders() {
    return { Authorization: `Bearer ${this.token}` };
  }

  async getAllTasks(): Promise<Task[]> {
    const response = await this.request.get('/api/tasks', {
      headers: this.authHeaders,
    });
    return response.json();
  }

  async getTask(id: number): Promise<Task> {
    const response = await this.request.get(`/api/tasks/${id}`, {
      headers: this.authHeaders,
    });
    return response.json();
  }

  async createTask(payload: CreateTaskPayload): Promise<{ response: APIResponse; task: Task }> {
    const response = await this.request.post('/api/tasks', {
      headers: this.authHeaders,
      data: payload,
    });
    const task = await response.json();
    return { response, task };
  }

  async updateTask(id: number, payload: UpdateTaskPayload): Promise<{ response: APIResponse; task: Task }> {
    const response = await this.request.patch(`/api/tasks/${id}`, {
      headers: this.authHeaders,
      data: payload,
    });
    const task = await response.json();
    return { response, task };
  }

  async deleteTask(id: number): Promise<APIResponse> {
    return this.request.delete(`/api/tasks/${id}`, {
      headers: this.authHeaders,
    });
  }

  async getTasksByStatus(status: Task['status']): Promise<Task[]> {
    const response = await this.request.get(`/api/tasks?status=${status}`, {
      headers: this.authHeaders,
    });
    return response.json();
  }
}

Fully typed. Every method returns the raw APIResponse where needed (so tests can assert on status codes) AND the parsed body. No hunting through raw responses in test files. 🎯

🧩 Building the API Fixture

Just like Part 1's auth.fixture.ts made POM setup invisible in tests, our api.fixture.ts makes API client setup invisible.

// fixtures/api.fixture.ts
import { test as base, APIRequestContext } from '@playwright/test';
import { TaskApiClient } from '../api/TaskApiClient';
import { AuthApiClient } from '../api/AuthApiClient';

type ApiFixtures = {
  authApi: AuthApiClient;
  adminTaskApi: TaskApiClient;
  userTaskApi: TaskApiClient;
};

export const test = base.extend<ApiFixtures>({
  // Auth API client — unauthenticated, used for login/logout tests
  authApi: async ({ request }, use) => {
    await use(new AuthApiClient(request));
  },

  // Task API client — authenticated as admin
  adminTaskApi: async ({ request }, use) => {
    const authApi = new AuthApiClient(request);
    const token = await authApi.getAdminToken();
    await use(new TaskApiClient(request, token));
  },

  // Task API client — authenticated as regular user
  userTaskApi: async ({ request }, use) => {
    const authApi = new AuthApiClient(request);
    const token = await authApi.getUserToken();
    await use(new TaskApiClient(request, token));
  },
});

export { expect } from '@playwright/test';

🛡️ Building the Schema Validator

One of the most common API testing gaps: everyone checks status codes, almost nobody validates response structure. A 200 with a broken body is still a broken API.

// utils/schema-validator.ts

export interface TaskSchema {
  id: number;
  title: string;
  status: string;
  assignee: string;
  createdAt: string;
  updatedAt: string;
}

export function validateTaskSchema(task: unknown): task is TaskSchema {
  if (typeof task !== 'object' || task === null) return false;

  const t = task as Record<string, unknown>;

  return (
    typeof t.id === 'number' &&
    typeof t.title === 'string' &&
    t.title.length > 0 &&
    ['pending', 'in_progress', 'completed'].includes(t.status as string) &&
    typeof t.assignee === 'string' &&
    typeof t.createdAt === 'string' &&
    typeof t.updatedAt === 'string' &&
    !isNaN(Date.parse(t.createdAt as string)) &&
    !isNaN(Date.parse(t.updatedAt as string))
  );
}

export function validateTaskListSchema(tasks: unknown): tasks is TaskSchema[] {
  return Array.isArray(tasks) && tasks.every(validateTaskSchema);
}

// Generic field presence checker — useful for partial response validation
export function assertRequiredFields(
  obj: Record<string, unknown>,
  fields: string[]
): void {
  const missing = fields.filter(field => !(field in obj) || obj[field] === undefined);
  if (missing.length > 0) {
    throw new Error(`Missing required fields in response: ${missing.join(', ')}`);
  }
}

✅ Task API Tests — CRUD, Status Codes & Schema Validation

// tests/api/tasks-api.spec.ts
import { test, expect } from '../../fixtures/api.fixture';
import { validateTaskSchema, validateTaskListSchema } from '../../utils/schema-validator';

test.describe('Tasks API — GET', () => {
  test('GET /api/tasks returns 200 and valid task list schema', async ({ adminTaskApi }) => {
    const tasks = await adminTaskApi.getAllTasks();

    expect(Array.isArray(tasks)).toBe(true);
    expect(validateTaskListSchema(tasks)).toBe(true);
  });

  test('GET /api/tasks/:id returns correct task', async ({ adminTaskApi }) => {
    // Create a task first so we have a known ID
    const { task: created } = await adminTaskApi.createTask({
      title: 'Schema validation test task',
    });

    const fetched = await adminTaskApi.getTask(created.id);

    expect(fetched.id).toBe(created.id);
    expect(fetched.title).toBe('Schema validation test task');
    expect(validateTaskSchema(fetched)).toBe(true);

    // Cleanup
    await adminTaskApi.deleteTask(created.id);
  });

  test('GET /api/tasks/:id returns 404 for non-existent task', async ({ request }) => {
    // Use raw request here since TaskApiClient throws on non-ok responses
    const response = await request.get('/api/tasks/999999', {
      headers: { Authorization: `Bearer ${process.env.ADMIN_TOKEN}` },
    });
    expect(response.status()).toBe(404);

    const body = await response.json();
    expect(body).toHaveProperty('error');
  });

  test('GET /api/tasks filters by status correctly', async ({ adminTaskApi }) => {
    const completedTasks = await adminTaskApi.getTasksByStatus('completed');

    expect(Array.isArray(completedTasks)).toBe(true);
    completedTasks.forEach(task => {
      expect(task.status).toBe('completed');
    });
  });
});

test.describe('Tasks API — POST', () => {
  test('POST /api/tasks creates task and returns 201', async ({ adminTaskApi }) => {
    const { response, task } = await adminTaskApi.createTask({
      title: 'New API test task',
      status: 'pending',
    });

    // Status code
    expect(response.status()).toBe(201);

    // Response body
    expect(task.title).toBe('New API test task');
    expect(task.status).toBe('pending');
    expect(task.id).toBeDefined();
    expect(typeof task.id).toBe('number');

    // Full schema
    expect(validateTaskSchema(task)).toBe(true);

    // Cleanup
    await adminTaskApi.deleteTask(task.id);
  });

  test('POST /api/tasks with missing title returns 400', async ({ request }) => {
    const response = await request.post('/api/tasks', {
      headers: {
        Authorization: `Bearer ${process.env.ADMIN_TOKEN}`,
        'Content-Type': 'application/json',
      },
      data: { status: 'pending' }, // no title
    });

    expect(response.status()).toBe(400);
    const body = await response.json();
    expect(body.error).toContain('title');
  });

  test('POST /api/tasks without auth returns 401', async ({ request }) => {
    const response = await request.post('/api/tasks', {
      data: { title: 'Unauthorized task' },
      // No Authorization header
    });

    expect(response.status()).toBe(401);
  });
});

test.describe('Tasks API — PATCH', () => {
  test('PATCH /api/tasks/:id updates task correctly', async ({ adminTaskApi }) => {
    // Create
    const { task: created } = await adminTaskApi.createTask({
      title: 'Task to update',
      status: 'pending',
    });

    // Update
    const { response, task: updated } = await adminTaskApi.updateTask(created.id, {
      status: 'completed',
      title: 'Updated task title',
    });

    expect(response.status()).toBe(200);
    expect(updated.status).toBe('completed');
    expect(updated.title).toBe('Updated task title');
    expect(updated.id).toBe(created.id); // ID should never change
    expect(updated.updatedAt).not.toBe(created.updatedAt); // timestamp should change

    // Cleanup
    await adminTaskApi.deleteTask(created.id);
  });

  test('regular user cannot update another user\'s task', async ({
    adminTaskApi,
    userTaskApi,
  }) => {
    // Admin creates a task
    const { task } = await adminTaskApi.createTask({
      title: 'Admin task — user should not update',
    });

    // User tries to update it
    const { response } = await userTaskApi.updateTask(task.id, {
      title: 'Hijacked title',
    });

    expect(response.status()).toBe(403);

    // Cleanup
    await adminTaskApi.deleteTask(task.id);
  });
});

test.describe('Tasks API — DELETE', () => {
  test('DELETE /api/tasks/:id returns 200 and task no longer exists', async ({
    adminTaskApi,
  }) => {
    const { task } = await adminTaskApi.createTask({ title: 'Task to delete via API' });

    const deleteResponse = await adminTaskApi.deleteTask(task.id);
    expect(deleteResponse.status()).toBe(200);

    // Try to fetch the deleted task — should be 404
    const tasks = await adminTaskApi.getAllTasks();
    const stillExists = tasks.find(t => t.id === task.id);
    expect(stillExists).toBeUndefined();
  });

  test('regular user cannot delete admin task — returns 403', async ({
    adminTaskApi,
    userTaskApi,
  }) => {
    const { task } = await adminTaskApi.createTask({ title: 'Protected admin task' });

    const deleteResponse = await userTaskApi.deleteTask(task.id);
    expect(deleteResponse.status()).toBe(403);

    // Cleanup as admin
    await adminTaskApi.deleteTask(task.id);
  });
});

🔐 Auth API Tests — Token Flows & Security

// tests/api/auth-api.spec.ts
import { test, expect } from '../../fixtures/api.fixture';

test('successful login returns valid token structure', async ({ authApi }) => {
  const tokens = await authApi.login({
    email: process.env.USER_EMAIL!,
    password: process.env.USER_PASSWORD!,
  });

  expect(typeof tokens.accessToken).toBe('string');
  expect(tokens.accessToken.length).toBeGreaterThan(0);
  expect(typeof tokens.refreshToken).toBe('string');
  expect(typeof tokens.expiresIn).toBe('number');
  expect(tokens.expiresIn).toBeGreaterThan(0);
});

test('invalid credentials return 401', async ({ request }) => {
  const response = await request.post('/api/auth/login', {
    data: {
      email: 'notauser@test.com',
      password: 'wrongpassword',
    },
  });

  expect(response.status()).toBe(401);
  const body = await response.json();
  expect(body).toHaveProperty('error');
});

test('expired or invalid token returns 401 on protected route', async ({ request }) => {
  const response = await request.get('/api/tasks', {
    headers: {
      Authorization: 'Bearer this.is.not.a.valid.token',
    },
  });

  expect(response.status()).toBe(401);
});

test('refresh token returns new access token', async ({ authApi }) => {
  // Login to get tokens
  const initial = await authApi.login({
    email: process.env.USER_EMAIL!,
    password: process.env.USER_PASSWORD!,
  });

  // Use refresh token to get a new access token
  const refreshed = await authApi.refreshToken(initial.refreshToken);

  expect(typeof refreshed.accessToken).toBe('string');
  // New access token should be different from the original
  expect(refreshed.accessToken).not.toBe(initial.accessToken);
});

test('logout invalidates the token', async ({ authApi, request }) => {
  // Login
  const tokens = await authApi.login({
    email: process.env.USER_EMAIL!,
    password: process.env.USER_PASSWORD!,
  });

  // Logout
  await authApi.logout(tokens.accessToken);

  // Try to use the token after logout — should be rejected
  const response = await request.get('/api/tasks', {
    headers: { Authorization: `Bearer ${tokens.accessToken}` },
  });

  expect(response.status()).toBe(401);
});

🔷 GraphQL API Tests

Many modern apps expose a GraphQL API alongside (or instead of) REST. Playwright handles it just as cleanly — GraphQL is just a POST to one endpoint.

// tests/api/graphql-api.spec.ts
import { test, expect } from '../../fixtures/api.fixture';
import { validateTaskSchema } from '../../utils/schema-validator';

// Helper to fire a GraphQL query/mutation
async function gql(
  request: import('@playwright/test').APIRequestContext,
  token: string,
  query: string,
  variables?: Record<string, unknown>
) {
  const response = await request.post('/graphql', {
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/json',
    },
    data: { query, variables },
  });
  return response;
}

test('GraphQL query — fetch all tasks', async ({ request, authApi }) => {
  const token = await authApi.getAdminToken();

  const response = await gql(request, token, `
    query GetTasks {
      tasks {
        id
        title
        status
        assignee
        createdAt
        updatedAt
      }
    }
  `);

  expect(response.status()).toBe(200);

  const body = await response.json();
  expect(body).not.toHaveProperty('errors');
  expect(Array.isArray(body.data.tasks)).toBe(true);
  expect(validateTaskSchema(body.data.tasks[0])).toBe(true);
});

test('GraphQL mutation — create task', async ({ request, authApi }) => {
  const token = await authApi.getAdminToken();

  const response = await gql(
    request,
    token,
    `
    mutation CreateTask($title: String!, $status: TaskStatus) {
      createTask(title: $title, status: $status) {
        id
        title
        status
        createdAt
      }
    }
  `,
    { title: 'GraphQL created task', status: 'PENDING' }
  );

  expect(response.status()).toBe(200);

  const body = await response.json();
  expect(body).not.toHaveProperty('errors');
  expect(body.data.createTask.title).toBe('GraphQL created task');
  expect(body.data.createTask.id).toBeDefined();
});

test('GraphQL returns errors array on bad query — not HTTP error', async ({
  request,
  authApi,
}) => {
  const token = await authApi.getAdminToken();

  // GraphQL always returns 200 — errors live in the body
  const response = await gql(request, token, `
    query {
      nonExistentField {
        id
      }
    }
  `);

  expect(response.status()).toBe(200); // ← this is the GraphQL gotcha
  const body = await response.json();
  expect(body).toHaveProperty('errors');
  expect(Array.isArray(body.errors)).toBe(true);
});

The last test is important — it catches the GraphQL testing gotcha that trips up most engineers. GraphQL almost always returns HTTP 200, even for errors. The actual error lives in body.errors. If you're only asserting on the status code, you'll miss half your bugs. 🎯

🔗 Chaining API + UI — The Most Powerful Pattern

This is the pattern that makes your entire test suite faster and more reliable.

The idea: Use the API to set up test data (fast, no UI overhead) — then use the UI to assert on what the user actually sees. Or flip it — interact via UI, then validate what hit the database via API.

// tests/api/api-ui-chain.spec.ts
import { test, expect } from '@playwright/test';
import { TaskApiClient } from '../../api/TaskApiClient';
import { AuthApiClient } from '../../api/AuthApiClient';
import { TaskPage } from '../../pages/TaskPage';
import { validateTaskSchema } from '../../utils/schema-validator';

test('seed task via API — verify it appears in UI', async ({ page, request }) => {
  // Step 1 — Get token and create task via API (fast, no browser)
  const authApi = new AuthApiClient(request);
  const token = await authApi.getAdminToken();
  const taskApi = new TaskApiClient(request, token);

  const { task } = await taskApi.createTask({
    title: 'API-seeded task for UI verification',
    status: 'pending',
  });

  // Step 2 — Navigate to UI and verify the task is visible
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  await expect(
    taskPage.getTaskLocator('API-seeded task for UI verification')
  ).toBeVisible();

  // Cleanup via API — no UI overhead for teardown either
  await taskApi.deleteTask(task.id);
});

test('create task via UI — verify it hit the API correctly', async ({
  page,
  request,
}) => {
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  // Listen for the POST and capture what was sent
  let capturedTaskId: number | null = null;

  const [response] = await Promise.all([
    page.waitForResponse(
      resp =>
        resp.url().includes('/api/tasks') &&
        resp.request().method() === 'POST' &&
        resp.status() === 201
    ),
    taskPage.createTask('UI-created task — verify via API'),
  ]);

  const createdTask = await response.json();
  capturedTaskId = createdTask.id;

  // Now verify via API that the correct data was persisted
  const authApi = new AuthApiClient(request);
  const token = await authApi.getAdminToken();
  const taskApi = new TaskApiClient(request, token);

  const fetchedTask = await taskApi.getTask(capturedTaskId!);

  expect(fetchedTask.title).toBe('UI-created task — verify via API');
  expect(fetchedTask.status).toBe('pending');
  expect(validateTaskSchema(fetchedTask)).toBe(true);

  // Cleanup
  await taskApi.deleteTask(capturedTaskId!);
});

test('bulk seed via API — UI pagination works correctly', async ({
  page,
  request,
}) => {
  const authApi = new AuthApiClient(request);
  const token = await authApi.getAdminToken();
  const taskApi = new TaskApiClient(request, token);

  // Create 25 tasks via API — in parallel for speed
  const createdTasks = await Promise.all(
    Array.from({ length: 25 }, (_, i) =>
      taskApi.createTask({ title: `Pagination test task ${i + 1}` })
    )
  );

  // Navigate to UI and check pagination renders correctly
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  // First page should show page size (assuming 10 per page)
  await expect(page.getByRole('listitem')).toHaveCount(10);
  await expect(page.getByTestId('pagination')).toBeVisible();
  await expect(page.getByTestId('total-count')).toContainText('25');

  // Cleanup all created tasks via API in parallel
  await Promise.all(
    createdTasks.map(({ task }) => taskApi.deleteTask(task.id))
  );
});

test('delete via UI — confirm gone via API', async ({ page, request }) => {
  // Seed task via API
  const authApi = new AuthApiClient(request);
  const token = await authApi.getAdminToken();
  const taskApi = new TaskApiClient(request, token);

  const { task } = await taskApi.createTask({
    title: 'Task to delete from UI',
  });

  // Delete via UI
  const taskPage = new TaskPage(page);
  await taskPage.goto();
  await taskPage.deleteTask('Task to delete from UI');

  // Confirm UI updated
  await expect(taskPage.getTaskLocator('Task to delete from UI')).not.toBeVisible();

  // Confirm API also reflects deletion
  const allTasks = await taskApi.getAllTasks();
  const stillExists = allTasks.find(t => t.id === task.id);
  expect(stillExists).toBeUndefined();
});

This is the pattern that catches the bugs your pure UI tests miss — when the UI says something is deleted but the database still has it. Or when the UI shows a task but the API returns it with wrong data. Catching the gap between what the user sees and what actually happened is where this shines. 🔥

📁 Final Project Structure After Part 4

Every file listed below has been fully built across Parts 1 through 4:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts                        ✅ Part 1
│   ├── tasks/
│   │   └── task-management.spec.ts              ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   │   ├── api-mocking.spec.ts
│   │   ├── error-simulation.spec.ts
│   │   └── network-assertions.spec.ts
│   ├── multi-user/                              ✅ Part 3
│   │   ├── role-permissions.spec.ts
│   │   └── realtime-collaboration.spec.ts
│   ├── multi-tab/                               ✅ Part 3
│   │   └── multi-tab-flows.spec.ts
│   └── api/                                     ✅ Part 4
│       ├── tasks-api.spec.ts
│       ├── auth-api.spec.ts
│       ├── graphql-api.spec.ts
│       └── api-ui-chain.spec.ts
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── api/                                         ✅ Part 4
│   ├── TaskApiClient.ts
│   └── AuthApiClient.ts
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   ├── multi-user.fixture.ts                    ✅ Part 3
│   └── api.fixture.ts                           ✅ Part 4
├── scripts/
│   └── record-har.ts                            ✅ Part 2
├── utils/                                       ✅ Part 4
│   └── schema-validator.ts
├── .auth/                                       ← git-ignored
│   ├── admin.json
│   └── user.json
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Part 1 (updated Parts 3 & 4)
├── .env                                         ← git-ignored
└── package.json

🗺️ What's Coming in This Series

Part 1 — Stop Writing Tests Like a Beginner              ✅ Done
Part 2 — Network Interception: The Complete Guide        ✅ Done
Part 3 — Multi-User, Multi-Tab & Context Testing         ✅ Done
Part 4 — API Testing (The Underrated Superpower)         ← You are here
Part 5 — Visual Regression Testing
Part 6 — Debugging Like a Pro: Trace Viewer & Inspector
Part 7 — The CI/CD Setup Nobody Shows You
Part 8 — Playwright Meets AI: Agents, MCP & Self-Healing Tests

In Part 5, we add visual regression testing — toHaveScreenshot(), masking dynamic content, pixel tolerance config, and running VRT in CI without flakiness. The kind of bugs that slip past every assertion you've written so far.

🔖 Before You Go

After four parts, you've built something real:

A POM-based UI test layer with proper auth
A network interception layer that owns the API at the browser level
A multi-user layer that tests collaboration and permissions simultaneously
A raw API testing layer with typed clients, schema validation, and API-UI chaining

Each layer is independent. Each layer builds on the last.

And you've done it all in one tool, one language, one pipeline. No Postman. No separate pytest suite. No context switching. 💪

Follow me so you don't miss Part 5 — where we catch the bugs that slip past every assertion: CSS regressions, layout shifts, and design inconsistencies — with Playwright's visual regression testing.

Drop a comment below 👇

Are you currently running API tests in a separate tool from your UI tests?
Did the GraphQL gotcha (200 status with errors in body) catch you off guard?
What's the first API endpoint you'd write a schema validation test for?

Let's talk in the comments. 🙌

Faizal Shaikh | Senior Automation Engineer | Playwright & AI Testing
Connect with me on LinkedIn

The Playwright Playbook — Part 3: Multi-User, Multi-Tab & Browser Context Testing

Faizal — Wed, 17 Jun 2026 06:27:40 +0000

The Playwright Playbook — Part 3: Multi-User, Multi-Tab & Browser Context Testing

"Most frameworks test one user at a time. Real apps don't work that way."

In Part 1, we built the full foundation — POM, storageState, fixtures, and a clean project structure. In Part 2, we took control of the network layer — mocking APIs, simulating failures, and asserting on real network calls.

Now we tackle the scenario that breaks most automation frameworks.

Multiple users. Simultaneously. In one test.

Think about your app for a moment. How many features actually involve just one user?

Admin assigns a task → regular user sees it appear in real time
User A edits a record → User B gets a notification
Admin revokes access → User's session should invalidate
Two users try to edit the same record → conflict resolution kicks in

These are real features. They need to be tested. And page.goto('/login') in a beforeEach won't get you there.

Playwright's browser context architecture was built exactly for this. Let's use it properly. 🎯

🏗️ Where We Left Off

After Part 2, our project looks like this:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts                   ✅ Part 1
│   ├── tasks/
│   │   └── task-management.spec.ts         ✅ Part 1
│   └── network/                            ✅ Part 2
│       ├── api-mocking.spec.ts
│       ├── error-simulation.spec.ts
│       └── network-assertions.spec.ts
├── pages/
│   ├── LoginPage.ts                        ✅ Part 1
│   └── TaskPage.ts                         ✅ Part 1
├── fixtures/
│   ├── auth.fixture.ts                     ✅ Part 1
│   ├── tasks.json                          ✅ Part 2
│   ├── empty-tasks.json                    ✅ Part 2
│   └── tasks-har.har                       ✅ Part 2
├── scripts/
│   └── record-har.ts                       ✅ Part 2
├── .auth/
│   ├── admin.json
│   └── user.json
├── global-setup.ts                         ✅ Part 1
├── playwright.config.ts                    ✅ Part 1
└── .env

By the end of Part 3, we add:

playwright-playbook/
├── tests/
│   ├── multi-user/                         ← NEW
│   │   ├── role-permissions.spec.ts
│   │   └── realtime-collaboration.spec.ts
│   └── multi-tab/                          ← NEW
│       └── multi-tab-flows.spec.ts
├── pages/
│   └── DashboardPage.ts                    ← NEW
├── fixtures/
│   └── multi-user.fixture.ts               ← NEW

Every one of these files will be fully built in this article. Let's understand the foundation first. 👇

🧠 Understanding Browser Contexts — The Mental Model

This is the concept everything in this part is built on. Read it carefully.

In a regular browser, you have one session. One set of cookies. One logged-in user at a time.

Playwright has browser contexts — isolated browser environments that each have their own:

Cookies and session storage
Local storage
Authentication state
Network state

Browser (one instance)
├── Context A  ← admin session (.auth/admin.json)
│   └── Page A → logged in as admin
└── Context B  ← user session (.auth/user.json)
    └── Page B → logged in as regular user

Both contexts run inside the same browser process. But they are completely isolated from each other — like two separate incognito windows, but programmable and controllable via Playwright.

This means you can have adminPage and userPage active in the same test, interacting with the same app simultaneously. 🤯

🏗️ Building the New Page Object — DashboardPage

Our Task Manager's dashboard is where task assignments, notifications, and real-time updates happen. We need a page object for it.

// pages/DashboardPage.ts
import { Page, Locator } from '@playwright/test';

export class DashboardPage {
  readonly page: Page;
  readonly taskList: Locator;
  readonly notificationBell: Locator;
  readonly notificationPanel: Locator;
  readonly welcomeMessage: Locator;
  readonly adminPanel: Locator;

  constructor(page: Page) {
    this.page = page;
    this.taskList = page.getByTestId('task-list');
    this.notificationBell = page.getByTestId('notification-bell');
    this.notificationPanel = page.getByTestId('notification-panel');
    this.welcomeMessage = page.getByTestId('welcome-message');
    this.adminPanel = page.getByTestId('admin-panel');
  }

  async goto() {
    await this.page.goto('/dashboard');
  }

  async waitForTaskToAppear(taskTitle: string) {
    await this.page
      .getByRole('listitem')
      .filter({ hasText: taskTitle })
      .waitFor({ state: 'visible' });
  }

  async openNotifications() {
    await this.notificationBell.click();
    await this.notificationPanel.waitFor({ state: 'visible' });
  }

  getNotificationLocator(text: string): Locator {
    return this.notificationPanel.getByText(text);
  }

  getTaskLocator(title: string): Locator {
    return this.page.getByRole('listitem').filter({ hasText: title });
  }

  async assignTaskToUser(taskTitle: string, userName: string) {
    const task = this.getTaskLocator(taskTitle);
    await task.hover();
    await task.getByRole('button', { name: 'Assign' }).click();
    await this.page.getByRole('option', { name: userName }).click();
  }
}

🧩 Building the Multi-User Fixture

The auth.fixture.ts from Part 1 gives one authenticated page. For multi-user tests, we need two authenticated contexts running simultaneously.

This is where multi-user.fixture.ts comes in — it creates two independent browser contexts, each loaded with a different storageState, and hands both to the test as named fixtures.

// fixtures/multi-user.fixture.ts
import { test as base, Browser, BrowserContext, Page } from '@playwright/test';
import { TaskPage } from '../pages/TaskPage';
import { DashboardPage } from '../pages/DashboardPage';

type MultiUserFixtures = {
  adminContext: BrowserContext;
  userContext: BrowserContext;
  adminPage: Page;
  userPage: Page;
  adminTaskPage: TaskPage;
  userTaskPage: TaskPage;
  adminDashboard: DashboardPage;
  userDashboard: DashboardPage;
};

export const test = base.extend<MultiUserFixtures>({
  // Admin browser context — loaded with admin session
  adminContext: async ({ browser }, use) => {
    const context = await browser.newContext({
      storageState: '.auth/admin.json',
    });
    await use(context);
    await context.close();
  },

  // User browser context — loaded with regular user session
  userContext: async ({ browser }, use) => {
    const context = await browser.newContext({
      storageState: '.auth/user.json',
    });
    await use(context);
    await context.close();
  },

  // Admin page — a new page inside the admin context
  adminPage: async ({ adminContext }, use) => {
    const page = await adminContext.newPage();
    await use(page);
  },

  // User page — a new page inside the user context
  userPage: async ({ userContext }, use) => {
    const page = await userContext.newPage();
    await use(page);
  },

  // Admin TaskPage POM — ready to use
  adminTaskPage: async ({ adminPage }, use) => {
    const taskPage = new TaskPage(adminPage);
    await taskPage.goto();
    await use(taskPage);
  },

  // User TaskPage POM — ready to use
  userTaskPage: async ({ userPage }, use) => {
    const taskPage = new TaskPage(userPage);
    await taskPage.goto();
    await use(taskPage);
  },

  // Admin DashboardPage POM — ready to use
  adminDashboard: async ({ adminPage }, use) => {
    const dashboard = new DashboardPage(adminPage);
    await dashboard.goto();
    await use(dashboard);
  },

  // User DashboardPage POM — ready to use
  userDashboard: async ({ userPage }, use) => {
    const dashboard = new DashboardPage(userPage);
    await dashboard.goto();
    await use(dashboard);
  },
});

export { expect } from '@playwright/test';

Now any test that needs two simultaneous users just imports from this fixture. The setup is invisible. The test reads like a story. 🎯

🔐 Testing Role-Based Permissions

The first multi-user scenario: ensuring admin and regular user see different things in the same app.

// tests/multi-user/role-permissions.spec.ts
import { test, expect } from '../../fixtures/multi-user.fixture';

test('admin sees admin panel — regular user does not', async ({
  adminDashboard,
  userDashboard,
}) => {
  // Admin should see the admin panel
  await expect(adminDashboard.adminPanel).toBeVisible();

  // Regular user should NOT see the admin panel
  await expect(userDashboard.adminPanel).not.toBeVisible();
});

test('admin can delete any task — regular user can only delete their own', async ({
  adminTaskPage,
  userTaskPage,
  adminPage,
  userPage,
}) => {
  // Admin creates a task
  await adminTaskPage.createTask('Admin-owned task');

  // Admin can see and delete the task
  await expect(adminTaskPage.getTaskLocator('Admin-owned task')).toBeVisible();
  const adminDeleteBtn = adminPage
    .getByRole('listitem')
    .filter({ hasText: 'Admin-owned task' })
    .getByRole('button', { name: 'Delete' });
  await expect(adminDeleteBtn).toBeVisible();

  // Regular user can see the task but NOT delete it
  await userTaskPage.page.reload(); // reload to fetch latest tasks
  await expect(userTaskPage.getTaskLocator('Admin-owned task')).toBeVisible();
  const userDeleteBtn = userPage
    .getByRole('listitem')
    .filter({ hasText: 'Admin-owned task' })
    .getByRole('button', { name: 'Delete' });
  await expect(userDeleteBtn).not.toBeVisible();
});

test('admin can access /admin route — user gets redirected', async ({
  adminPage,
  userPage,
}) => {
  // Admin navigates to admin-only route
  await adminPage.goto('/admin');
  await expect(adminPage).toHaveURL('/admin');
  await expect(adminPage.getByRole('heading', { name: 'Admin Dashboard' })).toBeVisible();

  // Regular user is redirected away
  await userPage.goto('/admin');
  await expect(userPage).toHaveURL('/dashboard'); // redirected to their dashboard
  await expect(userPage.getByTestId('access-denied')).toBeVisible();
});

test('user count badge updates correctly per role', async ({
  adminDashboard,
  userDashboard,
}) => {
  // Admin sees all tasks across all users
  const adminCount = await adminDashboard.page
    .getByTestId('task-count-badge')
    .textContent();

  // Regular user only sees their own tasks — count should be lower or equal
  const userCount = await userDashboard.page
    .getByTestId('task-count-badge')
    .textContent();

  expect(parseInt(adminCount ?? '0')).toBeGreaterThanOrEqual(
    parseInt(userCount ?? '0')
  );
});

🔄 Testing Real-Time Collaboration

This is the scenario that's genuinely hard to test without Playwright's context architecture.

Admin creates or assigns a task. Regular user — in a separate browser context, watching the same app — should see it appear without refreshing.

// tests/multi-user/realtime-collaboration.spec.ts
import { test, expect } from '../../fixtures/multi-user.fixture';

test('task assigned by admin appears in real time for user', async ({
  adminDashboard,
  userDashboard,
}) => {
  // Both users are on the dashboard simultaneously
  // User is watching — no action yet

  // Admin assigns a task to the user
  await adminDashboard.assignTaskToUser('Fix flaky test in CI', 'Regular User');

  // User should see the task appear WITHOUT refreshing
  // waitForTaskToAppear polls until the element is visible (up to default timeout)
  await userDashboard.waitForTaskToAppear('Fix flaky test in CI');

  await expect(
    userDashboard.getTaskLocator('Fix flaky test in CI')
  ).toBeVisible();
});

test('user receives notification when task is assigned', async ({
  adminDashboard,
  userDashboard,
}) => {
  // Admin assigns a task
  await adminDashboard.assignTaskToUser('Write unit tests', 'Regular User');

  // User opens their notification panel
  await userDashboard.openNotifications();

  // Notification should appear in real time
  await expect(
    userDashboard.getNotificationLocator('You have been assigned: Write unit tests')
  ).toBeVisible();
});

test('task status update by admin reflects instantly for user', async ({
  adminTaskPage,
  userTaskPage,
}) => {
  // Admin marks a task as completed
  const taskItem = adminTaskPage.getTaskLocator('Review pull request');
  await taskItem.hover();
  await taskItem.getByRole('checkbox', { name: 'Mark complete' }).check();

  // User should see the task status update — no page refresh
  await expect(
    userTaskPage.getTaskLocator('Review pull request')
      .getByTestId('status-badge')
  ).toHaveText('Completed');
});

test('admin deleting a task removes it from user view in real time', async ({
  adminTaskPage,
  userTaskPage,
}) => {
  // Confirm both users can see the task first
  await expect(adminTaskPage.getTaskLocator('Fix flaky test in CI')).toBeVisible();
  await expect(userTaskPage.getTaskLocator('Fix flaky test in CI')).toBeVisible();

  // Admin deletes the task
  await adminTaskPage.deleteTask('Fix flaky test in CI');

  // Admin's view updates immediately
  await expect(adminTaskPage.getTaskLocator('Fix flaky test in CI')).not.toBeVisible();

  // User's view should also update — real-time sync
  await expect(userTaskPage.getTaskLocator('Fix flaky test in CI')).not.toBeVisible();
});

This is what proper real-time feature testing looks like. Two users. One test. No mocking. No workarounds. Just Playwright doing what it was built for. 💪

🗂️ Testing Multi-Tab Flows

Browser contexts also power multi-tab testing — when your app opens links in new tabs, or when a workflow spans multiple browser windows.

The pattern is: listen for the new tab to open, wait for it, then interact with it just like any other page.

// tests/multi-tab/multi-tab-flows.spec.ts
import { test, expect } from '@playwright/test';
import { TaskPage } from '../../pages/TaskPage';
import { DashboardPage } from '../../pages/DashboardPage';

test('clicking task link opens detail view in new tab', async ({ page, context }) => {
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  // Listen for the new tab BEFORE clicking the link that opens it
  const newTabPromise = context.waitForEvent('page');

  // Click a task that opens its detail in a new tab
  await taskPage.getTaskLocator('Write unit tests')
    .getByRole('link', { name: 'Open details' })
    .click();

  // Wait for and capture the new tab
  const newTab = await newTabPromise;
  await newTab.waitForLoadState('domcontentloaded');

  // Assert on the new tab
  await expect(newTab).toHaveURL(/\/tasks\/\d+/);
  await expect(newTab.getByRole('heading', { name: 'Write unit tests' })).toBeVisible();
  await expect(newTab.getByTestId('task-detail-panel')).toBeVisible();
});

test('completing a task in new tab updates the count on original tab', async ({
  page,
  context,
}) => {
  const dashboard = new DashboardPage(page);
  await dashboard.goto();

  // Capture pending task count on the original tab
  const initialCountText = await page.getByTestId('pending-count').textContent();
  const initialCount = parseInt(initialCountText ?? '0');

  // Open a task in a new tab
  const newTabPromise = context.waitForEvent('page');
  await dashboard.getTaskLocator('Write unit tests')
    .getByRole('link', { name: 'Open details' })
    .click();
  const taskDetailTab = await newTabPromise;
  await taskDetailTab.waitForLoadState('domcontentloaded');

  // Complete the task in the new tab
  await taskDetailTab.getByRole('button', { name: 'Mark as Complete' }).click();
  await expect(taskDetailTab.getByTestId('status-badge')).toHaveText('Completed');

  // Switch back to the original tab — count should have updated
  await page.bringToFront();
  await expect(page.getByTestId('pending-count')).toHaveText(
    String(initialCount - 1)
  );
});

test('authentication persists across new tabs in same context', async ({
  page,
  context,
}) => {
  // Start on dashboard (already authenticated via storageState)
  await page.goto('/dashboard');
  await expect(page.getByTestId('welcome-message')).toBeVisible();

  // Open a new tab within the same context
  const newTab = await context.newPage();
  await newTab.goto('/tasks');

  // Should still be authenticated — same context, same session
  await expect(newTab.getByTestId('task-list')).toBeVisible();
  await expect(newTab).not.toHaveURL('/login');
});

test('popup window flows work correctly', async ({ page }) => {
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  // Some apps open popups (OAuth flows, print dialogs, confirm windows)
  // Listen for popup BEFORE triggering it
  const popupPromise = page.waitForEvent('popup');

  await page.getByRole('button', { name: 'Share task externally' }).click();
  const popup = await popupPromise;

  await popup.waitForLoadState('domcontentloaded');

  // Interact with the popup
  await expect(popup.getByRole('heading', { name: 'Share Task' })).toBeVisible();
  await popup.getByLabel('Recipient email').fill('colleague@company.com');
  await popup.getByRole('button', { name: 'Send' }).click();

  // Popup closes after sending
  await popup.waitForEvent('close');

  // Back on main page — confirm success message
  await expect(page.getByTestId('share-success-toast')).toBeVisible();
});

⚙️ Updating `playwright.config.ts` for Multi-Context Tests

Multi-context tests need one small config addition — we need to make sure the browser fixture is available to our multi-user fixture (it is by default), and that parallel workers don't share state.

// playwright.config.ts — updated section
import { defineConfig, devices } from '@playwright/test';
import * as dotenv from 'dotenv';

dotenv.config();

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 4 : undefined,
  globalSetup: './global-setup.ts',

  reporter: [
    ['html', { open: 'never' }],
    ['list'],
  ],

  use: {
    baseURL: process.env.BASE_URL || 'http://localhost:3000',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    trace: 'on-first-retry',
  },

  projects: [
    {
      name: 'admin',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/admin.json',
      },
      testMatch: ['**/auth/**', '**/tasks/**', '**/network/**'],
    },
    {
      name: 'user',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/user.json',
      },
      testMatch: ['**/tasks/**'],
    },
    {
      // Multi-user and multi-tab tests manage their own contexts internally
      // so they don't inherit a storageState at the project level
      name: 'multi-context',
      use: {
        ...devices['Desktop Chrome'],
      },
      testMatch: ['**/multi-user/**', '**/multi-tab/**'],
    },
  ],
});

The multi-context project has no storageState at the project level — because the multi-user.fixture.ts handles authentication per-context internally. 🔑

📁 Final Project Structure After Part 3

Every file listed below has been fully built across Parts 1, 2, and 3:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts                        ✅ Part 1
│   ├── tasks/
│   │   └── task-management.spec.ts              ✅ Part 1
│   ├── network/                                 ✅ Part 2
│   │   ├── api-mocking.spec.ts
│   │   ├── error-simulation.spec.ts
│   │   └── network-assertions.spec.ts
│   ├── multi-user/                              ✅ Part 3
│   │   ├── role-permissions.spec.ts
│   │   └── realtime-collaboration.spec.ts
│   └── multi-tab/                              ✅ Part 3
│       └── multi-tab-flows.spec.ts
├── pages/
│   ├── LoginPage.ts                             ✅ Part 1
│   ├── TaskPage.ts                              ✅ Part 1
│   └── DashboardPage.ts                         ✅ Part 3
├── fixtures/
│   ├── auth.fixture.ts                          ✅ Part 1
│   ├── tasks.json                               ✅ Part 2
│   ├── empty-tasks.json                         ✅ Part 2
│   ├── tasks-har.har                            ✅ Part 2
│   └── multi-user.fixture.ts                    ✅ Part 3
├── scripts/
│   └── record-har.ts                            ✅ Part 2
├── .auth/                                       ← git-ignored, auto-generated
│   ├── admin.json
│   └── user.json
├── global-setup.ts                              ✅ Part 1
├── playwright.config.ts                         ✅ Part 1 (updated Part 3)
├── .env                                         ← git-ignored
└── package.json

🗺️ What's Coming in This Series

Part 1 — Stop Writing Tests Like a Beginner              ✅ Done
Part 2 — Network Interception: The Complete Guide        ✅ Done
Part 3 — Multi-User, Multi-Tab & Context Testing         ← You are here
Part 4 — API Testing (The Underrated Superpower)
Part 5 — Visual Regression Testing
Part 6 — Debugging Like a Pro: Trace Viewer & Inspector
Part 7 — The CI/CD Setup Nobody Shows You
Part 8 — Playwright Meets AI: Agents, MCP & Self-Healing Tests

In Part 4, we go into Playwright's request context — making raw API calls without a browser, schema validation, chaining API setup with UI assertions, and GraphQL testing. If you've ever used Postman for test automation, Part 4 will replace it entirely.

🔖 Before You Go

Multi-user and multi-tab testing is the feature that most QA engineers don't even attempt — because they assume it's too complex.

It isn't. It's three things:

Two browser contexts with separate storageState
A fixture that wires them both up cleanly
Tests that treat both pages like normal Playwright pages

The complexity is in understanding the model. Once you have the model — the code is simple.

And now you have both. 💪

Follow me so you don't miss Part 4 — where we replace Postman with Playwright's API request context and build a proper API testing layer that talks directly to your backend without touching a browser.

Drop a comment below 👇

Have you ever tried testing real-time features in Playwright before?
What's the most painful multi-user scenario in your app right now?
Did the browser context mental model click for you — or do you have questions?

Let's talk in the comments. 🙌

Faizal Shaikh | Senior Automation Engineer | Playwright & AI Testing
Connect with me on LinkedIn

The Playwright Playbook — Part 2: Network Interception — The Complete Guide

Faizal — Tue, 16 Jun 2026 08:43:44 +0000

The Playwright Playbook — Part 2: Network Interception — The Complete Guide

"Your tests should own the network — not be at its mercy."

In Part 1, we built the full foundation — semantic locators, storageState, Page Object Model with LoginPage and TaskPage (including deleteTask), an auth fixture, and a clean playwright.config.ts. If you haven't read that, start there. Every file we reference today was built in Part 1.

Now we go to the feature that separates intermediate Playwright users from advanced ones.

Network interception.

Most people know page.route() exists. Most people use it once, mock one response, and move on.

But network interception is an entire testing strategy — not just a utility. When you fully understand it, you can:

Test your frontend completely independently of the backend
Simulate error states, timeouts, and edge cases in seconds
Assert on exactly what API calls your app is making — and what it's sending
Record real network traffic and replay it in tests without a live server
Catch silent API failures that your UI swallows gracefully

This is the part where Playwright stops feeling like a UI testing tool and starts feeling like a full testing platform. 🚀

Let's build it. 🎯

🏗️ Where We Left Off

At the end of Part 1, our project was fully built:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts
│   └── tasks/
│       └── task-management.spec.ts
├── pages/
│   ├── LoginPage.ts
│   └── TaskPage.ts              ← includes createTask + deleteTask
├── fixtures/
│   └── auth.fixture.ts
├── .auth/
│   ├── admin.json
│   └── user.json
├── global-setup.ts
├── playwright.config.ts
├── .env
└── package.json

By the end of Part 2, we add a network/ test folder, a scripts/ folder, and expand fixtures/ with mock data and HAR files:

playwright-playbook/
├── tests/
│   ├── auth/
│   ├── tasks/
│   └── network/                          ← NEW
│       ├── api-mocking.spec.ts
│       ├── error-simulation.spec.ts
│       └── network-assertions.spec.ts
├── pages/
├── fixtures/
│   ├── auth.fixture.ts
│   ├── tasks.json                        ← NEW
│   ├── empty-tasks.json                  ← NEW
│   └── tasks-har.har                     ← NEW (generated, not hand-written)
├── scripts/                              ← NEW
│   └── record-har.ts
├── .auth/
├── global-setup.ts
├── playwright.config.ts
└── .env

🔍 How Playwright Network Interception Works

Before we write code — let's understand the architecture.

Every time your browser makes a network request, Playwright can intercept it at the network layer — before it ever reaches the server.

Browser Action (click "Load Tasks")
        │
        ▼
Playwright Route Handler ← you control this
        │
   ┌────┴────────┐
   │             │             │
   ▼             ▼             ▼
Fulfill        Abort        Continue
(mock)       (kill it)    (real server,
                         optionally modified)

Three things you can do with an intercepted request:

Fulfill — return a fake response you define (mock)
Abort — kill the request entirely (simulate network failure)
Continue — let it go through to the real server, optionally with modified headers or body

That's the whole mental model. Everything we do in this part is a variation of those three options. Let's build each one. 👇

🧪 The Basic: `page.route()` — Your First Mock

Our Task Manager app calls GET /api/tasks to load the task list. In most test setups, this requires a running backend with seeded data.

With network interception — we don't need that.

First, create your mock data files:

// fixtures/tasks.json
[
  { "id": 1, "title": "Write unit tests", "status": "pending", "assignee": "admin" },
  { "id": 2, "title": "Review pull request", "status": "completed", "assignee": "user" },
  { "id": 3, "title": "Fix flaky test in CI", "status": "pending", "assignee": "admin" }
]

// fixtures/empty-tasks.json
[]

Now intercept and mock the API call in your test:

// tests/network/api-mocking.spec.ts
import { test, expect } from '@playwright/test';
import tasks from '../../fixtures/tasks.json';
import emptyTasks from '../../fixtures/empty-tasks.json';

test('task list renders mocked data correctly', async ({ page }) => {
  // ⚠️ Set up route BEFORE page.goto() — the handler must exist before requests fire
  await page.route('**/api/tasks', async route => {
    await route.fulfill({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify(tasks),
    });
  });

  await page.goto('/tasks');

  // Assert the UI rendered the mocked data
  await expect(page.getByRole('listitem')).toHaveCount(3);
  await expect(page.getByText('Write unit tests')).toBeVisible();
  await expect(page.getByText('Review pull request')).toBeVisible();
});

test('shows empty state when no tasks exist', async ({ page }) => {
  await page.route('**/api/tasks', async route => {
    await route.fulfill({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify(emptyTasks),
    });
  });

  await page.goto('/tasks');

  await expect(page.getByTestId('empty-state')).toBeVisible();
  await expect(page.getByText('No tasks yet')).toBeVisible();
  await expect(page.getByRole('listitem')).toHaveCount(0);
});

Key rule: Always set up page.route() BEFORE page.goto(). The route handler needs to be registered before the page makes any requests. Miss this and your mock silently does nothing. 🎯

❌ Simulating Error States

This is where network interception becomes genuinely powerful.

How do you test what happens when your API returns a 500? Or a 401? Or a timeout?

Without interception — you need to break your backend. With interception — one line.

// tests/network/error-simulation.spec.ts
import { test, expect } from '@playwright/test';

test('shows error banner when API returns 500', async ({ page }) => {
  await page.route('**/api/tasks', async route => {
    await route.fulfill({
      status: 500,
      contentType: 'application/json',
      body: JSON.stringify({ error: 'Internal Server Error' }),
    });
  });

  await page.goto('/tasks');

  await expect(page.getByTestId('error-banner')).toBeVisible();
  await expect(page.getByText('Something went wrong')).toBeVisible();
  // Task list should NOT render when the API fails
  await expect(page.getByTestId('task-list')).not.toBeVisible();
});

test('shows empty state when API returns 404', async ({ page }) => {
  await page.route('**/api/tasks', async route => {
    await route.fulfill({
      status: 404,
      contentType: 'application/json',
      body: JSON.stringify({ error: 'Not found' }),
    });
  });

  await page.goto('/tasks');

  await expect(page.getByTestId('empty-state')).toBeVisible();
  await expect(page.getByText('No tasks found')).toBeVisible();
});

test('shows loading skeleton when API is slow', async ({ page }) => {
  await page.route('**/api/tasks', async route => {
    // Delay the response — simulates a slow network or overloaded server
    await new Promise(resolve => setTimeout(resolve, 3000));
    await route.continue();
  });

  await page.goto('/tasks');

  // Skeleton should be visible WHILE the request is still in flight
  await expect(page.getByTestId('loading-skeleton')).toBeVisible();
});

test('shows retry button when network fails completely', async ({ page }) => {
  await page.route('**/api/tasks', async route => {
    // Abort kills the request entirely — simulates no internet
    await route.abort('failed');
  });

  await page.goto('/tasks');

  await expect(page.getByTestId('network-error')).toBeVisible();
  await expect(page.getByRole('button', { name: 'Retry' })).toBeVisible();
});

Think about what you just did. You tested four different failure scenarios — without touching a single line of backend code. Without spinning up a special error environment. Without asking a developer to temporarily break something. 💪

🔁 Modifying Requests on the Fly

Sometimes you don't want to fully mock a response. You want to intercept a real request, tweak it, and let it continue to the actual server.

// Inject custom test headers into every API call
test('injects test mode headers into requests', async ({ page }) => {
  await page.route('**/api/**', async route => {
    const headers = {
      ...route.request().headers(),
      'x-test-mode': 'true',
      'x-test-session': 'automation',
    };
    await route.continue({ headers });
  });

  await page.goto('/tasks');
  // Request goes to real server — but now carries test headers
  // Useful when your backend logs or behaves differently in test mode
});

// Modify query params before the request leaves the browser
test('filters to completed tasks via modified query param', async ({ page }) => {
  await page.route('**/api/tasks', async route => {
    const url = new URL(route.request().url());
    url.searchParams.set('status', 'completed');
    await route.continue({ url: url.toString() });
  });

  await page.goto('/tasks');

  // Only completed tasks show — filtered at the network layer
  await expect(page.getByText('Review pull request')).toBeVisible();
  await expect(page.getByText('Write unit tests')).not.toBeVisible();
});

📡 Asserting on Real Network Calls — `waitForResponse`

Everything so far has been about controlling the network. Now let's talk about observing it.

This is the pattern I use most in API-heavy applications — asserting that the correct API calls are made, with the correct payloads, at exactly the right time.

⚠️ Critical pattern: page.waitForResponse() must be set up BEFORE the action that triggers the request. If you await it first and THEN click the button — you'll miss the response. The correct pattern is Promise.all.

// tests/network/network-assertions.spec.ts
import { test, expect } from '@playwright/test';
import { TaskPage } from '../../pages/TaskPage';

test('creating a task fires a POST to /api/tasks', async ({ page }) => {
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  // ✅ Set up the listener and trigger the action AT THE SAME TIME
  const [response] = await Promise.all([
    page.waitForResponse(
      resp =>
        resp.url().includes('/api/tasks') &&
        resp.request().method() === 'POST' &&
        resp.status() === 201
    ),
    taskPage.createTask('Write integration tests'),
  ]);

  // Assert on the actual API response
  const body = await response.json();
  expect(body.title).toBe('Write integration tests');
  expect(body.status).toBe('pending');
  expect(body.id).toBeDefined();

  // Assert the UI also updated
  await expect(taskPage.getTaskLocator('Write integration tests')).toBeVisible();
});

test('deleting a task fires DELETE with the correct task ID', async ({ page }) => {
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  // Create a task first and capture its ID from the API response
  const [createResponse] = await Promise.all([
    page.waitForResponse(
      resp => resp.url().includes('/api/tasks') && resp.request().method() === 'POST'
    ),
    taskPage.createTask('Task to be deleted'),
  ]);

  const { id } = await createResponse.json();

  // Now watch for the DELETE call on that specific ID
  const [deleteResponse] = await Promise.all([
    page.waitForResponse(
      resp =>
        resp.url().includes(`/api/tasks/${id}`) &&
        resp.request().method() === 'DELETE'
    ),
    taskPage.deleteTask('Task to be deleted'),
  ]);

  expect(deleteResponse.status()).toBe(200);
  await expect(taskPage.getTaskLocator('Task to be deleted')).not.toBeVisible();
});

test('task creation sends the correct request payload', async ({ page }) => {
  const taskPage = new TaskPage(page);
  await taskPage.goto();

  let capturedPayload: Record<string, unknown> = {};

  // Intercept to read what the browser actually sends
  await page.route('**/api/tasks', async route => {
    if (route.request().method() === 'POST') {
      capturedPayload = JSON.parse(route.request().postData() || '{}');
    }
    // Let it continue to the real server
    await route.continue();
  });

  await taskPage.createTask('Validate this payload');

  expect(capturedPayload.title).toBe('Validate this payload');
  expect(capturedPayload.status).toBe('pending');
  expect(capturedPayload.assignee).toBeDefined();
});

This pattern is invaluable when testing forms — assert the correct payload is being sent without depending on what the backend does with it. And because we're using the TaskPage.deleteTask() method built in Part 1, the test stays clean and readable. 🎯

🎬 HAR Files — Record Real Traffic, Replay It in Tests

HAR (HTTP Archive) is the most underused feature in Playwright's network toolkit.

The idea: Record real network traffic from your app once — save it to a file — replay it in tests without ever hitting the server again.

Perfect for:

Third-party APIs you don't control (payment gateways, analytics, maps)
Expensive API calls you don't want to make on every test run
Flaky external dependencies you want to stabilize once and for all

Step 1 — Record the HAR file

Run this script once against your real running app. It records every network call made during the interaction.

// scripts/record-har.ts
import { chromium } from '@playwright/test';
import * as dotenv from 'dotenv';

dotenv.config();

(async () => {
  const browser = await chromium.launch({ headless: false }); // headless: false so you can watch
  const context = await browser.newContext();

  // This tells Playwright: record all network traffic to this file
  await context.routeFromHAR('./fixtures/tasks-har.har', { update: true });

  const page = await context.newPage();
  await page.goto(`${process.env.BASE_URL}/tasks`);

  // Interact with the app — every network call gets saved
  await page.getByRole('button', { name: 'New Task' }).click();
  await page.getByLabel('Task title').fill('Recorded task');
  await page.getByRole('button', { name: 'Save task' }).click();
  await page.waitForTimeout(1000); // let the response settle before closing

  await browser.close();
  console.log('✅ HAR recorded to fixtures/tasks-har.har');
})();

Run it once: npx ts-node scripts/record-har.ts

This creates fixtures/tasks-har.har — a JSON file containing every request and response. Commit it to your repo.

Step 2 — Replay it in your tests

// tests/network/api-mocking.spec.ts (add to existing file)
test('task flow using recorded HAR — no live server needed', async ({ page }) => {
  // All matching network calls get served from the HAR file
  await page.routeFromHAR('./fixtures/tasks-har.har', {
    notFound: 'abort',   // fail loudly if a request isn't in the HAR
    url: '**/api/**',    // only intercept API calls — let assets load normally
  });

  await page.goto('/tasks');

  // Runs entirely against recorded responses — deterministic every time
  await expect(page.getByRole('listitem')).toHaveCount(3);
});

Your tests are now completely decoupled from any live server. CI runs in half the time. Zero flakiness from external APIs. The HAR file is your source of truth. ✅

🧩 Putting It All Together — A Real-World Scenario

Here's a scenario you'll actually face: a dashboard that loads from multiple APIs, one of which is a flaky third-party service you don't control.

// tests/network/error-simulation.spec.ts (add to existing file)
test('dashboard renders correctly even when analytics API is down', async ({ page }) => {
  // Core tasks API — mock with known data
  await page.route('**/api/tasks', async route => {
    await route.fulfill({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify([
        { id: 1, title: 'Write unit tests', status: 'pending' },
        { id: 2, title: 'Deploy to staging', status: 'completed' },
      ]),
    });
  });

  // User profile API — mock with known user
  await page.route('**/api/users/me', async route => {
    await route.fulfill({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify({ name: 'Faizal', role: 'admin' }),
    });
  });

  // Third-party analytics API — simulate it being completely down
  await page.route('**/analytics.thirdparty.com/**', async route => {
    await route.abort('failed');
  });

  await page.goto('/dashboard');

  // Core content must still render — the page should not crash
  await expect(page.getByText('Write unit tests')).toBeVisible();
  await expect(page.getByText('Hello, Faizal')).toBeVisible();

  // Analytics widget should degrade gracefully
  await expect(page.getByTestId('analytics-widget')).toContainText('Analytics unavailable');

  // No full-page error banner — just the widget degrades
  await expect(page.getByTestId('error-banner')).not.toBeVisible();
});

This is the kind of test that catches real production bugs. Most teams never write it — because without network interception, it's too hard to set up. With Playwright — it's under 30 lines. 🔥

📁 Final Project Structure After Part 2

Every file listed below has been fully built across Part 1 and Part 2:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts                   ✅ Part 1
│   ├── tasks/
│   │   └── task-management.spec.ts         ✅ Part 1
│   └── network/                            ✅ Part 2
│       ├── api-mocking.spec.ts
│       ├── error-simulation.spec.ts
│       └── network-assertions.spec.ts
├── pages/
│   ├── LoginPage.ts                        ✅ Part 1
│   └── TaskPage.ts                         ✅ Part 1 (createTask + deleteTask)
├── fixtures/
│   ├── auth.fixture.ts                     ✅ Part 1
│   ├── tasks.json                          ✅ Part 2
│   ├── empty-tasks.json                    ✅ Part 2
│   └── tasks-har.har                       ✅ Part 2 (generated via script)
├── scripts/
│   └── record-har.ts                       ✅ Part 2
├── .auth/                                  ← git-ignored, auto-generated
│   ├── admin.json
│   └── user.json
├── global-setup.ts                         ✅ Part 1
├── playwright.config.ts                    ✅ Part 1
├── .env                                    ← git-ignored
└── package.json

🗺️ What's Coming in This Series

Part 1 — Stop Writing Tests Like a Beginner              ✅ Done
Part 2 — Network Interception: The Complete Guide        ← You are here
Part 3 — Multi-User, Multi-Tab & Context Testing
Part 4 — API Testing (The Underrated Superpower)
Part 5 — Visual Regression Testing
Part 6 — Debugging Like a Pro: Trace Viewer & Inspector
Part 7 — The CI/CD Setup Nobody Shows You
Part 8 — Playwright Meets AI: Agents, MCP & Self-Healing Tests

In Part 3, we tackle scenarios most automation frameworks simply can't handle — testing with multiple users simultaneously, multi-tab flows, and real-time features like live notifications. Playwright's browser context architecture makes this surprisingly clean. And the storageState files we created in Part 1 become the foundation for it.

🔖 Before You Go

Network interception is the feature that makes Playwright genuinely powerful — not just as a UI testing tool, but as a full-stack testing platform.

Once you start owning the network layer in your tests:

You stop depending on backend availability
Your tests stop failing because of external APIs
You start catching frontend bugs that only appear under specific API conditions
Your test suite becomes fast, deterministic, and actually reliable

That's the goal. Not just tests that run — tests that you can trust. 💪

Follow me so you don't miss Part 3 — where we go deep on multi-user and multi-tab testing. Two browser contexts. One test. Real-time feature validation that most frameworks can't even attempt.

Drop a comment below 👇

Are you using page.route() in your current test suite?
What's the most painful external dependency in your tests right now?
Have you used HAR files before — or is that new to you?

Let's talk in the comments. 🙌

Faizal Shaikh | Senior Automation Engineer | Playwright & AI Testing
Connect with me on LinkedIn

The Playwright Playbook — Part 1: Stop Writing Playwright Tests Like a Beginner

Faizal — Mon, 15 Jun 2026 02:50:37 +0000

The Playwright Playbook — Part 1: Stop Writing Playwright Tests Like a Beginner

"Most people aren't writing bad tests on purpose. They just never learned the right way."

I've reviewed a lot of Playwright test suites over 7.6 years in QA automation.

And I keep seeing the same mistakes. Over and over.

Not from junior engineers. From seniors. From teams that have been using Playwright for years.

The tests run. The CI is green. But the suite is secretly fragile — one UI change away from mass failure, one new feature away from a 40-minute login loop, one scale-up away from complete chaos.

This series is called The Playwright Playbook for a reason.

Playbooks aren't for beginners learning how to play. They're for players who already know the basics — and want to win. 🏆

By the end of this 8-part series, you'll have a production-grade Playwright framework in TypeScript — with network interception, multi-user testing, API integration, visual regression, a CI/CD pipeline, and AI-powered test agents on top.

But first — we need to tear down what's broken.

Let's go. 🎯

🏗️ What We're Building in This Series

Before we write a single line of code — let me show you the app we're going to test throughout all 8 parts.

We'll use a simple Task Manager app as our test target. It has:

Login / logout
Create, edit, delete tasks
A REST API underneath
Role-based access (admin vs regular user)
Real-time task updates

Every part of this series will add a new testing layer on top of this same project. By Part 8 — we'll have a complete, professional test framework built around it.

Here's the project structure we'll have by the end of Part 1:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts
│   └── tasks/
│       └── task-management.spec.ts
├── pages/
│   ├── LoginPage.ts
│   └── TaskPage.ts
├── fixtures/
│   └── auth.fixture.ts
├── .auth/
│   ├── admin.json
│   └── user.json
├── global-setup.ts
├── playwright.config.ts
├── .env
└── package.json

Clean. Scalable. Ready to grow. Let's build every file. 👇

❌ Mistake #1 — Fragile Selectors

This is the most common mistake in every codebase I've ever reviewed.

The bad way:

// 🔴 Brittle — breaks on any CSS refactor
await page.click('.btn-primary.submit-button-v2');

// 🔴 Brittle — breaks when DOM structure changes
await page.click('div > form > div:nth-child(3) > button');

// 🔴 Brittle — relies on auto-generated class names
await page.fill('#input_38', 'john@test.com');

These selectors are tied to implementation details. The moment a developer renames a class, restructures the DOM, or upgrades a UI library — your tests break. Not because the feature is broken. Because your test is brittle.

The right way — Playwright's built-in locators:

// ✅ Finds by accessible role — survives CSS changes
await page.getByRole('button', { name: 'Sign in' }).click();

// ✅ Finds by label — semantic and resilient
await page.getByLabel('Email address').fill('john@test.com');

// ✅ Finds by placeholder text
await page.getByPlaceholder('Enter your password').fill('secret123');

// ✅ Finds by visible text
await page.getByText('Welcome back, John').waitFor();

// ✅ The best option when you control the code — test IDs never change
await page.getByTestId('submit-btn').click();

getByRole, getByLabel, getByTestId — these are semantic locators. They describe what the element IS, not what it looks like. They survive redesigns. They survive framework upgrades. They survive the inevitable "quick CSS cleanup" from the frontend team.

Rule of thumb: If a non-technical person couldn't describe the element using your selector — it's probably fragile. 🎯

❌ Mistake #2 — Logging In Through the UI on Every Test

This one kills performance and introduces flakiness at the same time.

The bad way:

// 🔴 Every single test file has this at the top
test.beforeEach(async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill('user@test.com');
  await page.getByLabel('Password').fill('password123');
  await page.getByRole('button', { name: 'Sign in' }).click();
  await page.waitForURL('/dashboard');
});

Multiply this by 50 tests. That's 50 UI login flows. Each one making real network requests, waiting for page loads, and praying the login page doesn't have a hiccup today.

The right way — storageState + globalSetup:

Login once before the entire suite runs. Save the session to a file. Every test picks it up and starts already authenticated.

// global-setup.ts
import { chromium } from '@playwright/test';
import * as dotenv from 'dotenv';

dotenv.config();

async function globalSetup() {
  const browser = await chromium.launch();

  // Save admin session
  const adminContext = await browser.newContext();
  const adminPage = await adminContext.newPage();
  await adminPage.goto(`${process.env.BASE_URL}/login`);
  await adminPage.getByLabel('Email').fill(process.env.ADMIN_EMAIL!);
  await adminPage.getByLabel('Password').fill(process.env.ADMIN_PASSWORD!);
  await adminPage.getByRole('button', { name: 'Sign in' }).click();
  await adminPage.waitForURL('/dashboard');
  await adminContext.storageState({ path: '.auth/admin.json' });
  await adminContext.close();

  // Save regular user session
  const userContext = await browser.newContext();
  const userPage = await userContext.newPage();
  await userPage.goto(`${process.env.BASE_URL}/login`);
  await userPage.getByLabel('Email').fill(process.env.USER_EMAIL!);
  await userPage.getByLabel('Password').fill(process.env.USER_PASSWORD!);
  await userPage.getByRole('button', { name: 'Sign in' }).click();
  await userPage.waitForURL('/dashboard');
  await userContext.storageState({ path: '.auth/user.json' });
  await userContext.close();

  await browser.close();
}

export default globalSetup;

❌ Mistake #3 — No Project Structure (Everything in One File)

I've seen test files with 800 lines. Every test copy-pasting the same page.goto, page.fill, page.click sequences.

When the URL changes — you update it in 47 places. When the selector changes — same story.

The right way — Page Object Model (POM) in TypeScript:

// pages/LoginPage.ts
import { Page, Locator } from '@playwright/test';

export class LoginPage {
  readonly page: Page;
  readonly emailInput: Locator;
  readonly passwordInput: Locator;
  readonly signInButton: Locator;
  readonly errorMessage: Locator;

  constructor(page: Page) {
    this.page = page;
    this.emailInput = page.getByLabel('Email address');
    this.passwordInput = page.getByLabel('Password');
    this.signInButton = page.getByRole('button', { name: 'Sign in' });
    this.errorMessage = page.getByTestId('login-error');
  }

  async goto() {
    await this.page.goto('/login');
  }

  async login(email: string, password: string) {
    await this.emailInput.fill(email);
    await this.passwordInput.fill(password);
    await this.signInButton.click();
  }

  async loginAndWait(email: string, password: string) {
    await this.login(email, password);
    await this.page.waitForURL('/dashboard');
  }
}

// pages/TaskPage.ts
import { Page, Locator } from '@playwright/test';

export class TaskPage {
  readonly page: Page;
  readonly newTaskButton: Locator;
  readonly taskTitleInput: Locator;
  readonly saveTaskButton: Locator;

  constructor(page: Page) {
    this.page = page;
    this.newTaskButton = page.getByRole('button', { name: 'New Task' });
    this.taskTitleInput = page.getByLabel('Task title');
    this.saveTaskButton = page.getByRole('button', { name: 'Save task' });
  }

  async goto() {
    await this.page.goto('/tasks');
  }

  async createTask(title: string) {
    await this.newTaskButton.click();
    await this.taskTitleInput.fill(title);
    await this.saveTaskButton.click();
  }

  async deleteTask(title: string) {
    const taskItem = this.getTaskLocator(title);
    // Hover to reveal the delete button
    await taskItem.hover();
    await taskItem.getByRole('button', { name: 'Delete' }).click();
    // Confirm deletion in the dialog
    await this.page.getByRole('button', { name: 'Confirm' }).click();
  }

  getTaskLocator(title: string): Locator {
    return this.page.getByRole('listitem').filter({ hasText: title });
  }
}

Note the deleteTask() method — we'll need this in Part 2 when we assert on DELETE network calls. Build it once, use it everywhere. 💪

❌ Mistake #4 — Hard-Coded Waits

Nothing says "I don't trust my tests" like waitForTimeout.

The bad way:

// 🔴 Hoping 3 seconds is enough. It isn't. On a slow CI machine, it never is.
await page.waitForTimeout(3000);
await page.click('#submit');

// 🔴 Even worse — guessing at load time
await page.goto('/dashboard');
await page.waitForTimeout(5000);
await expect(page.locator('.tasks-list')).toBeVisible();

Hard-coded waits are the definition of flaky. Fast machine? Test passes. Slow CI? Fails. High network load? Fails. The test isn't measuring anything — it's just hoping.

The right way — Playwright's auto-wait and explicit conditions:

// ✅ Wait for a specific element to appear
await page.getByTestId('tasks-list').waitFor({ state: 'visible' });

// ✅ Wait for a network response AND the action that triggers it — together
await Promise.all([
  page.waitForResponse(resp => resp.url().includes('/api/tasks') && resp.status() === 200),
  page.getByRole('button', { name: 'Load tasks' }).click(),
]);

// ✅ Wait for URL to change — confirms navigation completed
await page.waitForURL('/dashboard');

// ✅ Wait for load state — page fully loaded
await page.waitForLoadState('networkidle');

// ✅ Playwright auto-waits before most actions — this just works
await expect(page.getByRole('heading', { name: 'My Tasks' })).toBeVisible();

Playwright's built-in locator methods already auto-wait. click(), fill(), check() — they all wait for the element to be actionable before acting. You rarely need to manually wait for anything.

If you find yourself writing waitForTimeout — stop. Find the condition you're actually waiting for. Wait for that instead. 🎯

❌ Mistake #5 — Hardcoded Config Everywhere

// 🔴 Scattered across 30 test files
await page.goto('http://localhost:3000/login');
await request.post('http://localhost:3000/api/tasks');

The moment you need to run tests against staging — you're doing a find-and-replace across your entire codebase.

The right way — centralized playwright.config.ts + .env:

# .env  (git-ignored)
BASE_URL=http://localhost:3000
API_URL=http://localhost:3000/api
ADMIN_EMAIL=admin@test.com
ADMIN_PASSWORD=admin123
USER_EMAIL=user@test.com
USER_PASSWORD=user123

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';
import * as dotenv from 'dotenv';

dotenv.config();

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 4 : undefined,
  globalSetup: './global-setup.ts',

  reporter: [
    ['html', { open: 'never' }],
    ['list'],
  ],

  use: {
    baseURL: process.env.BASE_URL || 'http://localhost:3000',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    trace: 'on-first-retry',
  },

  projects: [
    {
      name: 'admin',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/admin.json',
      },
    },
    {
      name: 'user',
      use: {
        ...devices['Desktop Chrome'],
        storageState: '.auth/user.json',
      },
    },
  ],
});

Now your tests use relative paths — works in any environment:

// ✅ baseURL resolved from config — one change switches environments
await page.goto('/login');
await page.goto('/dashboard');

❌ Mistake #6 — Assertions That Don't Actually Assert Anything

// 🔴 This passes even if the element is invisible, disabled, or wrong
const element = await page.locator('.task-item');
expect(element).toBeTruthy(); // The locator object ALWAYS exists!

// 🔴 No assertion after an action — just hoping it worked
await page.click('button[type="submit"]');

The right way — Playwright's web-first assertions:

// ✅ Checks actual visibility in the DOM
await expect(page.getByTestId('task-item')).toBeVisible();

// ✅ Checks the actual text content
await expect(page.getByRole('heading')).toHaveText('My Tasks');

// ✅ Checks element count
await expect(page.getByRole('listitem')).toHaveCount(3);

// ✅ Checks URL after navigation
await expect(page).toHaveURL('/dashboard');

// ✅ Checks input value
await expect(page.getByLabel('Task title')).toHaveValue('Write unit tests');

// ✅ Negative assertion
await expect(page.getByTestId('error-message')).not.toBeVisible();

// ✅ Checks element is disabled
await expect(page.getByRole('button', { name: 'Save' })).toBeDisabled();

Playwright's expect() assertions are web-first — they automatically retry until the condition is met or the timeout is reached. No manual waiting. No false passes.

❌ Mistake #7 — Tests That Depend on Each Other

// 🔴 Test 2 depends on Test 1 having run first
test('create a task', async ({ page }) => {
  await taskPage.createTask('Buy groceries');
});

test('delete the task', async ({ page }) => {
  // What if Test 1 failed? Or ran in a different order?
  await taskPage.deleteTask('Buy groceries');
});

Dependent tests are a maintenance nightmare. Run them in parallel — they break. Run them in a different order — they break. One failure cascades into many.

The right way — fully independent, self-contained tests:

// ✅ Each test owns its full lifecycle
test('user can delete a task', async ({ page }) => {
  const taskPage = new TaskPage(page);

  // Arrange — create the data this test needs
  await taskPage.goto();
  await taskPage.createTask('Temporary task for deletion test');
  await expect(taskPage.getTaskLocator('Temporary task for deletion test')).toBeVisible();

  // Act
  await taskPage.deleteTask('Temporary task for deletion test');

  // Assert
  await expect(taskPage.getTaskLocator('Temporary task for deletion test')).not.toBeVisible();
});

Every test owns its lifecycle. Setup → action → assertion → done. No shared state. No assumptions about what ran before. ✅

🏁 Building the Remaining Files

We have the POM and config. Now let's complete the rest of the project structure.

The Auth Fixture

Playwright fixtures are how you share setup logic cleanly across tests without beforeEach soup. Here we create a fixture that gives tests a pre-initialised TaskPage tied to a specific role.

// fixtures/auth.fixture.ts
import { test as base } from '@playwright/test';
import { TaskPage } from '../pages/TaskPage';
import { LoginPage } from '../pages/LoginPage';

type AuthFixtures = {
  taskPage: TaskPage;
  loginPage: LoginPage;
};

export const test = base.extend<AuthFixtures>({
  taskPage: async ({ page }, use) => {
    const taskPage = new TaskPage(page);
    await taskPage.goto();
    await use(taskPage);
  },

  loginPage: async ({ page }, use) => {
    const loginPage = new LoginPage(page);
    await loginPage.goto();
    await use(loginPage);
  },
});

export { expect } from '@playwright/test';

Now tests import from the fixture instead of @playwright/test directly:

// Instead of:
import { test, expect } from '@playwright/test';

// Use:
import { test, expect } from '../../fixtures/auth.fixture';

The Login Spec

// tests/auth/login.spec.ts
import { test, expect } from '../../fixtures/auth.fixture';
import { LoginPage } from '../../pages/LoginPage';

// This test deliberately does NOT use storageState
// It tests the login flow itself
test.use({ storageState: { cookies: [], origins: [] } });

test('successful login redirects to dashboard', async ({ page }) => {
  const loginPage = new LoginPage(page);
  await loginPage.goto();
  await loginPage.loginAndWait(
    process.env.USER_EMAIL!,
    process.env.USER_PASSWORD!
  );
  await expect(page).toHaveURL('/dashboard');
  await expect(page.getByRole('heading', { name: 'My Tasks' })).toBeVisible();
});

test('invalid credentials show error message', async ({ page }) => {
  const loginPage = new LoginPage(page);
  await loginPage.goto();
  await loginPage.login('wrong@test.com', 'wrongpassword');

  await expect(loginPage.errorMessage).toBeVisible();
  await expect(loginPage.errorMessage).toHaveText('Invalid email or password');
  // Should stay on login page — no redirect
  await expect(page).toHaveURL('/login');
});

test('empty fields show validation errors', async ({ page }) => {
  const loginPage = new LoginPage(page);
  await loginPage.goto();
  // Click sign in without filling anything
  await loginPage.signInButton.click();

  await expect(page.getByText('Email is required')).toBeVisible();
  await expect(page.getByText('Password is required')).toBeVisible();
});

Notice test.use({ storageState: { cookies: [], origins: [] } }) at the top — this clears the stored auth session for this file only, so we can actually test the login flow without being auto-authenticated. 🔑

The Task Management Spec

// tests/tasks/task-management.spec.ts
import { test, expect } from '../../fixtures/auth.fixture';

test('user can create a new task', async ({ taskPage }) => {
  await taskPage.createTask('Write integration tests');
  await expect(taskPage.getTaskLocator('Write integration tests')).toBeVisible();
});

test('user can delete a task', async ({ taskPage }) => {
  // Arrange
  await taskPage.createTask('Task to delete');
  await expect(taskPage.getTaskLocator('Task to delete')).toBeVisible();

  // Act
  await taskPage.deleteTask('Task to delete');

  // Assert
  await expect(taskPage.getTaskLocator('Task to delete')).not.toBeVisible();
});

test('task list shows correct count after creation', async ({ taskPage, page }) => {
  const initialCount = await page.getByRole('listitem').count();

  await taskPage.createTask('Count test task');

  await expect(page.getByRole('listitem')).toHaveCount(initialCount + 1);
});

Clean. No boilerplate. The fixture handles the navigation and page object setup. Tests just test. ✅

📁 Final Project Structure After Part 1

Every file listed here has been fully built in this article:

playwright-playbook/
├── tests/
│   ├── auth/
│   │   └── login.spec.ts          ✅ built above
│   └── tasks/
│       └── task-management.spec.ts ✅ built above
├── pages/
│   ├── LoginPage.ts               ✅ built above
│   └── TaskPage.ts                ✅ built above (includes deleteTask)
├── fixtures/
│   └── auth.fixture.ts            ✅ built above
├── .auth/                         ← git-ignored, auto-generated
│   ├── admin.json
│   └── user.json
├── global-setup.ts                ✅ built above
├── playwright.config.ts           ✅ built above
├── .env                           ← git-ignored
├── .env.example                   ← commit this to repo
└── package.json

🗺️ What's Coming in This Series

Part 1 — Stop Writing Tests Like a Beginner              ← You are here
Part 2 — Network Interception: The Complete Guide
Part 3 — Multi-User, Multi-Tab & Context Testing
Part 4 — API Testing (The Underrated Superpower)
Part 5 — Visual Regression Testing
Part 6 — Debugging Like a Pro: Trace Viewer & Inspector
Part 7 — The CI/CD Setup Nobody Shows You
Part 8 — Playwright Meets AI: Agents, MCP & Self-Healing Tests

In Part 2, we go deep on network interception — mocking APIs, simulating 500 errors, asserting on real API payloads, and recording HAR files. The TaskPage we built here (including deleteTask) will be used to assert on exactly what network calls Playwright fires when you interact with the app.

🎯 Who Is This Series For?

QA engineers who are already using Playwright but know their suite could be better
Automation engineers who want to level up from "it works" to "it scales"
Developers who write Playwright tests and want to do it properly
Anyone who has inherited a Playwright codebase and wondered why it keeps breaking

No beginner content here. We skip the "what is Playwright" intro.

If you can write a test — this series will make you dangerous. 🔥

🔖 Before You Go

Every pattern in this article is something I've fixed in a real codebase.

The hard selectors. The login loops. The waitForTimeout scattered everywhere. The 800-line test files. I've seen all of it — and I've cleaned all of it.

The good news: fixing these isn't complicated. It's just about knowing the right patterns.

Now you know them. 💪

Follow me so you don't miss Part 2 — where we get hands-on with Playwright's most underused feature: network interception. We'll mock APIs, simulate 500 errors, and test frontend behaviour without touching a single line of backend code.

Drop a comment below 👇

Which of these mistakes is your team making right now?
What's the most painful part of maintaining your Playwright suite?
Or are you starting fresh — building a new framework from scratch?

All levels welcome here. Let's build something solid together. 🙌

Faizal Shaikh | Senior Automation Engineer | Playwright & AI Testing
Connect with me on LinkedIn

How API Testing Levelled Up My QA Career (And Why Most Engineers Skip It)

Faizal — Sat, 13 Jun 2026 18:52:51 +0000

The Moment I Realised UI Testing Wasn't Enough

Three years into my QA career, I thought I was doing well.

I had a solid Selenium suite running. Regression coverage was green. Stakeholders were happy.

Then a production incident happened.

A payment API was returning incorrect amounts under a specific condition. The UI looked perfect — amounts displayed correctly after rounding. But the raw API response? Off by a significant margin.

My entire test suite missed it. Every single test.

Because I was only testing what users saw. Not what the system was actually doing.

That incident changed how I approached QA forever. 👇

Why API Testing Is the Most Underrated Skill in QA

Let me be direct about something.

Most QA engineers treat API testing as a secondary skill. Something you do with Postman when a developer asks you to verify an endpoint. A quick sanity check before moving on.

That's the wrong mental model entirely.

Here's the truth after 7.5 years:

The API layer is where your product actually lives.

The UI is a presentation layer. It shows users a version of the truth. But the API? That's the truth itself. Data contracts, business logic, validation rules, error handling — all of it lives at the API layer.

If you're only testing the UI, you're testing the packaging. Not the product.

My API Testing Journey — Tool by Tool

Let me walk you through exactly how my API testing practice evolved, and what each tool actually taught me.

Stage 1 — Postman: Learning to Think in Requests

Postman was my entry point. And it's still the tool I reach for first when exploring a new API.

But most people use Postman wrong.

They treat it like a manual testing tool — fire a request, check the response, move on. That's wasting 80% of what Postman can do.

Here's how I actually use it:

Collections + Environments = your real power combo

// Environment variables — not hardcoded values
{{base_url}}/api/v1/users/{{user_id}}

// Switch between dev/staging/prod by changing one environment
// Not by editing every single request

Pre-request scripts for dynamic data:

// Generate a unique email for every test run
const timestamp = Date.now();
pm.environment.set("test_email", `testuser_${timestamp}@example.com`);

Test scripts for automated assertions:

pm.test("Status code is 200", () => {
    pm.response.to.have.status(200);
});

pm.test("User ID matches", () => {
    const response = pm.response.json();
    pm.expect(response.data.id).to.eql(pm.environment.get("user_id"));
});

pm.test("Response time under 500ms", () => {
    pm.expect(pm.response.responseTime).to.be.below(500);
});

What Postman taught me:
Thinking in terms of contracts. Every API has an implied contract — here's what you send, here's what you get back. My job as a tester is to verify that contract holds under every condition.

Stage 2 — Charles Proxy: Seeing What the System Hides

Postman is great when you know what API you're testing.

But what about the APIs you don't know about?

Charles Proxy changed my debugging game completely. I started opening it before I even wrote my first test on any new feature.

Here's my actual workflow:

Open Charles, start capturing
Perform the user action in the app (or mobile device via proxy)
See every network call — including the ones nobody told me about
Build my test suite from what I actually observe, not just what's in the Swagger docs

The Swagger docs lie. Or they're outdated. Or they're incomplete. Charles never lies.

Where Charles became indispensable:

Scenario: Mobile checkout flow
What Swagger documented: 3 API calls
What Charles actually showed: 7 API calls

The undocumented calls included:
- A fraud scoring API call
- An inventory reservation call  
- A session refresh call

All three were failure points nobody had tested.

Breakpoints for on-the-fly request modification:

Set a breakpoint on any request. Intercept it before it hits the server. Modify the payload. Fire it. See what happens.

This is how I test edge cases that would take hours to set up in a test environment — in minutes.

Stage 3 — Playwright for API Testing: Automation That Actually Scales

Manual Postman and Charles get you far. But they don't scale.

When I needed repeatable, CI-friendly API tests, I moved to Playwright's API testing capabilities. Most people don't know Playwright does this — they think of it as a browser tool only.

import { test, expect } from '@playwright/test';

test.describe('User API', () => {

    let authToken;

    test.beforeAll(async ({ request }) => {
        // Authenticate once, reuse token across all tests
        const response = await request.post('/api/auth/login', {
            data: {
                email: 'test@example.com',
                password: 'testpassword'
            }
        });
        const body = await response.json();
        authToken = body.token;
    });

    test('GET user profile returns correct schema', async ({ request }) => {
        const response = await request.get('/api/v1/users/profile', {
            headers: {
                'Authorization': `Bearer ${authToken}`
            }
        });

        expect(response.status()).toBe(200);

        const body = await response.json();

        // Schema validation
        expect(body).toHaveProperty('id');
        expect(body).toHaveProperty('email');
        expect(body).toHaveProperty('role');
        expect(typeof body.id).toBe('number');
        expect(typeof body.email).toBe('string');
    });

    test('POST create user — validates required fields', async ({ request }) => {
        const response = await request.post('/api/v1/users', {
            headers: { 'Authorization': `Bearer ${authToken}` },
            data: {
                // Missing required 'email' field intentionally
                name: 'Test User'
            }
        });

        expect(response.status()).toBe(400);

        const body = await response.json();
        expect(body.error).toContain('email');
    });

    test('Response time SLA — under 800ms', async ({ request }) => {
        const start = Date.now();

        await request.get('/api/v1/users/profile', {
            headers: { 'Authorization': `Bearer ${authToken}` }
        });

        const duration = Date.now() - start;
        expect(duration).toBeLessThan(800);
    });
});

What makes Playwright API testing powerful:

Runs in the same framework as your UI tests — one suite, one report
Built-in request context — no extra HTTP client libraries
Parallel execution out of the box
Integrates cleanly with CI/CD

Stage 4 — DB Validation: The Layer Nobody Tests

This is the one most automation engineers completely skip.

Here's the problem with testing only at the API layer:

POST /api/v1/orders
Response: { "status": "success", "order_id": "ORD-123" }
Test result: ✅ PASS

But did the order actually get written to the database correctly?
Did all the fields persist as expected?
Did the foreign key relationships hold?
Did any triggers fire that modified related records?

Your API test doesn't know. Your DB test does.

import { test, expect } from '@playwright/test';
import { Client } from 'pg'; // PostgreSQL client

test('Order creation — validates DB persistence', async ({ request }) => {

    // Step 1: Create order via API
    const response = await request.post('/api/v1/orders', {
        headers: { 'Authorization': `Bearer ${authToken}` },
        data: {
            product_id: 42,
            quantity: 2,
            user_id: 101
        }
    });

    expect(response.status()).toBe(201);
    const { order_id } = await response.json();

    // Step 2: Validate at DB layer
    const db = new Client({ connectionString: process.env.DB_URL });
    await db.connect();

    const result = await db.query(
        'SELECT * FROM orders WHERE order_id = $1',
        [order_id]
    );

    expect(result.rows.length).toBe(1);

    const order = result.rows[0];
    expect(order.product_id).toBe(42);
    expect(order.quantity).toBe(2);
    expect(order.user_id).toBe(101);
    expect(order.status).toBe('pending'); // Business logic check

    await db.end();
});

This two-layer approach has caught bugs that API testing alone never would:

Records created with wrong foreign keys
Decimal precision lost during persistence
Audit log entries missing
Cache not invalidated after write

Stage 5 — JMeter: When "Does It Work?" Isn't Enough

All the tests above answer one question: does it work?

JMeter answers a different question: does it work under load?

I've been using JMeter for performance testing for years. And the pattern I see constantly is this — APIs that pass every functional test absolutely fall apart at scale.

My standard performance test structure for any API:

Thread Group: 100 concurrent users
Ramp-up: 60 seconds
Duration: 5 minutes

HTTP Request: POST /api/v1/orders
  → CSV Data Set Config (unique test data per user)
  → HTTP Header Manager (auth tokens)

Listeners:
  → Summary Report (throughput, error rate)
  → Response Time Graph
  → View Results Tree (for debugging failures)

Assertions:
  → Response time < 1000ms (95th percentile)
  → Error rate < 1%

The bugs JMeter found that nothing else did:

A search API that handled 10 concurrent users fine — degraded to 30-second responses at 50 users (missing database index)
A session API with a race condition that only appeared when 20+ users hit it simultaneously
A payment API that started returning 500s after sustained load for 10 minutes (connection pool exhaustion)

None of these would've been caught by functional tests alone.

The Career Impact — What API Testing Actually Changed

Here's what shifting my focus to API testing actually did for my career:

I became the person who caught things before production.

Not because I'm smarter than anyone else. Because I was looking at a layer most testers weren't looking at.

I got closer to developers.

API testing made me fluent in the language developers speak. Endpoints, payloads, status codes, contracts, performance SLAs. When you speak that language, developers start treating you as a peer — not a gatekeeper.

I became harder to replace.

A tester who only does UI automation is replaceable. A tester who understands the full stack — UI, API, performance, database — is a different kind of engineer entirely.

It prepared me for AI testing.

Everything I'm doing now with RAG-based testing and LLM validation is built on top of API testing fundamentals. AI systems are, at their core, APIs. The mental model transfers directly.

Where to Start If You're Behind on API Testing

If you're reading this and realising your API testing practice needs work — here's the honest starting point:

Week 1: Learn Postman properly. Collections, environments, test scripts. Newman for CLI runs.

Week 2: Set up Charles Proxy. Intercept a real app's traffic. Map every API call for one feature.

Week 3: Write your first Playwright API test suite for a real endpoint in your project.

Week 4: Add DB validation for your most critical write operations.

Month 2: Set up a JMeter test for your highest-traffic endpoint. Run it. See what breaks.

You don't need to do all of this overnight. But you need to start. Because the gap between teams that do this and teams that don't is enormous — and it shows up directly in production.

Final Thought

The best QA engineers I've ever worked with share one trait.

They're deeply curious about how things work — not just whether things work.

API testing is what happens when that curiosity meets a systematic approach.

It's not glamorous. There's no visual feedback, no screenshots, no UI to look at. Just raw data flowing between systems — and your job to make sure every byte of it is exactly right.

That's the work. And in my experience, it's the most valuable work in QA.

Faizal Shaikh | Senior Automation Engineer | AI & RAG-Based Testing
Connect with me on LinkedIn

Building a complete RAG Testing Series on Dev.to — follow along if you're curious about testing AI systems.

RAG-Based Testing Series — Part 6: Automating RAG Quality Checks in CI/CD

Faizal — Fri, 12 Jun 2026 02:38:07 +0000

RAG-Based Testing Series — Part 6: Automating RAG Quality Checks in CI/CD

"A test that only runs when you remember to run it isn't really a test. It's a hope."

We've built something real over this series.

In Part 2, we gave retrieval quality a number — Precision@K, Recall@K, MRR.

In Part 3, we gave hallucination detection a number — faithfulness scoring with RAGAS.

In Part 4, we tested the edge cases that break RAG systems in production.

In Part 5, we assembled all of that into a structured, reusable framework with one command to run everything.

But there's still a problem. 🔴

The framework only runs when someone decides to run it.

And in a real team, "someone will run the tests before deploying" is not a guarantee. It's an assumption. And assumptions fail at the worst possible moments.

Someone updates the knowledge base at 5pm on a Friday.
Someone tweaks the system prompt and doesn't realise it changed retrieval behaviour.
Someone upgrades the embedding model and the similarity scores shift quietly.

None of these trigger a test run. None of these get caught. And your users discover the regression before your team does.

Part 6 fixes this.

We're wiring the framework from Part 5 into a GitHub Actions CI/CD pipeline so that RAG quality checks run automatically — on every relevant change, without anyone having to remember. 🤖

🗺️ What We're Building

By the end of this article, you'll have:

.github/
└── workflows/
    └── rag_quality_checks.yml   ← GitHub Actions workflow

rag_test_framework/
├── config/
│   └── settings.py
├── core/
│   ├── retriever.py
│   ├── evaluator.py
│   └── rag_pipeline.py
├── tests/
│   ├── conftest.py
│   ├── test_retrieval.py
│   ├── test_faithfulness.py
│   └── test_edge_cases.py
├── data/
│   └── test_cases.json
├── reports/
│   └── (auto-generated, uploaded as CI artifacts)
├── run_tests.py
└── requirements.txt

The workflow will:

Trigger automatically on pushes that touch relevant files
Install dependencies
Run the full test suite
Upload the test report as a downloadable artifact
Post a summary to the GitHub Actions summary page
Block the pipeline if any test fails — no silent regressions

Let's build it step by step. 🛠️

⚙️ Step 1 — Store Secrets Safely

Your framework needs an OpenAI API key. You never hardcode secrets in a repository.

In GitHub:

Go to your repository → Settings → Secrets and variables → Actions
Click New repository secret
Name: OPENAI_API_KEY
Value: your actual OpenAI API key

That's it. GitHub encrypts it. Your workflow accesses it as ${{ secrets.OPENAI_API_KEY }} — never exposed in logs or code.

Now update config/settings.py to read from the environment variable (this already works locally too if you set it with export OPENAI_API_KEY=...):

# config/settings.py

import os

OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")

if not OPENAI_API_KEY:
    raise EnvironmentError(
        "OPENAI_API_KEY environment variable is not set.\n"
        "Set it locally with: export OPENAI_API_KEY=your-key\n"
        "In CI, add it as a GitHub Actions secret."
    )

Failing loudly with a clear message is better than failing cryptically with an authentication error three steps later. ✅

📄 Step 2 — The GitHub Actions Workflow

Create this file at .github/workflows/rag_quality_checks.yml:

name: RAG Quality Checks

on:
  push:
    paths:
      # Run when test cases or knowledge base changes
      - 'rag_test_framework/data/**'
      # Run when any core framework code changes
      - 'rag_test_framework/core/**'
      # Run when configuration (thresholds, models) changes
      - 'rag_test_framework/config/**'
      # Run when tests themselves change
      - 'rag_test_framework/tests/**'
      # Run when dependencies change
      - 'rag_test_framework/requirements.txt'

  pull_request:
    paths:
      - 'rag_test_framework/data/**'
      - 'rag_test_framework/core/**'
      - 'rag_test_framework/config/**'
      - 'rag_test_framework/tests/**'
      - 'rag_test_framework/requirements.txt'

  # Allow manual trigger from the GitHub Actions UI
  workflow_dispatch:

jobs:
  rag-quality-checks:
    name: RAG Quality Checks
    runs-on: ubuntu-latest

    steps:
      # ── 1. Check out the repository ──────────────────────────
      - name: Checkout repository
        uses: actions/checkout@v4

      # ── 2. Set up Python ─────────────────────────────────────
      - name: Set up Python 3.11
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'   # cache pip installs between runs to speed up the workflow

      # ── 3. Install dependencies ───────────────────────────────
      - name: Install dependencies
        working-directory: rag_test_framework
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      # ── 4. Run the RAG test suite ─────────────────────────────
      - name: Run RAG quality checks
        working-directory: rag_test_framework
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          mkdir -p reports
          pytest tests/ \
            -v \
            --tb=short \
            --json-report \
            --json-report-file=reports/rag_test_report.json \
            --json-report-summary

      # ── 5. Upload report as a downloadable artifact ───────────
      - name: Upload test report
        if: always()   # upload even if tests failed — you want the report either way
        uses: actions/upload-artifact@v4
        with:
          name: rag-test-report-${{ github.run_number }}
          path: rag_test_framework/reports/rag_test_report.json
          retention-days: 30

      # ── 6. Post summary to GitHub Actions summary page ────────
      - name: Post test summary
        if: always()
        working-directory: rag_test_framework
        run: python ci/post_summary.py reports/rag_test_report.json

📋 Step 3 — The Summary Script

The workflow calls ci/post_summary.py to write a clean summary to GitHub's built-in job summary page. Create that file now:

# ci/post_summary.py

import json
import os
import sys


def post_summary(report_path: str):
    """
    Read the pytest JSON report and write a markdown summary
    to the GitHub Actions step summary page (GITHUB_STEP_SUMMARY).
    """
    if not os.path.exists(report_path):
        print(f"Report not found at {report_path}")
        sys.exit(1)

    with open(report_path) as f:
        report = json.load(f)

    summary  = report.get("summary", {})
    passed   = summary.get("passed", 0)
    failed   = summary.get("failed", 0)
    error    = summary.get("error", 0)
    total    = summary.get("total", 0)
    duration = round(report.get("duration", 0), 2)

    # Determine overall status
    if failed > 0 or error > 0:
        status_icon  = "❌"
        status_label = "FAILED"
    else:
        status_icon  = "✅"
        status_label = "PASSED"

    # Build the markdown summary
    lines = [
        f"## {status_icon} RAG Quality Checks — {status_label}",
        "",
        "| Metric | Value |",
        "|--------|-------|",
        f"| Total tests | {total} |",
        f"| Passed      | {passed} |",
        f"| Failed      | {failed} |",
        f"| Duration    | {duration}s |",
        "",
    ]

    if failed > 0 or error > 0:
        lines.append("### ❌ Failed Tests")
        lines.append("")
        for test in report.get("tests", []):
            if test["outcome"] in ("failed", "error"):
                lines.append(f"- `{test['nodeid']}`")
                # Include the failure message if available
                if "call" in test and "longrepr" in test["call"]:
                    # Truncate long failure output for readability
                    longrepr = test["call"]["longrepr"]
                    preview  = longrepr[:500] + "..." if len(longrepr) > 500 else longrepr
                    lines.append(f"  ```
{% endraw %}
\n  {preview}\n
{% raw %}
  ```")
        lines.append("")

    lines += [
        "### Test Breakdown",
        "",
        "| Test File | Tests | Status |",
        "|-----------|-------|--------|",
    ]

    # Group tests by file for the breakdown table
    file_results: dict = {}
    for test in report.get("tests", []):
        file_name = test["nodeid"].split("::")[0]
        if file_name not in file_results:
            file_results[file_name] = {"total": 0, "failed": 0}
        file_results[file_name]["total"] += 1
        if test["outcome"] in ("failed", "error"):
            file_results[file_name]["failed"] += 1

    for file_name, counts in file_results.items():
        icon = "✅" if counts["failed"] == 0 else "❌"
        lines.append(f"| `{file_name}` | {counts['total']} | {icon} |")

    summary_text = "\n".join(lines)

    # Write to GitHub step summary if running in CI
    github_summary_path = os.environ.get("GITHUB_STEP_SUMMARY")
    if github_summary_path:
        with open(github_summary_path, "a") as f:
            f.write(summary_text)
        print("✅ Summary written to GitHub Actions step summary.")
    else:
        # Running locally — just print it
        print(summary_text)


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python ci/post_summary.py <path-to-report.json>")
        sys.exit(1)
    post_summary(sys.argv[1])

🔍 Step 4 — Understanding the Trigger Strategy

The paths filter in the workflow is one of the most important design decisions here. Let me explain why it's set up this way.

on:
  push:
    paths:
      - 'rag_test_framework/data/**'      # knowledge base changed
      - 'rag_test_framework/core/**'      # retrieval or pipeline logic changed
      - 'rag_test_framework/config/**'    # thresholds or models changed
      - 'rag_test_framework/tests/**'     # tests themselves changed
      - 'rag_test_framework/requirements.txt'

Why not trigger on every push?

RAG quality tests are expensive. Each test run calls the OpenAI API for embeddings and RAGAS evaluation. Running on every push to every file — including README changes, frontend code, unrelated scripts — wastes time and money.

What actually warrants a RAG quality check?

Change	Should trigger?	Why
`data/test_cases.json` updated	✅ Yes	Ground truth changed — verify scores still hold
New document added to knowledge base	✅ Yes	Retrieval behaviour may shift
`config/settings.py` thresholds changed	✅ Yes	You're redefining what "passing" means
Embedding model changed	✅ Yes	Similarity scores will shift
System prompt changed	✅ Yes	LLM behaviour may change
README.md updated	❌ No	Documentation only
Frontend code changed	❌ No	No impact on RAG pipeline

The paths filter implements exactly this logic. Only relevant changes trigger the quality gate. 🎯

🚦 Step 5 — What Happens When Tests Fail

This is important to understand clearly.

When pytest exits with a non-zero return code (i.e., any test fails), GitHub Actions automatically marks the job as failed. You don't need to add any special logic for this.

What that means in practice:

On a push to main:
The commit is recorded but the workflow run is marked ❌ Failed. Your team sees it immediately in the repository's commit history.

On a pull request:
The PR's status checks show ❌ RAG Quality Checks — Failed. You can configure branch protection rules to block merging until this passes.

Setting up branch protection (strongly recommended):

Go to repository → Settings → Branches
Add a branch protection rule for main
Enable Require status checks to pass before merging
Add RAG Quality Checks as a required check

Now no one can merge a change that breaks RAG quality — not accidentally, not under deadline pressure. The gate is automated. 🔒

📊 Step 6 — Viewing Results

After a workflow run you have three places to check results:

1. GitHub Actions job logs
Full pytest output, line by line. Best for debugging a specific failure.

2. GitHub Actions step summary
The clean markdown table from post_summary.py. Best for a quick pass/fail overview. Visible directly on the workflow run page without opening logs.

3. Downloaded artifact
The full rag_test_report.json. Best for tracking scores over time or doing deeper analysis. Download it from the workflow run's Artifacts section.

💰 Step 7 — Managing API Costs in CI

Running RAGAS evaluations in CI means calling the OpenAI API on every trigger. Here's how to keep costs under control.

Use a Smaller Evaluation Dataset in CI

Your full ground truth dataset might have 50+ test cases. In CI, you don't need to run all of them on every push.

Create a separate, smaller CI dataset:

// data/test_cases_ci.json
{
  "knowledge_base": [ ... ],

  "retrieval_test_cases": [
    // Keep your 5 highest-signal retrieval cases
    // These should represent the most common and most critical query types
  ],

  "faithfulness_test_cases": [
    // 3-4 cases that cover your main faithfulness scenarios
  ],

  "edge_case_queries": {
    "out_of_scope":    ["What is the capital of France?"],
    "empty_retrieval": ["What is the pricing for the enterprise plan?"],
    "leading_questions": [ ... ]
  }
}

Then in conftest.py, read from an environment variable to decide which dataset to use:

# tests/conftest.py

import json
import os
import pytest
from core.retriever import build_collection
from core.evaluator import build_evaluator

@pytest.fixture(scope="session")
def test_data():
    # In CI, use the smaller dataset. Locally, use the full one.
    ci_mode       = os.environ.get("CI", "false").lower() == "true"
    dataset_path  = "data/test_cases_ci.json" if ci_mode else "data/test_cases.json"

    with open(dataset_path) as f:
        return json.load(f)

@pytest.fixture(scope="session")
def collection(test_data):
    kb = test_data["knowledge_base"]
    return build_collection(
        collection_name="rag_test_kb",
        documents=[doc["text"] for doc in kb],
        doc_ids=[doc["id"] for doc in kb]
    )

@pytest.fixture(scope="session")
def evaluator():
    llm, embeddings = build_evaluator()
    return llm, embeddings

GitHub Actions automatically sets CI=true in every workflow run — no extra configuration needed.

Result: CI runs a fast, cost-efficient subset. Full runs happen locally or on scheduled nightly jobs (see below). ✅

🌙 Step 8 — Scheduled Full Runs

For a complete quality audit — run the full dataset on a schedule, not just on push:

# Add this to the `on:` section of your workflow

  schedule:
    # Run every day at 2 AM UTC
    # This uses the full dataset, not the CI subset
    - cron: '0 2 * * *'

And in your workflow, pass an environment variable to tell conftest to use the full dataset:

      - name: Run RAG quality checks
        working-directory: rag_test_framework
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          # Override CI mode for scheduled runs — use full dataset
          CI: ${{ github.event_name != 'schedule' && 'true' || 'false' }}
        run: |
          mkdir -p reports
          pytest tests/ -v --tb=short --json-report --json-report-file=reports/rag_test_report.json

The result:

Trigger	Dataset	Purpose
Push / PR	`test_cases_ci.json` (small)	Fast gate — catch obvious regressions
Scheduled (nightly)	`test_cases.json` (full)	Full quality audit — track score trends

🧩 The Complete Final Architecture

Here's the full picture — everything we've built across all six parts:

rag_test_framework/
│
├── .github/workflows/
│   └── rag_quality_checks.yml     ← CI/CD trigger + orchestration
│
├── ci/
│   └── post_summary.py            ← GitHub Actions summary writer
│
├── config/
│   └── settings.py                ← all thresholds, model names, API keys
│
├── core/
│   ├── retriever.py               ← retrieval + Precision@K, Recall@K, MRR
│   ├── evaluator.py               ← RAGAS faithfulness + answer_relevancy
│   └── rag_pipeline.py            ← end-to-end RAG call
│
├── tests/
│   ├── conftest.py                ← shared session-scoped fixtures
│   ├── test_retrieval.py          ← Part 2 tests
│   ├── test_faithfulness.py       ← Part 3 tests
│   └── test_edge_cases.py         ← Part 4 tests
│
├── data/
│   ├── test_cases.json            ← full ground truth dataset
│   └── test_cases_ci.json         ← smaller CI subset
│
├── reports/
│   └── (timestamped JSON reports)
│
├── run_tests.py                   ← local single-command runner
└── requirements.txt

One push. One workflow. Automated quality gate on every relevant change. 🎯

✅ End-to-End Flow — What Happens on Every Relevant Push

Let's walk through exactly what happens when a developer updates data/test_cases.json:

Developer pushes a commit that updates data/test_cases.json
    │
    ▼
GitHub detects the push matches a path filter
    │
    ▼
GitHub Actions spins up ubuntu-latest runner
    │
    ▼
Python 3.11 installed, pip cache restored
    │
    ▼
pip install -r requirements.txt
    │
    ▼
pytest tests/ runs with CI=true (uses test_cases_ci.json)
    │
    ├── test_retrieval.py   — Precision@K, Recall@K, MRR asserted
    ├── test_faithfulness.py — Faithfulness, no critical hallucinations
    └── test_edge_cases.py  — Empty retrieval, out-of-scope, leading questions
    │
    ▼
rag_test_report.json written to reports/
    │
    ▼
post_summary.py writes markdown table to GitHub step summary
    │
    ▼
Report uploaded as downloadable artifact (kept 30 days)
    │
    ├── All tests pass → ✅ Pipeline green, PR can merge
    └── Any test fails → ❌ Pipeline blocked, team notified

🔖 Key Takeaways From Part 6

Automation removes the "someone will remember" assumption — the gate runs regardless of deadline pressure or human error
paths filtering keeps costs under control — only trigger on changes that can actually affect RAG quality
Separate CI and full datasets — fast feedback on push, deep audit on schedule
scope="session" fixtures + CI dataset = fast CI runs — no repeated expensive setup, no unnecessary API calls
Branch protection rules complete the gate — automated tests mean nothing if merging is still allowed when they fail
Reports as artifacts — every run is recorded; you can track quality score trends over time
if: always() on artifact upload — you always want the report, especially when tests fail

🏁 Series Complete — What You've Built

Let's take a moment to look at how far we've come.

Six parts ago, most engineers testing RAG systems had no framework, no metrics, and no automated gate. They were hoping the final answer "looked right."

You now have something completely different. 👇

Part 1 ✅ — Understood what RAG is and why traditional testing breaks down
Part 2 ✅ — Gave retrieval quality a number: Precision@K, Recall@K, MRR
Part 3 ✅ — Gave hallucination detection a number: faithfulness scoring with RAGAS
Part 4 ✅ — Tested the edge cases that break RAG systems in production
Part 5 ✅ — Assembled everything into a structured, reusable framework
Part 6 ✅ — Automated the framework in CI/CD with GitHub Actions

You can plug this into any RAG system. Swap the vector database. Swap the LLM. The tests stay the same. The gate stays active. The quality stays measurable. 🎯

This is what production-grade RAG testing looks like.

🚀 What's Next — Beyond This Series

The framework you've built is a foundation, not a ceiling. Here's where to take it from here:

NDCG implementation — We covered NDCG conceptually in Part 2. Adding a proper implementation using sklearn.metrics.ndcg_score is a natural next step for more sophisticated retrieval ranking tests.

Alternative vector databases — The framework currently uses ChromaDB. If your production system uses Pinecone, Weaviate, or pgvector, the only change is in core/retriever.py. The tests are untouched.

Score trend tracking — Each run produces a JSON report. Building a simple script to parse historical reports and plot score trends over time will tell you if your RAG quality is improving or degrading with each knowledge base update.

Latency testing — We tested quality but not speed. Retrieval latency and end-to-end response time are worth measuring, especially as your knowledge base grows.

Custom RAGAS metrics — RAGAS supports custom metrics beyond faithfulness and answer relevancy. Context precision and context recall are worth exploring as your test suite matures.

Thank you for following this series all the way to the end. 🙏

Every part was built with real QA engineering principles — not just AI hype. The goal was always to make RAG testing feel like engineering, not magic.

I hope it does. 🎯

Drop a comment below 👇

Have you wired this into your own CI/CD pipeline? How did it go?
Which part of the series was most useful for your specific situation?
What would you like me to cover next — NDCG implementation, alternative vector DBs, score trend tracking?

All questions and feedback welcome. Let's keep building. 🙌

Faizal Shaikh | Senior Automation Engineer | AI & RAG-Based Testing
Connect with me on LinkedIn

RAG-Based Testing Series — Part 5: Building a RAG Test Framework from Scratch

Faizal — Fri, 12 Jun 2026 02:36:53 +0000

RAG-Based Testing Series — Part 5: Building a RAG Test Framework from Scratch

"Individual tests tell you what broke. A framework tells you the health of the whole system."

We've come a long way in this series.

In Part 2, we measured retrieval quality — Precision@K, Recall@K, MRR.

In Part 3, we detected hallucinations — faithfulness scoring with RAGAS.

In Part 4, we stress-tested edge cases — empty retrieval, conflicting context, adversarial queries.

Every single one of those was written as an individual, isolated test.

That was intentional. Learning each concept in isolation makes it easier to understand.

But in a real project, isolated tests are a problem. 🔴

You end up with:

Tests scattered across multiple files with no shared structure
Repeated setup code everywhere (embedding model, DB connection, LLM client)
No unified way to run everything and see the full picture
No scoring — just green/red with no sense of how good the system actually is
Nothing configurable — every change requires hunting through multiple files

This is what Part 5 fixes.

We're taking everything we've built and assembling it into a proper, structured, reusable RAG test framework. 🏗️

🗺️ What We're Building

By the end of this article, you'll have a framework with this structure:

rag_test_framework/
├── config/
│   └── settings.py          ← all configuration in one place
├── core/
│   ├── retriever.py         ← retrieval logic + scoring
│   ├── evaluator.py         ← RAGAS evaluation wrapper
│   └── rag_pipeline.py      ← end-to-end RAG call
├── tests/
│   ├── conftest.py          ← shared pytest fixtures
│   ├── test_retrieval.py    ← retrieval quality tests
│   ├── test_faithfulness.py ← faithfulness + hallucination tests
│   └── test_edge_cases.py   ← edge case tests
├── data/
│   └── test_cases.json      ← your ground truth dataset
├── reports/
│   └── (auto-generated)     ← test run reports saved here
├── run_tests.py             ← single entry point to run everything
└── requirements.txt

One command to run the entire suite:

python run_tests.py

Or through pytest for CI/CD integration (Part 6):

pytest tests/ -v --tb=short

Let's build it. 🛠️

📦 Step 1 — Install Dependencies

pip install ragas
pip install openai
pip install chromadb
pip install datasets
pip install langchain-openai
pip install pytest
pip install pytest-json-report

Save this as requirements.txt:

ragas>=0.1.0
openai>=1.0.0
chromadb>=0.4.0
datasets>=2.0.0
langchain-openai>=0.0.1
pytest>=7.0.0
pytest-json-report>=1.5.0

⚙️ Step 2 — Configuration in One Place

The single biggest improvement you can make over scattered one-off tests is centralising all configuration.

# config/settings.py

import os

# ── API Keys ──────────────────────────────────────────────
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "your-openai-api-key")

# ── Models ────────────────────────────────────────────────
EMBEDDING_MODEL = "text-embedding-ada-002"
EVALUATION_LLM  = "gpt-4o-mini"
RAG_LLM         = "gpt-4o-mini"

# ── Retrieval Settings ────────────────────────────────────
DEFAULT_TOP_K             = 3          # how many docs to retrieve
SIMILARITY_THRESHOLD      = 0.8        # max L2 distance to consider relevant
                                        # tune this for your embedding model

# ── Quality Thresholds ────────────────────────────────────
# These are the minimum acceptable scores for your system.
# Adjust based on your risk tolerance.
MIN_PRECISION_AT_K        = 0.6        # >= 60% of retrieved docs must be relevant
MIN_RECALL_AT_K           = 0.7        # >= 70% of relevant docs must be retrieved
MIN_MRR                   = 0.7        # first relevant doc should rank in top 2 on average
MIN_FAITHFULNESS          = 0.8        # >= 80% of answer claims must be grounded
CRITICAL_FAITHFULNESS     = 0.3        # below this = outright fabrication, always fail

# ── Reporting ─────────────────────────────────────────────
REPORTS_DIR               = "reports"
REPORT_FILENAME           = "rag_test_report.json"

Every number here is a starting point, not a gospel. Run your framework on your actual system, look at the score distributions, then set thresholds that match your system's risk level. High-stakes domains (medical, legal, financial) should have tighter thresholds. Internal tooling can be more lenient.

🔌 Step 3 — Core Modules

3a — Retriever

# core/retriever.py

import chromadb
from chromadb.utils import embedding_functions
from config.settings import (
    OPENAI_API_KEY,
    EMBEDDING_MODEL,
    DEFAULT_TOP_K,
    SIMILARITY_THRESHOLD
)


def build_collection(collection_name: str, documents: list[str], doc_ids: list[str]):
    """
    Build a ChromaDB collection from a list of documents.
    Call this once during test setup.
    """
    client = chromadb.Client()
    embedding_fn = embedding_functions.OpenAIEmbeddingFunction(
        api_key=OPENAI_API_KEY,
        model_name=EMBEDDING_MODEL
    )

    # Delete collection if it already exists (clean slate for each test run)
    try:
        client.delete_collection(collection_name)
    except Exception:
        pass

    collection = client.create_collection(
        name=collection_name,
        embedding_function=embedding_fn
    )

    collection.add(documents=documents, ids=doc_ids)
    return collection


def retrieve(collection, query: str, n_results: int = DEFAULT_TOP_K) -> dict:
    """
    Retrieve top N documents for a query.
    Returns documents, IDs, and distance scores.
    """
    results = collection.query(
        query_texts=[query],
        n_results=n_results,
        include=["documents", "distances", "ids"]
    )
    return {
        "documents": results["documents"][0],
        "ids":       results["ids"][0],
        "distances": results["distances"][0]
    }


def filter_by_threshold(retrieval_result: dict, threshold: float = SIMILARITY_THRESHOLD) -> dict:
    """
    Filter out documents whose distance exceeds the similarity threshold.
    Returns only the documents considered relevant.
    """
    filtered = [
        (doc, doc_id, dist)
        for doc, doc_id, dist in zip(
            retrieval_result["documents"],
            retrieval_result["ids"],
            retrieval_result["distances"]
        )
        if dist < threshold
    ]

    if not filtered:
        return {"documents": [], "ids": [], "distances": []}

    docs, ids, dists = zip(*filtered)
    return {"documents": list(docs), "ids": list(ids), "distances": list(dists)}


def calculate_precision_at_k(retrieved_ids: list, relevant_ids: list, k: int) -> float:
    retrieved_at_k = retrieved_ids[:k]
    hits = [doc_id for doc_id in retrieved_at_k if doc_id in relevant_ids]
    return len(hits) / k if k > 0 else 0.0


def calculate_recall_at_k(retrieved_ids: list, relevant_ids: list, k: int) -> float:
    retrieved_at_k = retrieved_ids[:k]
    hits = [doc_id for doc_id in retrieved_at_k if doc_id in relevant_ids]
    return len(hits) / len(relevant_ids) if relevant_ids else 0.0


def calculate_mrr(collection, test_cases: list, k: int = 5) -> float:
    reciprocal_ranks = []
    for test in test_cases:
        result = retrieve(collection, test["query"], n_results=k)
        rr = 0.0
        for rank, doc_id in enumerate(result["ids"], start=1):
            if doc_id in test["relevant_doc_ids"]:
                rr = 1.0 / rank
                break
        reciprocal_ranks.append(rr)
    return round(sum(reciprocal_ranks) / len(reciprocal_ranks), 4) if reciprocal_ranks else 0.0

3b — Evaluator

# core/evaluator.py

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from config.settings import OPENAI_API_KEY, EVALUATION_LLM


def build_evaluator():
    """
    Build and return the LLM + embeddings used by RAGAS.
    Call once and reuse — avoids re-initialising on every test.
    """
    llm = ChatOpenAI(
        model=EVALUATION_LLM,
        openai_api_key=OPENAI_API_KEY
    )
    embeddings = OpenAIEmbeddings(
        openai_api_key=OPENAI_API_KEY
    )
    return llm, embeddings


def evaluate_faithfulness(test_data: list, llm, embeddings) -> list[dict]:
    """
    Run RAGAS faithfulness + answer_relevancy evaluation on a list of test cases.

    Each test case must have:
        - question  (str)
        - answer    (str)
        - contexts  (list of str)

    Returns a list of dicts with per-case scores added.
    """
    dataset = Dataset.from_list(test_data)

    results = evaluate(
        dataset=dataset,
        metrics=[faithfulness, answer_relevancy],
        llm=llm,
        embeddings=embeddings
    )

    df = results.to_pandas()

    scored = []
    for i, row in df.iterrows():
        scored.append({
            **test_data[i],
            "faithfulness":     round(float(row["faithfulness"]), 4),
            "answer_relevancy": round(float(row["answer_relevancy"]), 4)
        })

    return scored

3c — RAG Pipeline

# core/rag_pipeline.py

from openai import OpenAI
from core.retriever import retrieve, filter_by_threshold
from config.settings import OPENAI_API_KEY, RAG_LLM, DEFAULT_TOP_K

_openai_client = OpenAI(api_key=OPENAI_API_KEY)

SYSTEM_PROMPT = """You are a helpful customer support assistant.
Answer questions using ONLY the information provided in the context below.
If the context does not contain enough information to answer the question,
say so clearly — do not make up information.
If the question is outside the scope of the provided context, say so politely."""


def run_rag(collection, question: str, n_results: int = DEFAULT_TOP_K) -> dict:
    """
    Run a full RAG call: retrieve context, then generate an answer.

    Returns:
        question   — the original question
        contexts   — list of retrieved document strings (after threshold filter)
        answer     — the LLM's generated answer
        raw_distances — distances for all retrieved docs (for debugging)
    """
    raw_result = retrieve(collection, question, n_results=n_results)
    filtered   = filter_by_threshold(raw_result)

    contexts = filtered["documents"]

    if not contexts:
        # No relevant documents found — return a structured "I don't know"
        return {
            "question":      question,
            "contexts":      [],
            "answer":        "I don't have relevant information to answer this question.",
            "raw_distances": raw_result["distances"]
        }

    context_block = "\n\n".join(contexts)
    prompt = f"{SYSTEM_PROMPT}\n\nContext:\n{context_block}\n\nQuestion: {question}\n\nAnswer:"

    response = _openai_client.chat.completions.create(
        model=RAG_LLM,
        messages=[{"role": "user", "content": prompt}]
    )

    return {
        "question":      question,
        "contexts":      contexts,
        "answer":        response.choices[0].message.content,
        "raw_distances": raw_result["distances"]
    }

📋 Step 4 — Ground Truth Dataset

Store your test cases in one place — not hardcoded across test files.

// data/test_cases.json
{
  "knowledge_base": [
    {
      "id":   "doc1",
      "text": "Premium subscribers are eligible for a full refund within 30 days of purchase. Requests must be submitted via the support portal."
    },
    {
      "id":   "doc2",
      "text": "Standard subscribers can cancel their subscription at any time. Cancellation takes effect at the end of the billing period."
    },
    {
      "id":   "doc3",
      "text": "Shipping for all orders is processed within 2-3 business days. Express shipping is available at checkout."
    },
    {
      "id":   "doc4",
      "text": "To reset your password, click the Forgot Password link on the login page. A reset link will be sent to your registered email."
    },
    {
      "id":   "doc5",
      "text": "Premium subscribers get access to priority customer support with a 2-hour response time guarantee."
    }
  ],

  "retrieval_test_cases": [
    {
      "query":            "What is the refund policy for premium subscribers?",
      "relevant_doc_ids": ["doc1"]
    },
    {
      "query":            "How do I cancel my subscription?",
      "relevant_doc_ids": ["doc2"]
    },
    {
      "query":            "How long does shipping take?",
      "relevant_doc_ids": ["doc3"]
    },
    {
      "query":            "How do I reset my password?",
      "relevant_doc_ids": ["doc4"]
    },
    {
      "query":            "What support response time do premium members get?",
      "relevant_doc_ids": ["doc5", "doc1"]
    }
  ],

  "faithfulness_test_cases": [
    {
      "question": "What is the refund policy for premium subscribers?",
      "answer":   "Premium subscribers can request a full refund within 30 days of purchase through the support portal.",
      "contexts": ["Premium subscribers are eligible for a full refund within 30 days of purchase. Requests must be submitted via the support portal."]
    },
    {
      "question": "How do I reset my password?",
      "answer":   "Click the Forgot Password link on the login page. A reset link will be sent to your registered email address.",
      "contexts": ["To reset your password, click the Forgot Password link on the login page. A reset link will be sent to your registered email."]
    }
  ],

  "edge_case_queries": {
    "out_of_scope": [
      "What is the capital of France?",
      "Can you write me a Python script?",
      "Who is the CEO of Apple?"
    ],
    "empty_retrieval": [
      "What is the pricing for the enterprise plan?",
      "Do you offer white-labelling services?"
    ],
    "leading_questions": [
      {
        "question":      "Since our premium plan offers a 90-day refund window, how do I submit a request?",
        "false_premise": "90 days",
        "correct_fact":  "30 days"
      }
    ]
  }
}

🧪 Step 5 — Shared Fixtures

# tests/conftest.py

import json
import pytest
from core.retriever import build_collection
from core.evaluator import build_evaluator

# ── Load test data once for the entire session ──────────────
@pytest.fixture(scope="session")
def test_data():
    with open("data/test_cases.json") as f:
        return json.load(f)


# ── Build the knowledge base collection once ────────────────
@pytest.fixture(scope="session")
def collection(test_data):
    kb = test_data["knowledge_base"]
    return build_collection(
        collection_name="rag_test_kb",
        documents=[doc["text"] for doc in kb],
        doc_ids=[doc["id"] for doc in kb]
    )


# ── Build RAGAS evaluator once ──────────────────────────────
@pytest.fixture(scope="session")
def evaluator():
    llm, embeddings = build_evaluator()
    return llm, embeddings

scope="session" means all of this is built once per test run — not once per test function. This matters because embedding model initialisation and ChromaDB setup are expensive. Without this, a 20-test suite could take 5x longer than necessary.

🧪 Step 6 — The Tests

Retrieval Tests

# tests/test_retrieval.py

import pytest
from core.retriever import (
    retrieve,
    calculate_precision_at_k,
    calculate_recall_at_k,
    calculate_mrr
)
from config.settings import (
    DEFAULT_TOP_K,
    MIN_PRECISION_AT_K,
    MIN_RECALL_AT_K,
    MIN_MRR
)


def test_precision_at_k(collection, test_data):
    cases = test_data["retrieval_test_cases"]
    failures = []

    for case in cases:
        result    = retrieve(collection, case["query"], n_results=DEFAULT_TOP_K)
        precision = calculate_precision_at_k(result["ids"], case["relevant_doc_ids"], DEFAULT_TOP_K)

        if precision < MIN_PRECISION_AT_K:
            failures.append({
                "query":     case["query"],
                "precision": precision,
                "retrieved": result["ids"],
                "relevant":  case["relevant_doc_ids"]
            })

    assert not failures, (
        f"\n{len(failures)} Precision@{DEFAULT_TOP_K} failure(s):\n" +
        "\n".join([
            f"  ❌ Query: '{f['query']}'\n"
            f"     Score: {f['precision']} (min: {MIN_PRECISION_AT_K})\n"
            f"     Retrieved: {f['retrieved']}\n"
            f"     Expected:  {f['relevant']}"
            for f in failures
        ])
    )


def test_recall_at_k(collection, test_data):
    cases = test_data["retrieval_test_cases"]
    failures = []

    for case in cases:
        result = retrieve(collection, case["query"], n_results=DEFAULT_TOP_K)
        recall = calculate_recall_at_k(result["ids"], case["relevant_doc_ids"], DEFAULT_TOP_K)

        if recall < MIN_RECALL_AT_K:
            failures.append({
                "query":  case["query"],
                "recall": recall
            })

    assert not failures, (
        f"\n{len(failures)} Recall@{DEFAULT_TOP_K} failure(s):\n" +
        "\n".join([
            f"  ❌ Query: '{f['query']}'\n"
            f"     Score: {f['recall']} (min: {MIN_RECALL_AT_K})"
            for f in failures
        ])
    )


def test_mrr(collection, test_data):
    cases = test_data["retrieval_test_cases"]
    mrr   = calculate_mrr(collection, cases)

    assert mrr >= MIN_MRR, (
        f"\nMRR too low: {mrr} (min: {MIN_MRR})\n"
        f"Relevant documents are not ranking high enough in retrieval results."
    )

Faithfulness Tests

# tests/test_faithfulness.py

import pytest
from core.evaluator import evaluate_faithfulness
from config.settings import MIN_FAITHFULNESS, CRITICAL_FAITHFULNESS


def test_faithfulness_above_threshold(evaluator, test_data):
    llm, embeddings = evaluator
    cases   = test_data["faithfulness_test_cases"]
    scored  = evaluate_faithfulness(cases, llm, embeddings)
    failures = [s for s in scored if s["faithfulness"] < MIN_FAITHFULNESS]

    assert not failures, (
        f"\n{len(failures)} faithfulness failure(s):\n" +
        "\n".join([
            f"  ❌ Question: '{f['question']}'\n"
            f"     Score: {f['faithfulness']} (min: {MIN_FAITHFULNESS})\n"
            f"     Answer: {f['answer'][:100]}..."
            for f in failures
        ])
    )


def test_no_critical_hallucinations(evaluator, test_data):
    """
    Any answer below CRITICAL_FAITHFULNESS is an outright fabrication.
    This test should never be allowed to pass in any environment.
    """
    llm, embeddings = evaluator
    cases   = test_data["faithfulness_test_cases"]
    scored  = evaluate_faithfulness(cases, llm, embeddings)
    critical = [s for s in scored if s["faithfulness"] < CRITICAL_FAITHFULNESS]

    assert not critical, (
        f"\n🚨 {len(critical)} CRITICAL hallucination(s) detected:\n" +
        "\n".join([
            f"  🔴 Question: '{f['question']}'\n"
            f"     Score: {f['faithfulness']} (critical threshold: {CRITICAL_FAITHFULNESS})\n"
            f"     Answer: {f['answer'][:100]}..."
            for f in critical
        ])
    )

Edge Case Tests

# tests/test_edge_cases.py

import pytest
from core.retriever import retrieve, filter_by_threshold
from core.rag_pipeline import run_rag
from config.settings import SIMILARITY_THRESHOLD


def test_empty_retrieval_detected(collection, test_data):
    """
    Queries with no relevant documents should return zero docs
    after threshold filtering.
    """
    out_of_kb_queries = test_data["edge_case_queries"]["empty_retrieval"]

    for query in out_of_kb_queries:
        raw      = retrieve(collection, query)
        filtered = filter_by_threshold(raw, threshold=SIMILARITY_THRESHOLD)

        assert len(filtered["documents"]) == 0, (
            f"\n❌ Expected no relevant docs for: '{query}'\n"
            f"   Got {len(filtered['documents'])} doc(s) below threshold.\n"
            f"   Distances: {raw['distances']}"
        )


def test_rag_response_on_empty_retrieval(collection, test_data):
    """
    When no relevant documents exist, the pipeline should return
    an uncertainty response — not a fabricated answer.
    """
    uncertainty_indicators = [
        "don't have", "no information", "unable to find",
        "not available", "contact", "cannot find"
    ]

    out_of_kb_queries = test_data["edge_case_queries"]["empty_retrieval"]

    for query in out_of_kb_queries:
        result       = run_rag(collection, query)
        answer_lower = result["answer"].lower()
        is_uncertain = any(ind in answer_lower for ind in uncertainty_indicators)

        assert is_uncertain, (
            f"\n❌ RAG pipeline did not express uncertainty for: '{query}'\n"
            f"   Answer: {result['answer']}\n"
            f"   Expected one of: {uncertainty_indicators}"
        )


def test_out_of_scope_handling(collection, test_data):
    """
    Out-of-scope queries should produce a decline or redirect — not a fabricated answer.
    """
    decline_indicators = [
        "outside", "scope", "unable to help", "don't have information",
        "not able to assist", "please contact", "beyond", "not within"
    ]

    for query in test_data["edge_case_queries"]["out_of_scope"]:
        result       = run_rag(collection, query)
        answer_lower = result["answer"].lower()
        declined     = any(ind in answer_lower for ind in decline_indicators)

        assert declined, (
            f"\n❌ System did not decline out-of-scope query: '{query}'\n"
            f"   Answer: {result['answer']}"
        )


def test_leading_question_correction(collection, evaluator, test_data):
    """
    When a query contains a false premise, the generated answer should
    contradict the false premise — not accept it.
    """
    from core.evaluator import evaluate_faithfulness
    from config.settings import MIN_FAITHFULNESS

    llm, embeddings = evaluator
    leading_cases   = test_data["edge_case_queries"]["leading_questions"]

    for case in leading_cases:
        result = run_rag(collection, case["question"])

        # The answer should NOT contain the false premise
        assert case["false_premise"] not in result["answer"], (
            f"\n❌ LLM accepted false premise in leading question.\n"
            f"   Question:      {case['question']}\n"
            f"   False premise: {case['false_premise']}\n"
            f"   Answer:        {result['answer']}"
        )

        # The answer SHOULD contain the correct fact
        assert case["correct_fact"] in result["answer"], (
            f"\n❌ LLM did not correct the false premise.\n"
            f"   Question:     {case['question']}\n"
            f"   Correct fact: {case['correct_fact']}\n"
            f"   Answer:       {result['answer']}"
        )

📊 Step 7 — Unified Runner with Reporting

# run_tests.py

import os
import json
import subprocess
from datetime import datetime
from config.settings import REPORTS_DIR, REPORT_FILENAME

def run():
    os.makedirs(REPORTS_DIR, exist_ok=True)

    timestamp   = datetime.now().strftime("%Y%m%d_%H%M%S")
    report_path = os.path.join(REPORTS_DIR, f"{timestamp}_{REPORT_FILENAME}")

    print("=" * 65)
    print("  RAG Test Framework — Full Suite")
    print(f"  Started: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
    print("=" * 65)

    result = subprocess.run(
        [
            "pytest", "tests/",
            "-v",
            "--tb=short",
            f"--json-report",
            f"--json-report-file={report_path}"
        ],
        capture_output=False  # stream output to terminal in real time
    )

    print("\n" + "=" * 65)

    # Parse and print summary from the JSON report
    if os.path.exists(report_path):
        with open(report_path) as f:
            report = json.load(f)

        summary  = report.get("summary", {})
        passed   = summary.get("passed", 0)
        failed   = summary.get("failed", 0)
        total    = summary.get("total", 0)
        duration = round(report.get("duration", 0), 2)

        print(f"  Results : {passed}/{total} passed | {failed} failed")
        print(f"  Duration: {duration}s")
        print(f"  Report  : {report_path}")

        if failed > 0:
            print("\n  ❌ Failed tests:")
            for test in report.get("tests", []):
                if test["outcome"] == "failed":
                    print(f"     • {test['nodeid']}")

        print("=" * 65)

    return result.returncode


if __name__ == "__main__":
    exit(run())

▶️ Running the Framework

# Run the full suite
python run_tests.py

# Run only retrieval tests
pytest tests/test_retrieval.py -v

# Run only faithfulness tests
pytest tests/test_faithfulness.py -v

# Run only edge case tests
pytest tests/test_edge_cases.py -v

# Run with full failure output
pytest tests/ -v --tb=long

Sample output:

=================================================================
  RAG Test Framework — Full Suite
  Started: 2025-06-01 14:32:01
=================================================================

tests/test_retrieval.py::test_precision_at_k    PASSED
tests/test_retrieval.py::test_recall_at_k       PASSED
tests/test_retrieval.py::test_mrr               PASSED
tests/test_faithfulness.py::test_faithfulness_above_threshold  PASSED
tests/test_faithfulness.py::test_no_critical_hallucinations    PASSED
tests/test_edge_cases.py::test_empty_retrieval_detected        PASSED
tests/test_edge_cases.py::test_rag_response_on_empty_retrieval PASSED
tests/test_edge_cases.py::test_out_of_scope_handling           PASSED
tests/test_edge_cases.py::test_leading_question_correction     PASSED

=================================================================
  Results : 9/9 passed | 0 failed
  Duration: 42.3s
  Report  : reports/20250601_143201_rag_test_report.json
=================================================================

🔌 Using This Framework on a Different RAG System

The whole point of a framework is reusability. Here's what you change to point it at a new system:

1. Update data/test_cases.json
Replace the knowledge base documents and ground truth test cases with your system's actual content.

2. Update config/settings.py
Adjust thresholds based on your new system's acceptable quality levels.

3. Update core/rag_pipeline.py
If your RAG system uses a different vector database (Pinecone, Weaviate, pgvector) or a different LLM — swap out those implementations in rag_pipeline.py and retriever.py. The tests themselves don't change.

4. Run

python run_tests.py

That's the power of the architecture. Tests are separated from infrastructure. Changing the underlying system doesn't mean rewriting your tests. 🎯

🧩 The Complete Testing Stack — All Five Layers

Layer 1 — RETRIEVAL QUALITY ✅
          tests/test_retrieval.py
          → Precision@K, Recall@K, MRR

Layer 2 — FAITHFULNESS & HALLUCINATION ✅
          tests/test_faithfulness.py
          → Faithfulness score, Answer Relevancy score

Layer 3 — EDGE CASES ✅
          tests/test_edge_cases.py
          → Empty retrieval, out-of-scope, leading questions

Layer 4 — FULL FRAMEWORK ✅  ← You are here
          All layers combined, one command to run

Layer 5 — CI/CD AUTOMATION ← Up next (Part 6)
          Running automatically on every knowledge base change

🔖 Key Takeaways From Part 5

Configuration in one place — settings.py is your single source of truth for thresholds, model names, and API keys
Shared fixtures with scope="session" — build your KB and evaluator once per run, not once per test
Ground truth in JSON — decouples your test data from your test logic; update cases without touching code
Modular structure — retriever, evaluator, and pipeline are separate; swap implementations without rewriting tests
One entry point — run_tests.py gives you a clean interface for humans and CI/CD alike
Reports are first-class — every run produces a timestamped JSON report; your team can track quality over time

🚀 What's Next

In Part 6 — the final part — we automate everything.

Right now, you run the framework manually. That's useful. But it's not enough.

Every time someone updates the knowledge base, changes the system prompt, or swaps the LLM — the quality of your RAG system could silently change.

Part 6 wires this framework into a GitHub Actions CI/CD pipeline so tests run automatically on every relevant change. Quality regressions get caught before they reach users. The team gets notified. The deployment is blocked if tests fail.

Part 1 — What Is RAG & Why It Needs Different Testing       ✅ Done
Part 2 — Testing Retrieval Quality: Are You Fetching Right? ✅ Done
Part 3 — Faithfulness & Hallucination Detection             ✅ Done
Part 4 — Edge Cases: What Breaks RAG & How to Catch It      ✅ Done
Part 5 — Building a RAG Test Framework from Scratch         ← You are here
Part 6 — Automating RAG Quality Checks in CI/CD             ← Up next

Follow me so you don't miss Part 6 — the final piece that turns this framework into a fully automated quality gate. 🚀

Drop a comment below 👇

Have you tried running this framework yet? What did your scores look like?
Are you using a different vector DB (Pinecone, Weaviate, pgvector)? Drop a comment — I'll cover alternative adapters in a bonus post.
What would you add to this framework for your specific use case?

All questions welcome. Let's learn this together. 🙌

Faizal Shaikh | Senior Automation Engineer | AI & RAG-Based Testing
Connect with me on LinkedIn

DEV Community: Faizal

The Playwright Playbook — Bonus: Refactoring Schema Validation with Zod

The Playwright Playbook — Bonus: Refactoring Schema Validation with Zod

🔍 Where We Left Off — The Original Validator

📦 Installing Zod

🔧 The Refactored Validator

🆚 Side-by-Side Comparison

✅ Using the Detailed Errors in Tests

🧩 Bonus — Composing Schemas (Something the Old Validator Couldn't Do)

🤔 So... Was the Hand-Rolled Validator Wrong?

📁 What Changed in the Project Structure

🔖 Before You Go

The Playwright Playbook — Part 8: Playwright Meets AI — Agents, MCP & Self-Healing Tests

The Playwright Playbook — Part 8: Playwright Meets AI — Agents, MCP & Self-Healing Tests

🏗️ Where We Left Off

🧠 Understanding the AI Layer — Three Distinct Tools

🔌 Playwright MCP — Giving AI Assistants a Browser

Setting up Playwright MCP

What MCP enables

How to use it with Claude

🤖 Building the Test Planner — AI-Powered Test Coverage Analysis

🔧 Building the Self-Healing Engine

📜 The Generate Test Plan Script

🔧 The Heal Selectors Script

🧪 AI-Generated Tests — Honest Evaluation

🤝 The Hybrid Strategy — Where AI Earns Its Place

📦 Final package.json Scripts

📁 The Complete Final Project Structure

🗺️ The Complete Series — Done

🔖 What You've Built

What's Next

The Playwright Playbook — Part 7: The CI/CD Setup Nobody Shows You

The Playwright Playbook — Part 7: The CI/CD Setup Nobody Shows You

🏗️ Where We Left Off

🧠 The CI Architecture — Mental Model First

⚙️ Updating playwright.config.ts — CI-Ready Final Version

🐳 Docker — Consistent Environments Everywhere

🔔 Slack Notification Script

🚀 The Main GitHub Actions Workflow

🎨 The Visual Regression Workflow — Separate Pipeline

📋 The Complete .gitignore

🔐 GitHub Secrets — What to Configure

📊 Understanding Sharding — Why It Matters

📁 Final Project Structure After Part 7

🗺️ What's Coming in This Series

🔖 Before You Go

The Playwright Playbook — Part 6: Debugging Like a Pro — Trace Viewer, Inspector & VS Code

The Playwright Playbook — Part 6: Debugging Like a Pro — Trace Viewer, Inspector & VS Code

🏗️ Where We Left Off

🧠 The Debugging Toolkit — Overview

🎬 Trace Viewer — The Most Powerful Debugging Tool in Playwright

Step 1 — Configure traces in playwright.config.ts

Step 2 — Open a trace locally

Step 3 — What you see in Trace Viewer

🔍 Playwright Inspector — Interactive Step-Through Debugging

Launch modes

Using Inspector to find the right locator

The Inspector toolbar

⏸️ page.pause() — Your Interactive Breakpoint

🛠️ Building the Debug Helpers Utility

🧪 Trace Examples — Tests That Demonstrate the Debug Toolkit

💻 VS Code Integration — Run & Debug Tests From Your IDE

Step 1 — Install the extension

Step 2 — VS Code debug launch config

What the VS Code extension gives you

📋 Reading CI Failures Like a Pro

The debugging workflow for CI failures

Reading the trace timeline — what to look for

🔧 Useful Debug Commands — Quick Reference

📁 Final Project Structure After Part 6

🗺️ What's Coming in This Series

🔖 Before You Go

The Playwright Playbook — Part 5: Visual Regression Testing

The Playwright Playbook — Part 5: Visual Regression Testing

🏗️ Where We Left Off

🧠 How Playwright Visual Regression Works — The Mental Model

⚙️ Updating playwright.config.ts for Visual Testing

🛠️ Building the Visual Helpers Utility

📸 Dashboard Visual Tests — Full Page & Component Level

🎨 Task Page Visual Tests — States & Error Handling

📦 Final `package.json` Scripts

⚙️ Updating `playwright.config.ts` — CI-Ready Final Version

📋 The Complete `.gitignore`

Step 1 — Configure traces in `playwright.config.ts`

⏸️ `page.pause()` — Your Interactive Breakpoint

⚙️ Updating `playwright.config.ts` for Visual Testing

⚙️ Configuring the API Base in `playwright.config.ts`

⚙️ Updating `playwright.config.ts` for Multi-Context Tests

🧪 The Basic: `page.route()` — Your First Mock

📡 Asserting on Real Network Calls — `waitForResponse`