How to Make AI Agents Prove Their Work with Screenshots (No More Hallucinations)

TL;DR

Replace subjective AI approvals with an evidence-based QA workflow: configure Playwright for desktop, tablet, and mobile breakpoints; capture screenshots, interactions, browser errors, network failures, and performance metrics; run a single evidence-collection script; then require your QA agent to return either PASS or NEEDS WORK with specific blocking issues.

Try Apidog today

Introduction

Stop accepting “looks great” from AI agents without proof.

You ask an AI agent to review your landing page. It responds:

The design looks premium and polished. The glassmorphism effects are well implemented. The page is fully responsive. Ready for production!

Then you open the page and find that:

• The “glassmorphism” is a solid gray background.
• The mobile layout is broken.
• The browser console contains errors.
• Nothing was actually measured or verified.

AI agents can produce confident conclusions without checking the implementation. An evidence-based QA workflow prevents that by requiring files, screenshots, metrics, and test results before approval.

The Reality Checker agent from The Agency collection follows this model:

Status: NEEDS WORK

Evidence:
- grep for "glassmorphism" returned NO PREMIUM FEATURES FOUND
- responsive-mobile.png shows a broken layout at 375px
- performance-metrics.json reports 3 console errors
- measured load time is 2.1s

Blocking issues: 4

No opinions. No unsupported approval. Just evidence.

In this tutorial, you’ll build a Playwright-based QA workflow that complements your API testing pipeline. Whether you’re validating frontend layouts or checking API responses in Apidog, the rule is the same: require proof before approval.

Why Evidence-Based QA Matters

Without verification requirements, an AI review can make claims such as:

• “The code looks solid” without running it.
• “Performance should be great” without measuring load time.
• “The page is fully responsive” without testing a mobile viewport.
• “Authentication is implemented” without locating the relevant code.

Evidence-based QA replaces those claims with:

• Screenshots at desktop, tablet, and mobile breakpoints
• Browser console error logs
• Failed network request logs
• Measured navigation timing
• Source-code searches proving that claimed features exist
• A final PASS or NEEDS WORK decision

The goal is not to eliminate AI-assisted reviews. It is to make those reviews auditable.

Step 1: Install and Configure Playwright

Install Playwright and Chromium:

npm install -D @playwright/test
npx playwright install chromium

Create qa-playwright.config.ts:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testMatch: '**/qa-screenshots.spec.ts',
  timeout: 30_000,

  // Run projects sequentially because the tests update one shared metrics file.
  workers: 1,

  use: {
    baseURL: process.env.BASE_URL || 'http://localhost:8000',
    headless: true,
    trace: 'on-first-retry',
  },

  projects: [
    {
      name: 'desktop',
      use: {
        viewport: { width: 1920, height: 1080 },
      },
    },
    {
      name: 'tablet',
      use: {
        viewport: { width: 768, height: 1024 },
      },
    },
    {
      name: 'mobile',
      use: {
        viewport: { width: 375, height: 667 },
      },
    },
  ],

  reporter: [
    ['list'],
    [
      'json',
      {
        outputFile: 'public/qa-screenshots/test-results.json',
      },
    ],
  ],

  outputDir: 'public/qa-screenshots/playwright-artifacts',
});

This configuration creates three Playwright projects. Every test runs once at each target viewport.

You can override the target application with BASE_URL:

BASE_URL=http://localhost:3000 npx playwright test \
  --config=qa-playwright.config.ts

Step 2: Create the Screenshot Test Suite

Create qa-screenshots.spec.ts:

import { test } from '@playwright/test';
import * as fs from 'node:fs';
import * as path from 'node:path';

const outputDir = 'public/qa-screenshots';

fs.mkdirSync(outputDir, { recursive: true });

function screenshotPath(fileName: string): string {
  return path.join(outputDir, fileName);
}

function writeProjectMetrics(
  projectName: string,
  data: {
    performance: {
      loadTime: number;
      domContentLoaded: number;
      jsHeapSize: number | null;
    };
    consoleErrors: string[];
    networkErrors: string[];
  }
): void {
  const metricsPath = screenshotPath('performance-metrics.json');

  let existing: Record<string, unknown> = {};

  if (fs.existsSync(metricsPath)) {
    try {
      existing = JSON.parse(fs.readFileSync(metricsPath, 'utf8'));
    } catch {
      existing = {};
    }
  }

  existing[projectName] = data;

  fs.writeFileSync(
    metricsPath,
    JSON.stringify(existing, null, 2)
  );
}

async function waitForPage(page: import('@playwright/test').Page): Promise<void> {
  await page.goto('/', { waitUntil: 'domcontentloaded' });

  // Some applications keep long-lived network connections open.
  // Continue after 10 seconds instead of failing the capture.
  await page
    .waitForLoadState('networkidle', { timeout: 10_000 })
    .catch(() => undefined);
}

test.describe('@screenshot Reality Check', () => {
  test('capture full page and browser evidence', async ({
    page,
    context,
  }, testInfo) => {
    const projectName = testInfo.project.name;
    const consoleErrors: string[] = [];
    const networkErrors: string[] = [];

    // Register listeners before navigation.
    page.on('console', message => {
      if (message.type() === 'error') {
        consoleErrors.push(message.text());
      }
    });

    page.on('requestfailed', request => {
      networkErrors.push(
        `${request.method()} ${request.url()}: ${
          request.failure()?.errorText || 'unknown error'
        }`
      );
    });

    await waitForPage(page);

    const navigationTiming = await page.evaluate(() => {
      const [navigation] = performance.getEntriesByType(
        'navigation'
      ) as PerformanceNavigationTiming[];

      return {
        loadTime: navigation?.loadEventEnd || 0,
        domContentLoaded: navigation?.domContentLoadedEventEnd || 0,
      };
    });

    // Chromium exposes heap metrics through the Chrome DevTools Protocol.
    const session = await context.newCDPSession(page);
    await session.send('Performance.enable');

    const { metrics } = await session.send('Performance.getMetrics');
    const jsHeapMetric = metrics.find(
      metric => metric.name === 'JSHeapUsedSize'
    );

    await page.screenshot({
      path: screenshotPath(`responsive-${projectName}.png`),
      fullPage: true,
    });

    writeProjectMetrics(projectName, {
      performance: {
        ...navigationTiming,
        jsHeapSize: jsHeapMetric?.value ?? null,
      },
      consoleErrors,
      networkErrors,
    });
  });

  test('capture navigation interactions', async ({ page }, testInfo) => {
    const projectName = testInfo.project.name;

    await waitForPage(page);

    const navigationSelector = 'nav a, header a, .nav a';
    const navigationItems = page.locator(navigationSelector);
    const itemCount = Math.min(await navigationItems.count(), 5);

    for (let index = 0; index < itemCount; index++) {
      await page.goto('/', { waitUntil: 'domcontentloaded' });

      const item = page.locator(navigationSelector).nth(index);

      await page.screenshot({
        path: screenshotPath(
          `nav-${projectName}-${index}-before.png`
        ),
      });

      await item.click();

      await page
        .waitForLoadState('networkidle', { timeout: 10_000 })
        .catch(() => undefined);

      await page.screenshot({
        path: screenshotPath(
          `nav-${projectName}-${index}-after.png`
        ),
      });
    }
  });

  test('capture form interactions', async ({ page }, testInfo) => {
    const projectName = testInfo.project.name;

    await waitForPage(page);

    const forms = page.locator('form');
    const formCount = await forms.count();

    for (let formIndex = 0; formIndex < formCount; formIndex++) {
      const form = forms.nth(formIndex);

      await form.screenshot({
        path: screenshotPath(
          `form-${projectName}-${formIndex}-initial.png`
        ),
      });

      const inputs = form.locator(
        'input[type="text"], input[type="email"], input[type="password"]'
      );

      const inputCount = await inputs.count();

      for (let inputIndex = 0; inputIndex < inputCount; inputIndex++) {
        await inputs.nth(inputIndex).fill('[e-mail]');
      }

      await form.screenshot({
        path: screenshotPath(
          `form-${projectName}-${formIndex}-filled.png`
        ),
      });
    }
  });

  test('capture accordion and dropdown interactions', async ({
    page,
  }, testInfo) => {
    const projectName = testInfo.project.name;

    await waitForPage(page);

    const components = page.locator(
      '[data-accordion], details, .accordion'
    );

    const componentCount = await components.count();

    for (let index = 0; index < componentCount; index++) {
      const component = components.nth(index);

      await component.screenshot({
        path: screenshotPath(
          `accordion-${projectName}-${index}-closed.png`
        ),
      });

      const trigger = component
        .locator('summary, [aria-expanded], button')
        .first();

      if (await trigger.count()) {
        await trigger.click();
      } else {
        await component.click();
      }

      await page.waitForTimeout(300);

      await component.screenshot({
        path: screenshotPath(
          `accordion-${projectName}-${index}-open.png`
        ),
      });
    }
  });
});

This suite collects four kinds of evidence:

1. Full-page screenshots at each configured breakpoint
2. Console and network errors
3. Navigation state screenshots
4. Form and accordion interaction screenshots

The @screenshot tag is important because the shell script in the next step filters tests using:

--grep "@screenshot"

Step 3: Create a One-Command Capture Script

Create qa-playwright-capture.sh:

#!/usr/bin/env bash
#
# Run Playwright evidence capture.
#
# Usage:
#   ./qa-playwright-capture.sh [BASE_URL] [OUTPUT_DIR]
#

set -euo pipefail

BASE_URL="${1:-http://localhost:8000}"
OUTPUT_DIR="${2:-public/qa-screenshots}"

echo "Starting Reality Check capture..."
echo "  Base URL: $BASE_URL"
echo "  Output:   $OUTPUT_DIR"

mkdir -p "$OUTPUT_DIR"

# Remove metrics from the previous run.
rm -f "$OUTPUT_DIR/performance-metrics.json"
rm -f "$OUTPUT_DIR/test-results.json"

export BASE_URL

npx playwright test \
  --config=qa-playwright.config.ts \
  --grep "@screenshot"

echo
echo "Generating evidence summary..."

SCREENSHOT_COUNT=$(
  find "$OUTPUT_DIR" -type f -name "*.png" | wc -l | tr -d ' '
)

echo "  Screenshots captured: $SCREENSHOT_COUNT"

if [ -f "$OUTPUT_DIR/performance-metrics.json" ]; then
  ERROR_COUNT=$(
    node -e "
      const fs = require('fs');
      const data = JSON.parse(
        fs.readFileSync('$OUTPUT_DIR/performance-metrics.json', 'utf8')
      );

      const total = Object.values(data).reduce(
        (sum, result) => sum + (result.consoleErrors?.length || 0),
        0
      );

      process.stdout.write(String(total));
    "
  )

  NETWORK_ERROR_COUNT=$(
    node -e "
      const fs = require('fs');
      const data = JSON.parse(
        fs.readFileSync('$OUTPUT_DIR/performance-metrics.json', 'utf8')
      );

      const total = Object.values(data).reduce(
        (sum, result) => sum + (result.networkErrors?.length || 0),
        0
      );

      process.stdout.write(String(total));
    "
  )

  MAX_LOAD_TIME=$(
    node -e "
      const fs = require('fs');
      const data = JSON.parse(
        fs.readFileSync('$OUTPUT_DIR/performance-metrics.json', 'utf8')
      );

      const loadTimes = Object.values(data).map(
        result => result.performance?.loadTime || 0
      );

      process.stdout.write(String(Math.max(...loadTimes)));
    "
  )

  echo "  Console errors:  $ERROR_COUNT"
  echo "  Network failures: $NETWORK_ERROR_COUNT"
  echo "  Max load time:    ${MAX_LOAD_TIME}ms"
fi

echo
echo "Reality Check complete."
echo "Review the evidence in: $OUTPUT_DIR"

Make it executable:

chmod +x qa-playwright-capture.sh

Run it against your local application:

./qa-playwright-capture.sh \
  http://localhost:8000 \
  public/qa-screenshots

Expected output:

Starting Reality Check capture...
  Base URL: http://localhost:8000
  Output:   public/qa-screenshots

Generating evidence summary...
  Screenshots captured: 18
  Console errors:  0
  Network failures: 0
  Max load time:    842ms

Reality Check complete.
Review the evidence in: public/qa-screenshots

The exact screenshot count depends on how many forms, navigation links, and interactive components your page contains.

Step 4: Verify the Implementation with Commands

Screenshots prove what the browser rendered. Source searches prove whether claimed features exist in the implementation.

Run these commands before asking an AI agent to approve the work.

1. Verify expected files

Adjust the paths for your framework:

ls -la resources/views/ 2>/dev/null || ls -la ./*.html
ls -la src/components/ 2>/dev/null || ls -la components/

2. Search for claimed visual features

grep -rE \
  "backdrop-filter|glassmorphism|blur\(" \
  . \
  --include="*.css" \
  --include="*.scss" \
  --include="*.html" \
  --include="*.tsx" \
  --exclude-dir=node_modules \
  --exclude-dir=.git \
  --exclude-dir=public \
  || echo "NO GLASSMORPHISM FOUND"

3. Search for responsive CSS

grep -rE \
  "@media|container-query|@container" \
  . \
  --include="*.css" \
  --include="*.scss" \
  --exclude-dir=node_modules \
  --exclude-dir=.git \
  --exclude-dir=public \
  || echo "NO RESPONSIVE CSS FOUND"

4. Search for authentication code

grep -rEi \
  "jsonwebtoken|jwt|authentication|authorization" \
  . \
  --include="*.ts" \
  --include="*.tsx" \
  --include="*.js" \
  --include="*.jsx" \
  --exclude-dir=node_modules \
  --exclude-dir=.git \
  --exclude-dir=public \
  || echo "NO AUTHENTICATION IMPLEMENTATION FOUND"

A grep match does not prove that a feature works. It only proves that relevant implementation code may exist. Combine source searches with runtime tests.

5. Capture browser evidence

./qa-playwright-capture.sh \
  http://localhost:8000 \
  public/qa-screenshots

6. Inspect the generated files

find public/qa-screenshots -maxdepth 1 -type f -print

Expected files include:

public/qa-screenshots/responsive-desktop.png
public/qa-screenshots/responsive-tablet.png
public/qa-screenshots/responsive-mobile.png
public/qa-screenshots/nav-desktop-0-before.png
public/qa-screenshots/nav-desktop-0-after.png
public/qa-screenshots/form-mobile-0-initial.png
public/qa-screenshots/form-mobile-0-filled.png
public/qa-screenshots/test-results.json
public/qa-screenshots/performance-metrics.json

7. Review the metrics

cat public/qa-screenshots/test-results.json
cat public/qa-screenshots/performance-metrics.json

For easier inspection, use jq if it is installed:

jq . public/qa-screenshots/performance-metrics.json

Step 5: Activate the Reality Checker Agent

Open a Claude Code session or your preferred coding-agent environment and provide a strict review prompt:

Activate Reality Checker mode.

Run the mandatory reality check process:

1. Verify expected files exist.
2. Cross-reference every claimed feature with source-code evidence.
3. Review all screenshots in public/qa-screenshots/.
4. Inspect test-results.json.
5. Inspect performance-metrics.json.
6. Report console errors and failed network requests.
7. Return exactly one final status: PASS or NEEDS WORK.

Project URL:
http://localhost:8000

Do not approve based on descriptions alone.
List every blocking issue with its supporting evidence.

A useful result should look like this:

## Reality Check Results

### File Verification: PASS

- Component files present: 12
- Expected project structure found

### Feature Verification: NEEDS WORK

- Claim: "Premium glassmorphism design"
- Evidence command: grep for `backdrop-filter`
- Result: no matching implementation found
- Status: CLAIM NOT SUPPORTED

### Screenshot Evidence: NEEDS WORK

- Desktop, 1920×1080: layout renders correctly
- Tablet, 768×1024: navigation overlap visible
- Mobile, 375×667: product grid remains two columns

### Runtime Evidence: NEEDS WORK

- Load time: 2.3s
- Project target: less than 1s
- Console errors: 3
- Network failures: 1

## Final Status: NEEDS WORK

### Blocking Issues

1. Claimed glassmorphism implementation was not found.
2. Mobile product grid is broken at 375px.
3. Load time exceeds the configured target.
4. Three browser console errors remain.
5. One network request fails.

Do not approve until the blocking issues are resolved.

A result such as “looks good overall” is not sufficient. Each conclusion should point to a command result, screenshot, or metrics file.

Step 6: Track Claims Against Evidence

Create a checklist for every project:

## Claims vs. Evidence Checklist

| Claim | Evidence | Result |
|---|---|---|
| Premium glassmorphism | Search for `backdrop-filter` | Not found |
| Fully responsive | `responsive-mobile.png` | Failed: broken grid |
| No console errors | `performance-metrics.json` | Failed: 3 errors |
| Fast load time | Navigation timing | 2.3s; target is under 1s |
| JWT authentication | Search for `jsonwebtoken` | Found |
| Rate limiting | Search for `rateLimit` | Not found |

Treat the checklist as part of the deliverable. If a claim has no evidence, mark it as unverified.

A practical review rule is:

No evidence → Unverified
Contradicting evidence → NEEDS WORK
Passing evidence → Eligible for PASS

Complete Reality Check Workflow

┌─────────────────────────────────────────────────────────────┐
│ 1. Developer or AI agent completes the implementation       │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│ 2. Collect evidence                                         │
│    - Verify files with ls/find                              │
│    - Verify implementation claims with grep                 │
│    - Capture screenshots with Playwright                    │
│    - Record console, network, and performance data          │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│ 3. Run the Reality Checker review                           │
│    - Cross-reference every claim                            │
│    - Review screenshots at each breakpoint                  │
│    - Inspect test and metrics files                         │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│ 4. Return PASS or NEEDS WORK                                │
│    - PASS: evidence supports the implementation             │
│    - NEEDS WORK: fix blockers and rerun the workflow        │
└─────────────────────────────────────────────────────────────┘

Integrate the Reality Check into GitHub Actions

Create .github/workflows/qa-reality-check.yml:

name: Reality Check

on:
  pull_request:

jobs:
  reality-check:
    runs-on: ubuntu-latest

    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright and system dependencies
        run: npx playwright install --with-deps chromium

      - name: Start application
        run: npm start > /tmp/app.log 2>&1 &
        env:
          PORT: 8000

      - name: Wait for application
        run: |
          for attempt in {1..30}; do
            if curl --fail --silent http://localhost:8000 > /dev/null; then
              echo "Application is ready"
              exit 0
            fi

            echo "Waiting for application: attempt $attempt/30"
            sleep 2
          done

          echo "Application did not start"
          cat /tmp/app.log
          exit 1

      - name: Capture QA evidence
        run: |
          ./qa-playwright-capture.sh \
            http://localhost:8000 \
            public/qa-screenshots

      - name: Validate runtime budgets
        env:
          MAX_LOAD_TIME_MS: 1000
        run: |
          node <<'NODE'
          const fs = require('fs');

          const file = 'public/qa-screenshots/performance-metrics.json';
          const maximumLoadTime = Number(
            process.env.MAX_LOAD_TIME_MS || 1000
          );

          if (!fs.existsSync(file)) {
            console.error(`Missing evidence file: ${file}`);
            process.exit(1);
          }

          const results = JSON.parse(fs.readFileSync(file, 'utf8'));

          let failed = false;

          for (const [project, result] of Object.entries(results)) {
            const consoleErrors = result.consoleErrors?.length || 0;
            const networkErrors = result.networkErrors?.length || 0;
            const loadTime = result.performance?.loadTime || 0;

            console.log(`Project: ${project}`);
            console.log(`  Console errors: ${consoleErrors}`);
            console.log(`  Network errors: ${networkErrors}`);
            console.log(`  Load time: ${loadTime}ms`);

            if (consoleErrors > 0) {
              console.error(
                `${project}: expected 0 console errors, found ${consoleErrors}`
              );
              failed = true;
            }

            if (networkErrors > 0) {
              console.error(
                `${project}: expected 0 network failures, found ${networkErrors}`
              );
              failed = true;
            }

            if (loadTime > maximumLoadTime) {
              console.error(
                `${project}: load time ${loadTime}ms exceeds ` +
                `${maximumLoadTime}ms`
              );
              failed = true;
            }
          }

          if (failed) {
            process.exit(1);
          }
          NODE

      - name: Upload QA evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: reality-check-evidence
          path: public/qa-screenshots/

This workflow:

1. Installs the application and Playwright dependencies.
2. Starts the application.
3. Waits until the server responds.
4. Captures screenshots and metrics.
5. Fails the pull request if runtime budgets are exceeded.
6. Uploads the evidence even when the job fails.

Set thresholds based on your application’s requirements. A universal one-second budget may not be appropriate for every project.

What You Built

Component	Purpose
Playwright configuration	Runs tests at desktop, tablet, and mobile breakpoints
Screenshot test suite	Captures layouts and common interactions
Runtime evidence capture	Records console errors, network failures, and load timing
Shell script	Runs evidence collection with one command
Claims checklist	Maps implementation claims to verifiable evidence
GitHub Actions workflow	Enforces evidence-based checks on pull requests

Troubleshooting

Playwright tests time out

Confirm that the application is running:

curl --fail http://localhost:8000

Increase the global timeout if the application needs more time:

export default defineConfig({
  timeout: 60_000,
});

Run Playwright in headed mode:

BASE_URL=http://localhost:8000 \
npx playwright test \
  --config=qa-playwright.config.ts \
  --headed

Open Playwright’s interactive debugger:

PWDEBUG=1 \
npx playwright test \
  --config=qa-playwright.config.ts

No screenshot tests are found

The shell script filters tests using @screenshot. Confirm that the test suite includes the tag:

test.describe('@screenshot Reality Check', () => {
  // Tests
});

You can verify discovered tests with:

npx playwright test \
  --config=qa-playwright.config.ts \
  --list

Screenshots are not created

Create the output directory manually:

mkdir -p public/qa-screenshots
chmod 755 public/qa-screenshots

Confirm that Chromium is installed:

npx playwright install chromium

Enable Playwright API logging:

DEBUG=pw:api \
npx playwright test \
  --config=qa-playwright.config.ts

Console errors are missing

Register the listener before calling page.goto():

page.on('console', message => {
  if (message.type() === 'error') {
    console.log(message.text());
  }
});

await page.goto('/');

Temporarily log every browser message:

page.on('console', message => {
  console.log(message.type(), message.text());
});

Also verify that the page rendered successfully. A blank page may indicate an application startup or routing failure.

Mobile screenshots still show the desktop layout

Set the viewport before navigation. Playwright projects do this automatically through the configuration.

Also verify that the page contains the responsive viewport meta tag:

<meta
  name="viewport"
  content="width=device-width, initial-scale=1"
/>

For device-level emulation, use Playwright’s built-in devices:

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  projects: [
    {
      name: 'iPhone 12',
      use: {
        ...devices['iPhone 12'],
      },
    },
  ],
});

CI fails while installing Chromium

Install Playwright with system dependencies:

npx playwright install --with-deps chromium

Alternatively, use the official Playwright container image:

jobs:
  reality-check:
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.40.0-jammy

Keep the container version aligned with the Playwright version in your project.

Advanced Pattern 1: Visual Regression Testing

Screenshots prove what a page currently looks like. Visual regression tests compare that output with an approved baseline.

import { test, expect } from '@playwright/test';

test('homepage matches approved baseline', async ({ page }) => {
  await page.goto('/', {
    waitUntil: 'networkidle',
  });

  await expect(page).toHaveScreenshot('homepage-base.png', {
    fullPage: true,
    maxDiffPixels: 100,
  });
});

Create or update the baseline:

npx playwright test --update-snapshots

Review baseline changes carefully. Do not automatically approve new screenshots simply because the UI changed.

Advanced Pattern 2: Accessibility Audits

Install the Playwright integration for axe-core:

npm install -D @axe-core/playwright

Add an accessibility test:

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
import * as fs from 'node:fs';

test('page has no serious accessibility violations', async ({ page }) => {
  await page.goto('/', {
    waitUntil: 'networkidle',
  });

  const results = await new AxeBuilder({ page }).analyze();

  fs.writeFileSync(
    'public/qa-screenshots/accessibility-results.json',
    JSON.stringify(results, null, 2)
  );

  const blockingViolations = results.violations.filter(
    violation =>
      violation.impact === 'critical' ||
      violation.impact === 'serious'
  );

  expect(blockingViolations).toHaveLength(0);
});

The generated JSON file gives your agent specific accessibility evidence to review.

Advanced Pattern 3: Performance Budget Enforcement

You can enforce a performance budget directly in Playwright:

import { test, expect } from '@playwright/test';

test('page stays within the performance budget', async ({
  page,
  context,
}) => {
  await page.goto('/', {
    waitUntil: 'networkidle',
  });

  const loadTime = await page.evaluate(() => {
    const [navigation] = performance.getEntriesByType(
      'navigation'
    ) as PerformanceNavigationTiming[];

    return navigation.loadEventEnd;
  });

  const session = await context.newCDPSession(page);

  await session.send('Performance.enable');

  const { metrics } = await session.send(
    'Performance.getMetrics'
  );

  const jsHeapSize =
    metrics.find(metric => metric.name === 'JSHeapUsedSize')
      ?.value || 0;

  expect(loadTime).toBeLessThan(2_000);
  expect(jsHeapSize).toBeLessThan(5 * 1024 * 1024);
});

Treat these values as examples. Choose budgets that match your application and deployment environment.

Next Steps

Extend the workflow with:

• Lighthouse reports for additional performance evidence
• axe-core accessibility checks
• Visual regression baselines
• API response validation
• Network response status checks
• Screenshot review requirements in pull requests

You can also maintain an evidence log:

Claim
Evidence source
Observed result
Expected result
Status
Reviewer
Timestamp

Over time, this gives your team a record of which claims were verified and which required rework.

The core rule remains simple:

Do not approve implementation claims without evidence.

Run the commands, inspect the screenshots, check the metrics, and require a clear PASS or NEEDS WORK.

FAQ

Why do AI agents hallucinate when reviewing code?

AI agents can generate plausible conclusions without executing the application or inspecting the relevant files. If the review prompt does not require evidence, the agent may treat descriptions and assumptions as facts.

Require command output, screenshots, metrics, and test results for every approval.

How do I set up Playwright for screenshot testing?

Install Playwright:

npm install -D @playwright/test
npx playwright install chromium

Then create a configuration with your target viewports and write tests that call page.screenshot() after loading the page.

Which reality check commands should I run before approval?

At minimum:

# Verify expected files.
find src -maxdepth 3 -type f

# Search for claimed implementation details.
grep -r "backdrop-filter" src

# Capture browser evidence.
./qa-playwright-capture.sh http://localhost:8000

# Review runtime evidence.
cat public/qa-screenshots/performance-metrics.json
cat public/qa-screenshots/test-results.json

Adapt the paths and search terms to the claims being reviewed.

What is the Reality Checker agent?

Reality Checker is a specialized AI agent from The Agency that validates work using evidence. It verifies files, searches for claimed features, reviews screenshots, checks metrics, and returns PASS or NEEDS WORK with specific blocking issues.

How do I integrate reality checks into CI/CD?

Add a pipeline that:

1. Installs Playwright and its browser dependencies.
2. Starts the application.
3. Runs the screenshot suite.
4. Uploads the generated evidence.
5. Fails when configured error or performance budgets are exceeded.

What if the screenshots show issues but the agent says PASS?

Treat the result as invalid. The final status must follow the evidence.

Update the review prompt so the agent must:

1. Cite source-code search results.
2. Cite specific screenshot files.
3. Report console and network errors.
4. Compare metrics with explicit thresholds.
5. Explain why each issue is blocking or non-blocking.

How do I get my team to adopt evidence-based QA?

Make evidence collection part of the normal pull request process:

• Upload screenshots as CI artifacts.
• Require screenshot review for UI changes.
• Define performance and error budgets.
• Include a claims-versus-evidence checklist in pull requests.
• Block approval when required evidence is missing.