How to Add Private CI Reports to Playwright Email Tests

An OTP test can pass for the wrong reason.

If several test runs share one inbox, a broad query such as "find the latest
verification email" can match a message created by another worker. The test is
green, but it has not proved that the application sent an email for the user it
just created.

Giving each run a unique recipient fixes the matching problem. There is still
one awkward question when the test runs in CI: how do you leave evidence of
what matched without printing an email address, subject, or body into a public
build log?

This article shows the approach I use with Playwright, GitHub Actions, and
@pntr/testkit.

If you only need recipient isolation, I covered that first in
Test Email OTP Flows in Playwright Without a Shared Gmail Inbox.
Here I will add an owner-only CI report to the same workflow.

The workflow

Each test run:

1. Generates its own catch-all address.
2. Submits that address through the real signup UI.
3. Waits for the exact recipient, subject, and start time.
4. Extracts the OTP from the matched message.
5. Adds a content-free report link to the GitHub Actions Step Summary.

The report is evidence that PNTR matched a message for this run. It is not a
copy of the message.

Install the test dependencies

npm install --save-dev @playwright/test @pntr/testkit

Create or select a PNTR hostname with its test inbox enabled, then create an API
token from MCP integration > API token in the dashboard.

Keep the values outside the test source:

PNTR_EMAIL_HOSTNAME=testbox.pntr.dev
PNTR_TOKEN=pntr_your_token

Use synthetic test accounts only. A catch-all test inbox should not receive
real customer mail.

Write the Playwright test

This is a complete TypeScript example. The selectors are deliberately explicit
so the test reads like the user journey it verifies.

import { appendFile } from "node:fs/promises";
import { expect, test } from "@playwright/test";
import {
  createRecipient,
  extractOtp,
  formatReportMarkdown,
  PntrTestKit,
} from "@pntr/testkit";

function required(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`${name} is required`);
  return value;
}

const inboxHostname = required("PNTR_EMAIL_HOSTNAME");
const pntr = new PntrTestKit({
  token: required("PNTR_TOKEN"),
});

test("a user can finish signup with the emailed OTP", async ({ page }) => {
  const recipient = createRecipient(inboxHostname, {
    prefix: "signup",
  });
  const startedAt = new Date();

  await page.goto("/signup");
  await page.getByTestId("signup-email").fill(recipient);
  await page.getByTestId("signup-submit").click();

  const match = await pntr.waitForEmailWithReport(inboxHostname, {
    recipient,
    subject: "verification",
    since: startedAt,
    timeoutSeconds: 25,
  });

  test.info().annotations.push({
    type: "pntr-report",
    description: match.report.url,
  });

  if (process.env.GITHUB_STEP_SUMMARY) {
    await appendFile(
      process.env.GITHUB_STEP_SUMMARY,
      formatReportMarkdown(match.report),
    );
  }

  const otp = extractOtp(match, /\b(\d{6})\b/);

  await page.getByTestId("otp-code").fill(otp);
  await page.getByTestId("otp-submit").click();
  await expect(page.getByTestId("account-home")).toBeVisible();
});

There are three details worth calling out.

1. The address belongs to one run

createRecipient() returns an address such as:

[e-mail]

The hostname has a catch-all inbox, so there is no mailbox-provisioning step.
Every parallel worker can use a different local part.

If your CI system already exposes a stable run identifier, you can include it:

const recipient = createRecipient(inboxHostname, {
  prefix: "signup",
  runId: process.env.GITHUB_RUN_ID,
});

2. The wait is narrow

The query combines the exact recipient with a subject fragment and a
startedAt boundary. An old message for the same address cannot satisfy the
test.

The wait happens server-side. The Playwright worker does not need to download
and scan an entire inbox every second.

3. The report is created only after a match

waitForEmailWithReport() returns the normal email result plus a report
object. If no message matches before the timeout, the wait fails and no report
is created.

That distinction matters. The report confirms a successful match. It is useful
when a later assertion fails or when you want an audit trail for a green run,
but it is not a postmortem for an email that never arrived.

Add it to GitHub Actions

Store the token as an Actions secret. The hostname is not sensitive, so it can
be a repository variable.

name: Playwright OTP

on:
  workflow_dispatch:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    env:
      PNTR_TOKEN: ${{ secrets.PNTR_TOKEN }}
      PNTR_EMAIL_HOSTNAME: ${{ vars.PNTR_EMAIL_HOSTNAME }}
      PLAYWRIGHT_BASE_URL: ${{ vars.PLAYWRIGHT_BASE_URL }}

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Install Chromium
        run: npx playwright install --with-deps chromium

      - name: Run the signup OTP test
        run: npx playwright test tests/signup-otp.spec.ts

GITHUB_STEP_SUMMARY is provided by GitHub Actions. The test appends a small
Markdown table to it after the match.

The summary contains:

• Match status
• Email or webhook report type
• PNTR hostname
• Match time
• Report expiry time
• A link to the owner-only report

The formatter deliberately ignores message content and even ignores the
report's summary object. This keeps a future metadata addition from
accidentally appearing in CI output.

What the private report does not store

The durable report does not copy:

• Sender or recipient addresses
• Email subject or body
• Webhook path, query, headers, source IP, or payload
• API tokens

Opening the report URL requires the PNTR account that owns the hostname. The
report follows the account's current retention tier: 48 hours on Free and 90
days on Premium.

The matched email still exists in the test inbox according to the inbox's own
retention rules. The report is a separate, deliberately smaller record.

Why not print the email into the job log?

CI logs tend to live longer and reach more people than expected. They may be
copied into bug reports, retained as artifacts, or exposed to contributors who
can read job output but should not read test messages.

For a synthetic OTP workflow, the minimum useful evidence is usually:

This run matched an email on this hostname at this time.

Anything more should be an explicit debugging decision, not the default output
of every successful build.

Try the same pattern

You can create a test hostname and API token from the
PNTR TestKit setup.
The package and its source are public:

• @pntr/testkit on npm
• oddunits/pntr-testkit on GitHub
• TestKit documentation and examples

The ordinary waitForEmail() and waitForWebhook() methods remain available
when a run does not need a durable report.

Disclosure: I built PNTR and @pntr/testkit. The workflow above works on the
free plan; the pricing and retention differences are listed on the linked
TestKit page.