DEV Community: Christian Potvin

Why Your CI Email Tests Are Flaky (And How to Fix Them)

Christian Potvin — Tue, 17 Mar 2026 19:01:04 +0000

TL;DR: If your Playwright/Cypress/Pytest tests involving email keep failing intermittently in CI, the cause is almost always shared inbox state. The fix is per-test inbox isolation, not longer timeouts.

The Symptom

Your test suite has a test that looks reasonable:

✓ user can register (3.2s)
✗ user receives welcome email (timeout after 30s)
✓ user receives welcome email (2.1s)  ← same test, re-run, now passes

The test passes locally. It fails in CI. It passes again on retry. Your team has learned to just re-run the pipeline and move on. The underlying problem is never fixed.

This is not an infrastructure problem. It's not "CI is slower than local." It's a state isolation problem.

The Root Cause

When email-dependent tests share an inbox, several things go wrong.

Race conditions in parallel runs

CI pipelines run tests concurrently. If you're using -n 4 in Pytest or --workers=4 in Playwright, four tests may be signing up simultaneously — all using the same test@yourcompany.com address.

Worker 1 signs up. Worker 2 signs up. Both trigger welcome emails. Worker 1 reads the inbox and finds two emails. Which one is its? It doesn't know. It reads the wrong one. The test either fails on assertion or false-passes.

Stale emails from previous runs

A test passes. The inbox now contains a "Welcome" email. The next test run starts. It signs up again, polls the inbox, and immediately finds the email from the previous run — before the new email has even been sent. The test passes — but it's asserting on stale data.

This is why "it works locally, fails in CI" is common: locally you likely run tests serially, so stale emails are less frequent. CI runs them fast and parallel.

The "longer timeout" trap

The instinct is to increase the wait:

time.sleep(10)  # give the email time to arrive

This doesn't fix the race condition. It makes your test suite slower and still non-deterministic.

Why the Obvious Solutions Don't Fully Work

Mocking the email service

If you control the email service, you can intercept emails before sending. But:

Many teams use third-party auth providers (Auth0, Cognito, Firebase) that send emails you can't intercept
Mocking removes confidence in the real integration
It doesn't test whether the email was actually sent, properly formatted, or delivered

A dedicated test Gmail account

Better than sharing, but still has problems:

Gmail IMAP is rate-limited; fast parallel tests will hit limits
You still need to clean up emails between runs (fragile teardown)
Shared between all test workers unless you manage multiple accounts

Mailhog / MailDev (self-hosted SMTP)

Great for testing your own email service, but:

Doesn't work when the email originates from a third party (Cognito SES, SendGrid, etc.)
Requires infrastructure in CI (Docker, port mapping)
Shared inbox per service instance (same race condition problem unless you implement per-test routing)

The Fix: Per-Test Inbox Isolation

The correct architecture is:

Before each test, create a fresh, unique email inbox
Use the inbox's address in the test
Poll that specific inbox for the expected email
The inbox expires automatically — no teardown

This eliminates shared state entirely. Each test is completely independent. Parallel execution is safe by design.

How to Implement It

Python (Pytest)

# conftest.py
import pytest, requests, time, os

API_KEY = os.environ["MINUTEMAIL_API_KEY"]
BASE = "https://api.minutemail.co/v1"
HDRS = {"Authorization": f"Bearer {API_KEY}"}

@pytest.fixture
def inbox():
    mb = requests.post(f"{BASE}/mailboxes", headers=HDRS, json={"domain": "minutemail.cc", "expiresIn": 10}).json()
    yield mb
    # expires automatically

def wait_for_email(mailbox_id, timeout=30):
    deadline = time.time() + timeout
    while time.time() < deadline:
        msgs = requests.get(f"{BASE}/mailboxes/{mailbox_id}/mails", headers=HDRS).json().get("items", [])
        if msgs: return msgs[0]
        time.sleep(2)
    raise TimeoutError("No email")

TypeScript (Playwright)

// fixtures/email.ts
import { test as base } from '@playwright/test';

const headers = { 'Authorization': `Bearer ${process.env.MINUTEMAIL_API_KEY}` };
const BASE = 'https://api.minutemail.co/v1';

export const test = base.extend({
  inbox: async ({}, use) => {
    const mb = await fetch(`${BASE}/mailboxes`, { method: 'POST', headers, body: JSON.stringify({ domain: 'minutemail.cc', expiresIn: 10 }) }).then(r => r.json());
    await use({
      address: mb.address,
      waitForEmail: async (timeout = 30000) => {
        const deadline = Date.now() + timeout;
        while (Date.now() < deadline) {
          const { items } = await fetch(`${BASE}/mailboxes/${mb.id}/mails`, { headers }).then(r => r.json());
          if (items.length) return items[0];
          await new Promise(r => setTimeout(r, 2000));
        }
        throw new Error('Email timeout');
      }
    });
  }
});

With this pattern, parallel CI becomes stable by design, not by accident.

What to Use for the Inbox Service

You need a hosted service with:

Per-inbox API: create a new inbox on demand, get a unique address back
TTL control: inbox expires automatically (no cleanup logic needed)
Message polling API: read received emails programmatically
Reliable delivery: must actually receive emails from third-party senders

A few options:

Service	API	Per-inbox TTL	Free tier	Notes
Mailinator	Limited	No	Yes	Known domains often blocked
Mailtrap	Yes	No	Yes (limited)	Better for SMTP testing
MinuteMail	Yes	Yes (1–60 min)	Yes (100 calls/day)	Built for this use case

MinuteMail was built specifically for developer/QA use cases: each POST /mailboxes returns a unique address with configurable TTL. Full docs at https://docs.minutemail.co.

The Outcome

After switching to per-test inbox isolation:

Tests that were "retry to fix" become deterministic
Parallel CI works correctly without coordination between workers
You get confidence that the full email flow works end-to-end, not just that your code tried to send

The pattern requires ~20 lines of fixture code. The payoff is a test suite you can trust.

Tags: #testing #devops #automation #ci

Created: 2026-02-26

Per-Test Email Inbox Isolation with Pytest and MinuteMail

Christian Potvin — Tue, 10 Mar 2026 19:00:52 +0000

TL;DR: Replace your shared test email account with a per-test disposable inbox. One Pytest fixture, zero infrastructure, works in parallel CI.

The Problem

You have a Pytest test that exercises a real sign-up or notification flow:

def test_registration_sends_welcome_email():
    # 1. Register a user
    # 2. Check the inbox for a welcome email
    # 3. Assert subject line and content

You're using a shared test@yourcompany.com inbox. It works in isolation but breaks the moment you run tests in parallel — multiple workers fight over the same inbox. One test reads the email meant for another. A stale email from the previous run triggers a false pass.

The fix is not "add a longer sleep." The fix is inbox isolation.

Why Shared Inboxes Break Parallel Tests

When two tests share the same email address:

Test A registers user_a@shared-inbox.com
Test B registers user_b@shared-inbox.com (same address, different user object)
Test A polls the inbox and reads Test B's email
Test A asserts on the wrong user's welcome message → false pass or false fail

This gets worse at scale. 8 workers, 8 tests, 1 inbox = chaos.

The Solution: Per-Test Disposable Inboxes

Create a fresh inbox before each test. Give that test's unique address to the sign-up form. Poll only that inbox. Let it expire when done.

Implementation with MinuteMail

MinuteMail has a REST API for creating short-lived mailboxes and reading their contents. There's also an open-source Python SDK.

Install the SDK:

pip install minutemail

Set your API key:

export MINUTEMAIL_API_KEY=your_key_here

The Fixture

# conftest.py

import pytest
import os
import time
import requests

API_KEY = os.environ["MINUTEMAIL_API_KEY"]
BASE_URL = "https://api.minutemail.co/v1"
HEADERS = {"Authorization": f"Bearer {API_KEY}"}


@pytest.fixture
def isolated_inbox():
    """Creates a fresh MinuteMail inbox for each test. Expires automatically."""
    response = requests.post(
        f"{BASE_URL}/mailboxes",
        headers=HEADERS,
        json={"domain": "minutemail.cc", "expiresIn": 10},  # 10-minute TTL — more than enough for a test
    )
    response.raise_for_status()
    mailbox = response.json()

    yield mailbox  # provide { "id": ..., "address": ..., "expiresAt": ... }

    # Inbox expires on its own — no teardown needed


def wait_for_email(mailbox_id: str, timeout: int = 30) -> dict:
    """Poll the inbox until an email arrives or timeout is reached."""
    deadline = time.time() + timeout
    while time.time() < deadline:
        response = requests.get(
            f"{BASE_URL}/mailboxes/{mailbox_id}/mails",
            headers=HEADERS,
        )
        response.raise_for_status()
        messages = response.json().get("items", [])
        if messages:
            return items[0]
        time.sleep(2)
    raise TimeoutError(f"No email received in mailbox {mailbox_id} within {timeout}s")

The Test

# test_registration.py

import re


def test_welcome_email_sent_after_registration(client, isolated_inbox):
    # isolated_inbox["address"] is a unique address for this test only
    email_address = isolated_inbox["address"]

    response = client.post("/register", json={
        "email": email_address,
        "password": "SecurePass123!"
    })
    assert response.status_code == 201

    # Poll for the welcome email
    email = wait_for_email(isolated_inbox["id"], timeout=30)

    assert "Welcome" in email["subject"]
    assert email_address in email["body"]


def test_password_reset_email_contains_link(client, isolated_inbox):
    # Each test gets its own inbox — no interference from other tests
    email_address = isolated_inbox["address"]

    # Create and then request reset for this user
    client.post("/register", json={"email": email_address, "password": "OldPass123!"})
    client.post("/reset-password", json={"email": email_address})

    email = wait_for_email(isolated_inbox["id"], timeout=30)

    # Extract reset link
    link = re.search(r'https?://\S+reset\S+', email["body"])
    assert link is not None, "Reset link not found in email body"

Running in Parallel

pip install pytest-xdist
pytest -n 8  # 8 parallel workers, each with its own inbox

Because each worker gets its own isolated_inbox fixture instance with a unique address, there is zero shared state between workers.

Using the Python SDK (Alternative)

If you'd rather use the official SDK than raw requests:

# conftest.py (SDK version)

import pytest
from minutemail import MinuteMail

client = MinuteMail(api_key=os.environ["MINUTEMAIL_API_KEY"])


@pytest.fixture
def isolated_inbox():
    mailbox = client.create_mailbox(ttl=10)
    yield mailbox


def wait_for_email(mailbox, timeout=30):
    import time
    deadline = time.time() + timeout
    while time.time() < deadline:
        messages = mailbox.get_messages()
        if messages:
            return items[0]
        time.sleep(2)
    raise TimeoutError("No email received")

SDK source: https://github.com/minutemailco/minutemail-python

CI/CD Integration

# .github/workflows/test.yml
- name: Run tests
  env:
    MINUTEMAIL_API_KEY: ${{ secrets.MINUTEMAIL_API_KEY }}
  run: pytest -n 4 --tb=short

Free tier: 100 API calls/day. Each test uses 2–5 calls (create + poll attempts). For larger CI pipelines with many test workers, the Pro plan gives 10,000 calls/day.

Handling Slow Email Providers

Some email providers (particularly during cold starts or heavy load) can take 10–30 seconds. Tune the timeout and poll interval:

def wait_for_email(mailbox_id, timeout=60, poll_interval=3):
    deadline = time.time() + timeout
    while time.time() < deadline:
        messages = requests.get(
            f"{BASE_URL}/mailboxes/{mailbox_id}/mails",
            headers=HEADERS
        ).json().get("items", [])
        if messages:
            return items[0]
        time.sleep(poll_interval)
    raise TimeoutError(...)

Summary

Shared inboxes cause non-deterministic failures in parallel test suites
Per-test inbox isolation eliminates the shared-state problem entirely
A pytest.fixture wrapping a disposable inbox API makes this a one-liner in each test
The MinuteMail API + Python SDK handles the mailbox lifecycle; tests just yield and move on

Tags: #testing #python #pytest #automation

Created: 2026-02-26

Persona-based testing with AI agents: find the UX gaps your E2E tests can't see

Christian Potvin — Tue, 03 Mar 2026 20:01:31 +0000

TL;DR: Give an AI agent a user persona and no map of your product. Make it state what it expects before every click. Compare expectation to result. Anything that doesn't match is a UX gap your functional tests will never find. At the end of the session, ask it for a score. The scores are humbling.

The blind spot you can't fix by writing more tests

Your E2E test suite passes. Every assertion is green. The sign-up flow works, the confirmation email arrives, the dashboard loads.

And then a real user lands on your product and gets confused at the third step.

Not because it's broken. Because it wasn't legible.

This is the gap that functional testing can't close. A test asserts that clicking a button triggers an event. It doesn't assert that a new user understood what the button was for before clicking it. That gap lives entirely outside the test runner — and it only surfaces in production, when someone who doesn't think like you tries to use something you built while thinking like yourself.

Persona-based AI testing is an attempt to close that gap before launch.

What persona-based testing actually is

It's not a new testing framework. It's not an AI that automatically generates test cases. It's closer to a think-aloud usability session — the kind a UX researcher would run with a real participant — except the participant is an AI loaded with a specific user identity and given access to your product cold.

The core idea:

Define a persona — who this person is, what they want, how patient they are, how technical they are
Give the agent that persona and a starting URL, nothing else
The agent navigates your product as that person would — reading, clicking, filling forms
Before every action, the agent states what it expects
After every action, it records what actually happened
Any mismatch between expectation and result gets flagged as an unclear behavior
At the end, the agent steps out of role and gives you a score + a list of what would have changed their experience

The output isn't a pass/fail report. It's a legibility audit from a specific type of user.

The personas that reveal the most

Not all personas are equally useful. The ones that find the most interesting issues are the ones furthest from your own mental model.

The power user of a similar tool — someone who has used a competitor and arrives with assumptions baked in. They'll try keyboard shortcuts that don't exist, expect features in locations where you didn't put them, and flag every terminology difference between your product and what they're used to.

The non-technical professional with a legitimate use case — a lawyer, an accountant, a project manager. They have real reasons to use your product but zero tolerance for developer vocabulary. They will tell you exactly which words stopped them and which interactions felt unsafe.

The impatient first-time visitor — five seconds to understand what this is. Ten seconds to decide whether to continue. Anything that requires reading a paragraph to understand is already a failure for this persona.

The compliance-minded user — this one surprised me the most when I ran it. The persona asks: can I export my data? Can I delete my account? Where is the privacy policy? What data do you store? These aren't power-user concerns. They're table stakes for anyone in a regulated environment — and they're the exact questions most developer-built products can't answer cleanly.

The expectation-before-action loop

This is the mechanism that makes the approach work.

Standard automated tests look like this:

do thing → assert result

Persona testing looks like this:

state expectation → do thing → compare result to expectation → flag gap if mismatch

The addition of the expectation step forces the agent to engage with the UI as a reader before engaging with it as an actor. That's what real users do — they read, form a mental model, then act based on that model. When the mental model is wrong, the UX failed.

The prompt instruction I've settled on:

Before every click, fill, navigation, or form submission, write one sentence 
starting with "I expect...". 

After every action, write one sentence starting with "The result was...". 

If the result contradicts the expectation in any way, write:
[UNCLEAR BEHAVIOR]: describe what you expected, what happened, and why the gap matters 
for someone with this persona.

An example output from a developer persona testing a product with a bulk-action feature absent:

I expect to be able to select multiple items and delete them in one action,
since I have accumulated a large number of records during testing.

[action: look for a select-all checkbox or bulk action menu]

The result was: no bulk selection exists. Each item has an individual delete button.

[UNCLEAR BEHAVIOR]: For a user managing high volumes — which is the core use case for 
a developer in a testing workflow — deleting items one by one is not a viable 
interaction. This will feel broken even though it technically works. A checkbox column 
and a "Delete selected" action would resolve this entirely.

That's not something a functional test would ever flag. The deletion works. The problem is the interaction cost at volume.

Handling registration flows: the email problem

Most products have a sign-up step that requires a real working email address. You want to test the full flow — the confirmation email arrives, the link works, the user lands on the right page. You can't mock this for a persona session; the whole point is to test the real path.

The cleanest solution I've found is to give the agent access to a disposable email MCP server alongside Playwright MCP. The agent creates a fresh inbox as part of the session, uses that address to register, reads the confirmation email natively, and continues — all in the same tool-calling context, no tab-switching, no clipboard gymnastics.

I used this approach when running persona sessions on my own platform, MinuteMail — which, conveniently, is a disposable email service itself. The agent used MinuteMail to generate the throwaway address it needed to register for a MinuteMail account. It's a bit recursive, but it worked cleanly.

The MCP server for MinuteMail is open source at chrptvn/mcp-servers if you want to look at how it's structured. The pattern generalises to any service with an API — the agent tool call looks roughly like:

[TOOL: minutemail_create_mailbox]
params: { expiresIn: 15 }
→ { id: "mbx_abc123", address: "qk7r2@minutemail.cc" }

I expect this address to function as a valid email recipient for the sign-up form.

[TOOL: playwright_fill] → input[name="email"] → "qk7r2@minutemail.cc"
[TOOL: playwright_click] → button[type="submit"]

I expect a confirmation email to arrive within 60 seconds.

[TOOL: minutemail_list_messages] → polling mailboxId "mbx_abc123"
→ { items: [{ subject: "Confirm your account", ... }] }

The result was: email arrived in approximately 9 seconds. Subject matched expectation.

[TOOL: minutemail_get_message] → extract confirmation URL from body
[TOOL: playwright_navigate] → confirmation URL

The session reads like a coherent narrative. Browser state and inbox state interleave naturally because both are just tool calls to the agent.

Real examples: what the personas actually found

I ran three persona sessions on MinuteMail during early development. Here are two findings that led to real product changes.

The developer persona and bulk delete

The developer persona was evaluating MinuteMail as a testing tool for a CI pipeline. The use case: spin up many mailboxes during a test run, clean up afterward.

After creating a dozen test mailboxes, the persona went to clean up:

I expect to be able to select all mailboxes and delete them at once, since 
managing a large number of test inboxes is the primary developer use case.

[action: look for a select-all or bulk delete control]

The result was: no bulk selection exists. Each mailbox has an individual delete button 
in its row.

[UNCLEAR BEHAVIOR]: A developer managing test infrastructure will accumulate many 
mailboxes. Deleting them individually is not a workable workflow. This makes the 
cleanup step — which is table stakes for any testing tool — unnecessarily painful. 
Bulk delete should be a first-class action.

I hadn't thought about this because I'd been testing with one or two mailboxes at a time. The persona was right: bulk delete is normal when you look at it from the developer's workflow. It got built.

The lawyer persona and a cluster of compliance gaps

The compliance-minded persona — a lawyer evaluating the platform for professional use — was the most thorough session I ran. It didn't find one problem. It found a list.

The session produced multiple [UNCLEAR BEHAVIOR] flags across the product, nearly all of them in the same category: the platform didn't give the user enough control over their own data and identity. Account deletion was the most concrete example:

I expect to be able to delete my account from the settings page, as this is 
a standard requirement for any platform handling personal data.

[action: navigate to settings, look for account deletion]

The result was: no account deletion option exists anywhere in the UI.

[UNCLEAR BEHAVIOR]: The absence of account deletion is a significant concern for any 
user in a professional or regulated context. It implies either that data is retained 
indefinitely against the user's wishes, or that the platform has not considered its 
obligations to users. This alone would prevent adoption in any environment where 
data governance matters.

But it was one of several flags in the same vein. By the end of the session, the persona had built up a picture of a product that worked technically but hadn't been evaluated through a compliance or data-governance lens at all.

End-of-session UX score: 2/10. The note: "I cannot recommend this platform to clients."

Each issue in the list got addressed. When the same persona session was re-run after the fixes, it scored 9/10.

That delta — from 2 to 9 — came entirely from things that were invisible to functional tests. The tests were green the whole time. What changed was the product's legibility to a specific type of user with specific concerns.

This is also the most useful property of persona-based testing: the issues aren't random. They cluster by persona. The developer persona found workflow efficiency problems. The lawyer persona found data governance problems. Each session is a coherent audit from a specific angle, not a random list of bugs.

Setting up a session

You need:

A Copilot CLI session (or any agent runtime that supports MCP tool use)
Playwright MCP configured
A disposable email MCP if your product has a sign-up flow (MinuteMail's is at chrptvn/mcp-servers, or use any inbox API you prefer)
A persona prompt

The persona prompt template I use:

You are [name], a [role/background]. 

You have heard about [product] and want to [goal].
You have no prior knowledge of how the platform works.

Your characteristics:
- Technical level: [high / medium / low]
- Patience: [high / medium / low]  
- Main concern: [speed / reliability / compliance / cost / simplicity]

Rules for this session:
1. Before every click, fill, navigation, or form submission: write "I expect..."
2. After every action: write "The result was..."
3. If result contradicts expectation: write "[UNCLEAR BEHAVIOR]: " and explain why 
   it matters for someone with your background.
4. Do not look for workarounds. If something isn't obvious, flag it and stop.

Starting URL: [your product URL]

At the end of the session, provide:
- A UX score from 1–10
- The single biggest friction point
- One change that would have most improved your confidence in the product

Keep the persona description short. The more you tell the agent about your product, the less useful the session is — you want cold navigation, not an informed walkthrough.

What this catches that normal testing misses

To be direct about where this fits in a testing strategy:

What	Caught by	Missed by
Feature doesn't work	E2E tests	—
Feature works but is confusing	Persona testing	E2E tests
Terminology is opaque to non-technical users	Persona testing	E2E tests
Missing feature obvious to target user	Persona testing	E2E tests
Compliance gap (no delete, no export)	Compliance persona	E2E tests
Interaction cost at volume (bulk actions)	Power-user persona	E2E tests

Use both. E2E tests tell you the product works. Persona sessions tell you the product is usable. You need both signals before you ship to real users.

Limitations

A few things this approach won't give you:

It won't simulate real emotional friction. An AI persona can report "this is unclear" but can't feel the frustration of a real person who closes the tab and never comes back.

The prompt shapes the output. If your persona prompt doesn't include compliance as a concern, the agent won't surface compliance gaps. Think carefully about what each persona actually cares about.

It's not a substitute for talking to real users. Persona sessions are fast and cheap — run them before user interviews to filter out the obvious issues. That way your user research time goes toward the subtle stuff.

If you run this on your own product, the non-technical persona will find the most things. Every product I've seen tested this way has at least one piece of vocabulary that makes complete sense to the builder and zero sense to anyone outside their industry. Finding it before launch is worth the hour.

Summary

Persona-based AI testing finds legibility gaps that functional tests can't — because tests assert on behavior, not on communication clarity
The expectation-before-action loop is the key mechanism: it forces the agent to engage as a reader before acting, the way real users do
Different personas surface different categories of issues — developers find workflow friction, compliance-minded users find governance gaps
For sign-up flows, pair Playwright MCP with a disposable inbox MCP so the agent handles the full registration sequence without manual intervention
Use persona sessions as a fast first pass before user interviews — they filter the obvious, so your research time goes toward the subtle
The compliance persona is the most surprising: if your product isn't ready for a lawyer, it's probably not ready for anyone in a professional context

Tags: #testing #ai #webdev #ux

Created: 2026-03-01

Testing Email Verification Flows with Playwright and a Disposable Inbox API

Christian Potvin — Sat, 28 Feb 2026 21:20:14 +0000

TL;DR: Per-test inbox isolation solves the flaky-email-test problem. This tutorial shows how to build a Playwright TypeScript fixture that creates a fresh disposable inbox per test, polls for the verification email, and extracts the OTP or magic link — using the MinuteMail API.

The Problem

Your sign-up flow looks like this:

User fills in the registration form
Your backend sends a verification email
User clicks the link (or enters the OTP)
User lands on the dashboard

You want to test this end-to-end in Playwright. So you write the test, use a shared test@yourdomain.com inbox, and it works — until you run two tests in parallel. Now test A reads the email that was meant for test B. Or test B reads a stale email from the previous run. The test suite becomes non-deterministic.

This is not a Playwright bug. It's a state isolation problem.

Why the Obvious Solutions Fall Short

Mocking the email service: Fast, but misses the real integration. If you're testing Auth0, Cognito, or Firebase Auth, you can't intercept the email before it's sent. The whole point is to verify the third-party service sends the right thing.

Shared Gmail account + IMAP polling: Fragile. Gmail rate-limits IMAP connections. You still have the shared-state problem. "Wait 5 seconds" fixes nothing when tests run in parallel.

Mailhog or MailDev (self-hosted SMTP): Great for testing your own email service, but useless when the email originates from a third-party provider that sends to real SMTP. You can't redirect Cognito's SES to your local Mailhog.

Mailinator: Known temp-email domain, increasingly blocked by sign-up forms and auth providers. No API ergonomics. No per-test TTL control.

The Right Approach: Per-Test Inbox Isolation

The fix is to give each test its own fresh inbox. Requirements:

Create a new inbox on demand (before each test)
Get a unique email address for that inbox
Poll the inbox for incoming mail
Extract OTP or link from the email body
Inbox expires automatically after the test (no cleanup needed)

This is exactly what a hosted disposable email API is for.

Implementation

We'll use MinuteMail — it has a REST API with per-mailbox TTL and a free tier (100 API calls/day) that's sufficient for local dev.

Get an API key from https://minutemail.co after signing up. Set it as an environment variable:

export MINUTEMAIL_API_KEY=your_key_here

Step 1: Create the helper utilities

// tests/helpers/email.ts

const BASE_URL = 'https://api.minutemail.co/v1';
const API_KEY = process.env.MINUTEMAIL_API_KEY!;

const headers = {
  'Authorization': `Bearer ${API_KEY}`,
  'Content-Type': 'application/json',
};

export interface Mailbox {
  id: string;
  address: string;
  expiresAt: string;
}

export interface Message {
  id: string;
  subject: string;
  from: string;
  body: string;
  received_at: string;
}

export async function createMailbox(ttlMinutes = 10): Promise<Mailbox> {
  const response = await fetch(`${BASE_URL}/mailboxes`, {
    method: 'POST',
    headers,
    body: JSON.stringify({ ttl: ttlMinutes }),
  });

  if (!response.ok) throw new Error(`Failed to create mailbox: ${response.status}`);
  return response.json();
}

export async function waitForEmail(
  mailboxId: string,
  { timeout = 30000, pollInterval = 2000 } = {}
): Promise<Message> {
  const deadline = Date.now() + timeout;

  while (Date.now() < deadline) {
    const response = await fetch(`${BASE_URL}/mailboxes/${mailboxId}/mails`, { headers });
    const { items }: { messages: Message[] } = await response.json();

    if (items.length > 0) return items[0];
    await new Promise(resolve => setTimeout(resolve, pollInterval));
  }

  throw new Error(`No email received within ${timeout}ms`);
}

Step 2: Create a Playwright fixture

// tests/fixtures/email-fixture.ts

import { test as base } from '@playwright/test';
import { createMailbox, waitForEmail, Mailbox, Message } from '../helpers/email';

type EmailFixtures = {
  inbox: {
    mailbox: Mailbox;
    waitForEmail: (options?: { timeout?: number }) => Promise<Message>;
  };
};

export const test = base.extend<EmailFixtures>({
  inbox: async ({}, use) => {
    const mailbox = await createMailbox(10); // 10-minute TTL

    await use({
      mailbox,
      waitForEmail: (options) => waitForEmail(mailbox.id, options),
    });

    // Mailbox expires automatically — no cleanup needed
  },
});

export { expect } from '@playwright/test';

Step 3: Write the test

// tests/registration.spec.ts

import { test, expect } from './fixtures/email-fixture';

test('user can complete email verification after sign-up', async ({ page, inbox }) => {
  await page.goto('/register');

  // Use the unique inbox address — isolated from other parallel tests
  await page.fill('[data-testid=email]', inbox.mailbox.address);
  await page.fill('[data-testid=password]', 'SecurePass123!');
  await page.click('[data-testid=submit]');

  await expect(page.getByText('Check your email')).toBeVisible();

  // Poll for the verification email
  const email = await inbox.waitForEmail({ timeout: 45000 });

  // Extract 6-digit OTP (adjust regex for your format)
  const otp = email.body.match(/\b\d{6}\b/)?.[0];
  expect(otp, 'OTP not found in email body').toBeDefined();

  await page.fill('[data-testid=otp]', otp!);
  await page.click('[data-testid=verify]');

  await expect(page).toHaveURL('/dashboard');
});

Step 4: Run in parallel with confidence

Each test worker gets its own inbox fixture — a completely separate MinuteMail address. No shared state. You can run with --workers=8 and every test reads only its own email.

npx playwright test --workers=8

CI/CD Configuration

Add the API key to your CI secrets:

# .github/workflows/e2e.yml
- name: Run E2E tests
  env:
    MINUTEMAIL_API_KEY: ${{ secrets.MINUTEMAIL_API_KEY }}
  run: npx playwright test

The free tier handles 100 API calls/day. Each test uses 1–2 calls (create mailbox + poll). For larger CI pipelines, the Pro plan ($15/month) gives 10,000 calls/day.

What About the Email Delivery Delay?

The waitForEmail helper polls every 2 seconds with a 30-second default timeout. In practice, transactional emails from services like Cognito or SendGrid arrive within 2–5 seconds. The 30-second timeout is a safety net, not the expected wait.

If your email provider is consistently slow, increase the timeout:

const email = await inbox.waitForEmail({ timeout: 60000 });

Summary

Shared inboxes cause flaky parallel tests — fix it with per-test isolation
A hosted disposable inbox API (MinuteMail, or similar) makes this clean and infrastructure-free
The Playwright fixture pattern wraps this in a reusable, type-safe abstraction
Free tier is enough for local development; scale up for CI

Showcasing MinuteMail: Secure Disposable Inboxes with API Access

Christian Potvin — Fri, 04 Jul 2025 02:18:37 +0000

I built MinuteMail, a temporary email service that lets you generate a disposable inbox secured by your browser session. All inboxes auto-expire, and a free API lets developers generate aliases for testing or CI workflows.

Features:

Generate @minutemail.co addresses in one click
Mailbox is deleted automatically after 1h
Read emails without manual login
Handles email attachments
Free REST API for automation

Check it out: MinuteMail

Would you find this useful in your testing workflows? Feedback is welcome!