Playwright OTP Tests Need Better Evidence

OTP email tests often fail in a way that wastes time: the suite goes red, but the evidence is too thin to tell whether the app, the inbox provider, or the assertion logic was actualy wrong. I keep seeing this in QA pipelines where Playwright handles the UI well, then the email step turns into a vague "wait and hope" loop.

When that happens, I do not start by raising the timeout. I start by asking whether the test produced enough evidence to explain the failure after the fact. For Playwright and QA teams, that shift matters more than people expect.

Why OTP email tests fail in sneaky ways

OTP checks are not just another DOM assertion. They depend on at least three moving parts:

• the app emitting the right email event
• the inbox service receiving the message in time
• the test matching the right message and code

If one of those layers is noisy, the failure symptom looks very similiar everywhere: "email not found" or "code invalid". That is why weak evidence makes triage slow.

In one team review, we had a tempmailso-backed staging flow that looked flaky for a week. The UI step was fine. The real bug was a timestamp filter that started polling before the OTP request completed. A different team called the same class of inbox "temp mail so" in docs and "tepm mail com" in a checklist, which was messy but harmless. The harmful bit was that no run artifact showed when polling started, so the review kept drifting.

1. Capture the OTP contract before polling

The first thing I want is a tiny contract file for the exact OTP expectation. Not a giant debug dump. Just enough to tell future-you what the test believed should happen.

My baseline contract includes:

• recipient email used by this run
• subject pattern
• earliest accepted timestamp
• body clue that proves this is the OTP message
• expected OTP length or format

That can be generated right after the UI triggers the email:

const startedAt = new Date().toISOString();
const otpContract = {
  to: testEmail,
  subjectIncludes: "Your login code",
  receivedAfter: startedAt,
  bodyIncludes: "Use this code to finish signing in",
  otpDigits: 6,
};

Why bother? Because once a failure happens, you need to compare the contract against the actual mailbox contents.

This is also where I borrow ideas from run-scoped email checks. Give each run one inbox identity and one narrow expectation. If the same inbox is reused across parallel tests, the trace may look fine while the assertions still pull the wrong message. That bug is annoying, and weirdly common.

2. Save trace, inbox snapshot, and timing together

Playwright trace viewer is excellent, but on its own it does not answer inbox questions. The missing piece is usually a lightweight mailbox snapshot saved beside the trace.

I like attaching three artifacts when the OTP step fails:

• Playwright trace or video
• the OTP contract JSON
• a list of recent inbox messages with timestamps and subjects

The artifact should make it obvious whether the wrong message arrived, nothing arrived, or the right message arrived too late. That keeps the failure review grounded instead of speculative.

test.afterEach(async ({}, testInfo) => {
  if (testInfo.status !== testInfo.expectedStatus) {
    await testInfo.attach("otp-contract", {
      body: JSON.stringify(otpContract, null, 2),
      contentType: "application/json",
    });

    await testInfo.attach("inbox-snapshot", {
      body: JSON.stringify(await inbox.listRecent(), null, 2),
      contentType: "application/json",
    });
  }
});

This can look a bit boring, but boring is good here. I want another engineer to open one failed run and understand the story in under two minutes.

The same mindset appears in contract-testing signup emails: treat the message step as a testable contract, not as an invisible side effect. Once you do that, the failure modes get much more legible.

3. Separate app delay from inbox delay

A lot of flaky OTP suites mix two timers:

• how long the app takes to send
• how long the inbox takes to surface the mail

That makes every red build feel random. I prefer to log those timings separately, even if the test still uses one overall timeout.

For example:

const requestStarted = Date.now();
await page.getByRole("button", { name: "Send code" }).click();

await expect(page.getByText("Check your email")).toBeVisible();
const uiConfirmedAt = Date.now();

const message = await inbox.waitForMessage(otpContract);
const inboxResolvedAt = Date.now();

Now your failure logs can show:

• UI confirmation latency
• inbox delivery latency
• total OTP wait time

That seems small, but it changes the conversation. If UI confirmation is instant and inbox delivery is slow only in CI, the app may be fine. If UI confirmation itself is delayed, the bug probably lives upstream.

If you use a temporary inbox in staging, keep it limited to QA traffic and make the lifetime short. The goal is to make one run explain itself, not to turn the inbox provider into a long-term source of truth.

A QA checklist for calmer failure reviews

Before I trust an OTP email test in Playwright, I want this checklist:

• each run creates or reserves a unique inbox identity
• the OTP contract is saved before polling begins
• the inbox query filters by recipient and receive time
• failed runs attach both trace evidence and inbox evidence
• timing logs separate UI confirmation from inbox arrival
• the OTP assertion checks the intended message, not merely any recent email

When these are in place, red builds are still possible, but the failure review gets calmer and more usefull.

Q&A

Should I just retry the inbox call more times?

Only after the contract and artifacts are good. More retries can hide race conditions and make the suite slower without making it smarter.

Is Playwright trace enough by itself?

Not for email-driven OTP flows. Trace explains the browser side very well, but it does not prove what was in the mailbox at the same moment.

What should I keep if artifact storage is limited?

Keep the contract JSON, a short inbox snapshot, and one trace for failing runs. That is usually enough to reconstruct what happend without creating a huge storage bill.

Good OTP tests are less about waiting longer and more about explaining themselves clearly. Once the run captures the contract, the timing, and the inbox evidence, QA review stops feeling like a guessing game.