DEV Community

Cover image for A Green Signup Test Can Verify The Wrong Account
Vipul Swarup
Vipul Swarup

Posted on Fully Autonomous

A Green Signup Test Can Verify The Wrong Account

Test A submits a signup form using qa@example.com. Its verification email is delayed, and the test times out. Test B starts with the same address, submits the same form, and waits for an email whose subject contains “Verify.” A's message arrives first. B reads it, follows the link, sees a success page, and goes green.

Every individual assertion can pass while B's signup is broken.

The failure is a correlation problem. Email delivery is asynchronous, and a shared mailbox has no idea which Playwright worker triggered a message. A subject filter narrows the search. It does not identify the transaction.

Disclosure: I build InboxRhino, a receive-only test inbox API. The test design below works with any mailbox that your CI job can read.

The timestamp trap

A common repair is to record a timestamp before clicking Submit, then query for messages received after that time. That helps with mail already stored in the inbox. It does not reject a delayed message from an earlier run that arrives after the timestamp.

The sequence can look like this:

10:00:00  Run A submits qa@example.com
10:00:30  Run A times out
10:00:31  Run B records received_after=10:00:31
10:00:32  Run B submits qa@example.com
10:00:34  Run A's email arrives
10:00:35  Run B reads A's email and passes
Enter fullscreen mode Exit fullscreen mode

The useful correlation key is the recipient. Give each test invocation a new address, including retries. Then A's late mail cannot appear in B's inbox. With a genuinely new inbox, an arrival-time filter adds little. With a reused inbox, it can exclude stored mail, but allow for clock skew between the CI runner and the receiver. It still cannot identify the action that caused a late arrival.

This also means the application under test must treat the address as a new account. Reusing an address can silently turn a signup test into a resend-verification test.

Wait for one message, then check whose message it is

This Playwright excerpt assumes a project fixture has created a ready inbox and will delete it after the test. inbox contains a unique id and address. The fixture should provision the address before the action that sends mail; some receiving systems take time before a new address accepts delivery.

The API call uses InboxRhino's query parameters as a concrete example. The page selectors and verificationUrlFrom function are application-specific.

import { expect } from '@playwright/test';
import { test } from './fixtures/test-inbox'; // extends Playwright's test with `inbox`

test('verifies the account created by this run', async ({ page, request, inbox }) => {
  const key = process.env.INBOXRHINO_API_KEY;
  if (!key) throw new Error('INBOXRHINO_API_KEY is required');

  await page.goto('https://staging.example.com/signup');
  await page.getByLabel('Email').fill(inbox.address);

  // Keep this for diagnostics. A fresh inbox provides the isolation.
  const submittedAt = new Date().toISOString();
  await page.getByRole('button', { name: 'Create account' }).click();

  const query = new URLSearchParams({
    wait_seconds: '120',
    limit: '1',
    include: 'content',
    sender: 'accounts@example.com',
    subject: 'Verify your email',
  });

  const response = await request.get(
    `https://api.inboxrhino.in/v1/inboxes/${inbox.id}/messages?${query}`,
    {
      headers: { Authorization: `Bearer ${key}` },
      timeout: 130_000,
    },
  );

  if (response.status() === 204) {
    throw new Error(`No verification email for ${inbox.address} within 120s`);
  }
  if (response.status() !== 200) {
    const requestId = response.headers()['x-request-id'] ?? 'none';
    throw new Error(`Mail API returned HTTP ${response.status()}; request ID: ${requestId}`);
  }

  const { data } = await response.json();
  expect(data).toHaveLength(1);
  const message = data[0];
  expect(message.to.toLowerCase()).toBe(inbox.address.toLowerCase());
  expect(message.from.address.toLowerCase()).toBe('accounts@example.com');
  console.info({ submittedAt, messageId: message.id, receivedAt: message.received_at });

  const link = verificationUrlFrom(message.text, message.html);
  const url = new URL(link);
  expect(url.origin).toBe('https://staging.example.com');
  expect(url.pathname).toBe('/verify');

  await page.goto(url.toString());
  await expect(page.getByTestId('verified-account-email')).toHaveText(inbox.address);
});
Enter fullscreen mode Exit fullscreen mode

wait_seconds holds the request until a matching message arrives or the wait expires. For this API, a timeout is HTTP 204 with an empty body, so parsing JSON before checking the status would produce a misleading error. include=content returns the body and requires limit=1. The 130-second client timeout gives the 120-second server wait room to finish.

The inbox fixture is important for more than cleanup. Each parallel worker needs its own address; each retry needs a new one. If the receiving service has a provisioning delay, create addresses early or wait for an explicit ready signal when one exists. Some services have no ready signal, so the fixture must account for that delay. If the sender gets a 550 because the address is not yet active, a longer message wait will not fix the test.

The success page is a weak final assertion

The test still needs to establish that the right account changed state. Many verification pages say “Success” without showing which account was verified. If B follows A's link, that generic text is a false positive.

Assert an account identifier in the UI or fetch the account through a test-only API. For a stronger persistence check, start a fresh session and verify that the address from this run is marked verified. Choose the boundary that matches your product's actual contract.

Link extraction also deserves an explicit rule. “Take the first href” can select a logo, tracking link, or unsubscribe URL. Parse the verification CTA or the plain-text URL for your app, then validate its origin and path before opening it. If the email uses a one-time link, do not fetch that URL from the test runner before the browser does; a preview request can consume the token.

What a green result still does not mean

One matching email proves that at least one matching email arrived. It does not prove exactly-once delivery. If duplicate sends matter, inspect the inbox again over a defined observation window and report that window in the test. No finite wait proves that another message will never arrive.

Keep enough evidence to locate a failure: run ID, recipient, submission time, sender, subject, message ID, and final account ID. Avoid logging the full email body, verification token, or API key. If your app and mail provider expose a shared message or trace ID, record it. That lets a developer follow the send path while a tester checks the receiving side.

The next time an email test flakes, check the recipient first. If parallel runs share it, tightening the subject regex or adding another sleep will not fix the underlying race.

Top comments (0)