DEV Community

DapperX
DapperX

Posted on

Scenario Files for Safer Email Smoke Tests

#devtools #automation #testing #api

When an email smoke test lives only inside one shell script, it works right up until somebody needs to change the environment, subject line, or timeout under pressure. Then the test becomes a tiny detective story. I have been moving these checks into scenario files instead, mostly because boring inputs make automation easier to trust.

This pattern is useful when you validate signup, password reset, or verification flows with a create temporary mail step or any other disposable email inbox. The point is not fancy abstraction. The point is making one run easy to read, rerun, and review later.

Why ad hoc email smoke tests get noisy fast

Many teams start with a single bash script that does everything:

  • create an inbox
  • trigger the app flow
  • poll for a message
  • grep for a few expected strings
  • exit non-zero if somthing looks wrong

That can be okay for a first draft. It gets messy once you have more than one flow. Subject lines drift. Different apps need different timeouts. One environment uses seeded users and another expects just-in-time signups. Soon every branch has a slightly differnt copy of the same check.

I like the same mindset behind reusable email API workflow checks: make the workflow explicit, not hidden inside incidental shell glue. A good scenario file gives reviewers the shape of the test before they even read the runner.

What I put inside a scenario file

My version is tiny on purpose. I usually store one JSON or YAML file per flow with:

  • flow name
  • trigger endpoint or UI action
  • expected subject fragment
  • required body strings
  • timeout and retry interval
  • cleanup rules

Here is a stripped-down JSON example:

{
  "name": "signup-verification",
  "inboxMode": "ephemeral",
  "trigger": {
    "method": "POST",
    "url": "https://api.example.com/test/signup"
  },
  "expect": {
    "subjectIncludes": "Verify your account",
    "bodyIncludes": [
      "verification code",
      "expires in 10 minutes"
    ]
  },
  "poll": {
    "timeoutSeconds": 90,
    "intervalSeconds": 5
  }
}
Enter fullscreen mode Exit fullscreen mode

That file becomes the contract. The runner only knows how to execute the contract. Because the inputs are separate, I can diff test intent without reading the whole script, which is prety nice during reviews.

A small runner that stays boring on purpose

The runner itself should do very little:

  1. Read the scenario file.
  2. Create the inbox for the run.
  3. Trigger the product flow.
  4. Poll until the matching message arrives or times out.
  5. Write a receipt artifact with the scenario name, run id, and assertions checked.

In Node.js pseudocode, that looks like this:

const scenario = await loadScenario("scenarios/signup-verification.json");
const inbox = await inboxProvider.create();

await triggerFlow(scenario.trigger, { email: inbox.address });

const message = await waitForMessage({
  inboxId: inbox.id,
  timeoutSeconds: scenario.poll.timeoutSeconds,
  intervalSeconds: scenario.poll.intervalSeconds,
  subjectIncludes: scenario.expect.subjectIncludes,
});

assertBodyIncludes(message.text, scenario.expect.bodyIncludes);
await writeReceipt({ scenario: scenario.name, inbox: inbox.address, ok: true });
Enter fullscreen mode Exit fullscreen mode

Nothing there is clever, and that is why I trust it. If I need UI coverage too, I keep the email wait logic isolated and let the browser code focus on the app state. That separation has the same feel as abortable inbox polling patterns, where cancellation and waiting rules are treated as first-class behavior instead of afterthoughts.

Where disposable inboxes actually help

I do not use temporary inboxes for every case. For production-like staging with audit requirements, a fixed mailbox can be better. But for fast smoke tests, preview deploys, and branch-level checks, a disposable email flow is hard to beat because each run gets isolated evidence.

That isolation matters more than people think. If a flaky job reuses one shared inbox, an old success can look like a new one. With a one-run inbox, the receipt becomes much more believable. This is especially helpful when teams still have random notes like tamp mail com in their old scripts and nobody remembers which provider those notes meant.

If you want evidence that simpler test design matters, the 2024 State of Testing report from mabl found that teams continue to prioritize reliability and maintainability in automated testing over sheer test count: https://www.mabl.com/state-of-testing. That matches what I see in practice, even if the exact tools vary a lot from shop to shop.

Q&A

Should scenario files live next to the runner?

Usually yes. I keep them in a scenarios/ folder near the runner so changes to the execution logic and test intent can be reviewed together. Splitting them across repos is posible, but it adds friction fast.

JSON or YAML?

Either works. I prefer JSON when machines write the files and YAML when humans edit them often. The important part is not the syntax, it is that the format stays stable and easy to lint.

When is this overkill?

If you only have one email check and it changes once a year, a single script is fine. Once there are multiple flows, multiple environments, or handoffs between engineers, scenario files save time and reduce weird mistakes later.

Top comments (0)