GitHub Actions Email Tests Need Evidence

Email verification tests often fail in CI with a useless message: “expected email, received none.” A retry may turn green, but it does not tell the team whether the message was late, the inbox was shared, or the API rejected the request.

In my CI workflows, the biggest productivity win was treating every email test as an evidence-producing job. The test still asserts the user-visible result, but it also saves enough context to explain a failure. That makes a flaky test a short investigation instead of a rerun lottery.

The failure signal is too small

An email flow crosses several boundaries:

1. The application accepts a signup request.
2. A worker queues and sends a message.
3. The inbox provider exposes the message.
4. The test finds the right message and follows its link.

One failed assertion covers all four stages. Add a throwaway email address or a shared test inbox and the ambiguity gets worse: another job might consume the same message.

Give each run a unique correlation ID, and include it in the recipient or subject when your test system allows it. Also log state transitions, not the full message body. This keeps logs useful without leaking tokens.

Build an evidence bundle

For each attempt, I save five small artifacts:

• request.json: safe request metadata and correlation ID
• events.jsonl: queue, delivery, and polling events with timestamps
• inbox.json: message IDs, subjects, and received times
• failure.txt: the final reason and elapsed time
• a screenshot or trace when the browser is involved

The key is to write these files even when a retry succeeds. Otherwise, the first failure disappears and the green result hides a slow or broken dependency. A little extra disk use is worth it.

If your flow uses React, it is also useful to separate UI state from delivery state. This guide on preview email tests without inbox collisions is a good reminder that inbox ownership is part of test design.

A GitHub Actions workflow

Here is the small part of a workflow that makes artifacts available after any test outcome:

- name: Run email tests
  id: email_tests
  continue-on-error: true
  run: npm run test:email -- --reporter=line

- name: Upload email evidence
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: email-evidence-${{ github.run_id }}
    path: |
      .artifacts/email/
      test-results/
    if-no-files-found: ignore

- name: Fail after evidence is uploaded
  if: steps.email_tests.outcome == 'failure'
  run: exit 1

if: always() is the important line. The upload must run after a failed test, and the final step preserves the correct job status. I prefer one artifact per workflow run because it is easy to find and compare.

Make retries useful

Retries should answer a question. Retry only the polling operation when delivery may be delayed; do not silently repeat the entire signup if that creates duplicate accounts. Record the attempt number, polling interval, and last observed event.

When a retry passes, compare its evidence with the failed attempt. A five-second difference suggests eventual consistency. Two different recipients suggest a fixture bug. No queue event suggests an application or worker failure. These distinctions save more time than increasing the retry count.

For invite flows, keep authentication and tenant context explicit too. The discussion of tenant-bound magic links for SaaS invites shows why a valid link is not enough if it is accepted in the wrong context.

Checklist

• Generate a unique correlation ID per test.
• Isolate each inbox or recipient.
• Log safe event metadata with timestamps.
• Upload artifacts with always().
• Retry polling, not side effects.
• Compare failed and successful attempts.
• Keep tokens and message bodies out of CI logs.

I still see teams searching for “temp mailid” or “temp org mail” while diagnosing a test. The label matters less than the contract: the test needs a known recipient, a bounded wait, and evidence when that contract breaks.

Q&A

Should every email test upload a screenshot?

Only browser-facing tests need one. API-only tests usually get more value from structured events and inbox metadata.

Are retries a bad idea?

No. A bounded retry is useful for asynchronous delivery. It becomes harmful when it hides failures or repeats non-idempotent actions.

What is the first improvement to make?

Upload the artifacts from the first failed run. Once failures are inspectable, you can tune timeouts and retry policy based on evidence.