DEV Community

Jonathan
Jonathan

Posted on

Contract-Test API Emails in GitHub Actions

#githubactions #api #automation #testing

Email tests in CI often fail for boring reasons, not exotic ones. The API sends the right event, but the workflow looks at the wrong inbox, waits too long for one step, or only asserts that "some message arrived". That is how teams end up shipping template regressions even though the pipeline was green all week.

Why email checks break in otherwise good CI

The weak spot is usually the contract between your API and the job that validates the notification. If the contract is fuzzy, the workflow gets fuzzy too.

I see the same issues over and over:

  1. The test only checks subject text, so an older message can still pass.
  2. Multiple workflow runs share one inbox, which makes failures weirdly random.
  3. The API emits an event id, but the test never uses it as a correlation key.
  4. The workflow waits on UI behavior when the real thing being tested is an API side effect.

That last one is a big time sink. A lot of email flows are triggered by backend mutations, so the most reliable check is usually closer to the API boundary than the browser. It sounds obvious, but teams still end up with brittle end-to-end coverage becuse it feels faster to bolt email checks onto an existing UI suite.

The contract I validate in GitHub Actions

For API-triggered mail, I want each run to prove four things:

  • the request produced the expected domain event
  • exactly one matching message appeared for that event
  • the message payload contains the correct actor or resource context
  • the workflow can explain failures with one run id

That is the whole game. If you can prove those four points, your CI is much more useful than a generic "email received" assertion.

My preferred setup is:

  • create a run-scoped inbox id at job start
  • pass that id into the test fixture or API seed step
  • call the API directly from the workflow
  • poll a mailbox reader with the same run id
  • assert message count, subject intent, and one key link or token

This lines up nicely with other patterns for stable email assertions across app states and with broader privacy checks for auth-related messages. The point is not just delivery. The point is proving the message belongs to this workflow run and no other.

You may still find old fixture names like tempail mail or tamp mail com in legacy repos. I would not use those names in new test helpers, but keeping an eye on them helps when you are cleaning up older automation.

A workflow that stays fast and debuggable

Here is the shape I like in GitHub Actions:

jobs:
  email-contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set run inbox id
        run: echo "RUN_INBOX_ID=email-${GITHUB_RUN_ID}-${GITHUB_JOB}" >> "$GITHUB_ENV"

      - name: Trigger API flow
        run: |
          curl -fsS -X POST https://api.example.test/v1/invitations \
            -H "Authorization: Bearer $API_TOKEN" \
            -H "Content-Type: application/json" \
            -d "{\"email\":\"${RUN_INBOX_ID}@example.test\",\"teamId\":\"team_123\"}"

      - name: Assert matching message
        run: node scripts/assert-invite-email.js
Enter fullscreen mode Exit fullscreen mode

The useful shortcut here is the env contract. Every later step can log RUN_INBOX_ID, and every failure message can include it. That alone makes triage much faster when several jobs run in parallel.

For the assertion script, I keep the rules narrow on purpose:

const messages = await inboxClient.list({ inboxId: process.env.RUN_INBOX_ID });
const matching = messages.filter((m) => m.subject.includes("Invitation"));

if (matching.length !== 1) {
  throw new Error(`expected 1 invitation email, got ${matching.length}`);
}
Enter fullscreen mode Exit fullscreen mode

This is not fancy, and that is why it works. The script fails fast, logs one identifier, and avoids dragging a whole browser stack into a check that belongs at the API layer. In many pipelines, that cuts minutes off feedback time and removes a lot of flaky reruns. GitHub's own guidance on faster workflow feedback loops is worth revisiting here: https://docs.github.com/en/actions/using-workflows/about-workflows.

Small checks that catch big regressions

The highest-value assertions are usually tiny:

  • count must be exactly one
  • recipient or alias must match the run scope
  • one expected link path or token must exist
  • one forbidden phrase or stale template fragment must not exist

If you already track email rendering changes, add a plain text assertion too. HTML can look fine while the fallback body is broken, which happends more often than people expect.

I also like to separate "delivery happened" from "business meaning is correct". Put both in the same script, but log them as distinct failures. That makes on-call debugging less messy and keeps the workflow honest.

Q&A

Should this live in the UI test suite?

Usually no. If the behavior starts from an API call or queue event, validate it near that boundary first. Keep one browser test for confidence if you need it, but do not make the browser own the whole email contract.

What if the provider is eventually consistent?

Use short polling with a firm timeout and include the run-scoped inbox id in every log line. You need enough patience to handle normal lag, but not so much that the workflow hides real regressions.

What is the biggest mistake?

Shared inboxes. They create "works on rerun" noise, waste review time, and make CI feel haunted. Run-scoped inbox ownership is a very small change with a prety big payoff.

Top comments (0)