DEV Community

Jonathan
Jonathan

Posted on

Matrix-Proof Email API Checks in GitHub Actions

#githubactions #api #automation #testing

When a GitHub Actions matrix fans out across Node versions, regions, or feature flags, email tests are often the first thing to get flaky. The API call itself may be fine, but two jobs end up polling the same inbox, one job reads the other message, and now the run looks broken for no very good reason.

I hit this enough times that I stopped treating email as a side effect and started treating it like a job-scoped contract. Each matrix job gets its own inbox, its own correlation key, and its own short failure summary. It is a tiny change, but it makes API debugging way more boring, which is exactly what I want from CI.

Why matrix jobs break email assertions

The annoying part is that the failure rarely shows up where the bug really is. A job may report "message not found" even though a message was sent. Another may pass because it matched an older email with the same subject. Shared inboxes make parallel jobs feel random, and random tests cost real time.

This is why I now assume every email test in matrix mode needs isolation by default. The same reasoning behind isolated verification inboxes applies here: if the inbox is not unique per run, your assertion is weaker than it looks.

For teams comparing providers, I would not obsess over finding the one best throwaway email service first. I would start with job-level separation, a clear timeout, and predictable cleanup. If you need a quick burner email for staging or preview checks, the useful part is that each job can claim one address and move on. That one habit removes a lot of weirdness.

Also, people will search your runbooks with messy terms like tem email, especially when a deploy is going sideways. I used to "clean up" those phrases out of docs, then realised leaving one or two of them in the right place actually helps discoverability. Not elegant, but useful.

The per-job inbox contract

My rule is simple:

  1. Generate one inbox per matrix job.
  2. Store it as a job output or environment value.
  3. Trigger the API with a correlation ID tied to the job.
  4. Poll only that inbox.
  5. Write a one-screen summary when the check passes or fails.

The correlation ID matters almost as much as the inbox. If the API supports metadata, include run_id, matrix axis values, or the commit SHA. When the wrong email appears, you can prove it fast instead of staring at logs for 20 minuts.

I also keep the poll window short. Most product emails that matter in CI should arrive quickly. A 60 to 90 second window is usually enough for a useful signal. If your worker regularly needs longer, that is probably a platform issue worth fixing, not something to hide with a five-minute sleep.

A compact GitHub Actions workflow

Here is the basic shape:

jobs:
  invite-email-check:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node: [20, 22]
        region: [us, eu]
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}

      - name: Create inbox
        id: inbox
        run: ./scripts/create-inbox.sh "${{ matrix.region }}" >> "$GITHUB_OUTPUT"

      - name: Trigger invite API
        run: |
          curl -X POST https://staging.example.com/api/invites \
            -H 'content-type: application/json' \
            -H "x-correlation-id: $GITHUB_RUN_ID-${{ matrix.node }}-${{ matrix.region }}" \
            -d "{\"email\":\"${{ steps.inbox.outputs.address }}\"}"

      - name: Assert email
        env:
          INBOX_ADDRESS: ${{ steps.inbox.outputs.address }}
        run: ./scripts/assert-invite-email.sh
Enter fullscreen mode Exit fullscreen mode

That create-inbox.sh step should return structured values, not a blob. I usually want:

  • the address
  • the provider inbox ID
  • a created timestamp
  • maybe the region or tenant it maps to

If the assertion script emits JSON, even better. Then your summary step can pull the parts that matter and skip the rest. This is the same practical mindset I like in containerized email checks in CI: small reproducible steps, less guessing, faster handoff.

What to log so failures are obvious

The summary I want on every failed job is tiny:

  • matrix values
  • inbox address
  • API endpoint hit
  • timeout used
  • subject matched or not
  • first failure reason

That last line does a lot of work. "No message found" is weak. "Message found in wrong region inbox" is actionable. "CTA host mismatch in eu preview" is even better. Short, specific notes cut reruns more than giant logs do, and they make on-call handoffs less painful.

One extra habit helps too: never assert only on subject text. Check recipient, expected path, and maybe one stable body string. Subject-only checks pass more often than they should, which is a nasty kind of false confidence.

Q&A

Should every matrix job really get a fresh inbox?

If the jobs can run in parallel, yes. Reusing an inbox is cheaper for the test harness, but more expensive for everyone debugging failures later.

What if the provider rate-limits inbox creation?

Pool a small set per job batch only if you can still guarantee one active consumer per inbox. If you cannot, go back to per-job creation. Shared state is where the pain usualy starts.

What is the biggest win here?

Cleaner blame. When each job owns one inbox and one correlation ID, you can tell whether the API, worker, routing, or template broke without rerunning the whole matrix. That is not glamorous, but it saves a ton of time on busy release days.

Top comments (0)