GitHub Actions API Tests Need Cheap Isolation

API smoke tests in GitHub Actions are easy to write and surprisingly hard to trust. A test can fail because the API is broken, because a verification email is late, or because yesterday's inbox state leaked into today's run. When all three look like the same red check, the workflow becomes expensive to maintain.

I have found that the useful shortcut is cheap isolation plus a small run receipt. The goal is not a huge end-to-end framework. It is a workflow where each run gets its own inputs, inbox identity, and evidence.

The failure mode

Consider a signup smoke test:

1. Create a user.
2. Wait for a verification email.
3. Open the link.
4. Call an authenticated API endpoint.

This crosses several boundaries. A reused address may already have an account. A shared mailbox may contain an old message. A retry may create a second user. Then the final assertion says only expected 200, got 409, which is technically accurate but not very useful.

Search terms such as tempmailso or facebook temp email often describe the kind of isolated address people want, but the engineering requirement is more specific: one test identity, a bounded lifetime, and a traceable message.

A small workflow contract

Start by making the run inputs explicit. The test should know its run ID and derive a unique address from it. Keep the value in a job-local environment variable, and never print the full address if it contains credentials or tokens.

name: API smoke

on:
  workflow_dispatch:
  schedule:
    - cron: "17 */6 * * *"

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run isolated smoke test
        env:
          RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
          API_BASE_URL: ${{ secrets.SMOKE_API_BASE_URL }}
        run: ./scripts/smoke-api.sh
      - name: Upload evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: smoke-evidence-${{ github.run_id }}
          path: artifacts/smoke/

The if: always() step matters. A failed test is exactly when its logs and response summaries are most valuable. Before adding retries, freeze this contract; freeze the test plan before a scheduled run so a retry does not silently change what you are measuring.

Isolate the inbox

Generate an address per run, then record only a safe identifier in the evidence file:

set -euo pipefail

run_id="${RUN_ID:?missing RUN_ID}"
mkdir -p artifacts/smoke
address="ci+${run_id}@example.test"
printf 'run_id=%s\n' "$run_id" > artifacts/smoke/context.txt

python scripts/create_user.py \
  --email "$address" \
  --output artifacts/smoke/signup.json
python scripts/wait_for_verification.py \
  --email "$address" \
  --timeout-seconds 45 \
  --output artifacts/smoke/inbox.json
python scripts/assert_api.py \
  --input artifacts/smoke/inbox.json \
  --output artifacts/smoke/api.json

In a real system, replace the example domain with a provider or test mailbox service that your team controls. A disposable email generator can be useful for temporary test identities, but it should not become a hidden dependency: document retention, rate limits, and whether messages are publicly readable.

The slighty annoying part is cleanup. Delete the test user when the API supports it, and set a short retention policy for inbox data. Never store verification URLs in ordinary build logs.

Keep evidence as artifacts

A run receipt should answer four questions: which identity was used, when the message arrived, what endpoint was called, and what the response status was. Redact tokens and message bodies unless the body is essential to diagnosis.

{
  "run_id": "12345-1",
  "mail_wait_ms": 1840,
  "verification": "received",
  "endpoint": "/api/me",
  "status": 200
}

Add the message subject, provider request ID, and retry count when they are safe to retain. These small fields turn a flaky check into something you can replay. For scheduled publishers, small inbox probes for publishing workflows use the same idea: test the boundary directly and leave evidence behind.

A practical checklist

• Create a fresh identity per workflow run.
• Pass RUN_ID through every helper.
• Bound inbox polling with a timeout and a clear error.
• Separate API errors from delivery errors.
• Upload artifacts with if: always().
• Redact tokens, cookies, and full verification URLs.
• Make retries idempotent with a stable run key.
• Remove test accounts and old inbox data.

If your team casually says temp org mail or temp mailid, translate that request into these explicit controls. The wording may vary; the safety properties should not.

Closing thought

Reliable CI is less about adding another retry and more about reducing ambiguity. Give each API test a cheap isolated identity, a bounded inbox wait, and a compact receipt. The next red build then tells you whether the API, the email boundary, or the workflow itself needs attention—and that is a much better productivity win than a green check you cannot explain.

Comments

[Vinh Nguyen]

The run-identity contract reintroduces one of the failure modes you open with. RUN_ID is ${{ github.run_id }}-${{ github.run_attempt }} and the address is derived from it, so re-running the workflow does not retry the identity, it mints a second one — which is the "a retry may create a second user" case from your own list, now guaranteed instead of possible, and it is what the checklist bullet "make retries idempotent with a stable run key" is asking you to avoid. The two pieces want opposite things from the same variable: the inbox wants an identity that is stable across attempts of one logical test, the evidence wants attempts kept apart. Splitting them costs nothing — derive the address from github.run_id alone, and move github.run_attempt into the artifact name, which is where it is missing right now, so attempt 2 does not land on a name attempt 1 already used.

[Jesse Gamble]

The run receipt is the part that makes this maintainable. If each test keeps its own inputs, inbox identity and timestamps, a failed 409 stops being a mystery and starts pointing to a specific boundary.