DEV Community: Jonathan

Run-Scoped Mailboxes for GitHub Actions API Tests

Jonathan — Fri, 02 Oct 2026 23:24:21 +0000

Email-dependent API tests often look random when the real problem is shared state. Two GitHub Actions jobs create the same user, one job reads an old verification message, and a retry consumes the only useful link. The test reports a timeout, but the root cause is an inbox that belonged to another run.

I use a run-scoped mailbox contract to make these failures easier to reason about. Each workflow run gets its own disposable address, a short lease, and a small evidence record. The approach is simple enough for a few API tests, but it also scales better than asking every test helper to invent its own cleanup rules.

Why shared inboxes make API tests nondeterministic

A shared test inbox creates three kinds of collisions:

Identity collisions: two jobs register the same email address and one receives a conflict response.
Message collisions: a test matches a verification email from a previous run.
Lifecycle collisions: one cleanup step deletes data while another test is still polling it.

The usual reaction is to increase the polling timeout. That makes the pipeline slower and doesnt fix a stale message. A better boundary is to give every run an address derived from a run ID, then refuse to read messages outside that boundary.

The address does not need to expose the branch name or a user email. A short random suffix, a workflow run identifier, and a provider-managed domain are enough. Store the full address in the test process, but put only a fixture label in CI logs.

On a busy morning, this is usualy the first signal that saves a developer from chasing the wrong job.

Define a mailbox lease for each workflow run

Think of the mailbox as a leased test resource. The lease has four values:

run_id: the workflow execution that owns the mailbox.
address: the disposable address used by the API test.
expires_at: the time after which the fixture is no longer valid.
cleanup_state: whether the test released it, timed out, or failed.

The test should always query by run_id or a unique message marker. Subject text alone is too broad. A useful marker can look like ci-verification-${run_id} and should be passed as an API field or a controlled email subject, not hidden in an undocumented helper.

This is also where teams should decide what a disposable address means. A temp mail so service can be useful for manual checks or isolated test fixtures, but it is not a substitute for ownership and retention rules. The API test still needs to prove that the message belongs to the current run.

Implement the fixture contract

Keep the fixture interface small. For example:

type MailboxLease = {
  runId: string;
  address: string;
  marker: string;
  expiresAt: string;
};

async function waitForVerification(
  lease: MailboxLease,
  timeoutMs = 30_000,
): Promise<string> {
  const deadline = Date.now() + timeoutMs;

  while (Date.now() < deadline) {
    const message = await inbox.find({
      to: lease.address,
      subjectContains: lease.marker,
    });

    if (message) return extractVerificationUrl(message);
    await delay(1_000);
  }

  throw new Error(`No message for run ${lease.runId} before timeout`);
}

The important detail is not the polling function. It is the contract around it: the mailbox address and marker are created together, and the failure names the run. If a test uses a fake value such as temp gamil com or temp org mail during a manual experiment, keep that value out of production fixtures and logs so it doesnt become a confusing search term later.

For an overview of keeping auth email state aligned, the same principle applies on the client side: the address, request, and verification state should describe one attempt. The backend and UI should not each guess which email is current.

Wire it into GitHub Actions

Create the lease before the test command and pass only the minimum values through the environment:

- name: Create mailbox lease
  id: mailbox
  run: node scripts/create-mailbox-lease.mjs > mailbox.json

- name: Run API tests
  env:
    MAILBOX_LEASE_FILE: mailbox.json
  run: npm test -- --runInBand

- name: Upload test evidence
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-mailbox-evidence-${{ github.run_id }}
    path: artifacts/api-mailbox
    retention-days: 7

- name: Release mailbox lease
  if: always()
  run: node scripts/release-mailbox-lease.mjs mailbox.json

The lease step should fail closed if it cannot create a unique fixture. Do not silently fall back to a team inbox. That fallback is convenient for one green run and painful when a parallel job starts.

Keep the ownership rule seperate from the provider client. The test should know who owns the fixture, even if the mail adapter changes later.

For teams comparing the best throwaway email option for a test address, the practical decision is less about a catchy provider name and more about whether the fixture can be isolated, queried, and expired predictably.

Clean up without hiding failures

Cleanup should run even when the test fails, but its result must not erase the original error. Save a small JSON receipt before release:

{
  "run_id": "1842",
  "fixture": "signup-verification",
  "messages_seen": 1,
  "test_status": "failed",
  "cleanup_status": "released"
}

Keep message bodies out of the artifact unless a specific review requires them. A message ID, subject hash, timestamp, and redacted error usually gives enough context. This makes the cleanup is easier to audit and reduces the chance that a real address or token survives in CI storage.

If the provider cannot release a mailbox immediately, mark the lease expired and let a scheduled janitor remove it later. Never reassign an unexpired lease to another run. Reuse is the shortcut that turns one stale message into a hard-to-reproduce incident.

The result are easier to review when expiry and release appear as explicit states instead of hidden cleanup behavior.

Common questions

Should every API test create a new mailbox?

Not always. A test suite can share one lease when its cases are deliberately sequential and use distinct markers. Parallel tests should get separate leases, especially when they create users or consume one-time verification URLs.

Is a disposable address enough for privacy?

No. It limits exposure of personal mail, but logs, artifacts, provider retention, and application databases still need rules. Record fixture IDs instead of full addresses wherever possible.

How long should a lease live?

Long enough for the slowest supported CI path plus a small buffer. The exact duration depends on the provider and test environment. The key is to make expiration explicit and to report it as a fixture failure, not as an unexplained API timeout.

A small rollout checklist

Start with one flaky email API test:

Generate a unique lease per workflow run.
Add a run marker to every message lookup.
Record status, message ID, and redacted timing evidence.
Upload evidence with if: always().
Release or expire the lease in a final step.
Verify two concurrent jobs cannot read each other's messages.

The before-and-after improvement is straightforward: a failed test now says which run owned the mailbox, which marker was expected, and whether cleanup completed. That is a much better starting point than “email timed out again.” It also makes isolated inbox checks in CI a reusable engineering pattern instead of a one-off workaround.

Git Worktrees for Isolated Email API Tests

Jonathan — Fri, 02 Oct 2026 02:23:49 +0000

When an email API test starts failing on one branch, the first instinct is often to rerun the same checkout and watch the inbox again. That gets messy when two fixes are being investigated at once. One branch changes retry timing, another changes the webhook parser, and both write into the same temporary mailbox. The result is a test report that is green-ish, confusing, and realy hard to trust.

I have found a simple combination more useful: one Git worktree per investigation, one fixture namespace per worktree, and one small evidence bundle per run. It turns parallel debugging from a shared-state problem into a few independent folders. The workflow is especially handy for GitHub Actions and other automation where a failed check should explain itself after the job is gone.

Why shared email fixtures make parallel branches noisy

An email API test usually has more state than the test file suggests. There is a recipient address, a message ID, a polling window, a subject or token assertion, and often a cleanup step. If two branches reuse any of those values, a passing assertion can read the wrong message.

That is why I avoid a generic address such as test@example.com for concurrent checks. Even a disposable address should have a run-scoped name, for example ci-${run_id}@inbox.test, or an equivalent fixture identifier supplied by the email test service. A search like fake e mail com should not be the process for choosing test data; the test needs a documented contract instead.

The same principle applies to signup flows that have several visible states. A small state machine for signup flows is a useful companion when the UI can show pending, expired, and verified states independently of the inbox lookup.

Create one worktree per investigation

Git worktrees let me keep multiple branches checked out at once without copying the whole repository. A short setup looks like this:

git fetch origin
git worktree add ../email-api-retry-fix -b debug/email-api-retry origin/main
git worktree add ../email-api-parser -b debug/email-api-parser origin/main

Now each investigation has its own source tree and its own .env.test or fixture configuration. I put the run identifier in the environment rather than in a global config file:

export EMAIL_TEST_RUN="$(git branch --show-current | tr '/' '-')-$(date +%s)"
export EMAIL_FIXTURE_PREFIX="${EMAIL_TEST_RUN}-"
./scripts/email-api-check.sh --fixture-prefix "$EMAIL_FIXTURE_PREFIX"

The branch name is not a security boundary, so I still keep credentials in the CI secret store. It is only a convenient human-readable label. Also, remove a worktree when the investigation is finished; leaving many stale dependancies around makes the local setup harder to understand.

Give every test run its own inbox contract

The check should produce a small contract before it sends anything. Mine normally records:

the run ID and commit SHA
the fixture or recipient identifier
the expected event type
the maximum polling duration
the message marker that proves the right email was found

The assertion should reject a message from another run, even if the subject looks correct. Matching both a run marker and an event marker is a small extra step with a big payoff. For high-concurrency suites, concurrency keys for email API checks can make this ownership explicit at the API boundary too.

Here is the shape of a result I want the shell script to write:

{
  "run_id": "debug-email-api-retry-1710000000",
  "commit": "abc1234",
  "event": "verification.sent",
  "matched": true,
  "poll_seconds": 8,
  "message_id": "msg_42"
}

If the check needs a generated address, make the generation step visible in that result. Searching for terms such as dummy e mail may describe the intent, but it does not tell a future reader which fixture was actually used.

Keep GitHub Actions evidence attached to the branch

The workflow becomes much easier to triage when every job uploads the same small set of files, including failures:

- name: Run email API check
  env:
    EMAIL_TEST_RUN: ${{ github.head_ref || github.ref_name }}-${{ github.run_id }}
  run: ./scripts/email-api-check.sh --out artifacts/email-result.json

- name: Upload email evidence
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: email-api-${{ github.run_id }}
    path: artifacts/

The before-and-after improvement is not fancy: before, a developer had to reconstruct which branch sent which message. After, the commit, fixture, assertion, and failure reason travel together. When a test fails, I can open the worktree, rerun the same command, and compare the artifact without guessing.

Q&A

Should every branch get a new inbox?

For parallel or long-running checks, yes. For a strictly sequential local test, a reusable fixture can be fine if cleanup is reliable. The important part is making that choice explicit.

Is a Git worktree required in CI?

No. CI already gives each job a fresh checkout in most setups. Worktrees are most valuable locally, where several branches are open and developers are tempted to reuse one directory.

What is the smallest useful artifact?

Commit SHA, run ID, fixture ID, expected event, matched message ID, and failure reason. Keep the payload small so people actualy read it.

The shortcut is simple: isolate code with worktrees, isolate test data with run-scoped fixtures, and isolate diagnosis with predictable artifacts. That gives Git and automation a cleaner handoff, while leaving fewer mysteries in the inbox.

A Failure Bundle for GitHub Actions API Tests

Jonathan — Thu, 01 Oct 2026 23:23:36 +0000

API tests usually fail with a red check, a stack trace, and a request to “run it again.” That is not much evidence. By the time someone opens the workflow, the relevant response body may be gone, the test fixture may have expired, and nobody remembers which commit produced the request.

When I build API integrations, I try to make every failed GitHub Actions job leave behind a small failure bundle. It is a boring improvement, but it turns a vague CI alert into something another developer can reproduce without a live debugging session.

Why a red CI job is not enough

A failed API test has at least three useful dimensions:

The request: method, route, safe headers, and a correlation ID.
The response: status, selected headers, and a redacted body.
The environment: commit, test command, dependency lockfile, and fixture state.

Most pipelines preserve only the test runner output. That output might say that an assertion expected 201 and received 409, but it will not explain whether the duplicate came from a reused fake email address, a retry, or a backend job that finished late.

The fix is not “log everything.” Logging everything creates privacy risks and makes the real signal hard to find. The fix is a predictable artifact with a narrow schema.

What belongs in a failure bundle

I use a directory with one small JSON file per failed case and a workflow summary that links to the artifact. A useful record looks like this:

{
  "test": "creates a verification session",
  "commit": "${GITHUB_SHA}",
  "request_id": "req-8d2c",
  "method": "POST",
  "path": "/v1/verification-sessions",
  "status": 409,
  "fixture": "fresh-address",
  "retry": 1,
  "body_redacted": "verification session already exists"
}

Do not store authorization headers, full cookies, or an entire inbound email in this file. A fixture label such as fresh-address is enough to tell a teammate what was intended. If a test needs an inbox-like value, generate it inside the test run and record only a hash or short label.

For teams that search for a tempail or a tem email while investigating, consistent field names matter more than the spelling in a query. Keep the canonical terms in documentation, then make the artifact easy to grep. This is a small detail, but it saves a realy annoying amount of time during an incident.

A GitHub Actions implementation

The test command should always write its diagnostics to a known directory, even when the process exits non-zero. Then upload the directory in an always() step:

name: API tests

on:
  pull_request:
  push:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm test -- --reporter=dot
        env:
          API_DIAGNOSTICS_DIR: artifacts/api-failures
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: api-failure-bundle-${{ github.run_id }}
          path: artifacts/api-failures
          if-no-files-found: ignore
          retention-days: 7

The important part is the contract between the test runner and the workflow: API_DIAGNOSTICS_DIR is stable, and the upload step runs whether the tests pass or fail. On a green run, the directory can be empty. On a red run, the bundle is attached to the exact workflow execution.

For quick investigation, append a short summary as well. A summary can say which test failed, link the artifact, and show the request ID. Keep it short; large logs belong in the artifact, not in the pull request timeline.

Make retries produce comparable evidence

Retries can hide a race condition. Every attempt should therefore have its own record, with an attempt number and timestamp. Do not overwrite failure.json on retry. Use names such as creates-session-attempt-1.json and creates-session-attempt-2.json.

This makes it possible to compare a 409 followed by a 201, or a timeout followed by a successful response. It also tells you when the test is not flaky at all: a fixture may simply be reused between attempts. The cleanup rule should be visible in the bundle, not implied in a test helper that only one person remembers.

I also keep the request ID from the API response, when one exists. That ID connects the CI artifact to server logs without copying sensitive payloads into GitHub. A redacted trace is usualy more useful than a perfect transcript that cannot be shared.

If your project already has notes on email API triage, this failure-bundle pattern fits naturally beside them. The same boundary applies when an email event is part of an API test: record the event ID and state, not private message contents.

Keep test data private and useful

Fake email addresses and temporary fixtures are helpful for isolating signup and verification flows, but they are not a privacy policy. Avoid real customer addresses, production tokens, and complete message bodies in CI artifacts. Set a short retention period, restrict who can read artifacts, and delete local diagnostics before packaging anything else.

An email deployment contract is a useful mental model here: define what the test promises to observe, then store only the evidence needed to check that promise. This keeps the bundle small and makes later cleanup much simpler.

A small adoption checklist

Start with one unreliable API test and add:

A stable diagnostics directory.
A JSON record for request, response, fixture, and attempt metadata.
Redaction before the record is written.
An always() artifact upload in GitHub Actions.
A short workflow summary with the test name and request ID.
A retention rule that matches the sensitivity of the data.

The result is not a fancy observability system. It is a repeatable handoff from a failing check to the next developer. That little bit of evidence often makes API failures faster to fix, easier to discuss, and less likely to be dismissed as “just CI being flaky.”

GitHub Actions API Tests Need a Debuggable Receipt

Jonathan — Thu, 01 Oct 2026 14:23:41 +0000

An API test that says only failed is not finished. It has detected a problem, but it has not helped the next developer decide whether the cause was a bad request, a slow dependency, a changed response, or a broken test fixture.

I have found a small receipt contract makes GitHub Actions workflows much easier to operate. Each test run gets a correlation ID, records the request shape without secrets, saves the response summary, and leaves a short artifact even when the assertion fails. The change takes minutes, and the debugging loop becomes much shorter.

Why a green API test can still be hard to debug

Most CI API checks produce a status and a few lines of console output. That is fine when everything is green. It is not enough when a test runs beside several deployments or when a provider returns an unexpected response.

The usual clues are scattered across logs:

Which commit and workflow run sent the request?
Was this a fresh test fixture or a reused one?
Did the API reject the payload, or did the response arrive too late?
Did a retry see the same result?

Without those facts, developers often rerun the job and hope the second result explains the first. That costs time and can hide a real race condition. A receipt gives the run an identity and puts the important evidence in one place.

Create a receipt contract before the request

Start with a tiny JSON document. Keep it safe to upload as a CI artifact:

{
  "run_id": "local-abc123",
  "endpoint": "/v1/verification",
  "method": "POST",
  "started_at": "2026-10-01T14:00:00Z",
  "status": "pending"
}

Do not put authorization headers, full email addresses, tokens, or response bodies in this file. A redacted request fingerprint is normally enough to compare retries. For test flows that use a tempmailso fixture, store only an opaque fixture ID and its lifecycle state. Test email data should be temporary and non-production.

The run_id can combine the GitHub run number and commit:

RUN_ID="${GITHUB_RUN_ID:-local}-$(git rev-parse --short HEAD)"
RECEIPT="artifacts/api-receipt-${RUN_ID}.json"
mkdir -p artifacts

Write the initial receipt before making the request. If the process crashes during setup, that first record still shows where the attempt stopped. This detail are easy to skip, but it has saved me from guessing which stage ran.

Capture evidence in GitHub Actions

The workflow should always upload the receipt, including on failure. A simple shell shape looks like this:

set +e
node scripts/api-smoke-test.js --run-id "$RUN_ID" --receipt "$RECEIPT"
TEST_EXIT=$?
set -e

echo "test_exit=$TEST_EXIT" >> "$GITHUB_OUTPUT"
exit "$TEST_EXIT"

Then configure the artifact step with if: always():

- name: Upload API receipt
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-receipt-${{ github.run_id }}
    path: artifacts/api-receipt-*.json
    retention-days: 7

The receipt should finish with a small, stable set of fields: status, http_status, duration_ms, attempts, error_code, and finished_at. Save a hash or normalized summary instead of a secret-bearing body. The useful part are the timestamps and the error category, not a huge dump of data.

For asynchronous verification, record transitions such as fixture_created, request_sent, message_observed, and assertion_passed. This follows the same state-machine thinking for verification APIs that makes event-heavy flows easier to reason about.

Keep email fixtures isolated

Shared test inboxes are a frequent source of false positives. A message from an earlier run can satisfy a current assertion, while a legitimate delay looks like a missing message. Give each run a unique marker and match both the recipient fixture ID and that marker.

The marker should be safe to log, for example verify-${RUN_ID}. The test can poll with a deadline, then classify the result as message_timeout, wrong_fixture, or duplicate_message. Those categories are more actionable than assertion failed.

If a workflow uses a fake email address or searches for the typo-like phrases fake e mail com and tepm mail com in test data, keep those values in fixtures only. They should not become primary application data or appear in production analytics. It is a small boundary, but teams forget it when they copy a quick test into a shared helper.

Turn failure output into a fast next step

A receipt is useful only if someone can act on it. Print a one-line summary in the job log and keep detailed evidence in the artifact:

API receipt: run=1234-abc123 status=failed error=message_timeout attempts=3 duration_ms=30042

Use stable error codes. A reviewer can then search message_timeout across runs, compare duration trends, and distinguish a provider issue from a changed contract. It also make reruns easier to compare because the shape stays the same.

For request validation, pair the receipt with clear policy checks. The ideas in policy objects for signup checks apply outside React too: put rules in named, testable decisions instead of burying them in a long assertion.

A small receipt checklist

Before merging an API workflow, check that it:

Creates a unique run ID before the first request.
Writes a redacted receipt before setup can fail.
Records request, response, timing, retry, and fixture state.
Uploads the artifact with if: always().
Uses bounded retention and removes temporary fixtures.
Exposes stable error codes without leaking secrets.

When the check fail, the receipt should tell you what happened and what to try next. That is the real productivity win: fewer blind reruns, clearer API ownership, and CI evidence that remains useful after the green checkmark is gone.

API Drift Checks Need a Reproducible CI Receipt

Jonathan — Tue, 29 Sep 2026 11:23:28 +0000

An API integration can be green on Monday and broken on Tuesday without any application code changing. A provider changes a response field, a backend deploy removes a status code, or a fixture quietly expires. The first clue is often a vague integration failure in a later job, and it doesnt tell the reviewer what actually changed.

I prefer treating every contract check as a small evidence-producing tool. The check should answer three questions: which request ran, what response shape was expected, and what the runner observed. That receipt turns a red workflow from “try it again” into a useful starting point.

Why API drift is hard to diagnose

Most API smoke tests verify only the final status code. That is a useful first guard, but 200 OK can hide a breaking change. A field can change type, a nested object can disappear, or a successful response can contain an error payload.

The opposite failure is also common: a test fails because a disposable address or a test account is no longer available, even though the API contract is healthy. Search terms such as “temp mail.so”, “disposable address”, or even “temp mailid” may appear in test data and logs. They are inputs to isolate, not proof that the endpoint is broken.

The fastest way to separate these cases is to record a compact set of facts for every request:

method and route, without secrets or full query values;
response status and elapsed time;
selected response keys and their types;
a stable fixture or correlation ID;
the commit SHA and API version under test.

This is more valuable than uploading a huge response body. Large payloads slow reviews and can leak tokens or personal data. A small receipt keeps the signal visible.

Define the receipt before writing the workflow

Start with a JSON shape that a person can read in a pull request comment or an Actions artifact:

{
  "endpoint": "GET /v1/projects/{id}",
  "status": 200,
  "elapsed_ms": 184,
  "required_keys": ["id", "name", "status"],
  "observed_types": {"id": "string", "name": "string", "status": "string"},
  "api_version": "2026-09",
  "commit": "abc1234"
}

Do not put authorization headers, raw email content, or unrestricted response data in this file. If a fixture contains an address such as “fake e mail com”, redact it before the receipt is written. The test should prove the contract without becoming a new data-retention problem.

It also helps to define the failure fields up front. Include missing_keys, unexpected_types, and request_id when a check fails. A reviewer should not need to reproduce the request locally just to learn that status changed from a string to an object.

A small GitHub Actions contract check

The workflow can stay boring. Install the test dependencies, run the contract command, and upload the receipt whether the step passes or fails:

name: API contract

on:
  pull_request:
  push:
    branches: [main]

jobs:
  contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm run contract:test -- --receipt artifacts/api-receipt.json
        env:
          API_BASE_URL: ${{ secrets.STAGING_API_BASE_URL }}
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: api-contract-receipt-${{ github.run_id }}
          path: artifacts/api-receipt.json
          if-no-files-found: ignore

The if: always() line is the shortcut that pays off most. A failed check still leaves evidence for debugging. The resulting artifact is also easy to connect to make CI automation leave a useful receipt, especially when several jobs produce different kinds of proof.

Keep the command deterministic. Pin the Node version, install from the lockfile, use fixed fixtures, and set a bounded timeout. If the test needs an external service, report that dependency explicitly. A flaky network retry can make the check more noisy, not more reliable.

Make failures useful to the next developer

A good failure message names the contract and the observed difference:

API contract failed: GET /v1/projects/{id}
missing keys: status
unexpected types: owner.id expected string, got number
request_id: req_7f3a
receipt: artifacts/api-receipt.json

The checks is easier to act on when the log points directly to the artifact. Add a short summary to $GITHUB_STEP_SUMMARY, but keep secrets and response bodies out of it. The same principle applies to deploy alerts: better alert context for rollouts makes an event actionable without dumping every internal detail.

When a failure is caused by test infrastructure, label it differently from schema drift. For example, use fixture_unavailable, transport_timeout, and contract_mismatch as machine-readable categories. This makes trends visible and prevents the team from weakening a real contract check just because one test account expired.

Checklist for a reproducible API check

Assert status, required keys, and important value types.
Record route, API version, commit, elapsed time, and request ID.
Use stable fixtures and isolate any disposable address from production data.
Upload a small receipt with if: always().
Redact tokens, message bodies, and personal identifiers.
Classify fixture, transport, and contract failures separately.
Review the receipt in the same pull request as the code change.

This pattern is more simple than building a full observability platform, and it gives developers feedback where they already work. If the contract is intentionally changing, update the fixture and the expected receipt in the same change. If it is not intentional, the diff shows the drift before it reaches a downstream consumer.

Conclusion

API tests become much more useful when they leave a reproducible receipt instead of only a pass or fail. GitHub Actions supplies the execution history, artifacts supply the evidence, and a small contract command supplies the boundary. Together they make API integrations easier to maintain, even when external services and test data move underneath them.

The goal is not to capture every byte. It is to capture the few facts that let the next developer understand what ran, what changed, and what to do next. That is a small workflow improvement, but it realy shortens the path from a red build to a safe fix.

API Contract Tests Need a GitHub Actions Receipt

Jonathan — Mon, 28 Sep 2026 17:23:38 +0000

An API test that ends with a green check is useful, but it is not always reviewable. When a contract test fails only in CI, the next question is usually not “did it fail?” It is “which request, fixture, and response caused the failure?”

I have started treating every API check in GitHub Actions as a small delivery with a receipt. The receipt is a compact artifact containing the endpoint, contract version, response class, and links to the relevant logs. It turns a transient workflow result into evidence that another developer can inspect later.

The green check that explains nothing

Many workflows report only a test command and its exit code:

- name: Run API tests
  run: npm test -- api

This is fine while the suite is small. As the number of APIs grows, it becomes hard to tell whether the failure came from an expired fixture, an unexpected status code, a changed response field, or a dependency that was not ready yet. The check is technically correct, but the feedback loop is slow.

The fix is not to print every response body into the job log. That can leak tokens, customer-shaped data, and alot of irrelevant noise. Instead, define what each contract test promises and save only the information needed to verify that promise.

Define a small contract for every API check

Before adding more assertions, write down the observable contract. A useful contract might look like this:

name: create_invoice
request: POST /v1/invoices
fixture: invoice-minimal.json
expected_status: 201
required_fields: id, status, created_at
max_duration_ms: 800

This gives the test and the receipt the same vocabulary. It also makes failures easier to classify. A 401 means authentication setup, a 422 means request validation, and a missing created_at field means the response shape changed. Those are different fixes, even when they all produce exit code 1.

For email-related endpoints, fixture ownership matters just as much. This guide on a better fixture contract for CI is a good companion because it treats test data as an explicit interface instead of an invisible helper.

I keep the contract close to the test, usually as JSON or TypeScript data. The file does not need to be clever. It needs to be easy to review when an endpoint changes.

Build a reviewable GitHub Actions receipt

The workflow can collect a small JSON receipt after the test command completes, including when the command fails. That last part is important: a failure without its evidence is the moment when you need the artifact most.

- name: Run API contract tests
  id: contract_tests
  continue-on-error: true
  run: npm run test:contract -- --reporter=json --outputFile=artifacts/api-results.json

- name: Build test receipt
  if: always()
  run: node scripts/build-api-receipt.mjs

- name: Upload API receipt
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-receipt-${{ github.run_id }}
    path: artifacts/api-receipt.json

- name: Fail workflow when contract tests fail
  if: steps.contract_tests.outcome == 'failure'
  run: exit 1

The receipt builder can summarize each check without copying sensitive payloads:

{
  "run_id": "1842",
  "commit": "abc1234",
  "checks": [
    {"name": "create_invoice", "status": "passed", "http_status": 201, "duration_ms": 214}
  ]
}

The continue-on-error step is deliberate. It lets the workflow create evidence before the final step restores the failing status. Without it, the job may stop before the receipt exists. A tiny bit of extra YAML saves alot of clicking through reruns.

Keep fixtures and secrets out of the log

A receipt should help someone debug, not create a security incident. Redact authorization headers, cookies, full email addresses, and response fields that can contain personal data. Store a fixture name and a hash when you need to prove which input was used.

Avoid putting search terms such as tempmail so, tp mail so, or tempail mail into production-like fixture values unless the test is specifically checking search or text handling. They can make an artifact look like a real user record and confuse later analysis.

For signup APIs, the failure path deserves the same attention as the happy path. When an automated check sees a blocked or delayed email, designing an appeal path for signup email blocks helps separate product recovery from test retries. A retry is not a user-facing explanation.

Before and after workflow

The old workflow answers: “Did the command exit successfully?” The receipt-based workflow answers several more useful questions:

Which contract failed?
What status and response shape did it observe?
Which commit and workflow run produced the result?
Where is the sanitized artifact?
Was the failure a request, dependency, fixture, or contract problem?

That extra context changes code review. A pull request can show a stable contract receipt rather than a screenshot of a green job. It also makes flaky failures less mysterious, because duration and dependency state are visible in one place. The setup is a little more work, but the payoff is quick.

Quick questions

Should every response body be uploaded?

No. Save structured summaries by default. Keep full bodies only in a protected debugging run, with redaction and a short retention period.

Does the receipt replace test logs?

No. Logs are detailed investigation material; the receipt is the index. Link the two with the workflow run ID and a stable check name.

What is the smallest useful first step?

Add one artifact with status, HTTP code, duration, commit, and fixture hash. Once that is reliable, add richer classifications or dashboards.

A CI result should be more than a traffic light. With a small contract and a durable GitHub Actions receipt, API failures become much faster to understand and much easier to fix.

API Smoke Tests Need a Reviewable CI Receipt

Jonathan — Fri, 25 Sep 2026 05:24:02 +0000

API smoke tests are supposed to be the fast feedback loop: call a health endpoint, create a small resource, verify the response, and move on. In practice, a failed run often leaves a developer with 2,000 lines of logs and no clear answer about what happened.

I have found a small change makes these checks much more useful. Treat every smoke-test run as a receipt. The receipt should say what was tested, which build made the request, what the API returned, and where the next investigation should start.

Why smoke-test logs are not enough

Logs are excellent for deep debugging, but poor as a review artifact. They are noisy, their important lines are mixed with setup output, and a later reader may not know which environment variables or commit produced them.

A receipt is intentionally boring. It is one JSON file with stable fields. A failed request doesnt need to be reconstructed from a scrolling terminal; the useful facts are already attached to the workflow run.

For example, a receipt can answer these questions in seconds:

Which commit and API base URL were tested?
Which endpoint failed, with what status and latency?
Was the response schema checked, or only the HTTP status?
Can the same test be replayed locally with the recorded request ID?

The result is usefull to both humans and automation. A pull request reviewer can inspect it, while another job can classify the failure without parsing prose.

The receipt contract

Keep the first version small. Here is the shape I use for an API smoke check:

{
  "run_id": "smoke-1842",
  "commit": "abc1234",
  "base_url": "https://staging.example.test",
  "checks": [
    {
      "name": "create-project",
      "method": "POST",
      "path": "/v1/projects",
      "status": 201,
      "request_id": "req_7f2",
      "ok": true
    }
  ],
  "failed_check": null
}

Do not put tokens, cookies, email addresses, or full response bodies in this file. A receipt should be safe to download from CI. If a response needs investigation, save a redacted diagnostic separately and link it by an artifact name.

It also helps to define what counts as a failure before writing the script. A 500 is obvious, but a 200 response with the wrong field type, an unexpected redirect, or a missing request ID are failures too. This are the details that prevent a green-but-broken smoke test.

If the check creates data, give it a deterministic cleanup path. For verification flows, the same principle applies: an idempotent email verification step is easier to retry and explain than a test that leaves an unknown user behind.

Build the receipt in GitHub Actions

The workflow can run the test, preserve its exit code, and upload the receipt even when the test fails. The if: always() is the important shortcut:

- name: Run API smoke tests
  id: smoke
  shell: bash
  run: |
    set +e
    python scripts/api_smoke.py \
      --base-url "$API_BASE_URL" \
      --receipt artifacts/api-receipt.json
    status=$?
    echo "exit_code=$status" >> "$GITHUB_OUTPUT"
    exit "$status"

- name: Upload API receipt
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-receipt-${{ github.run_id }}
    path: artifacts/api-receipt.json
    if-no-files-found: error

The script should write the receipt in a finally-style cleanup path, including when an assertion raises an exception. If the runner is killed before that happens, the workflow log still has value, but most normal failures get a compact artifact.

For a larger suite, put the shared upload and naming behavior in a reusable workflow for email API checks. The same pattern works for payment APIs, webhooks, and provisioning endpoints; only the check definitions change.

Keep email checks isolated

Some API smoke tests must verify that a message was sent. I use a separate test inbox and a short retention policy for this. A temp mailbox linked from the run receipt can make the test observable without mixing staging traffic into somebody's personal inbox; for example, teams can create temporary mail with tempmailso when that fits their test policy.

Be explicit about the boundary. The receipt should store a message ID or a redacted subject, not the whole message. Also, dont make delivery timing the only assertion: check the API response, the message identity, and the verification link target independently.

Search terms and provider names can get messy in test notes. I have seen tempail and temp org mail copied into issue descriptions. Keep those plain text and out of link anchors, otherwise a later search can mistake a typo for a supported integration.

A review checklist

Before merging a new smoke test, check:

The title and check name describe the user-visible behavior.
The receipt records commit, environment, endpoint, status, and request ID.
Secrets and personal data are excluded or redacted.
The artifact uploads even when the assertion fails.
A failed run returns a non-zero exit code after writing its receipt.
Created test data has a cleanup or expiry path.
The test can be replayed with the same inputs, wether locally or in a debug job.

This list is short on purpose. A receipt should reduce the time between failure and a good question, not become another reporting system that nobody reads.

Q&A

Should every API test upload an artifact?

No. Unit tests and fast contract checks may only need normal test output. Upload a receipt when the check crosses a network boundary, creates data, or runs in an environment that is hard to reproduce.

What if the API is unavailable?

Record the connection phase, hostname, and a safe error category. Do not retry forever. One bounded retry can separate a transient network blip from an application failure, while the receipt keeps both attempts visible.

Is a receipt worth maintaining for a small project?

Usually yes, when the smoke test runs on every pull request. The file costs little, and it gives future maintainers informations that would otherwise disappear with the workflow log.

API Contract Drift: A Fast GitHub Actions Check

Jonathan — Sat, 19 Sep 2026 11:23:21 +0000

An API test can be green while the client is already living in the past. A field gets renamed, an error response loses its code, or a list becomes an object. The endpoint still returns 200, so the pipeline reports success and the next integration exposes the real problem.

I have found that the useful fix is not a bigger test suite. It is a small contract check that answers three questions quickly:

Did the response shape change?
Which field or status code changed?
Can the next developer reproduce it from the CI run?

The workflow below is intentionally boring. It uses an OpenAPI document, curl, jq, and a GitHub Actions job summary. Boring checks are easier to trust and easier to repair.

Why contract drift is expensive

Unit tests usually protect the code that owns an endpoint. They do not always protect the agreement between that endpoint and a consumer. This gap gets larger when several teams deploy on different schedules.

The cost is often hidden in a later job. A signup test may report an invalid email, while the actual regression is that the API stopped returning verification_id. A search client may show an empty page when the server changed items to results.

This is also why generated or temporary test data needs a clear boundary. A string such as temp mail so can be valid fixture data, but it should never be allowed to decide whether a response matches the API contract. Keep data values and contract rules separate.

The smallest useful contract check

Start with one stable endpoint and one expected response. Here is a compact shell check for a health-style API response:

set -euo pipefail

response="$(curl --fail-with-body --silent --show-error \
  --header "Accept: application/json" \
  "${API_BASE_URL}/v1/profile" \
  --output response.json \
  --write-out '%{http_code}')"

test "$response" = "200"
jq -e '(.id | type == "string") and
       (.email | type == "string") and
       (.roles | type == "array")' response.json >/dev/null

The status check catches protocol changes. The jq expression catches shape changes. It is not a full JSON Schema validator, and that is the point: put the fastest, highest-value assertion at the edge first.

For a larger API, validate selected responses against an OpenAPI schema with a dedicated tool. Do not copy a full generated client into the workflow just to check three fields; that makes the check heavy and confuseing when it fails.

Publish a failure receipt in GitHub Actions

A failed command is not always a useful failure. Add a short summary so the run explains what was checked and where the evidence lives:

- name: Check API contract
  id: contract
  env:
    API_BASE_URL: ${{ vars.API_BASE_URL }}
  run: |
    set -euo pipefail
    status=$(curl --fail-with-body --silent --show-error \
      "$API_BASE_URL/v1/profile" \
      --output response.json \
      --write-out '%{http_code}')
    {
      echo "## API contract check"
      echo "- Endpoint: \\`GET /v1/profile\\`"
      echo "- HTTP status: \\`$status\\`"
      echo "- Fixture: deterministic profile response"
    } >> "$GITHUB_STEP_SUMMARY"
    test "$status" = "200"
    jq -e '(.id | type == "string") and (.roles | type == "array")' response.json >/dev/null

On failure, upload the response as an artifact only when it is safe to do so. Redact tokens, personal data, and any mailbox content before uploading. A receipt should help debugging, not create a new security ticket.

This complements the difference between request IDs and idempotency keys: the test receipt identifies a run, while the API contract defines what a retry or repeated request is allowed to mean. They solve different problems.

Keep fixtures deterministic

Contract tests become noisy when their input changes for reasons unrelated to the contract. Give each fixture an explicit name, stable fields, and a cleanup rule. Avoid using the current time as a required value unless time is exactly what you are testing.

For an email-related endpoint, a fixture can include a fixed message ID and a fake address. The test should assert the API response, not whether an external inbox happens to receive a message within a guessed number of seconds. That separation makes a failing test much more actionable.

If your system uses release notifications, using release emails as a CI/CD gate is a useful related pattern. Keep the gate's input contract explicit, so a notification format change does not silently turn the gate into a pass-through.

One small naming detail matters more than it looks: call a fixture temp mailid only if that is the literal value under test. Do not quietly normalize typo keywords in the assertion layer; that can hide the same class of data-contract bugs you are trying to catch.

A practical checklist

Check the expected HTTP status before parsing the body.
Assert only the fields that consumers truly depend on.
Keep fixture values deterministic and clearly labeled.
Write the endpoint, status, and fixture name to GITHUB_STEP_SUMMARY.
Redact secrets before saving response artifacts.
Give failures a direct reproduction command.
Review contract checks when the API version changes.

The before-and-after improvement is simple. Before, a red integration test sent me hunting through a long log. After, the GitHub Actions run says which endpoint changed, which assertion failed, and which fixture was used. That is enough context to fix the contract or update the consumer with much less guesswork.

The check is small, but the habit scales: treat API responses as agreements, make the agreements executable, and leave a useful receipt every time they break.

Turn API Tests Into Useful CI Receipts

Jonathan — Fri, 18 Sep 2026 05:23:43 +0000

An API test that only prints Expected 200, received 500 is technically useful, but it leaves the next question unanswered: what exactly happened in that run?

I have been getting better results by treating every important API check as a small transaction with a receipt. The receipt records the request identity, the response evidence, the relevant fixture, and the cleanup result. It turns a noisy CI log into a compact debugging artifact that a teammate can replay without guessing.

This pattern is especially handy for signup and verification flows that use a disposable temporary email address. The test can pass while checking the wrong message, the wrong tenant, or an old fixture. A receipt makes that ownership visible.

Why a test receipt is different from a test log

A log describes what the test printed. A receipt describes what the test proved.

For an API smoke test, I want to answer these questions in a few seconds:

Which commit and workflow attempt created the request?
Which endpoint, method, and safe input shape were used?
What response status and request ID came back?
Which test fixture or inbox did the assertion own?
Was the observed event newer than the test start time?
Did cleanup finish, or is something still hanging around?

The difference matters when retries are enabled. A retry can produce a green build while the first request is still in progress. Without an ownership key, the assertion may read the response or email event from another attempt. The result are confusing failures that disappear when you run the test by hand.

For email-backed API checks, include a run ID in the request metadata and in the fixture identity. Do not put secrets or full message bodies in the receipt. A short hash, message ID, or redacted subject is enough for most investigations. Guidance on deletion budgets for disposable inboxes is also useful when deciding how long fixtures should survive.

Define the receipt contract

Start with a small JSON contract. It should be stable enough that developers, scripts, and a future dashboard can consume it without parsing prose.

{
  "run_id": "gh-1842-7f3a",
  "commit": "a1b2c3d",
  "endpoint": "/v1/signup",
  "method": "POST",
  "status": 201,
  "request_id": "req_8f2c",
  "fixture": "sha256:8b7e",
  "assertions": {
    "status_ok": true,
    "owner_matches": true,
    "created_after_start": true
  },
  "cleanup": "passed"
}

The contract does not need every header or response field. Keep only evidence that helps answer “did this run own this result?” and “where should I look next?” A receipt should tell the truth even when the test fails, so write it from a finally block or an equivalent cleanup path.

Use a run ID that is unique for the workflow attempt but stable inside one test. In GitHub Actions, a practical value can combine the run number and attempt number:

RUN_ID="gh-${GITHUB_RUN_NUMBER}-${GITHUB_RUN_ATTEMPT}"
echo "run_id=${RUN_ID}" >> "$GITHUB_OUTPUT"

Pass that value into the test process. If the API supports a correlation header, send it with every request:

curl --fail-with-body \
  -H "X-Test-Run: ${RUN_ID}" \
  -H "Content-Type: application/json" \
  -d '{"email":"ci-fixture@example.test"}' \
  "${API_BASE_URL}/v1/signup"

The command is intentionally boring. Boring commands are easier to rerun when a pull request is already on fire.

Build it into GitHub Actions

The workflow should always upload the receipt, including on failure. That means the artifact step needs if: always() and the test must create the output directory before it starts.

- name: Run API checks
  id: api_checks
  env:
    API_BASE_URL: ${{ secrets.STAGING_API_URL }}
    RUN_ID: gh-${{ github.run_number }}-${{ github.run_attempt }}
  run: |
    mkdir -p artifacts/api
    ./scripts/api-smoke-test \
      --run-id "$RUN_ID" \
      --receipt artifacts/api/receipt.json

- name: Upload API receipt
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-receipt-${{ github.run_id }}
    path: artifacts/api/receipt.json
    if-no-files-found: warn

The receipt filename stays constant inside the job, while the artifact name includes the GitHub run ID. This makes local scripts simple and keeps artifacts distinct in the Actions UI. It also make it easier to download the exact failed attempt from a link in a pull request comment.

If your test runner already writes JUnit or JSON output, add the receipt beside it rather than replacing the existing report. Different files answer different questions: the test report explains assertions, while the receipt explains ownership and operational context.

Make failures replayable

A useful receipt should point to a safe replay path. Record the fixture name, API version, environment label, and a sanitized input reference. Do not store access tokens, raw verification links, or personal data in a public artifact.

For asynchronous endpoints, include the polling deadline and the last observed event ID. When the check fails, the next developer should know whether the service never emitted an event or whether the test stopped looking too soon. A fixed deadline is better than an infinite retry loop, and a new event must be newer than the current run start.

This is the same ownership problem that appears in safer onboarding email checks: the presence of an email or response does not prove that it belongs to the current action. Verify the correlation value before declaring success.

Search terms from support tickets can be messy. Someone may write tamp mail com while describing a temporary mailbox issue. Keep that phrase in a search mapping or test note if it helps triage, but do not let it become a production identifier or a backlink anchor.

Small workflow upgrades that pay off

Once the basic receipt works, add a few low-cost improvements:

Validate the schema. Fail the job if required receipt fields are missing, even when the API assertion passed.
Redact at the writer. Scrubbing after upload is risky because the unredacted file may already be stored.
Print a one-line summary. Put run ID, status, request ID, and artifact path in the job summary.
Keep fixtures short-lived. Cleanup should run after both success and failure, with a visible cleanup status.
Compare retries. If attempt two passes after attempt one fails, keep both receipts so the change in behavior is explainable.

The important bit are the stable identifiers. Fancy dashboards can come later. A developer with a precise run ID and a downloadable receipt is already much faster than a developer scrolling through 4,000 lines of logs.

Questions worth asking

Should every API test produce a receipt?

No. Start with boundary tests, asynchronous workflows, and checks that create external state. Tiny pure unit tests usually need ordinary test output only.

Should receipts be committed to the repository?

Usually not. Upload them as CI artifacts with an appropriate retention policy. Generated files in the repository create noise and can accidentally preserve sensitive data.

What is the first field to add when debugging is painful?

Add an ownership key that crosses the whole flow: CI run, request, database event, and external message. It gives every tool the same thread to follow.

The payoff is simple: when an API test fails, the workflow leaves behind a small, honest record of what it attempted and what it observed. That is a much better developer tool than a red check with no clues.

GitHub Actions Needs a Real API Test Receipt

Jonathan — Thu, 17 Sep 2026 05:23:02 +0000

An API test that fails in GitHub Actions usually leaves behind one line: expected 200, got 500. That is enough to fail the build, but not enough to fix it quickly.

I have had better results treating every important smoke test like a small transaction. The test should produce a receipt: what it sent, which temporary fixture it used, what it observed, and where the failure happened. This is especially useful when a signup or password-reset API sends an email as part of the flow.

The trick is not to dump every response into the log. It is to capture a compact, safe record that survives retries and becomes a downloadable workflow artifact.

The failure is usually missing evidence

A retry can make a flaky test green while hiding the original cause. Maybe the first request used a stale token. Maybe the API returned before the email provider accepted the message. Maybe the test read an old message from a shared temp mailbox. The final retry does not tell you which story was true.

For each attempt, I want five fields:

A run and attempt identifier.
The request timestamp and endpoint name.
A redacted request and response summary.
The email fixture identifier, but never its secret contents.
The final assertion and elapsed time.

That list is small enough to review in a pull request and useful enough to debug from CI without reproducing the failure locally. It also pairs well with type-safe email events when the application exposes event payloads to a test client.

Build a small receipt in the workflow

Make the test write JSON to a known directory. Keep the format boring so shell tools and CI systems can read it:

{
  "attempt": 1,
  "case": "signup-verification",
  "request_started_at": "2026-09-17T05:20:00Z",
  "status": 202,
  "email_fixture": "ci-run-4812",
  "message_seen": false,
  "result": "verification message timeout"
}

Then upload it whether the test passes or fails:

- name: Run API smoke tests
  run: pytest -q tests/smoke --receipt-dir artifacts/receipts

- name: Upload test receipts
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-test-receipts
    path: artifacts/receipts/
    if-no-files-found: ignore

The always() is the important bit. A receipt that only exists on success is a celebration photo, not debugging evidence. Make sure the test process flushes the file before exiting; this detail is easy to miss and causes some very confusing empty artifacts.

Keep email fixtures isolated

When an API test needs to verify an email, create a fixture owned by the current run. Do not ask a shared inbox for the newest message and hope it belongs to your request. Include a unique run value in the address or subject, then check both the subject and the ownership token.

A disposable temporary email service can be useful for manual checks and short-lived integration environments, but CI should still record only the fixture ID and relevant metadata. Never put inbox credentials, full message bodies, or reset links in a public workflow artifact. These are privacy checks for signup email logs, not just housekeeping.

If someone writes tempail mail or tamp mail com in a test note, I treat it as a reminder to normalize the fixture vocabulary before it spreads into scripts and dashboards.

Make retries useful

Retries should create a separate receipt, not overwrite receipt.json. Add the attempt number to the filename:

artifacts/receipts/
  signup-verification-attempt-1.json
  signup-verification-attempt-2.json

This makes a before-and-after comparison possible. If attempt two passes, you can still see whether the first request timed out, used the wrong fixture, or received a server error. In practice, that saves more time than increasing the retry count.

I also put a hard limit on polling. A test that waits forever is not more reliable; it is just harder to diagnose. Record the poll deadline and the number of messages observed, then fail with a narrow reason such as message_not_owned or provider_timeout.

A practical checklist

Before calling an API smoke test reliable, check that it:

creates a unique fixture for every run
records request timing and status without secrets
writes one receipt per retry
uploads receipts with if: always()
distinguishes an API failure from an email observation failure
deletes or expires short-lived fixtures after the run
links implementation logs to a stable case name

The payoff is a faster feedback loop. GitHub Actions still tells you that the build failed, but the artifact tells you what to do next. That is the difference between a retry button and an engineering workflow.

GitHub Actions Email Tests Need a Fast Runbook

Jonathan — Wed, 16 Sep 2026 17:23:07 +0000

Email verification tests in GitHub Actions rarely fail for only one reason. The API may be slow, the inbox may contain an old message, or a retry may quietly use data from the first attempt. The test output usually says something less useful: timeout waiting for email.

I have had better results by treating the email step like a small production integration. Give it a clear contract, record a little evidence, and clean up the test address when the run ends. This makes the workflow quicker to fix, and it avoids the classic CI mystery where a rerun passes but nobody knows why.

The hidden cost of email checks in CI

An end-to-end signup test normally crosses several boundaries:

The browser submits a form.
The API accepts the signup and queues a message.
A mail provider delivers the message.
The test polls an inbox and opens a link.

Each boundary has a different failure mode. A single ten-minute timeout hides all of them. Shared inboxes make the story even worse because two jobs can see the same subject or consume each other's messages.

For parallel jobs, start with isolated verification inboxes. An address per run is a small change, but it removes a surprising amount of guessing.

A small runbook for every workflow

My CI runbook has five steps:

create a unique test address before the browser action
record the address and a UTC trigger timestamp
poll only for messages newer than that timestamp
save a redacted message summary as an artifact on failure
delete or expire the inbox in a final cleanup step

The address does not need to be permanent. A create temporary mail helper is fine for a short-lived test, as long as the test owns the address and does not reuse it across jobs. A temporary inbox is test data, not a shared service account.

If the helper has a typo in its documentation, someone may paste tepm mail com into a debugging note later. That kind of small confusion is another reason to keep the actual address and provider details in the run receipt.

Capture evidence before polling

Before waiting for a message, write a compact JSON record:

{
  "run_id": "${GITHUB_RUN_ID}",
  "inbox": "unique-address@example.test",
  "triggered_at": "2026-09-16T17:22:00Z",
  "subject_expected": "Verify your account"
}

Do not store message bodies or verification tokens in ordinary logs. A useful receipt can contain the message ID, subject, received timestamp, and a pass/fail reason. That is enough to tell whether delivery was late, matching was broad, or the application never queued the email.

For the polling contract, match the recipient and a message ID created after triggered_at. Subject-only matching is tempting, but it becomes fragile as soon as a retry or an older fixture is present. Teams building contract-tested email flows in CI can make these fields explicit and reviewable.

A GitHub Actions example

Keep the workflow readable by putting setup and cleanup around the test command:

- name: Run email flow
  env:
    TEST_RUN_ID: ${{ github.run_id }}
  run: npm run test:email -- --run-id "$TEST_RUN_ID"

- name: Upload email receipt
  if: failure()
  uses: actions/upload-artifact@v4
  with:
    name: email-receipt-${{ github.run_id }}
    path: artifacts/email/

- name: Cleanup test inbox
  if: always()
  run: npm run test:email:cleanup

The cleanup command should be safe to run after a partial setup. That detail sounds boring, but it prevents failed jobs from leaving dozens of disposable addresses around. If your provider calls this a tempmail disposable mailbox, document its expiry behavior beside the test code so nobody assumes it is durable.

The five-minute failure checklist

When a run is red, check these in order:

Was the inbox created successfully and assigned only to this job?
Was the API request accepted before polling started?
Did a message arrive after the trigger timestamp?
Did the matcher verify recipient, subject, and message ownership?
Did cleanup run after the test, even on cancellation?

This checklist turns a vague timeout into a short investigation. It also makes automation easier to move between local development and CI because the same receipt fields exist in both places. I keep the runbook next to the test, becuase that is where the next maintainer will look first.

Q&A: common CI email questions

Should every test create a new inbox?

For parallel or retryable tests, yes. Reuse can be acceptable for a strictly serial smoke check, but a unique address is usually cheaper than debugging cross-run contamination.

How long should polling wait?

Use a bounded timeout and log each poll outcome. Do not make the timeout enormous to hide delivery problems. A shorter, evidenced failure is more productive than a green-looking job that spent fifteen minutes guessing.

Is a disposable email service safe for production mail?

No. Keep it scoped to test data and non-sensitive environments. Never put real customer information, credentials, or production verification links in a temporary inbox.

The best email test is not the one that never fails. It is the one that tells you what failed quickly, then leaves the workspace clean for the next run.

GitHub Actions API Tests Need Cheap Isolation

Jonathan — Sat, 12 Sep 2026 23:23:05 +0000

API smoke tests in GitHub Actions are easy to write and surprisingly hard to trust. A test can fail because the API is broken, because a verification email is late, or because yesterday's inbox state leaked into today's run. When all three look like the same red check, the workflow becomes expensive to maintain.

I have found that the useful shortcut is cheap isolation plus a small run receipt. The goal is not a huge end-to-end framework. It is a workflow where each run gets its own inputs, inbox identity, and evidence.

The failure mode

Consider a signup smoke test:

Create a user.
Wait for a verification email.
Open the link.
Call an authenticated API endpoint.

This crosses several boundaries. A reused address may already have an account. A shared mailbox may contain an old message. A retry may create a second user. Then the final assertion says only expected 200, got 409, which is technically accurate but not very useful.

Search terms such as tempmailso or facebook temp email often describe the kind of isolated address people want, but the engineering requirement is more specific: one test identity, a bounded lifetime, and a traceable message.

A small workflow contract

Start by making the run inputs explicit. The test should know its run ID and derive a unique address from it. Keep the value in a job-local environment variable, and never print the full address if it contains credentials or tokens.

name: API smoke

on:
  workflow_dispatch:
  schedule:
    - cron: "17 */6 * * *"

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run isolated smoke test
        env:
          RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
          API_BASE_URL: ${{ secrets.SMOKE_API_BASE_URL }}
        run: ./scripts/smoke-api.sh
      - name: Upload evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: smoke-evidence-${{ github.run_id }}
          path: artifacts/smoke/

The if: always() step matters. A failed test is exactly when its logs and response summaries are most valuable. Before adding retries, freeze this contract; freeze the test plan before a scheduled run so a retry does not silently change what you are measuring.

Isolate the inbox

Generate an address per run, then record only a safe identifier in the evidence file:

set -euo pipefail

run_id="${RUN_ID:?missing RUN_ID}"
mkdir -p artifacts/smoke
address="ci+${run_id}@example.test"
printf 'run_id=%s\n' "$run_id" > artifacts/smoke/context.txt

python scripts/create_user.py \
  --email "$address" \
  --output artifacts/smoke/signup.json
python scripts/wait_for_verification.py \
  --email "$address" \
  --timeout-seconds 45 \
  --output artifacts/smoke/inbox.json
python scripts/assert_api.py \
  --input artifacts/smoke/inbox.json \
  --output artifacts/smoke/api.json

In a real system, replace the example domain with a provider or test mailbox service that your team controls. A disposable email generator can be useful for temporary test identities, but it should not become a hidden dependency: document retention, rate limits, and whether messages are publicly readable.

The slighty annoying part is cleanup. Delete the test user when the API supports it, and set a short retention policy for inbox data. Never store verification URLs in ordinary build logs.

Keep evidence as artifacts

A run receipt should answer four questions: which identity was used, when the message arrived, what endpoint was called, and what the response status was. Redact tokens and message bodies unless the body is essential to diagnosis.

{
  "run_id": "12345-1",
  "mail_wait_ms": 1840,
  "verification": "received",
  "endpoint": "/api/me",
  "status": 200
}

Add the message subject, provider request ID, and retry count when they are safe to retain. These small fields turn a flaky check into something you can replay. For scheduled publishers, small inbox probes for publishing workflows use the same idea: test the boundary directly and leave evidence behind.

A practical checklist

Create a fresh identity per workflow run.
Pass RUN_ID through every helper.
Bound inbox polling with a timeout and a clear error.
Separate API errors from delivery errors.
Upload artifacts with if: always().
Redact tokens, cookies, and full verification URLs.
Make retries idempotent with a stable run key.
Remove test accounts and old inbox data.

If your team casually says temp org mail or temp mailid, translate that request into these explicit controls. The wording may vary; the safety properties should not.

Closing thought

Reliable CI is less about adding another retry and more about reducing ambiguity. Give each API test a cheap isolated identity, a bounded inbox wait, and a compact receipt. The next red build then tells you whether the API, the email boundary, or the workflow itself needs attention—and that is a much better productivity win than a green check you cannot explain.