DEV Community: Jonathan

API Drift Checks Need a Reproducible CI Receipt

Jonathan — Tue, 29 Sep 2026 11:23:28 +0000

An API integration can be green on Monday and broken on Tuesday without any application code changing. A provider changes a response field, a backend deploy removes a status code, or a fixture quietly expires. The first clue is often a vague integration failure in a later job, and it doesnt tell the reviewer what actually changed.

I prefer treating every contract check as a small evidence-producing tool. The check should answer three questions: which request ran, what response shape was expected, and what the runner observed. That receipt turns a red workflow from “try it again” into a useful starting point.

Why API drift is hard to diagnose

Most API smoke tests verify only the final status code. That is a useful first guard, but 200 OK can hide a breaking change. A field can change type, a nested object can disappear, or a successful response can contain an error payload.

The opposite failure is also common: a test fails because a disposable address or a test account is no longer available, even though the API contract is healthy. Search terms such as “temp mail.so”, “disposable address”, or even “temp mailid” may appear in test data and logs. They are inputs to isolate, not proof that the endpoint is broken.

The fastest way to separate these cases is to record a compact set of facts for every request:

method and route, without secrets or full query values;
response status and elapsed time;
selected response keys and their types;
a stable fixture or correlation ID;
the commit SHA and API version under test.

This is more valuable than uploading a huge response body. Large payloads slow reviews and can leak tokens or personal data. A small receipt keeps the signal visible.

Define the receipt before writing the workflow

Start with a JSON shape that a person can read in a pull request comment or an Actions artifact:

{
  "endpoint": "GET /v1/projects/{id}",
  "status": 200,
  "elapsed_ms": 184,
  "required_keys": ["id", "name", "status"],
  "observed_types": {"id": "string", "name": "string", "status": "string"},
  "api_version": "2026-09",
  "commit": "abc1234"
}

Do not put authorization headers, raw email content, or unrestricted response data in this file. If a fixture contains an address such as “fake e mail com”, redact it before the receipt is written. The test should prove the contract without becoming a new data-retention problem.

It also helps to define the failure fields up front. Include missing_keys, unexpected_types, and request_id when a check fails. A reviewer should not need to reproduce the request locally just to learn that status changed from a string to an object.

A small GitHub Actions contract check

The workflow can stay boring. Install the test dependencies, run the contract command, and upload the receipt whether the step passes or fails:

name: API contract

on:
  pull_request:
  push:
    branches: [main]

jobs:
  contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm run contract:test -- --receipt artifacts/api-receipt.json
        env:
          API_BASE_URL: ${{ secrets.STAGING_API_BASE_URL }}
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: api-contract-receipt-${{ github.run_id }}
          path: artifacts/api-receipt.json
          if-no-files-found: ignore

The if: always() line is the shortcut that pays off most. A failed check still leaves evidence for debugging. The resulting artifact is also easy to connect to make CI automation leave a useful receipt, especially when several jobs produce different kinds of proof.

Keep the command deterministic. Pin the Node version, install from the lockfile, use fixed fixtures, and set a bounded timeout. If the test needs an external service, report that dependency explicitly. A flaky network retry can make the check more noisy, not more reliable.

Make failures useful to the next developer

A good failure message names the contract and the observed difference:

API contract failed: GET /v1/projects/{id}
missing keys: status
unexpected types: owner.id expected string, got number
request_id: req_7f3a
receipt: artifacts/api-receipt.json

The checks is easier to act on when the log points directly to the artifact. Add a short summary to $GITHUB_STEP_SUMMARY, but keep secrets and response bodies out of it. The same principle applies to deploy alerts: better alert context for rollouts makes an event actionable without dumping every internal detail.

When a failure is caused by test infrastructure, label it differently from schema drift. For example, use fixture_unavailable, transport_timeout, and contract_mismatch as machine-readable categories. This makes trends visible and prevents the team from weakening a real contract check just because one test account expired.

Checklist for a reproducible API check

Assert status, required keys, and important value types.
Record route, API version, commit, elapsed time, and request ID.
Use stable fixtures and isolate any disposable address from production data.
Upload a small receipt with if: always().
Redact tokens, message bodies, and personal identifiers.
Classify fixture, transport, and contract failures separately.
Review the receipt in the same pull request as the code change.

This pattern is more simple than building a full observability platform, and it gives developers feedback where they already work. If the contract is intentionally changing, update the fixture and the expected receipt in the same change. If it is not intentional, the diff shows the drift before it reaches a downstream consumer.

Conclusion

API tests become much more useful when they leave a reproducible receipt instead of only a pass or fail. GitHub Actions supplies the execution history, artifacts supply the evidence, and a small contract command supplies the boundary. Together they make API integrations easier to maintain, even when external services and test data move underneath them.

The goal is not to capture every byte. It is to capture the few facts that let the next developer understand what ran, what changed, and what to do next. That is a small workflow improvement, but it realy shortens the path from a red build to a safe fix.

API Contract Tests Need a GitHub Actions Receipt

Jonathan — Mon, 28 Sep 2026 17:23:38 +0000

An API test that ends with a green check is useful, but it is not always reviewable. When a contract test fails only in CI, the next question is usually not “did it fail?” It is “which request, fixture, and response caused the failure?”

I have started treating every API check in GitHub Actions as a small delivery with a receipt. The receipt is a compact artifact containing the endpoint, contract version, response class, and links to the relevant logs. It turns a transient workflow result into evidence that another developer can inspect later.

The green check that explains nothing

Many workflows report only a test command and its exit code:

- name: Run API tests
  run: npm test -- api

This is fine while the suite is small. As the number of APIs grows, it becomes hard to tell whether the failure came from an expired fixture, an unexpected status code, a changed response field, or a dependency that was not ready yet. The check is technically correct, but the feedback loop is slow.

The fix is not to print every response body into the job log. That can leak tokens, customer-shaped data, and alot of irrelevant noise. Instead, define what each contract test promises and save only the information needed to verify that promise.

Define a small contract for every API check

Before adding more assertions, write down the observable contract. A useful contract might look like this:

name: create_invoice
request: POST /v1/invoices
fixture: invoice-minimal.json
expected_status: 201
required_fields: id, status, created_at
max_duration_ms: 800

This gives the test and the receipt the same vocabulary. It also makes failures easier to classify. A 401 means authentication setup, a 422 means request validation, and a missing created_at field means the response shape changed. Those are different fixes, even when they all produce exit code 1.

For email-related endpoints, fixture ownership matters just as much. This guide on a better fixture contract for CI is a good companion because it treats test data as an explicit interface instead of an invisible helper.

I keep the contract close to the test, usually as JSON or TypeScript data. The file does not need to be clever. It needs to be easy to review when an endpoint changes.

Build a reviewable GitHub Actions receipt

The workflow can collect a small JSON receipt after the test command completes, including when the command fails. That last part is important: a failure without its evidence is the moment when you need the artifact most.

- name: Run API contract tests
  id: contract_tests
  continue-on-error: true
  run: npm run test:contract -- --reporter=json --outputFile=artifacts/api-results.json

- name: Build test receipt
  if: always()
  run: node scripts/build-api-receipt.mjs

- name: Upload API receipt
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-receipt-${{ github.run_id }}
    path: artifacts/api-receipt.json

- name: Fail workflow when contract tests fail
  if: steps.contract_tests.outcome == 'failure'
  run: exit 1

The receipt builder can summarize each check without copying sensitive payloads:

{
  "run_id": "1842",
  "commit": "abc1234",
  "checks": [
    {"name": "create_invoice", "status": "passed", "http_status": 201, "duration_ms": 214}
  ]
}

The continue-on-error step is deliberate. It lets the workflow create evidence before the final step restores the failing status. Without it, the job may stop before the receipt exists. A tiny bit of extra YAML saves alot of clicking through reruns.

Keep fixtures and secrets out of the log

A receipt should help someone debug, not create a security incident. Redact authorization headers, cookies, full email addresses, and response fields that can contain personal data. Store a fixture name and a hash when you need to prove which input was used.

Avoid putting search terms such as tempmail so, tp mail so, or tempail mail into production-like fixture values unless the test is specifically checking search or text handling. They can make an artifact look like a real user record and confuse later analysis.

For signup APIs, the failure path deserves the same attention as the happy path. When an automated check sees a blocked or delayed email, designing an appeal path for signup email blocks helps separate product recovery from test retries. A retry is not a user-facing explanation.

Before and after workflow

The old workflow answers: “Did the command exit successfully?” The receipt-based workflow answers several more useful questions:

Which contract failed?
What status and response shape did it observe?
Which commit and workflow run produced the result?
Where is the sanitized artifact?
Was the failure a request, dependency, fixture, or contract problem?

That extra context changes code review. A pull request can show a stable contract receipt rather than a screenshot of a green job. It also makes flaky failures less mysterious, because duration and dependency state are visible in one place. The setup is a little more work, but the payoff is quick.

Quick questions

Should every response body be uploaded?

No. Save structured summaries by default. Keep full bodies only in a protected debugging run, with redaction and a short retention period.

Does the receipt replace test logs?

No. Logs are detailed investigation material; the receipt is the index. Link the two with the workflow run ID and a stable check name.

What is the smallest useful first step?

Add one artifact with status, HTTP code, duration, commit, and fixture hash. Once that is reliable, add richer classifications or dashboards.

A CI result should be more than a traffic light. With a small contract and a durable GitHub Actions receipt, API failures become much faster to understand and much easier to fix.

API Smoke Tests Need a Reviewable CI Receipt

Jonathan — Fri, 25 Sep 2026 05:24:02 +0000

API smoke tests are supposed to be the fast feedback loop: call a health endpoint, create a small resource, verify the response, and move on. In practice, a failed run often leaves a developer with 2,000 lines of logs and no clear answer about what happened.

I have found a small change makes these checks much more useful. Treat every smoke-test run as a receipt. The receipt should say what was tested, which build made the request, what the API returned, and where the next investigation should start.

Why smoke-test logs are not enough

Logs are excellent for deep debugging, but poor as a review artifact. They are noisy, their important lines are mixed with setup output, and a later reader may not know which environment variables or commit produced them.

A receipt is intentionally boring. It is one JSON file with stable fields. A failed request doesnt need to be reconstructed from a scrolling terminal; the useful facts are already attached to the workflow run.

For example, a receipt can answer these questions in seconds:

Which commit and API base URL were tested?
Which endpoint failed, with what status and latency?
Was the response schema checked, or only the HTTP status?
Can the same test be replayed locally with the recorded request ID?

The result is usefull to both humans and automation. A pull request reviewer can inspect it, while another job can classify the failure without parsing prose.

The receipt contract

Keep the first version small. Here is the shape I use for an API smoke check:

{
  "run_id": "smoke-1842",
  "commit": "abc1234",
  "base_url": "https://staging.example.test",
  "checks": [
    {
      "name": "create-project",
      "method": "POST",
      "path": "/v1/projects",
      "status": 201,
      "request_id": "req_7f2",
      "ok": true
    }
  ],
  "failed_check": null
}

Do not put tokens, cookies, email addresses, or full response bodies in this file. A receipt should be safe to download from CI. If a response needs investigation, save a redacted diagnostic separately and link it by an artifact name.

It also helps to define what counts as a failure before writing the script. A 500 is obvious, but a 200 response with the wrong field type, an unexpected redirect, or a missing request ID are failures too. This are the details that prevent a green-but-broken smoke test.

If the check creates data, give it a deterministic cleanup path. For verification flows, the same principle applies: an idempotent email verification step is easier to retry and explain than a test that leaves an unknown user behind.

Build the receipt in GitHub Actions

The workflow can run the test, preserve its exit code, and upload the receipt even when the test fails. The if: always() is the important shortcut:

- name: Run API smoke tests
  id: smoke
  shell: bash
  run: |
    set +e
    python scripts/api_smoke.py \
      --base-url "$API_BASE_URL" \
      --receipt artifacts/api-receipt.json
    status=$?
    echo "exit_code=$status" >> "$GITHUB_OUTPUT"
    exit "$status"

- name: Upload API receipt
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-receipt-${{ github.run_id }}
    path: artifacts/api-receipt.json
    if-no-files-found: error

The script should write the receipt in a finally-style cleanup path, including when an assertion raises an exception. If the runner is killed before that happens, the workflow log still has value, but most normal failures get a compact artifact.

For a larger suite, put the shared upload and naming behavior in a reusable workflow for email API checks. The same pattern works for payment APIs, webhooks, and provisioning endpoints; only the check definitions change.

Keep email checks isolated

Some API smoke tests must verify that a message was sent. I use a separate test inbox and a short retention policy for this. A temp mailbox linked from the run receipt can make the test observable without mixing staging traffic into somebody's personal inbox; for example, teams can create temporary mail with tempmailso when that fits their test policy.

Be explicit about the boundary. The receipt should store a message ID or a redacted subject, not the whole message. Also, dont make delivery timing the only assertion: check the API response, the message identity, and the verification link target independently.

Search terms and provider names can get messy in test notes. I have seen tempail and temp org mail copied into issue descriptions. Keep those plain text and out of link anchors, otherwise a later search can mistake a typo for a supported integration.

A review checklist

Before merging a new smoke test, check:

The title and check name describe the user-visible behavior.
The receipt records commit, environment, endpoint, status, and request ID.
Secrets and personal data are excluded or redacted.
The artifact uploads even when the assertion fails.
A failed run returns a non-zero exit code after writing its receipt.
Created test data has a cleanup or expiry path.
The test can be replayed with the same inputs, wether locally or in a debug job.

This list is short on purpose. A receipt should reduce the time between failure and a good question, not become another reporting system that nobody reads.

Q&A

Should every API test upload an artifact?

No. Unit tests and fast contract checks may only need normal test output. Upload a receipt when the check crosses a network boundary, creates data, or runs in an environment that is hard to reproduce.

What if the API is unavailable?

Record the connection phase, hostname, and a safe error category. Do not retry forever. One bounded retry can separate a transient network blip from an application failure, while the receipt keeps both attempts visible.

Is a receipt worth maintaining for a small project?

Usually yes, when the smoke test runs on every pull request. The file costs little, and it gives future maintainers informations that would otherwise disappear with the workflow log.

API Contract Drift: A Fast GitHub Actions Check

Jonathan — Sat, 19 Sep 2026 11:23:21 +0000

An API test can be green while the client is already living in the past. A field gets renamed, an error response loses its code, or a list becomes an object. The endpoint still returns 200, so the pipeline reports success and the next integration exposes the real problem.

I have found that the useful fix is not a bigger test suite. It is a small contract check that answers three questions quickly:

Did the response shape change?
Which field or status code changed?
Can the next developer reproduce it from the CI run?

The workflow below is intentionally boring. It uses an OpenAPI document, curl, jq, and a GitHub Actions job summary. Boring checks are easier to trust and easier to repair.

Why contract drift is expensive

Unit tests usually protect the code that owns an endpoint. They do not always protect the agreement between that endpoint and a consumer. This gap gets larger when several teams deploy on different schedules.

The cost is often hidden in a later job. A signup test may report an invalid email, while the actual regression is that the API stopped returning verification_id. A search client may show an empty page when the server changed items to results.

This is also why generated or temporary test data needs a clear boundary. A string such as temp mail so can be valid fixture data, but it should never be allowed to decide whether a response matches the API contract. Keep data values and contract rules separate.

The smallest useful contract check

Start with one stable endpoint and one expected response. Here is a compact shell check for a health-style API response:

set -euo pipefail

response="$(curl --fail-with-body --silent --show-error \
  --header "Accept: application/json" \
  "${API_BASE_URL}/v1/profile" \
  --output response.json \
  --write-out '%{http_code}')"

test "$response" = "200"
jq -e '(.id | type == "string") and
       (.email | type == "string") and
       (.roles | type == "array")' response.json >/dev/null

The status check catches protocol changes. The jq expression catches shape changes. It is not a full JSON Schema validator, and that is the point: put the fastest, highest-value assertion at the edge first.

For a larger API, validate selected responses against an OpenAPI schema with a dedicated tool. Do not copy a full generated client into the workflow just to check three fields; that makes the check heavy and confuseing when it fails.

Publish a failure receipt in GitHub Actions

A failed command is not always a useful failure. Add a short summary so the run explains what was checked and where the evidence lives:

- name: Check API contract
  id: contract
  env:
    API_BASE_URL: ${{ vars.API_BASE_URL }}
  run: |
    set -euo pipefail
    status=$(curl --fail-with-body --silent --show-error \
      "$API_BASE_URL/v1/profile" \
      --output response.json \
      --write-out '%{http_code}')
    {
      echo "## API contract check"
      echo "- Endpoint: \\`GET /v1/profile\\`"
      echo "- HTTP status: \\`$status\\`"
      echo "- Fixture: deterministic profile response"
    } >> "$GITHUB_STEP_SUMMARY"
    test "$status" = "200"
    jq -e '(.id | type == "string") and (.roles | type == "array")' response.json >/dev/null

On failure, upload the response as an artifact only when it is safe to do so. Redact tokens, personal data, and any mailbox content before uploading. A receipt should help debugging, not create a new security ticket.

This complements the difference between request IDs and idempotency keys: the test receipt identifies a run, while the API contract defines what a retry or repeated request is allowed to mean. They solve different problems.

Keep fixtures deterministic

Contract tests become noisy when their input changes for reasons unrelated to the contract. Give each fixture an explicit name, stable fields, and a cleanup rule. Avoid using the current time as a required value unless time is exactly what you are testing.

For an email-related endpoint, a fixture can include a fixed message ID and a fake address. The test should assert the API response, not whether an external inbox happens to receive a message within a guessed number of seconds. That separation makes a failing test much more actionable.

If your system uses release notifications, using release emails as a CI/CD gate is a useful related pattern. Keep the gate's input contract explicit, so a notification format change does not silently turn the gate into a pass-through.

One small naming detail matters more than it looks: call a fixture temp mailid only if that is the literal value under test. Do not quietly normalize typo keywords in the assertion layer; that can hide the same class of data-contract bugs you are trying to catch.

A practical checklist

Check the expected HTTP status before parsing the body.
Assert only the fields that consumers truly depend on.
Keep fixture values deterministic and clearly labeled.
Write the endpoint, status, and fixture name to GITHUB_STEP_SUMMARY.
Redact secrets before saving response artifacts.
Give failures a direct reproduction command.
Review contract checks when the API version changes.

The before-and-after improvement is simple. Before, a red integration test sent me hunting through a long log. After, the GitHub Actions run says which endpoint changed, which assertion failed, and which fixture was used. That is enough context to fix the contract or update the consumer with much less guesswork.

The check is small, but the habit scales: treat API responses as agreements, make the agreements executable, and leave a useful receipt every time they break.

Turn API Tests Into Useful CI Receipts

Jonathan — Fri, 18 Sep 2026 05:23:43 +0000

An API test that only prints Expected 200, received 500 is technically useful, but it leaves the next question unanswered: what exactly happened in that run?

I have been getting better results by treating every important API check as a small transaction with a receipt. The receipt records the request identity, the response evidence, the relevant fixture, and the cleanup result. It turns a noisy CI log into a compact debugging artifact that a teammate can replay without guessing.

This pattern is especially handy for signup and verification flows that use a disposable temporary email address. The test can pass while checking the wrong message, the wrong tenant, or an old fixture. A receipt makes that ownership visible.

Why a test receipt is different from a test log

A log describes what the test printed. A receipt describes what the test proved.

For an API smoke test, I want to answer these questions in a few seconds:

Which commit and workflow attempt created the request?
Which endpoint, method, and safe input shape were used?
What response status and request ID came back?
Which test fixture or inbox did the assertion own?
Was the observed event newer than the test start time?
Did cleanup finish, or is something still hanging around?

The difference matters when retries are enabled. A retry can produce a green build while the first request is still in progress. Without an ownership key, the assertion may read the response or email event from another attempt. The result are confusing failures that disappear when you run the test by hand.

For email-backed API checks, include a run ID in the request metadata and in the fixture identity. Do not put secrets or full message bodies in the receipt. A short hash, message ID, or redacted subject is enough for most investigations. Guidance on deletion budgets for disposable inboxes is also useful when deciding how long fixtures should survive.

Define the receipt contract

Start with a small JSON contract. It should be stable enough that developers, scripts, and a future dashboard can consume it without parsing prose.

{
  "run_id": "gh-1842-7f3a",
  "commit": "a1b2c3d",
  "endpoint": "/v1/signup",
  "method": "POST",
  "status": 201,
  "request_id": "req_8f2c",
  "fixture": "sha256:8b7e",
  "assertions": {
    "status_ok": true,
    "owner_matches": true,
    "created_after_start": true
  },
  "cleanup": "passed"
}

The contract does not need every header or response field. Keep only evidence that helps answer “did this run own this result?” and “where should I look next?” A receipt should tell the truth even when the test fails, so write it from a finally block or an equivalent cleanup path.

Use a run ID that is unique for the workflow attempt but stable inside one test. In GitHub Actions, a practical value can combine the run number and attempt number:

RUN_ID="gh-${GITHUB_RUN_NUMBER}-${GITHUB_RUN_ATTEMPT}"
echo "run_id=${RUN_ID}" >> "$GITHUB_OUTPUT"

Pass that value into the test process. If the API supports a correlation header, send it with every request:

curl --fail-with-body \
  -H "X-Test-Run: ${RUN_ID}" \
  -H "Content-Type: application/json" \
  -d '{"email":"ci-fixture@example.test"}' \
  "${API_BASE_URL}/v1/signup"

The command is intentionally boring. Boring commands are easier to rerun when a pull request is already on fire.

Build it into GitHub Actions

The workflow should always upload the receipt, including on failure. That means the artifact step needs if: always() and the test must create the output directory before it starts.

- name: Run API checks
  id: api_checks
  env:
    API_BASE_URL: ${{ secrets.STAGING_API_URL }}
    RUN_ID: gh-${{ github.run_number }}-${{ github.run_attempt }}
  run: |
    mkdir -p artifacts/api
    ./scripts/api-smoke-test \
      --run-id "$RUN_ID" \
      --receipt artifacts/api/receipt.json

- name: Upload API receipt
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-receipt-${{ github.run_id }}
    path: artifacts/api/receipt.json
    if-no-files-found: warn

The receipt filename stays constant inside the job, while the artifact name includes the GitHub run ID. This makes local scripts simple and keeps artifacts distinct in the Actions UI. It also make it easier to download the exact failed attempt from a link in a pull request comment.

If your test runner already writes JUnit or JSON output, add the receipt beside it rather than replacing the existing report. Different files answer different questions: the test report explains assertions, while the receipt explains ownership and operational context.

Make failures replayable

A useful receipt should point to a safe replay path. Record the fixture name, API version, environment label, and a sanitized input reference. Do not store access tokens, raw verification links, or personal data in a public artifact.

For asynchronous endpoints, include the polling deadline and the last observed event ID. When the check fails, the next developer should know whether the service never emitted an event or whether the test stopped looking too soon. A fixed deadline is better than an infinite retry loop, and a new event must be newer than the current run start.

This is the same ownership problem that appears in safer onboarding email checks: the presence of an email or response does not prove that it belongs to the current action. Verify the correlation value before declaring success.

Search terms from support tickets can be messy. Someone may write tamp mail com while describing a temporary mailbox issue. Keep that phrase in a search mapping or test note if it helps triage, but do not let it become a production identifier or a backlink anchor.

Small workflow upgrades that pay off

Once the basic receipt works, add a few low-cost improvements:

Validate the schema. Fail the job if required receipt fields are missing, even when the API assertion passed.
Redact at the writer. Scrubbing after upload is risky because the unredacted file may already be stored.
Print a one-line summary. Put run ID, status, request ID, and artifact path in the job summary.
Keep fixtures short-lived. Cleanup should run after both success and failure, with a visible cleanup status.
Compare retries. If attempt two passes after attempt one fails, keep both receipts so the change in behavior is explainable.

The important bit are the stable identifiers. Fancy dashboards can come later. A developer with a precise run ID and a downloadable receipt is already much faster than a developer scrolling through 4,000 lines of logs.

Questions worth asking

Should every API test produce a receipt?

No. Start with boundary tests, asynchronous workflows, and checks that create external state. Tiny pure unit tests usually need ordinary test output only.

Should receipts be committed to the repository?

Usually not. Upload them as CI artifacts with an appropriate retention policy. Generated files in the repository create noise and can accidentally preserve sensitive data.

What is the first field to add when debugging is painful?

Add an ownership key that crosses the whole flow: CI run, request, database event, and external message. It gives every tool the same thread to follow.

The payoff is simple: when an API test fails, the workflow leaves behind a small, honest record of what it attempted and what it observed. That is a much better developer tool than a red check with no clues.

GitHub Actions Needs a Real API Test Receipt

Jonathan — Thu, 17 Sep 2026 05:23:02 +0000

An API test that fails in GitHub Actions usually leaves behind one line: expected 200, got 500. That is enough to fail the build, but not enough to fix it quickly.

I have had better results treating every important smoke test like a small transaction. The test should produce a receipt: what it sent, which temporary fixture it used, what it observed, and where the failure happened. This is especially useful when a signup or password-reset API sends an email as part of the flow.

The trick is not to dump every response into the log. It is to capture a compact, safe record that survives retries and becomes a downloadable workflow artifact.

The failure is usually missing evidence

A retry can make a flaky test green while hiding the original cause. Maybe the first request used a stale token. Maybe the API returned before the email provider accepted the message. Maybe the test read an old message from a shared temp mailbox. The final retry does not tell you which story was true.

For each attempt, I want five fields:

A run and attempt identifier.
The request timestamp and endpoint name.
A redacted request and response summary.
The email fixture identifier, but never its secret contents.
The final assertion and elapsed time.

That list is small enough to review in a pull request and useful enough to debug from CI without reproducing the failure locally. It also pairs well with type-safe email events when the application exposes event payloads to a test client.

Build a small receipt in the workflow

Make the test write JSON to a known directory. Keep the format boring so shell tools and CI systems can read it:

{
  "attempt": 1,
  "case": "signup-verification",
  "request_started_at": "2026-09-17T05:20:00Z",
  "status": 202,
  "email_fixture": "ci-run-4812",
  "message_seen": false,
  "result": "verification message timeout"
}

Then upload it whether the test passes or fails:

- name: Run API smoke tests
  run: pytest -q tests/smoke --receipt-dir artifacts/receipts

- name: Upload test receipts
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-test-receipts
    path: artifacts/receipts/
    if-no-files-found: ignore

The always() is the important bit. A receipt that only exists on success is a celebration photo, not debugging evidence. Make sure the test process flushes the file before exiting; this detail is easy to miss and causes some very confusing empty artifacts.

Keep email fixtures isolated

When an API test needs to verify an email, create a fixture owned by the current run. Do not ask a shared inbox for the newest message and hope it belongs to your request. Include a unique run value in the address or subject, then check both the subject and the ownership token.

A disposable temporary email service can be useful for manual checks and short-lived integration environments, but CI should still record only the fixture ID and relevant metadata. Never put inbox credentials, full message bodies, or reset links in a public workflow artifact. These are privacy checks for signup email logs, not just housekeeping.

If someone writes tempail mail or tamp mail com in a test note, I treat it as a reminder to normalize the fixture vocabulary before it spreads into scripts and dashboards.

Make retries useful

Retries should create a separate receipt, not overwrite receipt.json. Add the attempt number to the filename:

artifacts/receipts/
  signup-verification-attempt-1.json
  signup-verification-attempt-2.json

This makes a before-and-after comparison possible. If attempt two passes, you can still see whether the first request timed out, used the wrong fixture, or received a server error. In practice, that saves more time than increasing the retry count.

I also put a hard limit on polling. A test that waits forever is not more reliable; it is just harder to diagnose. Record the poll deadline and the number of messages observed, then fail with a narrow reason such as message_not_owned or provider_timeout.

A practical checklist

Before calling an API smoke test reliable, check that it:

creates a unique fixture for every run
records request timing and status without secrets
writes one receipt per retry
uploads receipts with if: always()
distinguishes an API failure from an email observation failure
deletes or expires short-lived fixtures after the run
links implementation logs to a stable case name

The payoff is a faster feedback loop. GitHub Actions still tells you that the build failed, but the artifact tells you what to do next. That is the difference between a retry button and an engineering workflow.

GitHub Actions Email Tests Need a Fast Runbook

Jonathan — Wed, 16 Sep 2026 17:23:07 +0000

Email verification tests in GitHub Actions rarely fail for only one reason. The API may be slow, the inbox may contain an old message, or a retry may quietly use data from the first attempt. The test output usually says something less useful: timeout waiting for email.

I have had better results by treating the email step like a small production integration. Give it a clear contract, record a little evidence, and clean up the test address when the run ends. This makes the workflow quicker to fix, and it avoids the classic CI mystery where a rerun passes but nobody knows why.

The hidden cost of email checks in CI

An end-to-end signup test normally crosses several boundaries:

The browser submits a form.
The API accepts the signup and queues a message.
A mail provider delivers the message.
The test polls an inbox and opens a link.

Each boundary has a different failure mode. A single ten-minute timeout hides all of them. Shared inboxes make the story even worse because two jobs can see the same subject or consume each other's messages.

For parallel jobs, start with isolated verification inboxes. An address per run is a small change, but it removes a surprising amount of guessing.

A small runbook for every workflow

My CI runbook has five steps:

create a unique test address before the browser action
record the address and a UTC trigger timestamp
poll only for messages newer than that timestamp
save a redacted message summary as an artifact on failure
delete or expire the inbox in a final cleanup step

The address does not need to be permanent. A create temporary mail helper is fine for a short-lived test, as long as the test owns the address and does not reuse it across jobs. A temporary inbox is test data, not a shared service account.

If the helper has a typo in its documentation, someone may paste tepm mail com into a debugging note later. That kind of small confusion is another reason to keep the actual address and provider details in the run receipt.

Capture evidence before polling

Before waiting for a message, write a compact JSON record:

{
  "run_id": "${GITHUB_RUN_ID}",
  "inbox": "unique-address@example.test",
  "triggered_at": "2026-09-16T17:22:00Z",
  "subject_expected": "Verify your account"
}

Do not store message bodies or verification tokens in ordinary logs. A useful receipt can contain the message ID, subject, received timestamp, and a pass/fail reason. That is enough to tell whether delivery was late, matching was broad, or the application never queued the email.

For the polling contract, match the recipient and a message ID created after triggered_at. Subject-only matching is tempting, but it becomes fragile as soon as a retry or an older fixture is present. Teams building contract-tested email flows in CI can make these fields explicit and reviewable.

A GitHub Actions example

Keep the workflow readable by putting setup and cleanup around the test command:

- name: Run email flow
  env:
    TEST_RUN_ID: ${{ github.run_id }}
  run: npm run test:email -- --run-id "$TEST_RUN_ID"

- name: Upload email receipt
  if: failure()
  uses: actions/upload-artifact@v4
  with:
    name: email-receipt-${{ github.run_id }}
    path: artifacts/email/

- name: Cleanup test inbox
  if: always()
  run: npm run test:email:cleanup

The cleanup command should be safe to run after a partial setup. That detail sounds boring, but it prevents failed jobs from leaving dozens of disposable addresses around. If your provider calls this a tempmail disposable mailbox, document its expiry behavior beside the test code so nobody assumes it is durable.

The five-minute failure checklist

When a run is red, check these in order:

Was the inbox created successfully and assigned only to this job?
Was the API request accepted before polling started?
Did a message arrive after the trigger timestamp?
Did the matcher verify recipient, subject, and message ownership?
Did cleanup run after the test, even on cancellation?

This checklist turns a vague timeout into a short investigation. It also makes automation easier to move between local development and CI because the same receipt fields exist in both places. I keep the runbook next to the test, becuase that is where the next maintainer will look first.

Q&A: common CI email questions

Should every test create a new inbox?

For parallel or retryable tests, yes. Reuse can be acceptable for a strictly serial smoke check, but a unique address is usually cheaper than debugging cross-run contamination.

How long should polling wait?

Use a bounded timeout and log each poll outcome. Do not make the timeout enormous to hide delivery problems. A shorter, evidenced failure is more productive than a green-looking job that spent fifteen minutes guessing.

Is a disposable email service safe for production mail?

No. Keep it scoped to test data and non-sensitive environments. Never put real customer information, credentials, or production verification links in a temporary inbox.

The best email test is not the one that never fails. It is the one that tells you what failed quickly, then leaves the workspace clean for the next run.

GitHub Actions API Tests Need Cheap Isolation

Jonathan — Sat, 12 Sep 2026 23:23:05 +0000

API smoke tests in GitHub Actions are easy to write and surprisingly hard to trust. A test can fail because the API is broken, because a verification email is late, or because yesterday's inbox state leaked into today's run. When all three look like the same red check, the workflow becomes expensive to maintain.

I have found that the useful shortcut is cheap isolation plus a small run receipt. The goal is not a huge end-to-end framework. It is a workflow where each run gets its own inputs, inbox identity, and evidence.

The failure mode

Consider a signup smoke test:

Create a user.
Wait for a verification email.
Open the link.
Call an authenticated API endpoint.

This crosses several boundaries. A reused address may already have an account. A shared mailbox may contain an old message. A retry may create a second user. Then the final assertion says only expected 200, got 409, which is technically accurate but not very useful.

Search terms such as tempmailso or facebook temp email often describe the kind of isolated address people want, but the engineering requirement is more specific: one test identity, a bounded lifetime, and a traceable message.

A small workflow contract

Start by making the run inputs explicit. The test should know its run ID and derive a unique address from it. Keep the value in a job-local environment variable, and never print the full address if it contains credentials or tokens.

name: API smoke

on:
  workflow_dispatch:
  schedule:
    - cron: "17 */6 * * *"

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run isolated smoke test
        env:
          RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
          API_BASE_URL: ${{ secrets.SMOKE_API_BASE_URL }}
        run: ./scripts/smoke-api.sh
      - name: Upload evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: smoke-evidence-${{ github.run_id }}
          path: artifacts/smoke/

The if: always() step matters. A failed test is exactly when its logs and response summaries are most valuable. Before adding retries, freeze this contract; freeze the test plan before a scheduled run so a retry does not silently change what you are measuring.

Isolate the inbox

Generate an address per run, then record only a safe identifier in the evidence file:

set -euo pipefail

run_id="${RUN_ID:?missing RUN_ID}"
mkdir -p artifacts/smoke
address="ci+${run_id}@example.test"
printf 'run_id=%s\n' "$run_id" > artifacts/smoke/context.txt

python scripts/create_user.py \
  --email "$address" \
  --output artifacts/smoke/signup.json
python scripts/wait_for_verification.py \
  --email "$address" \
  --timeout-seconds 45 \
  --output artifacts/smoke/inbox.json
python scripts/assert_api.py \
  --input artifacts/smoke/inbox.json \
  --output artifacts/smoke/api.json

In a real system, replace the example domain with a provider or test mailbox service that your team controls. A disposable email generator can be useful for temporary test identities, but it should not become a hidden dependency: document retention, rate limits, and whether messages are publicly readable.

The slighty annoying part is cleanup. Delete the test user when the API supports it, and set a short retention policy for inbox data. Never store verification URLs in ordinary build logs.

Keep evidence as artifacts

A run receipt should answer four questions: which identity was used, when the message arrived, what endpoint was called, and what the response status was. Redact tokens and message bodies unless the body is essential to diagnosis.

{
  "run_id": "12345-1",
  "mail_wait_ms": 1840,
  "verification": "received",
  "endpoint": "/api/me",
  "status": 200
}

Add the message subject, provider request ID, and retry count when they are safe to retain. These small fields turn a flaky check into something you can replay. For scheduled publishers, small inbox probes for publishing workflows use the same idea: test the boundary directly and leave evidence behind.

A practical checklist

Create a fresh identity per workflow run.
Pass RUN_ID through every helper.
Bound inbox polling with a timeout and a clear error.
Separate API errors from delivery errors.
Upload artifacts with if: always().
Redact tokens, cookies, and full verification URLs.
Make retries idempotent with a stable run key.
Remove test accounts and old inbox data.

If your team casually says temp org mail or temp mailid, translate that request into these explicit controls. The wording may vary; the safety properties should not.

Closing thought

Reliable CI is less about adding another retry and more about reducing ambiguity. Give each API test a cheap isolated identity, a bounded inbox wait, and a compact receipt. The next red build then tells you whether the API, the email boundary, or the workflow itself needs attention—and that is a much better productivity win than a green check you cannot explain.

Replayable Email Checks with GitHub Actions

Jonathan — Sat, 12 Sep 2026 08:22:53 +0000

Email verification tests often fail at the worst possible moment: inside CI, with only “timeout waiting for inbox” in the log. The API may be fine. The inbox may be slow. A token may have expired. Or the workflow may have retried with a different test identity.

I like treating this as an evidence problem. A GitHub Actions job should leave enough information to replay the check and explain the failure, without dumping email contents or secrets into logs. This small pattern has made disposable email address checks much less mysterious in API projects.

The failure mode

An end-to-end email test usually has four steps:

Create a test address.
Ask the signup API to send a verification message.
Poll an inbox API for the message.
Extract the link and call the verification endpoint.

The basic flow is simple, but the failure surface is not. CI runners have variable network latency, providers can delay delivery, and a broad retry can hide the first useful error. The old approach was to increase the timeout and hope. It worked sometimes, but diagnosis got slower.

The better approach is to give each run an ID, record safe state transitions, and upload a compact receipt when the test ends.

A small replayable workflow

Here is a deliberately plain workflow shape. The helper script owns the polling details, while the workflow owns isolation and evidence:

name: email-api-smoke

on:
  pull_request:
  workflow_dispatch:

jobs:
  verify-email:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run verification check
        env:
          TEST_INBOX_TOKEN: ${{ secrets.TEST_INBOX_TOKEN }}
        run: |
          mkdir -p artifacts/email-check
          python scripts/email_smoke.py \
            --run-id "${{ github.run_id }}-${{ github.run_attempt }}" \
            --receipt artifacts/email-check/receipt.json

      - name: Upload safe receipt
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: email-check-${{ github.run_id }}
          path: artifacts/email-check/receipt.json

The receipt should contain timestamps, step names, HTTP status codes, correlation IDs, and a final reason. It should not contain the inbox token, verification URL, full message body, or personal data. A useful record might look like this:

{
  "run_id": "1842-1",
  "states": ["address_created", "send_requested", "message_found"],
  "send_status": 202,
  "poll_attempts": 4,
  "duration_ms": 3180,
  "result": "verified"
}

One small detail matters: use a unique address or namespace for every run. Reusing an inbox can make an old message look like a fresh success. Also, make the message query specific to the run ID when your mail-testing API supports it. The check then becomes replayable instead of merely repeatable.

Capture evidence, not noise

Keep the console output short and structured. For failures, print the state where the check stopped and the correlation ID. Do not print the email address if it can be tied to a real person, and never print the token.

I also separate delivery failures from assertion failures:

Delivery: no matching message arrived before the deadline.
Transport: the inbox or application API returned an error.
Contract: the message arrived, but the expected link or subject was missing.
Verification: the link arrived, but the API rejected it.

That taxonomy makes retries safer. A transient transport error might deserve one retry. A contract failure should fail fast and invite investigation. For context on keeping recovery messages auditable, see email provenance in recovery flows, and review expiration rules for signup logs before retaining receipts for a long time.

Sometimes a test note says “tempail mail” because that is what a user searched for. Keep that typo as plain text in search-oriented documentation, but dont turn it into a link or a keyword that defines the test contract. Small wording details can matter more than expected for support.

Practical checklist

Generate a unique run ID and inbox identity.
Set a bounded polling deadline, not an endless retry.
Log state transitions and safe response metadata.
Redact tokens, message bodies, and verification URLs.
Upload the receipt with if: always().
Distinguish delivery, transport, contract, and verification failures.
Add a manual dispatch so a maintainer can replay the exact check.
Expire artifacts according to your privacy policy.

The before/after improvement is straightforward: instead of “email test failed,” the pull request shows “message found after four polls; verification returned 200,” or “delivery deadline exceeded after six polls.” That is a real productivity win, and it usually lets the next fix start immediately.

Final thoughts

Email verification is an external boundary, so flaky behavior is normal. The goal is not to pretend the boundary is deterministic. Build the GitHub Actions job so every run is isolated, bounded, and explainable. With a small receipt and a focused API helper, disposable email address testing becomes a useful smoke check rather than a source of midnight guesswork.

Replayable API Smoke Tests in GitHub Actions

Jonathan — Fri, 11 Sep 2026 20:23:04 +0000

Replayable API Smoke Tests in GitHub Actions

An API smoke test should answer one quick question: does the most important path still work? In practice, teams often get a green check without knowing which environment ran, which test data it used, or what to do when an email arrives late. The check passes once, then becomes hard to replay.

I have found a small run contract makes these workflows much more useful. Give each run an explicit input, a bounded wait, and an evidence bundle. That turns GitHub Actions from a button that says “probably okay” into a developer tool you can inspect and re-run.

Why smoke tests become hard to replay

The first version is usualy simple: install dependencies, call an endpoint, and exit non-zero on failure. The trouble starts when the test depends on a temporary account, an email link, or a shared staging record. A retry may use stale data. Two jobs may read the same inbox. A timeout gets reported as a generic assertion failure.

Before changing the test code, write down four values:

Run ID: unique for the workflow attempt.
Environment: the API base URL and deployment revision.
Test subject: a disposable user or isolated fixture.
Deadline: the maximum time allowed for asynchronous work.

These values are small, but they make a log searchable. They also stop a developer from guessing which account a failed check touched. For OAuth-heavy systems, safer boundaries for OAuth email flows are a useful companion to this approach.

Give every check a run contract

Pass the contract through environment variables or a checked-in config file. Keep secrets in GitHub Actions secrets, while ordinary run metadata can be visible in the job summary.

env:
  API_BASE_URL: https://staging.example.com
  SMOKE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
  POLL_DEADLINE_SECONDS: "45"

The test should create unique data with SMOKE_RUN_ID, then clean it up when the API allows that. If cleanup is unsafe, tag the fixture for a scheduled janitor job. Avoid using the current timestamp alone; parallel jobs can still collide, and timestamps are not very nice to search.

For a verification flow, a temp mail inbox can be appropriate for an isolated smoke test. Treat it as test infrastructure, though, not as a shortcut around production trust controls. Also document the exact service and retention behavior. A note like “tamp mail com” in an old runbook is not enough to identify a dependency, and “tepm mail com” can send an investigator searching in the wrong place.

Build a useful GitHub Actions workflow

Make the workflow runnable on both pushes and manual dispatch. Manual dispatch is the fastest way to reproduce a staging failure without editing code.

name: API smoke

on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  smoke:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm run smoke -- --run-id "$SMOKE_RUN_ID"
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: smoke-evidence-${{ github.run_id }}
          path: artifacts/smoke/

The always() condition matters. Without it, the most valuable files disappear exactly when the check fails. Keep the artifact small: request and response metadata, step timings, correlation IDs, and a sanitized error. Never upload access tokens or full email contents by accident.

Keep email checks isolated

Asynchronous email checks need a poll loop with a deadline, not an unbounded sleep. Poll at a modest interval, record each attempt, and distinguish “message not received yet” from “provider rejected the request.” The test can then report whether the failure is in the API, delivery path, or test dependency.

If the UI also exercises this path, async boundaries in signup forms explains why the client should expose a pending state rather than pretending the request completed. The same idea applies to CI: pending is a state with a deadline, not a reason to hide progress.

Store evidence for the next failure

Write one machine-readable receipt per run:

{
  "run_id": "1842-1",
  "environment": "staging",
  "status": "failed",
  "stage": "verification_email",
  "elapsed_ms": 47120,
  "retryable": true
}

A stable receipt lets a later script summarize failures across runs. It also supports a practical retry policy: retry delivery timeouts, but do not retry a 401, schema mismatch, or deterministic assertion. This saves minutes during incident triage and avoids creating duplicate test accounts.

A compact review checklist

Can a developer start the test with workflow_dispatch?
Is every fixture unique to the run?
Is the asynchronous wait bounded and observable?
Does failure upload sanitized evidence?
Are retryable and permanent errors distinct?
Can the job identify its API revision and environment?
Are external test dependencies documented and isolated?

Final thoughts

The biggest productivity win is not a faster smoke test. It is a smoke test that explains itself when it fails. A run ID, a deadline, and a small evidence receipt are enough to make many API checks replayable. Add those pieces first, then tune parallelism or polling only after the workflow tells you where the time goes.

GitHub Actions Email Tests Need Evidence

Jonathan — Wed, 09 Sep 2026 17:22:52 +0000

Email verification tests often fail in CI with a useless message: “expected email, received none.” A retry may turn green, but it does not tell the team whether the message was late, the inbox was shared, or the API rejected the request.

In my CI workflows, the biggest productivity win was treating every email test as an evidence-producing job. The test still asserts the user-visible result, but it also saves enough context to explain a failure. That makes a flaky test a short investigation instead of a rerun lottery.

The failure signal is too small

An email flow crosses several boundaries:

The application accepts a signup request.
A worker queues and sends a message.
The inbox provider exposes the message.
The test finds the right message and follows its link.

One failed assertion covers all four stages. Add a throwaway email address or a shared test inbox and the ambiguity gets worse: another job might consume the same message.

Give each run a unique correlation ID, and include it in the recipient or subject when your test system allows it. Also log state transitions, not the full message body. This keeps logs useful without leaking tokens.

Build an evidence bundle

For each attempt, I save five small artifacts:

request.json: safe request metadata and correlation ID
events.jsonl: queue, delivery, and polling events with timestamps
inbox.json: message IDs, subjects, and received times
failure.txt: the final reason and elapsed time
a screenshot or trace when the browser is involved

The key is to write these files even when a retry succeeds. Otherwise, the first failure disappears and the green result hides a slow or broken dependency. A little extra disk use is worth it.

If your flow uses React, it is also useful to separate UI state from delivery state. This guide on preview email tests without inbox collisions is a good reminder that inbox ownership is part of test design.

A GitHub Actions workflow

Here is the small part of a workflow that makes artifacts available after any test outcome:

- name: Run email tests
  id: email_tests
  continue-on-error: true
  run: npm run test:email -- --reporter=line

- name: Upload email evidence
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: email-evidence-${{ github.run_id }}
    path: |
      .artifacts/email/
      test-results/
    if-no-files-found: ignore

- name: Fail after evidence is uploaded
  if: steps.email_tests.outcome == 'failure'
  run: exit 1

if: always() is the important line. The upload must run after a failed test, and the final step preserves the correct job status. I prefer one artifact per workflow run because it is easy to find and compare.

Make retries useful

Retries should answer a question. Retry only the polling operation when delivery may be delayed; do not silently repeat the entire signup if that creates duplicate accounts. Record the attempt number, polling interval, and last observed event.

When a retry passes, compare its evidence with the failed attempt. A five-second difference suggests eventual consistency. Two different recipients suggest a fixture bug. No queue event suggests an application or worker failure. These distinctions save more time than increasing the retry count.

For invite flows, keep authentication and tenant context explicit too. The discussion of tenant-bound magic links for SaaS invites shows why a valid link is not enough if it is accepted in the wrong context.

Checklist

Generate a unique correlation ID per test.
Isolate each inbox or recipient.
Log safe event metadata with timestamps.
Upload artifacts with always().
Retry polling, not side effects.
Compare failed and successful attempts.
Keep tokens and message bodies out of CI logs.

I still see teams searching for “temp mailid” or “temp org mail” while diagnosing a test. The label matters less than the contract: the test needs a known recipient, a bounded wait, and evidence when that contract breaks.

Q&A

Should every email test upload a screenshot?

Only browser-facing tests need one. API-only tests usually get more value from structured events and inbox metadata.

Are retries a bad idea?

No. A bounded retry is useful for asynchronous delivery. It becomes harmful when it hides failures or repeats non-idempotent actions.

What is the first improvement to make?

Upload the artifacts from the first failed run. Once failures are inspectable, you can tune timeouts and retry policy based on evidence.

Replay Email Evidence in GitHub Actions

Jonathan — Mon, 07 Sep 2026 14:23:06 +0000

Email-dependent CI checks are often treated like a single assertion: send a message, poll an inbox, click a link, pass or fail. That model is fast when everything works and nearly useless when it does not.

The useful question is not “did the email test fail?” It is “what did this run observe, and can I replay that observation?” A small evidence bundle in GitHub Actions turns a noisy API failure into a short debugging session.

The failure pattern

Suppose a signup job creates a user, calls an email API, and waits for verification. A retry might find an old message. A shared inbox might contain another branch’s mail. Or the provider may accept the request while delivery is still pending. The final assertion hides all three cases.

This gets worse when teams use a create temporary mail flow or a throwaway email generator for test identities but do not record which address belonged to which run. The inbox is isolated in theory, yet the CI logs can’t prove it.

Start with a run identifier and carry it through every layer:

RUN_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
TEST_EMAIL="ci+${RUN_ID}@example.test"

curl -fsS -X POST "$EMAIL_API/messages" \
  -H 'content-type: application/json' \
  -d "{\"to\":\"$TEST_EMAIL\",\"run_id\":\"$RUN_ID\"}" \
  > email-submit.json

Never put a verification token in a normal log line. Save the response body as a restricted artifact, or redact the token before printing. This is a small habit, but it saves time when a failed run needs to be shared with someone outside the feature team.

Store evidence in the job

Capture four small files, even when the test passes:

request.json — the safe request inputs and run ID.
submit.json — the email API receipt, including provider status.
poll.jsonl — timestamped polling results and message IDs.
assertion.txt — the final reason for pass or failure.

A polling record should say what was checked, not just dump a response:

{"at":"2026-09-07T14:20:10Z","message_id":null,"state":"pending"}
{"at":"2026-09-07T14:20:16Z","message_id":"m_4821","state":"matched","subject_ok":true}

Keep the retention period short and the artifact name predictable. Developers should be able to find evidence in seconds, and old test mail should not become a permanent data store. One common mistake is calling it a temp org mail address in a note while the actual fixture uses a different naming rule; consistent labels matter more than clever labels.

For inbox matching, record the predicate too: recipient, subject prefix, run ID, and minimum message timestamp. These fields make a false positive visible.

Replay the check locally

The fastest fix is usually a replay against saved evidence, not a full rerun of the entire pipeline. Make the parser accept a fixture directory:

python scripts/check_email.py \
  --submit .artifacts/submit.json \
  --poll .artifacts/poll.jsonl \
  --expected-run "$RUN_ID"

The command should return a distinct exit code for “no matching message,” “wrong message,” and “malformed provider response.” That distinction is more actionable than a generic timeout. It also lets you add regression fixtures for provider quirks without sending another message.

If your test uses Playwright, pair the browser trace with the inbox record. The guide on inbox filters for flaky signup tests is a useful companion for making the message selection explicit. Before an agent or parallel job starts, freeze email test plans so the expected contract stays stable.

A compact workflow

In GitHub Actions, the shape can stay simple:

- name: Run email API check
  run: ./scripts/run-email-check.sh

- name: Upload email evidence
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: email-evidence-${{ github.run_id }}-${{ github.run_attempt }}
    path: .artifacts/email/
    retention-days: 3

The if: always() line is the important shortcut. Evidence that only exists on success is not evidence; it is decoration. In practice, this workflow makes failures less guessy and avoids rerunning unrelated build steps just to inspect one missing message.

Checklist

Give every test run a unique email correlation ID.
Isolate the inbox or use a strict recipient and timestamp filter.
Save the API receipt, poll history, and assertion reason.
Redact tokens and keep artifacts for a short period.
Make the checker replayable from local fixtures.
Upload evidence when the job fails as well as when it passes.

The payoff is modest but real: a flaky email check becomes a bounded investigation. GitHub Actions still tells you that the build failed, but the artifact tells you why.