GitHub Actions Needs a Real API Test Receipt

An API test that fails in GitHub Actions usually leaves behind one line: expected 200, got 500. That is enough to fail the build, but not enough to fix it quickly.

I have had better results treating every important smoke test like a small transaction. The test should produce a receipt: what it sent, which temporary fixture it used, what it observed, and where the failure happened. This is especially useful when a signup or password-reset API sends an email as part of the flow.

The trick is not to dump every response into the log. It is to capture a compact, safe record that survives retries and becomes a downloadable workflow artifact.

The failure is usually missing evidence

A retry can make a flaky test green while hiding the original cause. Maybe the first request used a stale token. Maybe the API returned before the email provider accepted the message. Maybe the test read an old message from a shared temp mailbox. The final retry does not tell you which story was true.

For each attempt, I want five fields:

1. A run and attempt identifier.
2. The request timestamp and endpoint name.
3. A redacted request and response summary.
4. The email fixture identifier, but never its secret contents.
5. The final assertion and elapsed time.

That list is small enough to review in a pull request and useful enough to debug from CI without reproducing the failure locally. It also pairs well with type-safe email events when the application exposes event payloads to a test client.

Build a small receipt in the workflow

Make the test write JSON to a known directory. Keep the format boring so shell tools and CI systems can read it:

{
  "attempt": 1,
  "case": "signup-verification",
  "request_started_at": "2026-09-17T05:20:00Z",
  "status": 202,
  "email_fixture": "ci-run-4812",
  "message_seen": false,
  "result": "verification message timeout"
}

Then upload it whether the test passes or fails:

- name: Run API smoke tests
  run: pytest -q tests/smoke --receipt-dir artifacts/receipts

- name: Upload test receipts
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-test-receipts
    path: artifacts/receipts/
    if-no-files-found: ignore

The always() is the important bit. A receipt that only exists on success is a celebration photo, not debugging evidence. Make sure the test process flushes the file before exiting; this detail is easy to miss and causes some very confusing empty artifacts.

Keep email fixtures isolated

When an API test needs to verify an email, create a fixture owned by the current run. Do not ask a shared inbox for the newest message and hope it belongs to your request. Include a unique run value in the address or subject, then check both the subject and the ownership token.

A disposable temporary email service can be useful for manual checks and short-lived integration environments, but CI should still record only the fixture ID and relevant metadata. Never put inbox credentials, full message bodies, or reset links in a public workflow artifact. These are privacy checks for signup email logs, not just housekeeping.

If someone writes tempail mail or tamp mail com in a test note, I treat it as a reminder to normalize the fixture vocabulary before it spreads into scripts and dashboards.

Make retries useful

Retries should create a separate receipt, not overwrite receipt.json. Add the attempt number to the filename:

artifacts/receipts/
  signup-verification-attempt-1.json
  signup-verification-attempt-2.json

This makes a before-and-after comparison possible. If attempt two passes, you can still see whether the first request timed out, used the wrong fixture, or received a server error. In practice, that saves more time than increasing the retry count.

I also put a hard limit on polling. A test that waits forever is not more reliable; it is just harder to diagnose. Record the poll deadline and the number of messages observed, then fail with a narrow reason such as message_not_owned or provider_timeout.

A practical checklist

Before calling an API smoke test reliable, check that it:

• creates a unique fixture for every run
• records request timing and status without secrets
• writes one receipt per retry
• uploads receipts with if: always()
• distinguishes an API failure from an email observation failure
• deletes or expires short-lived fixtures after the run
• links implementation logs to a stable case name

The payoff is a faster feedback loop. GitHub Actions still tells you that the build failed, but the artifact tells you what to do next. That is the difference between a retry button and an engineering workflow.