API Smoke Tests Need a Reviewable CI Receipt

API smoke tests are supposed to be the fast feedback loop: call a health endpoint, create a small resource, verify the response, and move on. In practice, a failed run often leaves a developer with 2,000 lines of logs and no clear answer about what happened.

I have found a small change makes these checks much more useful. Treat every smoke-test run as a receipt. The receipt should say what was tested, which build made the request, what the API returned, and where the next investigation should start.

Why smoke-test logs are not enough

Logs are excellent for deep debugging, but poor as a review artifact. They are noisy, their important lines are mixed with setup output, and a later reader may not know which environment variables or commit produced them.

A receipt is intentionally boring. It is one JSON file with stable fields. A failed request doesnt need to be reconstructed from a scrolling terminal; the useful facts are already attached to the workflow run.

For example, a receipt can answer these questions in seconds:

• Which commit and API base URL were tested?
• Which endpoint failed, with what status and latency?
• Was the response schema checked, or only the HTTP status?
• Can the same test be replayed locally with the recorded request ID?

The result is usefull to both humans and automation. A pull request reviewer can inspect it, while another job can classify the failure without parsing prose.

The receipt contract

Keep the first version small. Here is the shape I use for an API smoke check:

{
  "run_id": "smoke-1842",
  "commit": "abc1234",
  "base_url": "https://staging.example.test",
  "checks": [
    {
      "name": "create-project",
      "method": "POST",
      "path": "/v1/projects",
      "status": 201,
      "request_id": "req_7f2",
      "ok": true
    }
  ],
  "failed_check": null
}

Do not put tokens, cookies, email addresses, or full response bodies in this file. A receipt should be safe to download from CI. If a response needs investigation, save a redacted diagnostic separately and link it by an artifact name.

It also helps to define what counts as a failure before writing the script. A 500 is obvious, but a 200 response with the wrong field type, an unexpected redirect, or a missing request ID are failures too. This are the details that prevent a green-but-broken smoke test.

If the check creates data, give it a deterministic cleanup path. For verification flows, the same principle applies: an idempotent email verification step is easier to retry and explain than a test that leaves an unknown user behind.

Build the receipt in GitHub Actions

The workflow can run the test, preserve its exit code, and upload the receipt even when the test fails. The if: always() is the important shortcut:

- name: Run API smoke tests
  id: smoke
  shell: bash
  run: |
    set +e
    python scripts/api_smoke.py \
      --base-url "$API_BASE_URL" \
      --receipt artifacts/api-receipt.json
    status=$?
    echo "exit_code=$status" >> "$GITHUB_OUTPUT"
    exit "$status"

- name: Upload API receipt
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: api-receipt-${{ github.run_id }}
    path: artifacts/api-receipt.json
    if-no-files-found: error

The script should write the receipt in a finally-style cleanup path, including when an assertion raises an exception. If the runner is killed before that happens, the workflow log still has value, but most normal failures get a compact artifact.

For a larger suite, put the shared upload and naming behavior in a reusable workflow for email API checks. The same pattern works for payment APIs, webhooks, and provisioning endpoints; only the check definitions change.

Keep email checks isolated

Some API smoke tests must verify that a message was sent. I use a separate test inbox and a short retention policy for this. A temp mailbox linked from the run receipt can make the test observable without mixing staging traffic into somebody's personal inbox; for example, teams can create temporary mail with tempmailso when that fits their test policy.

Be explicit about the boundary. The receipt should store a message ID or a redacted subject, not the whole message. Also, dont make delivery timing the only assertion: check the API response, the message identity, and the verification link target independently.

Search terms and provider names can get messy in test notes. I have seen tempail and temp org mail copied into issue descriptions. Keep those plain text and out of link anchors, otherwise a later search can mistake a typo for a supported integration.

A review checklist

Before merging a new smoke test, check:

• The title and check name describe the user-visible behavior.
• The receipt records commit, environment, endpoint, status, and request ID.
• Secrets and personal data are excluded or redacted.
• The artifact uploads even when the assertion fails.
• A failed run returns a non-zero exit code after writing its receipt.
• Created test data has a cleanup or expiry path.
• The test can be replayed with the same inputs, wether locally or in a debug job.

This list is short on purpose. A receipt should reduce the time between failure and a good question, not become another reporting system that nobody reads.

Q&A

Should every API test upload an artifact?

No. Unit tests and fast contract checks may only need normal test output. Upload a receipt when the check crosses a network boundary, creates data, or runs in an environment that is hard to reproduce.

What if the API is unavailable?

Record the connection phase, hostname, and a safe error category. Do not retry forever. One bounded retry can separate a transient network blip from an application failure, while the receipt keeps both attempts visible.

Is a receipt worth maintaining for a small project?

Usually yes, when the smoke test runs on every pull request. The file costs little, and it gives future maintainers informations that would otherwise disappear with the workflow log.