API Smoke Tests Need Receipts

I like small smoke tests, but I do not like mystery failures. A lot of CI pipelines run one API call, get one red mark, and leave the next engineer to guess what changed. That is not really a smoke test anymore. It is a slot machine with logs.

The fix that helped me most was boring: every smoke test writes a receipt. Not a huge artifact dump, just a compact record of what was called, what came back, how long it took, and why the job decided pass or fail. In GitHub Actions, this turns noisy checks into something you can triage in a minute, not half an hour.

Why smoke tests fail twice

The first failure is the actual product issue. The second failure is the pipeline not explaining itself.

This happens a lot with APIs because the request path, auth state, and response shape can all drift independently. A job might tell you curl exited non-zero, but that still leaves a bunch of questions:

• did DNS fail or did auth fail?
• was the status code wrong or was the body wrong?
• was latency climbing for a few runs before it broke?

GitHub Actions gives us step summaries, annotations, and artifacts that are made for this kind of thing (https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions). If you use them together, the pipeline starts reading more like an incident receipt and less like a random terminal capture.

I have seen the same benefit in adjacent workflows like type-safe invite email checks, where the main win is not more tooling, but better evidence around the tooling.

The receipt pattern I add first

My default shape is three files:

1. request.json
2. response.json
3. receipt.json

The first two are raw enough for debugging. The third one is the human layer. It should be tiny, stable, and easy to diff between runs.

Here is the sort of receipt I mean:

{
  "check": "billing-health",
  "status": "warn",
  "http_status": 200,
  "schema_ok": true,
  "latency_ms": 1842,
  "reason": "response passed but latency crossed warning budget",
  "run_id": "smoke-4821"
}

That one object lets the workflow print a clean summary, raise a warning when needed, and archive the detailed files without making the main log unreadable. It sounds tiny because it is tiny, but it saves a suprising amount of time.

A GitHub Actions shape that stays readable

I prefer a single shell step that gathers evidence and then a second step that formats the verdict. Keeping those seperate makes reruns easier when a team wants to inspect the raw output.

- name: Run billing smoke test
  run: |
    mkdir -p artifacts
    echo "::group::Call health endpoint"
    curl -sS \
      -H "Authorization: Bearer $API_TOKEN" \
      -o artifacts/response.json \
      -w "%{http_code}" \
      https://api.example.com/health > artifacts/status.txt
    echo "::endgroup::"

    node scripts/write-receipt.mjs \
      --status-file artifacts/status.txt \
      --response-file artifacts/response.json \
      --out artifacts/receipt.json

- name: Summarize result
  run: |
    status="$(jq -r '.status' artifacts/receipt.json)"
    reason="$(jq -r '.reason' artifacts/receipt.json)"
    latency="$(jq -r '.latency_ms' artifacts/receipt.json)"

    if [ "$status" != "ok" ]; then
      echo "::warning::${reason}"
    fi

    {
      echo "## API smoke test"
      echo ""
      echo "- Status: \`$status\`"
      echo "- Latency: ${latency}ms"
      echo "- Reason: $reason"
      echo "- Files: response.json, receipt.json"
    } >> "$GITHUB_STEP_SUMMARY"

This is close to the same thinking behind keeping rollout alerts tied to one deploy: each run should tell one coherent story. If the evidence spans several retries or environments, the workflow needs to stitch that into one obvious verdict.

Where temporary inbox checks fit

A lot of product smoke tests are not just API calls. They are API calls plus side effects: invite mail, password reset mail, receipt mail, and trial onboarding mail. That is where teams often lose the clean story, because they validate the endpoint but not the outcome.

When I need to add mailbox validation, I keep it as one bounded sub-check and give it the same receipt treatment. For example, the workflow might record inbox polling attempts, final message latency, and the subject line that matched. If I need a free temp email flow for a disposable verification path, I treat that as supporting evidence, not the entire test strategy.

The useful bit is consistency. API step, inbox step, and final verdict should all share the same run id. If they do not, engineers end up grepping for weird strings like tempail mail and hoping they found the correct attempt. That is the sort of mess a receipt pattern avoids.

One more practical tip: keep warning budgets distinct from failure budgets. A 2-second response might still pass the smoke test today but deserve a warning, while a schema mismatch should fail imediately. Mixing those together makes dashboards look stable right untill they are not.

Q&A

Should every smoke test write JSON artifacts?

Not every single one, but any check that can fail for multiple reasons probably should. The more branches in the diagnosis tree, the more value you get from a compact receipt.

Do you put the full response body in the step summary?

No. The summary should stay human-sized. Put the verdict there, and keep the raw body in artifacts. Otherwise the summary becomes another wall of text.

What is the biggest payoff?

Faster reruns, faster triage, and fewer Slack threads asking what a red job even means. For developer tools and APIs work, that is a pretty high return for a very smal pattern.