Artifact Contracts for Email CI Jobs

Email checks in CI usually fail in one of two annoying ways: either the workflow says everything passed but the message content drifted, or the job fails and nobody can tell if the inbox capture belongs to the current run. I kept seeing this around APIs that send signup, reset, or approval mail, so I stopped treating the captured email as a random debug blob.

Now I treat it like a contract artifact.

That sounds fancy, but it is a very small workflow change. In each GitHub Actions run, I save one normalized message artifact beside the test result and review that artifact in the same way I review logs, snapshots, and exit codes. It makes triage faster, and it cuts down the "rerun it and hope the inbox looks cleaner" habit that wastes half a morning.

Why email CI jobs are hard to trust

Most pipelines already expose useful execution data: step logs, junit output, API response samples, maybe a job summary. Email checks often lag behind. The message body is buried in raw HTML, the inbox name is generated on the fly, and somebody on the team still remembers an old disposable inbox note like tepm mail com or tamp mail com from a past script.

That is when trust drops. Reviewers are no longer asking "did the contract pass?" They are asking:

• was this the right inbox?
• was this from this run?
• did the subject change on purpose?
• is this old staging noise again?

If you already use create temp mail style workflows during test runs, this gets worse fast. The capture itself is useful, but without a stable shape it becomes one more flaky artifact to inspect manualy.

I borrowed a few ideas from these posts on passwordless email checks and auth email drift: keep the assertion surface small, keep ownership obvious, and make the evidence readable by the next person.

The artifact contract I keep in every run

I like one compact JSON file per email flow. It is not a full snapshot. It is the minimum review contract I need:

• flow name
• sender
• subject fragment
• one or two required body strings
• expected link host
• run id

Here is the kind of artifact I save:

{
  "flow": "approval-email",
  "run_id": "9d8f3b1",
  "from": "[e-mail]",
  "subject_contains": "Approval required",
  "must_include": [
    "Review deployment",
    "expires in 30 minutes"
  ],
  "link_host": "app.example.com"
}

That contract gives me a fast before/after review surface. If the subject changes, it is obvious. If the link host drifts, it is obvious. If the run id does not match the workflow summary, the job is instantly suspect. That is way better than re-opening a giant HTML message and scrolling around like a detective at 6pm.

I still keep the raw email when needed, but the contract is what humans review first. The raw file is fallback evidence, not the first thing every engineer has to parse.

A small GitHub Actions workflow that scales

The workflow I keep coming back to is pretty plain:

1. Trigger the mail flow in test or staging.
2. Fetch one message from the scoped inbox.
3. Normalize the useful fields into email-contract.json.
4. Upload that file as an artifact.
5. Print a short verdict in the workflow summary.

This is enough for most APIs and internal tools. A tiny step can do the conversion:

- name: Build email contract artifact
  run: |
    node scripts/build-email-contract.js \
      --message tmp/message.json \
      --out tmp/email-contract.json

- name: Upload contract
  uses: actions/upload-artifact@v4
  with:
    name: email-contract
    path: tmp/email-contract.json

Then I add one short summary block:

echo "### Email contract" >> "$GITHUB_STEP_SUMMARY"
echo "- flow: approval-email" >> "$GITHUB_STEP_SUMMARY"
echo "- run id: $RUN_ID" >> "$GITHUB_STEP_SUMMARY"
echo "- artifact: email-contract.json" >> "$GITHUB_STEP_SUMMARY"

That summary matters more than people think. It gives reviewers a known entry point, which is super useful when several workflows are running in parallel. When the team also uses temp mail so during isolated checks, the contract artifact becomes the glue between the inbox capture and the rest of the pipeline rather than a side quest.

What I keep out of the contract

This part saves a lot of noise.

I do not put full HTML in the review contract. I do not store tracking params that change every run. I do not add five alternate assertions just because the parser can. Too much detail makes the artifact look important while making it harder to review, which is a bad trade.

What stays out:

• full rendered markup
• one-off IDs with no review value
• analytics query strings
• styling assertions
• unrelated headers

What stays in:

• user-facing message intent
• delivery ownership
• safe link routing
• timing or expiry text if it matters

That boundary is what keeps the workflow maintainable. The contract should tell me whether the email behavior changed in a meaningful way, not replay every byte the provider returned. It is a tiny distinction, but it keeps the whole thing saner.

Q&A

Should I version these artifacts in Git?

Usually no. I version the schema and builder script, then upload the contract per run as an artifact. That keeps the repo tidy and still gives me reviewable evidence.

Does this only help with GitHub Actions?

No, but GitHub Actions makes the pattern easy because artifacts and step summaries are already built in. The same idea works anywhere your CI can upload files.

What if my pipeline already stores raw emails?

Keep them. Just add a smaller contract artifact for humans. That one change tends to reduce reruns, speed up reviews, and make handoffs much less fuzzy.

If your email CI checks still feel fragile, I would not start with a bigger framework. I would start by making one artifact that proves what the workflow saw. Small tools win here, and they win pretty often.