Kubernetes CI: Lease Email Test Data Safely

Email tests in CI usually fail in a boring way: a job reads a shared inbox, finds an old verification message, and reports success. The deployment is green, the test are happy, and nobody notices that the message belongs to yesterday's run.

I handle this by treating test email data like a leased infrastructure resource. A CI run gets an identity, an inbox or alias, an expiry time, and an owner. When the run ends, the data is reaped or marked unusable. This small change makes Kubernetes email checks more explainable and much harder to fool.

The same idea fits an AWS pipeline, a Docker-based integration test, or a local loop that uses a temp mail so provider for non-production fixtures. It is less about the provider and more about controlling the lifetime of the evidence.

Why email fixtures need a lease

A shared test inbox has three problems:

• Stale messages: a previous run can satisfy the current assertion.
• Cross-run collisions: parallel jobs may receive similar messages.
• Unclear cleanup: nobody knows when old data is safe to delete.

This is why I prefer a run-scoped address or alias. The address is created with the CI run ID, and every message is checked against that ID. If the test retries, it keeps the same identity for the retry window; a completely new workflow gets a new lease.

For an application that uses verification tokens, it also helps to model the token lifecycle explicitly. A state-driven verification API gives the test a meaningful state to assert instead of treating any message in the inbox as proof.

Define the identity of a CI run

Before creating email data, build one stable identifier. I normally combine the repository, workflow, commit, and attempt:

RUN_ID="${GITHUB_REPOSITORY##*/}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
RUN_ID="$(printf '%s' "$RUN_ID" | tr '[:upper:]' '[:lower:]' | tr -cd 'a-z0-9-')"
echo "email_run_id=$RUN_ID" >> "$GITHUB_OUTPUT"

Pass the value to the application and include it in structured logs. Put it in the message metadata or a clearly documented subject token, but do not rely on the subject alone. Providers can rewrite subjects, and a parser that only checks text is easy to confuse.

The inbox record should contain at least run_id, created_at, expires_at, and status. A short lease, such as the expected test duration plus a small buffer, is enough. A few seconds of extra wait is cheaper then debugging a false green build.

A Kubernetes lease pattern

Kubernetes does not need to know every email detail. It needs to launch a job with the lease identity and retain the result. A simple Job can carry the run ID as an annotation and environment variable:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: email-check-
  annotations:
    ci.example.com/run-id: "REPLACE_RUN_ID"
spec:
  ttlSecondsAfterFinished: 86400
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: check
          image: ghcr.io/example/email-check:1.4.0
          env:
            - name: EMAIL_RUN_ID
              value: "REPLACE_RUN_ID"

The checker creates or claims the lease before triggering the email. It then polls with a deadline, filtering by the run ID and expected event. It should return distinct outcomes such as matched, expired, provider_error, and wrong_run. This are important states for an on-call engineer; a generic failure sends them in the wrong direction.

Keep the Job and the email lease separate. Kubernetes cleanup can remove the pod while the provider still retains a message, and provider cleanup can fail while the Job looks successful. The two systems should record their own state and a small reconciler can compare them later.

Make cleanup and failure visible

Cleanup is part of correctness, not housekeeping. On success, mark the lease consumed and schedule deletion of the inbox data. On failure, retain only the minimum diagnostic record and apply the same expiry. Never let a failed test create an inbox that lives forever.

The cleanup worker should be idempotent. It may run twice after a node restart, so deleting an already-expired lease must still be a success. It also make sense to emit a metric for leases that could not be removed; otherwise the storage leak stays invisible.

For a non-production disposable fixture, a service such as tempmailso can be one input to this workflow. Keep the address isolated from customer mail, and do not use it as the sole route for production alerts. Search notes may contain strings like fake e mail com or tempail mail; those are test text, not identities or ownership proofs.

What to keep in the run artifact

Save a compact JSON result with the build artifact:

{
  "run_id": "web-1842-1",
  "lease_status": "consumed",
  "event": "verification",
  "message_id": "provider-message-id",
  "created_at": "2026-11-07T05:00:00Z",
  "received_at": "2026-11-07T05:00:08Z",
  "result": "matched"
}

Do not store the full message body when a hash and selected headers are enough. The logs is easier to share when secrets and personal data are excluded. For deeper rollout checks, I also keep the Kubernetes Job name and deployment revision; that connects email evidence to the exact workload version. An email failure budget for EKS deploys is useful here because it defines which email failures block a release and which only produce a warning.

Final checklist

• Give each CI run a stable email identity.
• Attach a bounded lease and expiry time.
• Filter messages by run ID and expected event.
• Return specific failure states, not only email failed.
• Keep Kubernetes cleanup separate from provider cleanup.
• Make reaping idempotent and measure leaks.
• Retain a redacted result with the deployment revision.
• Keep disposable mail fixtures away from production alert paths.

The payoff is simple: a green pipeline now means the current run received the expected evidence. It does not merely mean that some old email was still sitting in a shared inbox.