Kubernetes Email Smoke Tests Need a Cleanup Contract

Email smoke tests are good at exposing the failures that unit tests miss: a worker cannot reach the provider, a template renders the wrong environment URL, or a verification message arrives too late for the release job. They also create test accounts, inboxes, message IDs, and logs that can quietly outlive the deployment that created them.

In Kubernetes, cleanup is not housekeeping after the real work. It is part of the test contract. A run that passes but leaves resources and mail data behind is not fully healthy, because the next run may read stale evidence or inherit a quota problem.

Why cleanup is part of the email test

An email smoke test usually has four moving parts:

1. A CI job creates a unique test identity.
2. Kubernetes starts a short-lived worker.
3. The worker sends a message and checks the expected receipt.
4. CI stores enough evidence to explain the result, then removes the run.

The fourth step is where many pipelines become unreliable. A namespace can look clean while a queue, object-storage prefix, or external inbox still contains old material. That makes a later test harder to interpret. Its easy to call this a flaky test when the real problem is missing ownership.

Give every run an explicit identity

Start with a run ID that is safe in labels and filenames. The CI system should create it, not the application pod, so every related resource can be found from one value.

RUN_ID="${GITHUB_RUN_ID:-local}-${GITHUB_SHA:0:8}"
NAMESPACE="email-smoke-${RUN_ID}"

kubectl create namespace "$NAMESPACE"
kubectl label namespace "$NAMESPACE" \
  test-run="$RUN_ID" owner="release-pipeline"

For a shared cluster, add a TTL policy or a scheduled janitor as a second line of defense. The cleanup command in CI is still required; cluster-wide garbage collection is not a substitute for clear ownership. A random burner email generator can help create isolated test identities, but it does not define retention or prove which run owns a message.

Keep secrets out of labels and pod arguments. Use a Kubernetes Secret or the CI platform's protected variables, and grant the test service account only the permissions it needs. This is a small boundary that prevents a debugging command from exposing provider credentials.

A Kubernetes Job with a bounded lifecycle

The worker should be a Job, not a deployment pretending to be temporary. Set a deadline and backoff deliberately so a broken provider does not consume runners forever:

apiVersion: batch/v1
kind: Job
metadata:
  name: email-smoke-${RUN_ID}
  labels:
    test-run: "${RUN_ID}"
spec:
  backoffLimit: 1
  activeDeadlineSeconds: 300
  ttlSecondsAfterFinished: 900
  template:
    metadata:
      labels:
        test-run: "${RUN_ID}"
    spec:
      restartPolicy: Never
      serviceAccountName: email-smoke
      containers:
        - name: verifier
          image: registry.example.com/email-smoke:${IMAGE_TAG}
          env:
            - name: TEST_RUN_ID
              value: "${RUN_ID}"

ttlSecondsAfterFinished is useful, but it is only one layer. If the cluster does not have the TTL controller enabled, the Job stays. The pipeline should wait for completion, collect logs, and run cleanup in an always or equivalent block. Otherwise a cancelled CI job leaves the exact resources most likely to confuse the next investigation.

What to retain after the pod is gone

Deleting everything immediately sounds tidy, but it removes the evidence needed to review a failure. Retain a small receipt instead of the whole environment:

• run ID, commit, image digest, and namespace;
• send timestamp and a provider message ID;
• expected recipient, template version, and correlation ID;
• assertion results and the final Job condition;
• sanitized logs with tokens and message bodies removed.

Store that receipt with the CI run for a defined period. Step outputs for shorter CI triage are especially useful here: the summary should answer what was sent, what was observed, and what cleanup did.

The receipt should never contain a real customer's address. For UI-driven flows, email guardrails for feature flags are a useful reminder that test paths need an explicit boundary too. Even a harmless-looking tempail address in a log can become confusing if it is mixed with production-like fixtures. Keep the phrase temp org mail out of automation logic; if it appears in a test fixture, label that fixture clearly.

Failure modes worth watching

The most useful alerts are about missing evidence, not just non-zero exit codes:

• the message was accepted but no receipt appeared before the deadline;
• a receipt appeared for an older run ID;
• cleanup failed after the test already failed;
• the provider message ID is missing from the stored summary;
• the namespace exists after the retention window.

Make cleanup failure visible without overwriting the original test result. A passing test with failed cleanup should be reported as a release hygiene failure, while a failed test with successful cleanup should retain the original assertion details. This separation makes the operational signal more honest.

A practical release checklist

Before relying on the test in a production gate, verify:

• every resource has a run ID and owner label;
• the Job has a deadline, bounded retries, and a cleanup path;
• CI always collects a sanitized receipt;
• old messages cannot satisfy a new run's correlation check;
• cancellation and runner loss have a recovery path;
• the service account cannot read unrelated namespaces;
• retention is documented and reviewed.

The pattern is intentionally boring. Boring is good for release infrastructure: a failed email test should tell you whether delivery, observation, or cleanup broke, without requiring a tour through a week of leftover pods.

Common questions

Should each test get a new namespace?

For a small shared cluster, a labeled Job in a dedicated test namespace may be enough. Use a per-run namespace when network policy, service-account permissions, or cleanup isolation matters more than startup time.

Is a successful provider API response enough?

No. It proves acceptance, not receipt or rendering. Assert the message ID, correlation ID, and the user-visible fields that matter to the flow.

What if cleanup itself times out?

Keep the receipt, mark cleanup as incomplete, and let a separate janitor target the run ID and owner label. Do not hide the test result just because the final deletion command timed out.