Kubernetes Email Tests Need Artifact Retention

Email tests often fail in a Kubernetes pipeline for reasons that are hard to see after the pod is gone. The fixture was created, the verification message arrived, and then the namespace cleanup removed the only useful evidence. A fake email generator is still useful for this workflow, but the generated address and the resulting message need an artifact contract around them.

This pattern keeps test evidence traceable without turning every CI run into a permanent storage bill. It uses a short-lived namespace, a small failure receipt, and an object-storage lifecycle rule. The goal is not to save every email forever. The goal is to save enough context to answer: what was tested, which message arrived, and why did the assertion fail?

Why email test artifacts disappear too early

Kubernetes makes cleanup easy. A job finishes, its namespace is deleted, and the temporary mailbox may expire soon after. That is good for isolation, but rough for diagnosis. A log line saying expected verification email, got timeout does not tell an on-call engineer whether the application never sent the message, the worker was delayed, or the test read a message belonging to another run.

The first version of this pattern I shipped was to optimistic about logs. Container logs showed the polling loop, but not the message identity or the fixture that was used. When a parallel test failed, the evidence was basicly gone.

The fix is to define the test output before writing the test. Each run gets a stable run ID, and every artifact carries that ID:

ci-20260920-1722/
  manifest.json
  request.json
  received-message.json
  test.log
  trace.zip

The files do not need to contain full message bodies. Redact tokens and personal data. A subject, message ID, timestamp, recipient hash, and assertion result are usualy enough to locate the failure.

A retention pattern for Kubernetes CI

I use three storage layers for this kind of pipeline:

1. The pod filesystem for fast writes during the test.
2. A CI artifact upload for the normal pass/fail summary.
3. Object storage for failed runs that need a longer debugging window.

The test writes to a path derived from CI_RUN_ID, never from a user-provided email address. An upload sidecar or post-test step copies the files before the namespace is deleted. The cleanup step must run after collection, even when the test command exits non-zero.

For example, a simplified job command can look like this:

set -o pipefail
run_id="${CI_RUN_ID:-local-$(date +%s)}"
mkdir -p "artifacts/${run_id}"

pytest tests/email_flow.py \
  --junitxml="artifacts/${run_id}/junit.xml" \
  2>&1 | tee "artifacts/${run_id}/test.log"
test_status=${PIPESTATUS[0]}

./scripts/write-email-manifest.sh "${run_id}" "artifacts/${run_id}"
./scripts/upload-failed-artifacts.sh "${run_id}" "artifacts/${run_id}" || true
exit "${test_status}"

The || true belongs only on the evidence upload, not on the test command. Losing an upload should be visible as a warning, while changing a failed test into a green build is a much worse outcome.

The artifact contract

Keep the manifest small and boring. A useful example is:

{
  "run_id": "ci-20260920-1722",
  "namespace": "email-test-ci-20260920-1722",
  "fixture_provider": "fake-email-generator",
  "recipient_hash": "sha256:...",
  "expected_subject": "Verify your account",
  "observed_message_id": "msg-...",
  "created_at": "2026-09-20T17:22:10Z",
  "result": "failed",
  "failure_class": "wrong_message"
}

The namespace and run ID make it possible to correlate Kubernetes events with the CI job. The failure_class is more useful than a free-form sentence when you need to group failures later. Keep secrets out of this file; even a temporary test email can contain a live verification token.

If the test checks several messages, record the ownership decision for each one. This avoids the common mistake of treating the first message in a shared inbox as the right message. For browser-driven checks, clock boundaries in Playwright tests are also worth making explicit, because time windows and retention windows tend to fail together.

Implementation details that prevent noisy failures

Use labels as the cleanup interface

Label every namespace and job with ci.run_id, ci.repository, and ci.retention_class. Cleanup can then select resources without guessing from names. A missing label can make a clean up job delete nothing, or worse, select an unrelated namespace.

Upload before deleting the namespace

Make collection a finalizer in the pipeline, not an optional developer habit. Give it a short timeout and report its status separately. If the upload times out, retain the namespace for a second, controlled cleanup attempt when the cluster policy allows it.

Separate pass and failure retention

Passing runs usually need only the JUnit summary. Failed runs may need the manifest, relevant logs, and a browser trace. Saving all raw messages for every green build is expensive and gives little extra signal.

For deeper browser failures, trace-based debugging for flaky email tests is a good companion pattern. The same run ID should be included in the trace name so the two systems can be joined without manual searching.

Redact at the producer

Do not upload first and redact later. Remove bearer tokens, reset links, and message bodies at the point where the receipt is generated. Also check that shell tracing is disabled around commands containing addresses or credentials. This is easy to forget in a rushed incident.

What to retain and for how long

A practical starting point is seven days for failed-run artifacts and one day for successful summaries. Adjust this after looking at how long failures take to investigate. Use an object-storage lifecycle rule rather than a cron job that walks prefixes; lifecycle policies are simpler to audit and harder to forget.

The same rule applies to fixture addresses. A test address that includes a typo such as tamp mail com should be treated as untrusted input, not silently normalized into a real destination. Test fixtures should never be allowed to send mail to a human address by accident.

A small operational checklist

• Does every email fixture have a CI run ID and namespace ID?
• Is the failure receipt uploaded before namespace deletion?
• Are verification tokens and message bodies redacted?
• Can a reviewer distinguish a timeout from a wrong-message failure?
• Are successful and failed artifacts on different retention paths?
• Does the upload failure remain visible without masking the test result?
• Is the object-storage lifecycle policy tested in a non-production bucket?

This is a small amount of infrastructure, but it save hours when an email test fails only once in a hundred parallel jobs. Kubernetes still provides the isolation, while the retention contract provides the memory. That combination makes CI failures less mysterious and much more actionable.