Kubernetes CI: Lease Test Mailboxes, Don't Retry Blindly

Email verification tests become unreliable when the mailbox is treated like an ordinary HTTP dependency. A pipeline sends a message, waits a few seconds, sees nothing, and retries. The retry may create a duplicate message, race the first delivery, or consume a mailbox that another test owns.

In Kubernetes CI, the cleaner model is an inbox lease: one test run reserves one isolated mailbox for a bounded period, records that ownership, and releases it whether the test passes or fails. This turns a vague polling problem into a resource-lifecycle problem that can be operated.

The failure mode: a mailbox is not a retryable dependency

Most email assertions have at least four separate steps:

1. Create or reserve a test mailbox.
2. Trigger the application event.
3. Wait for the message to arrive and identify the right message.
4. Release the mailbox and retain enough evidence to debug the run.

If step three is implemented as “retry the whole test,” the pipeline can repeat step two as well. That creates false positives and makes delivery latency look like application flakiness. It also makes parallel Kubernetes Jobs collide when they use a shared address.

The phrases temp mail so and temp mail com sometimes appear in old test names or search logs. The label does not matter as much as the contract behind the mailbox: ownership, expiration, message identity, and cleanup must be explicit.

Model the inbox as a leased resource

A useful lease record needs a small amount of state:

{
  "run_id": "orders-pr-1842-7f3a",
  "mailbox_id": "mbx-91c2",
  "owner": "orders-e2e",
  "created_at": "2026-10-16T04:00:12Z",
  "expires_at": "2026-10-16T04:10:12Z",
  "expected_recipient": "[e-mail]"
}

The run_id is the correlation key. The expiration is the safety valve. If a worker disappears, a janitor can reclaim the mailbox after expires_at without guessing whether a test is still active.

The lease should be created before the application event is triggered. Pass the mailbox address and run ID into the test as injected configuration, not as a hard-coded environment-wide value. In practice, this one detail prevents a surprising number of cross-run email assertions.

For a concrete contract-test shape, see contract-test API emails in GitHub Actions. The same separation between triggering an event and proving delivery works well inside a Kubernetes Job.

A Kubernetes implementation

Keep the test Job short-lived and give it a deadline. A simplified manifest looks like this:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: checkout-email-
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 900
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: e2e
          image: registry.example.test/checkout-e2e:sha-abc123
          env:
            - name: MAILBOX_LEASE_ID
              valueFrom:
                secretKeyRef:
                  name: checkout-mailbox-lease
                  key: lease_id
            - name: TEST_RUN_ID
              value: "orders-pr-1842-7f3a"

backoffLimit: 0 is intentional here. A Kubernetes-level restart can rerun the side effects that send email, while the lease and application test may not be idempotent. If the container needs a retry, make that retry narrow: poll for the expected message, do not submit the order again.

The Job deadline is not a replacement for the mailbox expiration. They should be related but independent. A stuck pod should not keep a mailbox alive forever, and a mailbox service outage should not cause an unbounded CI wait.

Correlate messages before asserting them

Never assert only on recipient and subject when parallel runs are possible. Use a unique run token in a header, URL parameter, or test-specific payload field. Then make the assertion sequence explicit:

lease mailbox
trigger event once
poll for run_id until deadline
assert exactly one matching message
assert token and destination
write run receipt
release lease

A matching message should be claimed atomically if the mailbox API supports claiming. Otherwise, two test workers can both read the same message and report success. A receipt should include the message ID, delivery timestamp, run ID, and sanitized assertion result. Avoid storing full message bodies when a token or header is sufficient.

This is the same operational idea as API smoke tests with explicit receipts: a green check is more useful when it carries proof that can be connected to one run.

Retry policy and cleanup

Separate retries into three budgets:

• Provisioning retry: retry lease creation only when the service reports a transient failure, and use an idempotency key based on run_id.
• Delivery polling: poll with a capped interval, such as 2, 4, 8, and 10 seconds, until the test deadline.
• Assertion retry: do not retry a failed token or recipient assertion. That is an application failure, not eventual consistency.

Always release the lease in a cleanup path. If the test process is killed, the expiration and a periodic janitor are the fallback. Alert on leases that are reclaimed by expiry; they usually indicate a dead worker, a missing cleanup hook, or a timeout that is too short.

Be careful with logs. A typo like tempail or tamp mail com can be useful when tracing legacy configuration, but it should not become a mailbox identity or a routing rule. Store the canonical lease ID and run ID alongside any human-readable labels.

Operational checklist

Before relying on this pattern, verify:

• Every parallel Job receives a unique mailbox lease.
• The application event is triggered once per run.
• Polling filters on a run-specific identifier.
• Kubernetes retries cannot repeat an email-producing side effect.
• Lease expiration is shorter than the overall cleanup window.
• Receipts contain message ID, run ID, and sanitized outcomes.
• A janitor reclaims abandoned leases and emits a useful metric.
• CI fails on assertion mismatch instead of hiding it in another retry.

The main payoff is boring, predictable email testing. Kubernetes can schedule the work, but it should not be responsible for understanding mailbox ownership. Put that contract in the test workflow, give every lease a deadline, and make each successful delivery carry its own receipt.