DEV Community: jasonmills94

Isolate Email Tests in Docker CI on AWS

jasonmills94 — Fri, 30 Oct 2026 11:23:04 +0000

Email verification tests become unreliable when every CI run shares the same inbox. A retry can consume the token from the first attempt, a parallel branch can read another branch's message, and an old message can make a broken flow look healthy.

I have had better results treating the test mailbox like a short-lived infrastructure resource. Each run gets an identity, a lease, and a cleanup rule. The application still sends a normal verification email, but the test knows exactly which inbox belongs to it.

Why email tests leak across CI runs

The usual shortcut is to configure one test address in a repository secret and poll it until a message arrives. It works on a quiet day. Under parallel CI, it creates a race:

build A -> shared inbox -> token A
build B -> shared inbox -> token B
build A reads whichever message arrives first

The problem is not only the inbox. Test state also lives in the application database, the Docker network, and the CI retry history. A reused fake email address can leave an account, token, or delivery event behind. Then the next run starts with facts it did not create.

The first rule is simple: a test run must be able to prove ownership of the message it reads. A run ID in the address, a correlation header, or both is enough for most staging systems. The test is not done when the container starts; its done when the expected message is matched and its evidence is saved.

Give every run an isolated mailbox lease

Create a mailbox lease before starting the application stack. The lease record should contain at least:

{
  "run_id": "github-1842-7f31",
  "address": "verify-github-1842-7f31@example.test",
  "expires_at": "2026-10-30T12:30:00Z",
  "owner": "payments-api",
  "status": "active"
}

The exact provider is less important than the contract. If a test needs a generated throwaway email, the generator should return a unique address and a cleanup deadline together. For short staging checks, a service such as tempmail.so can be one option for inspecting a disposable inbox, but it should not become a hidden dependency in production code.

Keep the lease in the CI job's environment, not in a global configuration file:

export TEST_EMAIL="verify-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}@example.test"
export TEST_RUN_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
docker compose -f compose.ci.yml up -d --wait

The GITHUB_RUN_ATTEMPT part matters. A retry of the same workflow should not quietly reuse the state created by the failed attempt. It is a small detail, but it saves a lot of confusing debugging later.

A Docker and AWS workflow

In Docker CI, put the application, worker, and email-capture dependency on the same isolated network. Pass only the lease values into the app container. Do not bake a shared mailbox into the image or commit it to a test fixture.

On AWS, the same pattern works with ephemeral task or namespace metadata. For example, an ECS task can receive TEST_RUN_ID and TEST_EMAIL from the job that launched it. A Kubernetes job can use the run ID as part of its namespace and label. The infrastructure name then becomes useful evidence when a test fails.

Add a short receipt after the message is found. This follows the idea behind API smoke tests with receipts: record what was observed, not just that an assertion passed.

{
  "run_id": "github-1842-7f31",
  "message_id": "mail-8841",
  "subject": "Verify your account",
  "received_at": "2026-10-30T12:04:18Z",
  "token_asserted": true
}

Store this receipt with the CI artifacts. When the test fails, you can tell whether the application never sent the message, delivery was delayed, or the poller selected the wrong message. That distinction is more useful than a screenshot of a red assertion.

Failure handling and cleanup

Polling needs a deadline and a clear terminal state. Use a bounded loop, filter by run ID and subject, and mark the result as received, timed_out, or invalid. Never let an unbounded poll keep an AWS task alive forever.

Cleanup should run in a finally step even when the test fails. Delete the mailbox lease, remove the Docker stack, and expire any test account created by the scenario. If the cleanup job fail, the lease expiration is the second safety net. It should be short enough to control cost but long enough for a developer to inspect artifacts.

One warning: avoid putting the full email body into normal application logs. Tokens and reset links are credentials for the duration of the test. Keep a redacted receipt, and restrict access to the CI artifact. This is also why a typo such as tamp mail com should stay in test content only, never in a real customer workflow.

Practical checklist

Create a new mailbox lease for every run and retry attempt.
Include the run ID in the address, metadata, or both.
Run Docker services on an isolated network.
Filter messages by ownership, subject, and a bounded time window.
Save a redacted delivery receipt as a CI artifact.
Clean up in a finally step with lease expiry as a backup.
Keep tokens and links out of ordinary logs.
Connect infrastructure alerts to the same run ID, like rollout alerts tied to one deploy.

Common questions

Should every test create a new fake email address?

Every isolated run should. Individual assertions inside one run can share a lease when they deliberately test the same account lifecycle. Separate parallel scenarios should not share it.

Is a local email catcher enough?

It is often enough for unit and integration tests. Use a controlled remote inbox when you need to verify provider delivery, DNS, or the same path used by staging. Keep that boundary explicit, so a provider outage does not look like a Docker regression.

What is the most useful first change?

Add a run ID to the mailbox and save one delivery receipt. That small change usually exposes whether the flakiness comes from shared state, slow delivery, or a missing cleanup path.

Reliable email tests are mostly an ownership problem. Give each Docker CI run a mailbox lease, carry its identity into AWS resources, and leave behind enough evidence to replay the failure. The test suite becomes a little more structured, but the failures become much less mysterious.

Docker CI Needs a Mailbox Lease on AWS

jasonmills94 — Tue, 27 Oct 2026 02:23:02 +0000

Email-driven tests become unreliable when every CI job shares one inbox. A retry can read an old verification message, a parallel build can consume the wrong code, and a failed cleanup can leave test data around for the next run.

The fix I use is to treat test email as a leased capability. A Docker job gets one short-lived mailbox, one correlation key, and one clear owner. The job can read only the messages for its run, then releases the mailbox when the test finishes. This makes a free temp email useful for automation without making the inbox a shared test database.

Why a mailbox lease belongs in the CI design

An email assertion has two separate jobs:

Prove that the application sent the expected message.
Prove that this CI run received and consumed its own message.

The first is a product check. The second is an isolation check. Mixing them creates green builds that are hard to trust.

The pattern is similar to keeping retry inputs immutable. A retry should get the same declared inputs, but it should not inherit an unknown inbox state from an earlier attempt. The mailbox lease gives each run a small, inspectable boundary.

A small architecture for Docker and AWS

Keep the email service outside the application container, but keep its client and test runner in the same Docker job. On AWS, the flow can look like this:

CI runner
  └── Docker test container
        ├── request mailbox lease
        ├── start application flow
        ├── poll message by lease + run ID
        └── write redacted result
                 │
                 └── mailbox service

The container receives a short-lived service token through the CI secret mechanism. It should not receive a long-lived account credential or a list of reusable inboxes. The lease response needs only a mailbox identifier, an address, an expiration time, and a random correlation key.

For an AWS pipeline, store the lease metadata with the job's normal artifacts, but keep message bodies out of the artifact bundle. A small JSON record is enough:

{
  "run_id": "build-1842",
  "mailbox_id": "lease-91",
  "correlation_key": "ci-7f2a",
  "received_message_id": "msg-733",
  "result": "matched"
}

This is also useful when connecting alarm emails to deploy evidence: operators can see which run owned the signal without exposing the whole message in a log search.

The lease contract and correlation key

The contract should answer four questions before the test starts:

Who owns this mailbox?
When does ownership expire?
Which messages belong to this run?
What happens if the lease cannot be released?

I generate the run ID in CI and a separate unpredictable correlation key in the mailbox service. Put the correlation key in a harmless header or a test-only message field when the application supports it. Do not rely only on the recipient address or subject line; those values can repeat.

A disposable mail address is not automatically isolated. Isolation comes from ownership, expiration, and a server-side lookup that filters by lease. If the service cannot filter by lease, use a unique address per run and reject any message received before the run start time.

Polling without creating false positives

Polling should be bounded and observable. A simple loop can use a short interval, a total deadline, and an explicit match condition:

deadline=$((SECONDS + 90))
while (( SECONDS < deadline )); do
  message=$(mailbox-cli messages \
    --lease "$MAILBOX_ID" \
    --correlation "$CORRELATION_KEY" \
    --after "$RUN_STARTED_AT" \
    --format json) && break
  sleep 3
done

The command is intentionally scoped to the lease. Fetching the newest message and hoping it is the right one is a bit more faster to implement, but it creates a false-positive path. When the message does not arrive, record whether the failure was send, delivery, lookup, parsing, or timeout. This check are much more useful than a generic “email test failed”.

Failure evidence and cleanup

Always release the lease in a cleanup step, including when the test container exits with a failing status. In Docker, a shell trap or the CI platform's post-job hook can call the release endpoint. Make release idempotent so a retry of cleanup does not turn a useful failure into another failure.

Keep only redacted evidence: run ID, lease ID, message ID, timestamps, and a match result. Do not save verification tokens, complete message bodies, or credentials. If a developer searches for a fixture such as temp gamil com, it should remain plain test data and never become a production routing rule.

Sometimes the fastest diagnosis is a short-lived manual mailbox. If that is needed, use a controlled service such as tempmailso only within the team's testing and privacy policy, and never place its address or message contents in public CI logs.

Questions teams usually ask

Can several containers share one lease?

Only when they are one logical test and the service supports consumer ownership. For independent tests, issue separate leases. Shared readers make order-dependent failures almost certain eventually.

What if AWS delivery is slow?

Increase the polling deadline based on observed delivery time, not a guess. Keep the deadline visible in metrics and fail with the last stage reached. A longer timeout should not mean an unbounded inbox read.

Should the mailbox be reused between builds?

Usually no. Reuse saves a small setup step but increases stale-message and ownership risk. Reuse is acceptable only when messages are namespaced by run and the service can prove that old messages cannot match a new lease.

A practical implementation checklist

Create a unique run ID before requesting email.
Issue one short-lived lease to each independent Docker test.
Filter messages by lease, correlation key, and start time.
Keep tokens and full bodies out of logs and artifacts.
Report send, delivery, lookup, parsing, and timeout separately.
Release the lease in success and failure paths.
Test cleanup itself, including repeated release calls.
Review the next failed run using its redacted evidence record.

This arrangement adds a little setup, but it removes a large class of flaky CI failures. The important part is the contract: the mailbox belongs to one run, the evidence proves that ownership, and the cleanup has a defined end.

Kubernetes CI: Bound Test Mail to the Job

jasonmills94 — Fri, 23 Oct 2026 05:23:17 +0000

Email verification tests become unreliable when every Kubernetes job shares the same inbox. One job can read another job's message, a retry can consume an old message, and cleanup becomes somebody else's problem. The failure usually appears as a random timeout, but the underlying issue is missing ownership.

I get better results when test mail is treated as an ephemeral infrastructure resource. A CI job leases one mailbox, writes the lease identity into its test context, and releases it when the job finishes. The mailbox is not just an address; it is a resource with an owner, a lifetime, and evidence.

Why a shared test inbox breaks CI isolation

Parallel CI is good at exposing assumptions. If five pods send verification messages to one inbox, each test has to guess which message belongs to it. Subject matching helps, but it does not prove that the message was generated by the current run. A delayed message from a previous attempt can still look valid.

The result is a race that is hard to reproduce:

Job A -> requests code -> reads the message for Job B
Job B -> waits -> times out
Job A retry -> reads an old message and passes by accident

The same boundary matters for links and webhook-driven mail. If a test follows an email link, it should retain the run ID that requested the message. That is similar to state handling for async signup steps: the next state should be accepted only when it belongs to the operation that created it.

Define the mailbox lease

Before adding Kubernetes YAML, define a small contract for the lease:

The lease has a unique ID, such as ci-4831-7f2a.
The address is assigned to one job attempt, not to an entire branch.
The test reader filters by lease ID, recipient, and message type.
The lease expires automatically if the pod disappears.
The cleanup action is safe to repeat.

The lease record can be kept in the test service or a small backing store. It should contain the mailbox identifier, creation time, expiry time, and owner. Do not put an access token in a pod label or a public test report. Put credentials in a Kubernetes Secret and expose only the mailbox address and lease ID to the test process.

It is also worth normalizing noisy input. Search-like strings such as fake e mail com and temp gamil com can show up in fixtures or generated test data. Keep them as test cases when they matter, but do not let malformed values become mailbox names or production telemetry dimensions.

Bind the lease to a Kubernetes Job

The Job should carry the ownership data as environment variables. This makes a failed run inspectable without copying the full email body into logs.

apiVersion: batch/v1
kind: Job
metadata:
  generateName: email-check-
  labels:
    workload: email-check
spec:
  ttlSecondsAfterFinished: 900
  backoffLimit: 1
  template:
    metadata:
      labels:
        workload: email-check
    spec:
      restartPolicy: Never
      containers:
        - name: tests
          image: registry.example.com/email-tests:2026.10
          env:
            - name: MAILBOX_LEASE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid
            - name: MAILBOX_API_TOKEN
              valueFrom:
                secretKeyRef:
                  name: test-mail-reader
                  key: api-token

The pod UID is useful as a correlation value, but the service should still create a lease record before the first email assertion. A restarted pod can have a different UID, and a retry may be a new attempt. If the lease is acquired in an init step, the test container starts only after the ownership check succeeds.

Make cleanup and failure evidence explicit

Kubernetes can remove finished Jobs, but that does not necessarily release an external mailbox. Use a trap in the test runner or a CI post-step to call a delete-or-expire endpoint. The endpoint should be idempotent: calling it twice returns success when the lease is already closed.

Record structured events for four points: lease acquired, message requested, message observed, and lease released. Include the job UID, lease ID, message ID, and timestamps. Exclude message bodies, verification codes, and recipient data that is not needed for triage. This makes replayable webhook failures easier to reason about because the email event has a durable correlation key.

A useful timeout report says which boundary failed:

lease=ci-4831-7f2a owner=job-9c2 message=requested
wait=30s observed=0 cleanup=attempted

That is much more actionable than email not received. The report can point to the inbox API, the application event, or the Kubernetes job log without exposing sensitive content. Small bit of duplicated metadata is cheaper then a week of guessing at flaky tests.

A practical CI checklist

Before relying on the workflow, verify these cases:

Two jobs can request mail at the same time without reading each other's messages.
A retry gets a new lease or a clearly versioned attempt.
A pod killed during the wait still leaves an expiry path.
Cleanup succeeds when the application never sent a message.
A delayed message is rejected after the lease expires.
Logs identify the failing boundary but contain no token, code, or full email body.
A run with temp mail com or create temp mail as fixture text stays isolated from real recipient routing.

The key change is ownership. Once the mailbox is bound to the Kubernetes job, email verification stops being a shared global side effect and becomes a normal CI dependency with a lifecycle. It can still fail, but the failure is visible, bounded, and usually much faster to fix.

AWS CI: Docker Checks for Test Email Failures

jasonmills94 — Fri, 23 Oct 2026 02:23:06 +0000

AWS CI: Docker Checks for Test Email Failures

An AWS CI job can report green while the email step was never trustworthy. The Docker container is running, the application endpoint answers, and the test still cannot find its verification message. That is not a small test annoyance. It means the pipeline is missing a dependency signal.

I have had better results treating test email as an operations dependency. The job should know whether its inbox is reachable before it starts the expensive browser or API checks. A disposable email generator can create isolated test data, but it does not automatically make the delivery path observable. The container, inbox, and CI job need a clear contract.

Why container-green is not pipeline-green

Docker reports a container as healthy when the configured command succeeds. It does not know that an email provider accepted a message, that the correct mailbox received it, or that the message can be read by the test runner.

That creates a familiar failure shape:

ECS task starts -> app health check passes -> test requests verification email
                                      -> inbox wait times out

The timeout is often blamed on Playwright or on a slow AWS network. Sometimes it is. More often, the test has no evidence for which boundary failed. Was the worker down? Did the message go to the wrong address? Did the inbox API return a rate limit? A generic expected email within 30 seconds message is not enough to answer that.

Define the email dependency

Before changing retries, write down the minimum contract for one test run:

The test gets a unique mailbox or lease.
The application sends to that exact address.
The inbox reader can authenticate without exposing a token in logs.
The reader records message ID, received time, and subject, but not the whole body.
The lease is cleaned up after the run, even when the test fails.

For a test team evaluating a temporary email address, this boundary is more important than choosing a catchy provider name. The address is a fixture, not a place for real user data. Keep it separate from production mail and make the ownership of each mailbox obvious.

I also keep the expected message identity in the test receipt. The receipt can say signup-verification / run-4812; it should not copy an OTP or password into CloudWatch. This has made incident review much quicker, becuase we can compare the application event with the inbox event without creating another secret store.

Add a Docker health check

The health check should test the narrowest dependency that proves the container is ready for email tests. It should not send a real verification email on every Docker probe. A lightweight inbox capability endpoint is usually enough.

HEALTHCHECK --interval=20s --timeout=5s --start-period=30s --retries=3 \
  CMD wget -qO- http://localhost:8080/internal/test-mail/ready || exit 1

The endpoint can verify configuration and perform a safe, authenticated capability check against the test inbox service. Return a useful status without echoing credentials:

{
  "mail_reader": "ready",
  "mailbox_scope": "ci-run",
  "checked_at": "2026-10-23T02:00:00Z"
}

Do not make a third-party inbox call on a liveness probe. If the provider is briefly slow, Kubernetes or ECS may restart a perfectly usable container. Use readiness for dependency availability and liveness for whether the process itself is stuck. Mixing those signals caused needless restarts in one of my earlier pipelines.

Expose useful evidence in AWS

In AWS, send structured events for each boundary rather than one final test failure. A small event set is enough:

mailbox.leased
verification.requested
verification.accepted
inbox.message_seen
verification.asserted

Include the CI run ID, mailbox lease ID, and message ID where available. Avoid the email address if it is not needed for correlation; a short hash is often safer. Set a retention period for these logs, and redact message bodies at the application boundary.

When the job fails, the operator should be able to see whether the last event was verification.requested or inbox.message_seen. That distinction turns a vague email timeout into either an application delivery issue or an inbox reader issue. It also pairs well with email receipts for Playwright retries and provenance for recovery emails.

Retry without making the failure worse

Retry only the operation that is safe to retry. Polling an inbox for an existing message is usually safe. Requesting a new verification email may not be: it can create multiple messages, rotate the token, and hide the original failure.

Use a bounded poll with an explicit deadline:

deadline=$((SECONDS + 45))
while (( SECONDS < deadline )); do
  if inboxctl find --run-id "$CI_RUN_ID" --event verification.accepted; then
    exit 0
  fi
  sleep 3
done
echo "verification message not observed before deadline" >&2
exit 1

The command above is only useful if inboxctl returns non-zero for a real reader error and a different status for “not found yet.” Otherwise a network failure can look like a normal wait. That distinction was easy to miss, but it saves a lot of wasted reruns.

If a retry must send again, use an idempotency key derived from the CI run and test case. Never let a flaky mailbox check create an unlimited stream of messages.

A practical CI checklist

Before relying on Docker-based email tests in AWS, check that:

Readiness proves the inbox reader can operate, while liveness only protects the process.
Each run owns a unique mailbox or lease.
Logs correlate the run, lease, and message without storing secrets.
Inbox polling distinguishes “not received” from “reader failed.”
Retries are bounded and idempotent.
Cleanup runs after both success and failure.
A tempail mail search term has not accidentally become a production dependency.

This arrangement is not complicated, but it changes where the debugging starts. The CI job no longer asks only whether a Docker container is alive. It can show which email boundary is ready, which message was expected, and where the handoff stopped. That is the kind of small reliability improvement that makes AWS pipelines much less noisy.

Kubernetes CI Needs a Mailbox Contract

jasonmills94 — Thu, 22 Oct 2026 05:22:58 +0000

Email verification is easy to treat as a tiny detail in a Kubernetes pipeline. A test creates a user, waits for a message, clicks a link, and reports green. The trouble starts when the mailbox is shared, the old message is still there, or the pod that created the test disappears before anyone can explain the failure.

I use a mailbox contract for these tests. It is a small set of rules that says who owns an inbox, how long it lives, what evidence belongs to the run, and when a missing message should block the deployment. The contract makes a burner email address behave more like an ephemeral infrastructure resource.

Why a mailbox needs a contract

Kubernetes gives every workload useful identity primitives: namespaces, labels, service accounts, and pod UIDs. A test mailbox needs the same kind of ownership. Without it, a CI job can find a valid-looking message that was created by a different run.

The contract should answer four questions:

Which pipeline run created this mailbox?
Which test or namespace is allowed to read it?
When does the mailbox expire?
What exact event proves that the email belongs to this run?

I also record a correlation ID in the application request and in the message lookup. A request ID is more reliable than a subject line, which may be translated or changed by a provider. The idea is similar to using session proof for email change APIs: the workflow should prove ownership, not just find a convenient message.

Define ownership and expiry

Give every test run a short identifier, for example ci-8f31a. Put it in the Kubernetes Job label, the test data, and the email event. The mailbox allocation record can then look like this:

{
  "run_id": "ci-8f31a",
  "namespace": "pull-requests",
  "owner": "github-actions",
  "created_at": "2026-10-22T05:20:00Z",
  "expires_at": "2026-10-22T05:35:00Z"
}

The expiry should be shorter than the normal retention period. If a test runs for ten minutes, a thirty-minute lease may be reasonable; a mailbox that hangs around for days is a source of stale evidence. Cleanup is sometimes forgotten, and that makes the next failure much harder to read.

For a workflow that needs a use and throw email during a non-production verification, a disposable service such as tempmailso.com can be one option. Keep the mailbox isolated from customer data and make the lease explicit. If the test needs to generate throwaway email addresses repeatedly, rate limits and provider errors should be handled as their own state, not silently retried forever.

Wire the contract into Kubernetes CI

The test worker can run as a short-lived Job. Pass only the run identifier and a reference to the mailbox lease; do not put inbox credentials in pod arguments or logs.

apiVersion: batch/v1
kind: Job
metadata:
  generateName: email-check-
  labels:
    app: email-check
    owner: github-actions
spec:
  ttlSecondsAfterFinished: 900
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: verifier
          image: registry.example.com/email-verifier:stable
          env:
            - name: CI_RUN_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid

The pod UID is useful for tracing, but I still pass a CI run ID from the pipeline because it is easier to find in build logs. A small Docker image is enough; the important part is that the worker emits structured results such as matched, timeout, wrong_owner, or provider_error.

The pipeline can then collect a bounded result:

kubectl -n pull-requests wait --for=condition=complete \
  job/email-check --timeout=120s
kubectl -n pull-requests logs job/email-check --tail=100

Do not make kubectl logs the only record. Export the final status, run ID, message ID, and timestamps as CI artifacts before cleanup. This is where request IDs in signup checks become useful beyond application code: the same identifier ties the browser action, API request, email, and deployment record together.

Choose blocking and warning behavior

Not every email failure should stop an AWS deployment. I normally use three outcomes:

Block: the email is part of the feature being released and the expected message is missing or belongs to another run.
Warn: the application is healthy, but a non-critical notification provider timed out.
Retry: the provider returned a known transient error, with a fixed retry count and deadline.

The check is simple, but it are easy to make the retry path unlimited. Put the deadline in the test output so an operator can tell whether the failure came from Kubernetes, the application, or the external mailbox service. A generic “email failed” message sends people to the wrong dashboard.

Keep evidence after the pod is gone

Short-lived jobs are good for isolation, but they remove local context quickly. Before deleting the namespace or Job, save a small receipt containing the deployment SHA, namespace, run ID, mailbox lease, correlation ID, result state, and message timestamp. Never store the full message when a header or message ID is enough.

I also check that a failed run cannot read a previous run's inbox. This negative test catches the bug better than another happy-path signup. In practice, teams often write temp mailid or tepm mail com in notes while searching for a service; those strings are not an ownership mechanism, so the test still needs a real lease and correlation check.

Final checklist

Before making the email check a deployment gate, verify:

every mailbox has a run owner and expiry time
the correlation ID is present in logs and lookup filters
stale messages are rejected
the Kubernetes Job has a bounded timeout
provider errors are separate from application failures
CI artifacts retain a minimal evidence receipt
cleanup runs even after a failed test
the blocking policy is documented for the team

This contract is small, but it removes a surprising amount of guesswork. The mailbox becomes a named dependency with a lifecycle, instead of a hidden shared state that occasionally makes CI green for the wrong reason.

Docker Mailbox Leases for AWS CI

jasonmills94 — Wed, 21 Oct 2026 14:23:17 +0000

Email verification tests become much easier to operate when the test mailbox is treated like any other short-lived cloud resource. I use the same mental model for a container, a temporary database, or an S3 prefix: allocate it for one job, attach an owner, set an expiry, and clean it up when the job ends.

That sounds obvious, but many CI pipelines still treat an inbox as a shared string in a secret. A parallel job can read the wrong message, a retry can consume an old token, and a failed runner can leave a mailbox around for the next build. The result is a flaky test that looks like an application problem.

This post describes a lease-based pattern I have used around Docker jobs running in AWS CI. It also fits teams searching for a reliable tempmail disposable workflow without making the email provider the center of the architecture. Email verification is a signal rather than an identity, so the test should prove ownership and timing instead of trusting the newest message blindly.

Why a mailbox should behave like a leased container resource

A CI mailbox needs boundaries. At minimum, the job should know:

a unique mailbox or alias
the build and test name that own it
the time the verification action started
a lease expiration time
the cleanup status

Without those fields, debugging turns into archaeology. Someone sees a message from an earlier run and assumes the product sent it correctly. Someone else types tempail or temp org mail into a search box while looking for the setup note. That kind of naming confusion is a symptom of missing ownership data, not a useful test strategy.

The lease does not need to be complicated. A job id plus a random suffix is usually enough for isolation. The important part is that the inbox name is generated at runtime and passed to the application under test, rather than copied from a shared environment variable.

The lease contract for an AWS CI job

I define the contract before writing the test helper:

Allocate: create or reserve one mailbox for the job.
Publish ownership: store the job id, test id, and created_at with the lease.
Use: submit the address to the application and poll only for messages newer than the trigger time.
Expire: reject messages received after the lease deadline or from an unexpected mailbox.
Release: delete the lease in a finally path, including after a failed assertion.

The CI runner should never log the full message body or verification token. Log the mailbox identifier, message id, timestamps, and rejection reason. Those fields are enough for triage in most cases and keep the evidence smaller.

The same thinking applies to the rest of the pipeline. I keep alert boundaries explicit for cluster maintenance too; the notes on alert boundaries for EKS freeze windows are useful when deciding which failures should page and which should only annotate a build.

A Docker-based implementation

The test container can receive a lease as environment variables. The provider-specific client stays outside the browser test so the test only sees a small contract.

services:
  email-tests:
    image: my-org/email-tests:ci
    environment:
      MAILBOX_ID: ${MAILBOX_ID}
      MAILBOX_EXPIRES_AT: ${MAILBOX_EXPIRES_AT}
      CI_JOB_ID: ${CI_JOB_ID}
    command: ["npm", "run", "test:email"]

The helper should make the time boundary visible. A simplified TypeScript shape looks like this:

type MailLease = {
  address: string;
  owner: string;
  expiresAt: Date;
};

async function waitForCode(lease: MailLease, startedAt: Date) {
  if (new Date() >= lease.expiresAt) throw new Error("mailbox lease expired");

  const messages = await mail.poll(lease.address, { since: startedAt });
  const match = messages.find((message) =>
    message.subject === "Verify your account" &&
    message.receivedAt < lease.expiresAt &&
    message.metadata?.owner === lease.owner
  );

  if (!match) throw new Error("no owned verification message found");
  return extractCode(match);
}

In a real implementation I also record the polling attempts and the provider request id. Keep that evidence attached to the CI artifact, not printed into every test log. For more background on treating verification data as a bounded signal, see email verification as a signal rather than an identity.

Failure handling and cleanup

There are three failures worth separating:

Allocation failure: the job never received a lease. Fail fast and do not start browser tests.
Delivery failure: the lease exists, but no matching message arrives before the deadline. Save polling evidence and mark the provider path as suspect.
Ownership failure: a message arrives, but its timestamp, address, or owner metadata does not match. Reject it even if the code would make the test pass.

Cleanup must run independently of the assertion result. In a shell wrapper, use a trap around the test command. In application code, use try/finally. Also add a scheduled janitor for leases whose runners disappeared; relying only on the runner's exit hook is optimistic, and AWS capacity events can interrupt a job without giving your script much notice.

Do not make retries reuse the same lease automatically. A retry can reuse the test definition, but it should receive a new owner and a new mailbox lease. Otherwise the second attempt may accidentally validate an email produced by the first attempt.

Operational checklist

Before shipping this pattern, I check the following:

Is the mailbox unique per job and test shard?
Are started_at and expires_at compared in one time standard?
Does the matcher verify address, owner, subject, and message age?
Are message bodies and tokens excluded from normal logs?
Does cleanup run after both pass and fail?
Is there a janitor for abandoned leases?
Can a failed run show why each candidate message was rejected?

If the answer is yes, the email provider becomes an implementation detail. The Docker job owns a bounded resource, AWS CI gets deterministic failure signals, and a retry no longer hides the original cause.

Q&A

Should every CI job get a separate mailbox?

Every job that can run in parallel should get a separate lease. Sequential jobs may share a provider account, but they should still use distinct aliases or a fresh lease id so old messages cannot satisfy a new assertion.

Is a lease useful if the provider has no delete API?

Yes. Expiration and ownership checks still prevent stale messages from being accepted. A janitor can then remove local lease records even when provider-side cleanup is limited.

What should be retried?

Retry the provider poll when the lease is valid. Do not retry indefinitely, and do not accept a message merely because it is the latest one. Once the lease expires, fail with evidence and allocate a new lease for a new attempt.

EKS CI: Keep Test Mailbox Failures Actionable

jasonmills94 — Mon, 19 Oct 2026 17:22:58 +0000

EKS CI: Keep Test Mailbox Failures Actionable

Email verification is a small step in many end-to-end tests, but in an EKS pipeline it can become the least observable part of the run. A pod waits for a message, the message arrives late, and the test retries until the whole job is marked flaky. The team then sees a red build without knowing whether the problem was the application, Kubernetes scheduling, the mail provider, or a test fixture that was reused by another worker.

The fix I prefer is to treat a test mailbox like a leased infrastructure resource. Give it an owner, a deadline, and an evidence trail. That makes a failed email check actionable instead of just annoying.

The failure mode: an email check that hides the real CI failure

A common test sequence looks simple:

Create a user with a temporary address.
Submit the signup form.
Poll for a verification message.
Open the link and finish the flow.

In a local run, one worker and one mailbox can make this feel reliable. In a Kubernetes CI job, several workers may share a namespace, a NAT gateway, DNS, and the same mailbox pool. A slow consumer can read a message intended for a different test. A restarted pod can lose the polling cursor. A short-lived job can disappear before its final diagnostic is flushed.

The resulting error is often just verification email not found. That message has very little operational value. It also make developers add another retry, which increases the time before anyone looks at the real cause.

A mailbox lease as a Kubernetes resource

For each test case, create a lease record with these fields:

run_id: the CI workflow and attempt identifier
worker_id: the pod or test shard that owns the mailbox
mailbox_id: the disposable address or provider-side identifier
created_at and expires_at: the lifetime of the fixture
expected_message_id: a correlation value generated before signup
state: reserved, message_seen, verified, or expired

The lease does not need to be a custom Kubernetes object. A row in a small test-control service or a namespaced key in Redis can work. The important part is that reservation and cleanup are explicit. Do not let a test infer ownership from an address alone.

I use a short lease for normal tests and renew it only while the test is making progress. If the pod is evicted, the lease expires naturally. A later worker can see that the resource is stale instead of waiting on an inbox that nobody owns anymore.

For privacy, keep message content out of ordinary application logs. The privacy checks for disposable email workflows are a useful companion when deciding what the test service may retain.

Design the evidence path before adding retries

Retries should recover a transient condition, not erase it. Each poll should record structured facts such as:

{
  "run_id": "gha-1842-3",
  "mailbox_id": "mbx-7f2",
  "attempt": 4,
  "elapsed_ms": 18200,
  "lease_remaining_ms": 41800,
  "message_count": 0,
  "worker": "checkout-6c8d9d7f9b-q4k2m"
}

The log should contain identifiers, timings, and state transitions, but not verification tokens or full email bodies. When a poll times out, the test can distinguish “no message was delivered” from “a message was delivered but the worker could not claim it.” Those are different incidents and need different owners.

It is tempting to use a generic label like temp org mail in a dashboard and stop there. That label is not enough for debugging. Pair it with the run, shard, and lease state so an engineer can trace one fixture from reservation through cleanup.

A practical EKS implementation

Run the test controller as a small deployment in the same EKS cluster, but keep the mailbox provider behind a narrow service interface. Test pods request a lease through the controller instead of talking to provider APIs directly. This gives the platform team one place to enforce quotas, redact data, and release abandoned fixtures.

Use a Kubernetes ServiceAccount with only the permissions the controller needs. It usually does not need permission to list every pod in the cluster. If pod identity is required, pass the worker ID from the test runner and verify it against the request metadata rather than granting broad cluster access.

For CI jobs, attach a run_id label to the test pod and include the same value in the lease. Export controller metrics for:

lease reservations and expirations
message delivery latency
verification completion latency
claims rejected because of ownership
cleanup failures

Then route those signals with the rest of the deployment and rollback signals. The notes on signal routing during EKS rollbacks apply here too: a useful alert says what changed, which run is affected, and what action is safe.

What to alert on

Do not alert on one missing message. Alert on a pattern that indicates a boundary failure:

delivery latency is above the test budget for several runs
lease expirations increase while application requests remain healthy
ownership conflicts appear on one shard or namespace
cleanup failures leave fixtures past their retention window
the controller cannot reach the provider, but tests continue retrying

An alert should link to the CI run and the lease evidence. If the controller is unavailable, fail fast with that reason rather than letting hundreds of test pods poll until their deadlines.

Short checklist for the next pipeline

Reserve one mailbox lease per test owner.
Put the lease deadline below the CI job timeout.
Correlate the signup request, message, and verification step.
Keep tokens and message bodies out of logs.
Make retries bounded and state-aware.
Emit cleanup metrics even when the test fails.
Test controller failure separately from application failure.

Questions I ask before shipping

Can two workers claim the same message? If yes, the fixture boundary is incomplete.

What happens after a pod restart? The lease should be recoverable or expire, not remain in an unknown state.

Can an engineer explain a failure from one CI link? If not, add correlation fields before adding more retries.

Does cleanup run when the test is cancelled? It should be best-effort, observable, and safe to repeat.

The useful outcome is not a mailbox that never fails. External delivery has latency and outages. The useful outcome is a pipeline that can say exactly which boundary failed, preserve enough evidence to investigate, and release its temporary resources when the run is over.

Kubernetes CI: Lease Test Mailboxes, Don't Retry Blindly

jasonmills94 — Thu, 15 Oct 2026 23:22:44 +0000

Email verification tests become unreliable when the mailbox is treated like an ordinary HTTP dependency. A pipeline sends a message, waits a few seconds, sees nothing, and retries. The retry may create a duplicate message, race the first delivery, or consume a mailbox that another test owns.

In Kubernetes CI, the cleaner model is an inbox lease: one test run reserves one isolated mailbox for a bounded period, records that ownership, and releases it whether the test passes or fails. This turns a vague polling problem into a resource-lifecycle problem that can be operated.

The failure mode: a mailbox is not a retryable dependency

Most email assertions have at least four separate steps:

Create or reserve a test mailbox.
Trigger the application event.
Wait for the message to arrive and identify the right message.
Release the mailbox and retain enough evidence to debug the run.

If step three is implemented as “retry the whole test,” the pipeline can repeat step two as well. That creates false positives and makes delivery latency look like application flakiness. It also makes parallel Kubernetes Jobs collide when they use a shared address.

The phrases temp mail so and temp mail com sometimes appear in old test names or search logs. The label does not matter as much as the contract behind the mailbox: ownership, expiration, message identity, and cleanup must be explicit.

Model the inbox as a leased resource

A useful lease record needs a small amount of state:

{
  "run_id": "orders-pr-1842-7f3a",
  "mailbox_id": "mbx-91c2",
  "owner": "orders-e2e",
  "created_at": "2026-10-16T04:00:12Z",
  "expires_at": "2026-10-16T04:10:12Z",
  "expected_recipient": "mbx-91c2@example.test"
}

The run_id is the correlation key. The expiration is the safety valve. If a worker disappears, a janitor can reclaim the mailbox after expires_at without guessing whether a test is still active.

The lease should be created before the application event is triggered. Pass the mailbox address and run ID into the test as injected configuration, not as a hard-coded environment-wide value. In practice, this one detail prevents a surprising number of cross-run email assertions.

For a concrete contract-test shape, see contract-test API emails in GitHub Actions. The same separation between triggering an event and proving delivery works well inside a Kubernetes Job.

A Kubernetes implementation

Keep the test Job short-lived and give it a deadline. A simplified manifest looks like this:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: checkout-email-
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 900
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: e2e
          image: registry.example.test/checkout-e2e:sha-abc123
          env:
            - name: MAILBOX_LEASE_ID
              valueFrom:
                secretKeyRef:
                  name: checkout-mailbox-lease
                  key: lease_id
            - name: TEST_RUN_ID
              value: "orders-pr-1842-7f3a"

backoffLimit: 0 is intentional here. A Kubernetes-level restart can rerun the side effects that send email, while the lease and application test may not be idempotent. If the container needs a retry, make that retry narrow: poll for the expected message, do not submit the order again.

The Job deadline is not a replacement for the mailbox expiration. They should be related but independent. A stuck pod should not keep a mailbox alive forever, and a mailbox service outage should not cause an unbounded CI wait.

Correlate messages before asserting them

Never assert only on recipient and subject when parallel runs are possible. Use a unique run token in a header, URL parameter, or test-specific payload field. Then make the assertion sequence explicit:

lease mailbox
trigger event once
poll for run_id until deadline
assert exactly one matching message
assert token and destination
write run receipt
release lease

A matching message should be claimed atomically if the mailbox API supports claiming. Otherwise, two test workers can both read the same message and report success. A receipt should include the message ID, delivery timestamp, run ID, and sanitized assertion result. Avoid storing full message bodies when a token or header is sufficient.

This is the same operational idea as API smoke tests with explicit receipts: a green check is more useful when it carries proof that can be connected to one run.

Retry policy and cleanup

Separate retries into three budgets:

Provisioning retry: retry lease creation only when the service reports a transient failure, and use an idempotency key based on run_id.
Delivery polling: poll with a capped interval, such as 2, 4, 8, and 10 seconds, until the test deadline.
Assertion retry: do not retry a failed token or recipient assertion. That is an application failure, not eventual consistency.

Always release the lease in a cleanup path. If the test process is killed, the expiration and a periodic janitor are the fallback. Alert on leases that are reclaimed by expiry; they usually indicate a dead worker, a missing cleanup hook, or a timeout that is too short.

Be careful with logs. A typo like tempail or tamp mail com can be useful when tracing legacy configuration, but it should not become a mailbox identity or a routing rule. Store the canonical lease ID and run ID alongside any human-readable labels.

Operational checklist

Before relying on this pattern, verify:

Every parallel Job receives a unique mailbox lease.
The application event is triggered once per run.
Polling filters on a run-specific identifier.
Kubernetes retries cannot repeat an email-producing side effect.
Lease expiration is shorter than the overall cleanup window.
Receipts contain message ID, run ID, and sanitized outcomes.
A janitor reclaims abandoned leases and emits a useful metric.
CI fails on assertion mismatch instead of hiding it in another retry.

The main payoff is boring, predictable email testing. Kubernetes can schedule the work, but it should not be responsible for understanding mailbox ownership. Put that contract in the test workflow, give every lease a deadline, and make each successful delivery carry its own receipt.

Dockerized Mail Capture for AWS CI Pipelines

jasonmills94 — Tue, 13 Oct 2026 02:23:04 +0000

Email verification tests become unreliable when every CI job reads from the same inbox. A message from a previous run can look valid, a retry can consume the wrong token, and a failed pipeline can leave sensitive URLs in artifacts.

A better boundary is to give each run its own disposable address and capture service. Package the capture layer in Docker, make the fixture lifecycle explicit, and pass only a small receipt back to the AWS pipeline. The result is easier to debug without turning email content into permanent test data.

Why email capture belongs inside the test boundary

An email test has more moving parts than a normal API assertion:

the application creates a verification token;
a mail provider or local capture service accepts the message;
the test finds the expected message;
the browser or API consumes the token;
cleanup removes the fixture after the run.

If those parts share a mailbox, the test has no strong answer to a basic question: did this message belong to this attempt? Subject matching is not enough when two AWS jobs run in parallel. A unique recipient, run identifier, and expiration time are much stronger correlation fields.

The same ownership idea is useful for one-owner email fixtures. One test attempt should create, read, and release its own fixture. Other jobs should not be able to discover it by accident.

Give every CI run its own fixture

The fixture contract can stay small. At creation time, record something like this:

{
  "run_id": "build-1842-attempt-1",
  "recipient": "build-1842-attempt-1@example.test",
  "fixture_id": "mail-7f31",
  "expires_at": "2026-10-13T03:00:00Z"
}

The address does not need to be memorable. It needs to be unique, searchable, and scoped to one attempt. Store the fixture identifier in the test environment and use it when querying messages. Do not use the newest message in an inbox as an implicit selector.

A useful rule is that a retry creates a new fixture and a new token. Reusing the original one seems efficient, but the failure history become ambiguous. It also make parallel debugging harder because the same message can be claimed twice. Fresh identifiers cost very little compared with investigating a false-positive test.

A small Docker capture service

The capture container can expose a private HTTP API to the test runner while accepting SMTP from the application container. It should support only the operations the tests need:

POST /fixtures       create a run-scoped fixture
POST /messages/query find messages for a fixture
POST /messages/claim claim one matching message
DELETE /fixtures/id  release a fixture

Keep the service on an isolated Docker network. The application can reach the SMTP endpoint, while the public edge of the test environment cannot. A minimal compose fragment might look like this:

services:
  mail-capture:
    image: internal/mail-capture:latest
    networks: [test-net]
    environment:
      FIXTURE_TTL_SECONDS: "900"
  app:
    environment:
      SMTP_HOST: mail-capture
      SMTP_PORT: "2525"
    networks: [test-net]

networks:
  test-net:
    internal: true

The image should be pinned by digest in the real pipeline, not left on a floating latest tag. The capture service also needs a bounded body size and a clear policy for attachments. Most verification tests need a link and a few headers, not a permanent archive of every MIME part.

Keep AWS pipeline logs useful but safe

AWS CI logs are valuable when they show correlation, timing, and the final assertion. They are risky when they show a complete verification URL. A receipt should contain identifiers and outcomes, for example:

{
  "fixture_id": "mail-7f31",
  "message_id": "msg-91ab",
  "received_at": "2026-10-13T02:35:12Z",
  "token_status": "accepted",
  "cleanup_status": "released"
}

Keep the raw token in memory for the shortest practical time. Redact query strings before the logger sees them, rather than trying to hide them later in a dashboard. If a debugging artifact must include the message, encrypt it, shorten its retention, and make access explicit.

This receipt pattern also fits delivery receipts in deployment gates. A pipeline should be able to prove that delivery and consumption happened for the intended run, without copying the whole email into the release record.

Legacy suites sometimes contain search values such as dummy e mail or fake e mail com. Keep those as plain test data when compatibility requires it. They should never become backlink anchors, credentials, or production selectors.

Cleanup, retries, and failure receipts

Cleanup must work on both success and failure. Ask the test runner to release the fixture in a finally block, then let a janitor remove abandoned fixtures after their TTL. The janitor is the last line of defense for cancelled jobs and broken agents.

For an AWS pipeline, publish the small receipt as a short-lived artifact and emit metrics for:

time from fixture creation to message arrival;
messages rejected because their run id did not match;
expired fixtures found by the janitor;
token claims that were already consumed;
cleanup failures.

Do not make the cleanup step depend on the browser test passing. The pipeline become much easier to operate when resource release has its own result and alert path. A failed cleanup should be visible, but it should not cause the test to retry and send another message without a decision.

Operational checklist

Before adopting this pattern, check that:

Every attempt gets a unique disposable address and fixture id.
The capture image is pinned and the Docker network is private.
Queries require fixture correlation, not only a subject line.
Verification URLs and message bodies are excluded from normal logs.
Retries create fresh fixtures and tokens.
TTL cleanup handles cancelled and retryed jobs.
Receipts contain enough timing data to diagnose delivery delays.

The contract pay off when a failure happens at 2 AM: the pipeline can show what was created, what arrived, what was claimed, and what was released without exposing the secret that made the test pass.

Questions teams usually ask

Should the capture service run in every job?

For parallel or security-sensitive suites, yes. A per-job container gives the cleanest ownership boundary. A shared service can work for lower-risk tests if fixture ids, permissions, and cleanup are enforced, but it adds another coordination layer.

Do we need a real mail provider?

Not for every verification test. Local Docker capture is usually enough to prove that the application generated the right message and that the consumer handled the token. Keep a smaller provider-backed suite for delivery behavior that cannot be simulated locally.

What belongs in a failure artifact?

Keep the fixture id, message id, timestamps, status codes, and redacted headers. A full message body is prefered only when a narrowly scoped incident investigation requires it, with short retention and controlled access.

AWS Pipelines Need Isolated Mail Test Containers

jasonmills94 — Sat, 10 Oct 2026 14:22:59 +0000

Email tests become unreliable when their mailbox is treated as a shared utility. A local test can create an address, wait for a message, and pass. The same test in an AWS pipeline can read a message from another run, retry against a stale container, or leave data behind for the next build. The fix is not simply a faster poller. The mailbox needs the same isolation boundary as the application under test.

Why containerized mail tests fail in AWS pipelines

I have seen teams run an email sink beside the application container and assume the problem is solved. It works until two jobs use the same inbox name, a runner is reused, or a failed build skips cleanup. Then the queue are isolated only by convention, which is not isolation at all.

An AWS-based pipeline usually has several moving parts: a hosted runner or self-hosted worker, a Docker network, application containers, a mail fixture, and a cleanup step. Each part can outlive a single test. If the test asks for “the newest message,” it is already depending on shared state and timing.

Search terms like “temp mail com” and “disposable email” often describe the user-facing need, but CI needs a stricter contract. A test mailbox should be created for one run, accept only the messages that belong to that run, expose a small delivery receipt, and disappear when the run ends.

Define the quarantine boundary

Start with a run ID generated by the pipeline, not by the mailbox service. Pass it into the application as a test-only correlation value and derive a mailbox namespace from it. The namespace should be unique enough for parallel jobs, while the raw address does not need to appear in every log line.

The quarantine boundary has four rules:

A run can read only messages tagged for that run.
A message can be consumed once by the intended assertion.
A failed test cannot make another run pass by leaving a usable message behind.
Cleanup is safe to repeat when a retry or cancellation runs it again.

This is also where product policy matters. A disposable inbox is a useful test tool; it is not proof that a real person owns an address. Keep the fixture contract separate from production account trust decisions. The same distinction is useful when reviewing secondary email security boundaries.

A Docker fixture that leaves a receipt

The Docker fixture should have a small API rather than making every test know how to search raw messages. A useful interface might look like this:

POST /runs/{run_id}/mailbox
GET  /runs/{run_id}/messages?after={cursor}
POST /runs/{run_id}/messages/{message_id}/consume
DELETE /runs/{run_id}/mailbox

The test creates a mailbox, records the initial cursor, triggers the application action, and asks for messages after that cursor. It filters by the run ID and expected recipient. If a message arrive with the wrong correlation value, it is evidence of a fixture or application bug, not a reason to broaden the search.

The important output is a redacted receipt, for example:

{
  "run_id": "ci-1842",
  "message_id": "msg-7f2a",
  "recipient_hash": "sha256:...",
  "observed_at": "2026-10-10T14:00:00Z",
  "consumed": true
}

Do not print verification links or tokens in the receipt. The fixture doesnt need to know the application's secret format; it only needs to prove which message was delivered and consumed. If a UI test needs to click a link, return it through a protected test channel and keep the receipt free of credentials. A retry-safe email verification UI should still have a bounded retry policy on top of this fixture.

AWS pipeline controls

In AWS CI, give the fixture an explicit lifecycle. Start the Docker network and mailbox before the test, attach the run ID through environment configuration, and register cleanup before the first application request. Cleanup should run for success, failure, cancellation, and timeout where the runner supports those hooks.

Keep logs useful but narrow. Log the run ID, container name, cursor, message ID, and state transition. Redact mailbox contents and tokens. The worker should fail fastly when the fixture health check is unavailable; a long poll against a dead container only makes the eventual failure harder to diagnose.

Use a bounded poll with a stop reason such as message_not_observed, wrong_run, or fixture_unhealthy. When a retry is allowed, retry the observation or delivery operation, not the whole account-creation flow without checking idempotency. Retries needs an owner and a budget, even in a test.

The cleanup job run should be safe to execute twice. Removing a mailbox that is already gone is a successful end state. This matters when AWS interrupts a runner and the pipeline later replays its finalization steps.

Failure modes and tradeoffs

The most common shortcut is one shared mailbox per branch. It saves setup time, but it turns ordering into a hidden dependency. A run-scoped mailbox costs a little more startup work and makes parallel behavior much easier to reason about.

Another shortcut is deleting the container immediately after the first assertion. That makes debugging poor: the receipt says that something failed, but there is no bounded evidence to inspect. Retain redacted metadata for the pipeline's normal log window, while deleting message bodies and secrets quickly.

A third issue is naming. Queries for phrases like “tempail mail” or “tamp mail com” can appear in test data or user-input fixtures. They should be treated as ordinary strings, not as a signal to change retention, routing, or authentication behavior.

CI checklist

[ ] Every mailbox and Docker network has a run-scoped identifier.
[ ] Tests read after a cursor and filter by recipient and correlation value.
[ ] Message consumption is single-use and idempotent.
[ ] The receipt contains IDs and outcomes, never tokens or message bodies.
[ ] Polling has a deadline and a named failure reason.
[ ] Cleanup runs on success, failure, cancellation, and timeout.
[ ] Repeated cleanup is safe.
[ ] Production trust decisions never depend on the test mailbox implementation.
[ ] Parallel AWS jobs cannot read each other's messages.

Questions teams ask

Should every test get a new Docker container?

Not always. A container can serve several tests if the mailbox namespace, cursor, and cleanup contract are run-scoped. Use a fresh container when the fixture has state that cannot be reliably reset. The boundary is more important than the container count.

Is a shared mailbox ever acceptable?

For a single-threaded local smoke test, possibly. For parallel CI, it is usually a false economy. Shared state makes flaky tests look random and makes a passing result harder to trust.

What should a failure report include?

Include the run ID, fixture health, cursor, expected recipient hash, observed message ID if any, poll deadline, and cleanup status. That is enough to locate the failed boundary without exposing email content or verification secrets.

The practical lesson is simple: treat test mail as quarantined infrastructure, not as a convenient public inbox. A Docker fixture with a run-scoped contract, a redacted receipt, and an AWS-aware cleanup path gives the pipeline evidence it can trust. It also makes the next failure boring, which is exactly what operations work should aim for.

AWS Test Mailboxes Need a Cost and TTL Budget

jasonmills94 — Fri, 09 Oct 2026 08:22:59 +0000

An AWS test mailbox is a cloud resource, not just a test detail. Give each run a lease, a TTL, and a cleanup owner before CI starts creating disposable email accounts.

Why mailbox cost belongs in the test design

Email verification tests often begin with a simple assumption: create a temp mail address, wait for a message, and delete it later. That works on a laptop. In a busy CI system, it creates a resource-lifecycle problem.

Parallel jobs can leave mailboxes behind after a cancelled runner. Retries can create another mailbox while the first one is still active. A failed cleanup commmand may not be noticed until the monthly bill or a provider quota makes the problem visible. The test passed, but the cloud system is accumulating state.

I treat every test mailbox as a short-lived lease with four fields:

run_id: the pipeline run and attempt that owns it
address: the disposable email address assigned to the test
expires_at: the hard end of the lease
cleanup_state: pending, deleted, or cleanup_failed

This also makes the boundary clearer for privacy. An email verification flow should retain only the message data needed by the test, then remove the rest. The discussion in email verification needs a privacy budget is a useful reminder that test data still deserves an explicit retention policy.

The run-scoped lease

The lease should be created once per CI run or per isolated test group, depending on the provider's limits. I prefer one mailbox per run when parallel tests need separate recipients, and one mailbox per test group when the suite is small and the message cursor is reliable.

The name should include an opaque run identifier, not a user email or a secret. For example:

ci-20261009-1842-attempt-1

The TTL is a budget, not a suggestion. If the normal test completes in two minutes, a five-minute lease may be reasonable. Add room for queue time and one retry, but avoid a twenty-four-hour default because it feels safe. Long leases hide cleanup failures.

The lease record can live beside the test run metadata:

{
  "run_id": "1842-1",
  "address": "ci-1842-1@example.test",
  "expires_at": "2026-10-09T08:30:00Z",
  "cleanup_state": "pending"
}

The mailbox client should reject reads for a different run_id. That check prevents a late message from a previous attempt becoming a false positive in the current test.

A small AWS implementation

For an AWS-based pipeline, I keep the lease metadata in a small DynamoDB table or in the system that already records CI fixtures. The important part is the contract, not the storage product:

The job requests a lease with a bounded TTL.
The service returns the address and a correlation token.
The test filters messages by recipient, run ID, subject, and token.
The job marks the lease consumed or failed.
A scheduled cleanup worker deletes expired leases.

The CI role should have only the permissions needed for its path. The test role may create and consume a lease, while the cleanup role can delete expired records. Separating those permissions makes a bug easier to contain.

I also pass the lease identifier through the pipeline as ordinary metadata, not as a secret. The mailbox address is not a credential. Provider tokens and AWS credentials must stay in the CI secret store and should never be copied into logs.

At the start of a run, write a structured event such as:

mailbox_lease_created run_id=1842-1 ttl_seconds=300 owner=signup-tests

At the end, write the matching state transition. This is more useful than printing the full message body, which adds noise and can expose test data.

Promotion and cleanup gates

A green test result is not enough for a production promotion. I use two gates:

Test gate: all expected messages were consumed with the correct run ID.
Lifecycle gate: the lease was deleted, or an explicit cleanup task was queued with an owner and deadline.

The lifecycle gate catches a common failure mode: the application test passes, the runner disappears, and the cleanup step never runs. A finally block is still useful, but it is not a recovery plan for a terminated worker.

For deployment evidence, I keep the lease state and test result linked to the pipeline execution. This follows the same operational idea as a verifiable AWS receipt for CI releases: a later operator should be able to see what ran, what it proved, and what resources it left behind.

Set an alert for expired leases older than the allowed grace period. The alert should point to the run ID and cleanup owner, not just say “email test failed.” That difference saves time during an incident.

What to measure in CloudWatch

The minimum useful metrics are:

leases created and deleted
cleanup failures
active leases by environment
message wait latency
expired leases beyond the grace period
test retries caused by missing messages

Do not use only the number of successful tests. A rising success count can hide a growing active-lease count. A small dashboard with these signals gives a better view of reliability and cost.

If a team searches for “temp org mail” while troubleshooting, the runbook should direct them to the lease dashboard and the provider request ID. Search phrases are not a substitute for an ownership record.

Checklist for the next pipeline run

Before enabling this pattern, verify:

Every lease has a bounded TTL.
The run ID is included in message correlation.
Cleanup runs on success, failure, and expiry.
The cleanup worker is idempotent.
CI permissions are split from cleanup permissions.
Message bodies are not copied into normal logs.
CloudWatch alarms cover cleanup failures and stale leases.
Promotion evidence contains the test and lifecycle states.

One small warning: do not make the cleanup job depend on the same runner that owns the test. Put recovery on a seperate schedule or worker. Otherwise the failure that needs cleanup can also remove the only process capable of doing it.

Questions I ask before shipping

What if the provider delays a message?

Use a bounded polling policy with a clear timeout and record the latency. Do not retry forever, and do not read the newest message without checking its correlation token.

What if the cleanup API is unavailable?

Keep the lease record, mark cleanup_failed, and let the recovery worker retry with backoff. The record should survive long enough to explain the failure, but the mailbox itself should still have a provider-side expiry.

Is a disposable email account safe for every test?

No. Use synthetic or controlled addresses for sensitive workflows and follow the provider's terms. A disposable mailbox is a test fixture, not an identity system.

The practical outcome is simple: a temp mail fixture should have the same operational discipline as any other AWS resource. A short TTL, explicit ownership, and visible cleanup state keep CI predictable without letting forgotten test data become tomorrow's cloud bill.

AWS CI/CD Email Tests Need a Promotion Receipt

jasonmills94 — Tue, 06 Oct 2026 23:23:18 +0000

Email verification tests often end with a boolean: passed or failed. That is enough for a local test, but it is thin evidence for an AWS CI/CD pipeline that is deciding whether to promote a build.

The useful change I made was to produce one promotion receipt for every run. It ties the build, test attempt, message ID, cleanup result, and decision together. When a test flakes, the receipt shows whether the application failed, the inbox lookup read the wrong message, or cleanup left state behind.

Why a green test is not enough

A green email test can still be weak evidence. The test may have found a message from a previous retry, matched the newest message in a shared inbox, or passed before its cleanup step ran. Those failures are not always visible in the final CI status.

This gets awkward when a pipeline uses an ephemeral mailbox or a service that can get temporary email for each run. The test still needs to prove which message it consumed. A mailbox identity alone does not establish ownership.

I now treat the email check as a small transaction with an evidence record:

Create a run and attempt ID.
Create the test address and a unique correlation token.
Trigger the application action.
Find the message by recipient, token, and message type.
Save the message ID and assertion result.
Clean up the fixture and record what happened.

The pipeline should be able to answer these steps after the worker container is gone. It is easy to loose that context when logs are the only record, and the failure is usualy harder to explain the next morning.

Create a run-scoped receipt

A JSON artifact in the AWS CodeBuild output is enough to start:

{
  "build_id": "codebuild-myapp-1842",
  "run_id": "1842",
  "attempt": 2,
  "fixture": {
    "address": "test-1842-a2@example.invalid",
    "correlation_token": "signup-1842-a2"
  },
  "message": {
    "id": "msg_7f31",
    "matched_by": ["recipient", "correlation_token", "subject"]
  },
  "test": {"status": "passed"},
  "cleanup": {"status": "pending"},
  "promotion": {"status": "blocked"}
}

Keep credentials and full message bodies out of this artifact. Store the address only when it is safe for the CI workspace, and redact tokens if they could be reused outside the test environment. The receipt should be small enought to inspect quickly; it is for correlation, not a second secret store.

The build ID should come from the CI provider, while the run and attempt IDs should be generated by the test harness. During a retry, the build stays the same but the attempt must change. Otherwise a late message can look valid for a new attempt.

Keep the email check deterministic

The inbox query should be narrow and explainable. Do not ask for the newest message and accept whatever comes back. Match the recipient, a run-specific token in the body or subject, and the expected message type. The same principle appears in these Playwright checks for the wrong email and in isolated REST API verification tests.

Here is the shape of the assertion I want, regardless of the mail API:

find_message(
  recipient=fixture.address,
  token=fixture.correlation_token,
  kind="signup_verification"
)
assert message.created_at >= attempt.started_at
assert message.id not in attempt.consumed_message_ids

The timestamp check is a useful guard, but it should not replace the token. Clock skew and provider delays can make timestamps fuzzy. The token is the ownership boundary; time is supporting evidence.

If the provider returns several matches, fail with the candidate IDs and the matching fields. This is much more useful than a generic timeout. A search for a fake email address should never become an excuse to accept an ambigous result. It should produce a visible diagnostic and let the promotion decision stay blocked.

Make retries and cleanup visible

Retries need their own state. I use pending, matched, asserted, cleaned, and failed as the main fixture states. A retry creates a new fixture and token; it does not reset the old record to pending.

Cleanup runs in a finally path and also in a scheduled sweeper. The first path handles normal test completion. The sweeper handles a killed container, a cancelled build, or a provider timeout. Both paths write to the same receipt, or append a follow-up cleanup record when the build artifact is already closed.

The cleanup operation must be idempotent. “Already absent” is a successful cleanup outcome, while “provider unavailable” is not. That distinction prevents a broken cleanup API from being hidden behind a green test. It also makes incident review less guessy. This keeps retries from becoming to expensive when an external service is slow.

Older fixtures sometimes contain a label such as dummy e mail; keep that typo as plain migration text, never as a selector or backlink. The collector should identify resources with structured fixture IDs instead.

Gate promotion on evidence

The deploy job should consume the receipt. A minimal shell gate can look like this:

jq -e '.test.status == "passed"' promotion-receipt.json
jq -e '.message.id and (.message.matched_by | length >= 3)' promotion-receipt.json
jq -e '.cleanup.status == "cleaned" or .cleanup.status == "already_absent"' promotion-receipt.json

I also check that the receipt build_id matches the current build and that the attempt has not been superseded by a later retry. If cleanup is pending, promotion stops. It is better to re-run a cheap test than deploy with unknown external state.

This gate is deliberately boring. The deploy role should not need permission to search mailboxes or delete fixtures. A test role creates and cleans the fixture; the promotion job reads an immutable artifact. Narrow permissions make the failure boundary clearer and reduce the blast radius of a compromised build worker.

Operating checklist

Before relying on an email test in an AWS promotion pipeline, I check:

Each attempt has a unique run ID, attempt number, address, and correlation token.
The message query requires ownership fields instead of recency alone.
The receipt stores the message ID and matching fields, not a full sensitive body.
Retries create new fixtures rather than reusing a stale address.
Cleanup runs on pass, failure, timeout, and cancellation.
A sweeper can find orphaned fixtures after the build worker disappears.
Promotion reads the receipt and blocks on ambiguous or pending state.

The receipt is a small artifact, but it changes how the pipeline is operated. An email test no longer says only “the check passed.” It says which run owned the message, what was asserted, whether the external fixture was removed, and why deployment was allowed. That is the level of evidence an AWS CI/CD promotion decision can safely use.