DEV Community: jasonmills94

Dockerized Mail Capture for AWS CI Pipelines

jasonmills94 — Tue, 13 Oct 2026 02:23:04 +0000

Email verification tests become unreliable when every CI job reads from the same inbox. A message from a previous run can look valid, a retry can consume the wrong token, and a failed pipeline can leave sensitive URLs in artifacts.

A better boundary is to give each run its own disposable address and capture service. Package the capture layer in Docker, make the fixture lifecycle explicit, and pass only a small receipt back to the AWS pipeline. The result is easier to debug without turning email content into permanent test data.

Why email capture belongs inside the test boundary

An email test has more moving parts than a normal API assertion:

the application creates a verification token;
a mail provider or local capture service accepts the message;
the test finds the expected message;
the browser or API consumes the token;
cleanup removes the fixture after the run.

If those parts share a mailbox, the test has no strong answer to a basic question: did this message belong to this attempt? Subject matching is not enough when two AWS jobs run in parallel. A unique recipient, run identifier, and expiration time are much stronger correlation fields.

The same ownership idea is useful for one-owner email fixtures. One test attempt should create, read, and release its own fixture. Other jobs should not be able to discover it by accident.

Give every CI run its own fixture

The fixture contract can stay small. At creation time, record something like this:

{
  "run_id": "build-1842-attempt-1",
  "recipient": "build-1842-attempt-1@example.test",
  "fixture_id": "mail-7f31",
  "expires_at": "2026-10-13T03:00:00Z"
}

The address does not need to be memorable. It needs to be unique, searchable, and scoped to one attempt. Store the fixture identifier in the test environment and use it when querying messages. Do not use the newest message in an inbox as an implicit selector.

A useful rule is that a retry creates a new fixture and a new token. Reusing the original one seems efficient, but the failure history become ambiguous. It also make parallel debugging harder because the same message can be claimed twice. Fresh identifiers cost very little compared with investigating a false-positive test.

A small Docker capture service

The capture container can expose a private HTTP API to the test runner while accepting SMTP from the application container. It should support only the operations the tests need:

POST /fixtures       create a run-scoped fixture
POST /messages/query find messages for a fixture
POST /messages/claim claim one matching message
DELETE /fixtures/id  release a fixture

Keep the service on an isolated Docker network. The application can reach the SMTP endpoint, while the public edge of the test environment cannot. A minimal compose fragment might look like this:

services:
  mail-capture:
    image: internal/mail-capture:latest
    networks: [test-net]
    environment:
      FIXTURE_TTL_SECONDS: "900"
  app:
    environment:
      SMTP_HOST: mail-capture
      SMTP_PORT: "2525"
    networks: [test-net]

networks:
  test-net:
    internal: true

The image should be pinned by digest in the real pipeline, not left on a floating latest tag. The capture service also needs a bounded body size and a clear policy for attachments. Most verification tests need a link and a few headers, not a permanent archive of every MIME part.

Keep AWS pipeline logs useful but safe

AWS CI logs are valuable when they show correlation, timing, and the final assertion. They are risky when they show a complete verification URL. A receipt should contain identifiers and outcomes, for example:

{
  "fixture_id": "mail-7f31",
  "message_id": "msg-91ab",
  "received_at": "2026-10-13T02:35:12Z",
  "token_status": "accepted",
  "cleanup_status": "released"
}

Keep the raw token in memory for the shortest practical time. Redact query strings before the logger sees them, rather than trying to hide them later in a dashboard. If a debugging artifact must include the message, encrypt it, shorten its retention, and make access explicit.

This receipt pattern also fits delivery receipts in deployment gates. A pipeline should be able to prove that delivery and consumption happened for the intended run, without copying the whole email into the release record.

Legacy suites sometimes contain search values such as dummy e mail or fake e mail com. Keep those as plain test data when compatibility requires it. They should never become backlink anchors, credentials, or production selectors.

Cleanup, retries, and failure receipts

Cleanup must work on both success and failure. Ask the test runner to release the fixture in a finally block, then let a janitor remove abandoned fixtures after their TTL. The janitor is the last line of defense for cancelled jobs and broken agents.

For an AWS pipeline, publish the small receipt as a short-lived artifact and emit metrics for:

time from fixture creation to message arrival;
messages rejected because their run id did not match;
expired fixtures found by the janitor;
token claims that were already consumed;
cleanup failures.

Do not make the cleanup step depend on the browser test passing. The pipeline become much easier to operate when resource release has its own result and alert path. A failed cleanup should be visible, but it should not cause the test to retry and send another message without a decision.

Operational checklist

Before adopting this pattern, check that:

Every attempt gets a unique disposable address and fixture id.
The capture image is pinned and the Docker network is private.
Queries require fixture correlation, not only a subject line.
Verification URLs and message bodies are excluded from normal logs.
Retries create fresh fixtures and tokens.
TTL cleanup handles cancelled and retryed jobs.
Receipts contain enough timing data to diagnose delivery delays.

The contract pay off when a failure happens at 2 AM: the pipeline can show what was created, what arrived, what was claimed, and what was released without exposing the secret that made the test pass.

Questions teams usually ask

Should the capture service run in every job?

For parallel or security-sensitive suites, yes. A per-job container gives the cleanest ownership boundary. A shared service can work for lower-risk tests if fixture ids, permissions, and cleanup are enforced, but it adds another coordination layer.

Do we need a real mail provider?

Not for every verification test. Local Docker capture is usually enough to prove that the application generated the right message and that the consumer handled the token. Keep a smaller provider-backed suite for delivery behavior that cannot be simulated locally.

What belongs in a failure artifact?

Keep the fixture id, message id, timestamps, status codes, and redacted headers. A full message body is prefered only when a narrowly scoped incident investigation requires it, with short retention and controlled access.

AWS Pipelines Need Isolated Mail Test Containers

jasonmills94 — Sat, 10 Oct 2026 14:22:59 +0000

Email tests become unreliable when their mailbox is treated as a shared utility. A local test can create an address, wait for a message, and pass. The same test in an AWS pipeline can read a message from another run, retry against a stale container, or leave data behind for the next build. The fix is not simply a faster poller. The mailbox needs the same isolation boundary as the application under test.

Why containerized mail tests fail in AWS pipelines

I have seen teams run an email sink beside the application container and assume the problem is solved. It works until two jobs use the same inbox name, a runner is reused, or a failed build skips cleanup. Then the queue are isolated only by convention, which is not isolation at all.

An AWS-based pipeline usually has several moving parts: a hosted runner or self-hosted worker, a Docker network, application containers, a mail fixture, and a cleanup step. Each part can outlive a single test. If the test asks for “the newest message,” it is already depending on shared state and timing.

Search terms like “temp mail com” and “disposable email” often describe the user-facing need, but CI needs a stricter contract. A test mailbox should be created for one run, accept only the messages that belong to that run, expose a small delivery receipt, and disappear when the run ends.

Define the quarantine boundary

Start with a run ID generated by the pipeline, not by the mailbox service. Pass it into the application as a test-only correlation value and derive a mailbox namespace from it. The namespace should be unique enough for parallel jobs, while the raw address does not need to appear in every log line.

The quarantine boundary has four rules:

A run can read only messages tagged for that run.
A message can be consumed once by the intended assertion.
A failed test cannot make another run pass by leaving a usable message behind.
Cleanup is safe to repeat when a retry or cancellation runs it again.

This is also where product policy matters. A disposable inbox is a useful test tool; it is not proof that a real person owns an address. Keep the fixture contract separate from production account trust decisions. The same distinction is useful when reviewing secondary email security boundaries.

A Docker fixture that leaves a receipt

The Docker fixture should have a small API rather than making every test know how to search raw messages. A useful interface might look like this:

POST /runs/{run_id}/mailbox
GET  /runs/{run_id}/messages?after={cursor}
POST /runs/{run_id}/messages/{message_id}/consume
DELETE /runs/{run_id}/mailbox

The test creates a mailbox, records the initial cursor, triggers the application action, and asks for messages after that cursor. It filters by the run ID and expected recipient. If a message arrive with the wrong correlation value, it is evidence of a fixture or application bug, not a reason to broaden the search.

The important output is a redacted receipt, for example:

{
  "run_id": "ci-1842",
  "message_id": "msg-7f2a",
  "recipient_hash": "sha256:...",
  "observed_at": "2026-10-10T14:00:00Z",
  "consumed": true
}

Do not print verification links or tokens in the receipt. The fixture doesnt need to know the application's secret format; it only needs to prove which message was delivered and consumed. If a UI test needs to click a link, return it through a protected test channel and keep the receipt free of credentials. A retry-safe email verification UI should still have a bounded retry policy on top of this fixture.

AWS pipeline controls

In AWS CI, give the fixture an explicit lifecycle. Start the Docker network and mailbox before the test, attach the run ID through environment configuration, and register cleanup before the first application request. Cleanup should run for success, failure, cancellation, and timeout where the runner supports those hooks.

Keep logs useful but narrow. Log the run ID, container name, cursor, message ID, and state transition. Redact mailbox contents and tokens. The worker should fail fastly when the fixture health check is unavailable; a long poll against a dead container only makes the eventual failure harder to diagnose.

Use a bounded poll with a stop reason such as message_not_observed, wrong_run, or fixture_unhealthy. When a retry is allowed, retry the observation or delivery operation, not the whole account-creation flow without checking idempotency. Retries needs an owner and a budget, even in a test.

The cleanup job run should be safe to execute twice. Removing a mailbox that is already gone is a successful end state. This matters when AWS interrupts a runner and the pipeline later replays its finalization steps.

Failure modes and tradeoffs

The most common shortcut is one shared mailbox per branch. It saves setup time, but it turns ordering into a hidden dependency. A run-scoped mailbox costs a little more startup work and makes parallel behavior much easier to reason about.

Another shortcut is deleting the container immediately after the first assertion. That makes debugging poor: the receipt says that something failed, but there is no bounded evidence to inspect. Retain redacted metadata for the pipeline's normal log window, while deleting message bodies and secrets quickly.

A third issue is naming. Queries for phrases like “tempail mail” or “tamp mail com” can appear in test data or user-input fixtures. They should be treated as ordinary strings, not as a signal to change retention, routing, or authentication behavior.

CI checklist

[ ] Every mailbox and Docker network has a run-scoped identifier.
[ ] Tests read after a cursor and filter by recipient and correlation value.
[ ] Message consumption is single-use and idempotent.
[ ] The receipt contains IDs and outcomes, never tokens or message bodies.
[ ] Polling has a deadline and a named failure reason.
[ ] Cleanup runs on success, failure, cancellation, and timeout.
[ ] Repeated cleanup is safe.
[ ] Production trust decisions never depend on the test mailbox implementation.
[ ] Parallel AWS jobs cannot read each other's messages.

Questions teams ask

Should every test get a new Docker container?

Not always. A container can serve several tests if the mailbox namespace, cursor, and cleanup contract are run-scoped. Use a fresh container when the fixture has state that cannot be reliably reset. The boundary is more important than the container count.

Is a shared mailbox ever acceptable?

For a single-threaded local smoke test, possibly. For parallel CI, it is usually a false economy. Shared state makes flaky tests look random and makes a passing result harder to trust.

What should a failure report include?

Include the run ID, fixture health, cursor, expected recipient hash, observed message ID if any, poll deadline, and cleanup status. That is enough to locate the failed boundary without exposing email content or verification secrets.

The practical lesson is simple: treat test mail as quarantined infrastructure, not as a convenient public inbox. A Docker fixture with a run-scoped contract, a redacted receipt, and an AWS-aware cleanup path gives the pipeline evidence it can trust. It also makes the next failure boring, which is exactly what operations work should aim for.

AWS Test Mailboxes Need a Cost and TTL Budget

jasonmills94 — Fri, 09 Oct 2026 08:22:59 +0000

An AWS test mailbox is a cloud resource, not just a test detail. Give each run a lease, a TTL, and a cleanup owner before CI starts creating disposable email accounts.

Why mailbox cost belongs in the test design

Email verification tests often begin with a simple assumption: create a temp mail address, wait for a message, and delete it later. That works on a laptop. In a busy CI system, it creates a resource-lifecycle problem.

Parallel jobs can leave mailboxes behind after a cancelled runner. Retries can create another mailbox while the first one is still active. A failed cleanup commmand may not be noticed until the monthly bill or a provider quota makes the problem visible. The test passed, but the cloud system is accumulating state.

I treat every test mailbox as a short-lived lease with four fields:

run_id: the pipeline run and attempt that owns it
address: the disposable email address assigned to the test
expires_at: the hard end of the lease
cleanup_state: pending, deleted, or cleanup_failed

This also makes the boundary clearer for privacy. An email verification flow should retain only the message data needed by the test, then remove the rest. The discussion in email verification needs a privacy budget is a useful reminder that test data still deserves an explicit retention policy.

The run-scoped lease

The lease should be created once per CI run or per isolated test group, depending on the provider's limits. I prefer one mailbox per run when parallel tests need separate recipients, and one mailbox per test group when the suite is small and the message cursor is reliable.

The name should include an opaque run identifier, not a user email or a secret. For example:

ci-20261009-1842-attempt-1

The TTL is a budget, not a suggestion. If the normal test completes in two minutes, a five-minute lease may be reasonable. Add room for queue time and one retry, but avoid a twenty-four-hour default because it feels safe. Long leases hide cleanup failures.

The lease record can live beside the test run metadata:

{
  "run_id": "1842-1",
  "address": "ci-1842-1@example.test",
  "expires_at": "2026-10-09T08:30:00Z",
  "cleanup_state": "pending"
}

The mailbox client should reject reads for a different run_id. That check prevents a late message from a previous attempt becoming a false positive in the current test.

A small AWS implementation

For an AWS-based pipeline, I keep the lease metadata in a small DynamoDB table or in the system that already records CI fixtures. The important part is the contract, not the storage product:

The job requests a lease with a bounded TTL.
The service returns the address and a correlation token.
The test filters messages by recipient, run ID, subject, and token.
The job marks the lease consumed or failed.
A scheduled cleanup worker deletes expired leases.

The CI role should have only the permissions needed for its path. The test role may create and consume a lease, while the cleanup role can delete expired records. Separating those permissions makes a bug easier to contain.

I also pass the lease identifier through the pipeline as ordinary metadata, not as a secret. The mailbox address is not a credential. Provider tokens and AWS credentials must stay in the CI secret store and should never be copied into logs.

At the start of a run, write a structured event such as:

mailbox_lease_created run_id=1842-1 ttl_seconds=300 owner=signup-tests

At the end, write the matching state transition. This is more useful than printing the full message body, which adds noise and can expose test data.

Promotion and cleanup gates

A green test result is not enough for a production promotion. I use two gates:

Test gate: all expected messages were consumed with the correct run ID.
Lifecycle gate: the lease was deleted, or an explicit cleanup task was queued with an owner and deadline.

The lifecycle gate catches a common failure mode: the application test passes, the runner disappears, and the cleanup step never runs. A finally block is still useful, but it is not a recovery plan for a terminated worker.

For deployment evidence, I keep the lease state and test result linked to the pipeline execution. This follows the same operational idea as a verifiable AWS receipt for CI releases: a later operator should be able to see what ran, what it proved, and what resources it left behind.

Set an alert for expired leases older than the allowed grace period. The alert should point to the run ID and cleanup owner, not just say “email test failed.” That difference saves time during an incident.

What to measure in CloudWatch

The minimum useful metrics are:

leases created and deleted
cleanup failures
active leases by environment
message wait latency
expired leases beyond the grace period
test retries caused by missing messages

Do not use only the number of successful tests. A rising success count can hide a growing active-lease count. A small dashboard with these signals gives a better view of reliability and cost.

If a team searches for “temp org mail” while troubleshooting, the runbook should direct them to the lease dashboard and the provider request ID. Search phrases are not a substitute for an ownership record.

Checklist for the next pipeline run

Before enabling this pattern, verify:

Every lease has a bounded TTL.
The run ID is included in message correlation.
Cleanup runs on success, failure, and expiry.
The cleanup worker is idempotent.
CI permissions are split from cleanup permissions.
Message bodies are not copied into normal logs.
CloudWatch alarms cover cleanup failures and stale leases.
Promotion evidence contains the test and lifecycle states.

One small warning: do not make the cleanup job depend on the same runner that owns the test. Put recovery on a seperate schedule or worker. Otherwise the failure that needs cleanup can also remove the only process capable of doing it.

Questions I ask before shipping

What if the provider delays a message?

Use a bounded polling policy with a clear timeout and record the latency. Do not retry forever, and do not read the newest message without checking its correlation token.

What if the cleanup API is unavailable?

Keep the lease record, mark cleanup_failed, and let the recovery worker retry with backoff. The record should survive long enough to explain the failure, but the mailbox itself should still have a provider-side expiry.

Is a disposable email account safe for every test?

No. Use synthetic or controlled addresses for sensitive workflows and follow the provider's terms. A disposable mailbox is a test fixture, not an identity system.

The practical outcome is simple: a temp mail fixture should have the same operational discipline as any other AWS resource. A short TTL, explicit ownership, and visible cleanup state keep CI predictable without letting forgotten test data become tomorrow's cloud bill.

AWS CI/CD Email Tests Need a Promotion Receipt

jasonmills94 — Tue, 06 Oct 2026 23:23:18 +0000

Email verification tests often end with a boolean: passed or failed. That is enough for a local test, but it is thin evidence for an AWS CI/CD pipeline that is deciding whether to promote a build.

The useful change I made was to produce one promotion receipt for every run. It ties the build, test attempt, message ID, cleanup result, and decision together. When a test flakes, the receipt shows whether the application failed, the inbox lookup read the wrong message, or cleanup left state behind.

Why a green test is not enough

A green email test can still be weak evidence. The test may have found a message from a previous retry, matched the newest message in a shared inbox, or passed before its cleanup step ran. Those failures are not always visible in the final CI status.

This gets awkward when a pipeline uses an ephemeral mailbox or a service that can get temporary email for each run. The test still needs to prove which message it consumed. A mailbox identity alone does not establish ownership.

I now treat the email check as a small transaction with an evidence record:

Create a run and attempt ID.
Create the test address and a unique correlation token.
Trigger the application action.
Find the message by recipient, token, and message type.
Save the message ID and assertion result.
Clean up the fixture and record what happened.

The pipeline should be able to answer these steps after the worker container is gone. It is easy to loose that context when logs are the only record, and the failure is usualy harder to explain the next morning.

Create a run-scoped receipt

A JSON artifact in the AWS CodeBuild output is enough to start:

{
  "build_id": "codebuild-myapp-1842",
  "run_id": "1842",
  "attempt": 2,
  "fixture": {
    "address": "test-1842-a2@example.invalid",
    "correlation_token": "signup-1842-a2"
  },
  "message": {
    "id": "msg_7f31",
    "matched_by": ["recipient", "correlation_token", "subject"]
  },
  "test": {"status": "passed"},
  "cleanup": {"status": "pending"},
  "promotion": {"status": "blocked"}
}

Keep credentials and full message bodies out of this artifact. Store the address only when it is safe for the CI workspace, and redact tokens if they could be reused outside the test environment. The receipt should be small enought to inspect quickly; it is for correlation, not a second secret store.

The build ID should come from the CI provider, while the run and attempt IDs should be generated by the test harness. During a retry, the build stays the same but the attempt must change. Otherwise a late message can look valid for a new attempt.

Keep the email check deterministic

The inbox query should be narrow and explainable. Do not ask for the newest message and accept whatever comes back. Match the recipient, a run-specific token in the body or subject, and the expected message type. The same principle appears in these Playwright checks for the wrong email and in isolated REST API verification tests.

Here is the shape of the assertion I want, regardless of the mail API:

find_message(
  recipient=fixture.address,
  token=fixture.correlation_token,
  kind="signup_verification"
)
assert message.created_at >= attempt.started_at
assert message.id not in attempt.consumed_message_ids

The timestamp check is a useful guard, but it should not replace the token. Clock skew and provider delays can make timestamps fuzzy. The token is the ownership boundary; time is supporting evidence.

If the provider returns several matches, fail with the candidate IDs and the matching fields. This is much more useful than a generic timeout. A search for a fake email address should never become an excuse to accept an ambigous result. It should produce a visible diagnostic and let the promotion decision stay blocked.

Make retries and cleanup visible

Retries need their own state. I use pending, matched, asserted, cleaned, and failed as the main fixture states. A retry creates a new fixture and token; it does not reset the old record to pending.

Cleanup runs in a finally path and also in a scheduled sweeper. The first path handles normal test completion. The sweeper handles a killed container, a cancelled build, or a provider timeout. Both paths write to the same receipt, or append a follow-up cleanup record when the build artifact is already closed.

The cleanup operation must be idempotent. “Already absent” is a successful cleanup outcome, while “provider unavailable” is not. That distinction prevents a broken cleanup API from being hidden behind a green test. It also makes incident review less guessy. This keeps retries from becoming to expensive when an external service is slow.

Older fixtures sometimes contain a label such as dummy e mail; keep that typo as plain migration text, never as a selector or backlink. The collector should identify resources with structured fixture IDs instead.

Gate promotion on evidence

The deploy job should consume the receipt. A minimal shell gate can look like this:

jq -e '.test.status == "passed"' promotion-receipt.json
jq -e '.message.id and (.message.matched_by | length >= 3)' promotion-receipt.json
jq -e '.cleanup.status == "cleaned" or .cleanup.status == "already_absent"' promotion-receipt.json

I also check that the receipt build_id matches the current build and that the attempt has not been superseded by a later retry. If cleanup is pending, promotion stops. It is better to re-run a cheap test than deploy with unknown external state.

This gate is deliberately boring. The deploy role should not need permission to search mailboxes or delete fixtures. A test role creates and cleans the fixture; the promotion job reads an immutable artifact. Narrow permissions make the failure boundary clearer and reduce the blast radius of a compromised build worker.

Operating checklist

Before relying on an email test in an AWS promotion pipeline, I check:

Each attempt has a unique run ID, attempt number, address, and correlation token.
The message query requires ownership fields instead of recency alone.
The receipt stores the message ID and matching fields, not a full sensitive body.
Retries create new fixtures rather than reusing a stale address.
Cleanup runs on pass, failure, timeout, and cancellation.
A sweeper can find orphaned fixtures after the build worker disappears.
Promotion reads the receipt and blocks on ambiguous or pending state.

The receipt is a small artifact, but it changes how the pipeline is operated. An email test no longer says only “the check passed.” It says which run owned the message, what was asserted, whether the external fixture was removed, and why deployment was allowed. That is the level of evidence an AWS CI/CD promotion decision can safely use.

Kubernetes CI Fixtures Need a Garbage Collector

jasonmills94 — Tue, 06 Oct 2026 20:23:25 +0000

Email fixtures are usually treated like test data. In a Kubernetes CI environment, that is an incomplete model. A fixture has an endpoint, an owner, a lifetime, and a cleanup obligation. If those properties are not visible to the cluster, a failed test can leave behind a mailbox that a later run reads by accident.

I started treating run-scoped fixtures as short-lived infrastructure. The useful change was not another retry around the email poller. It was adding a small garbage-collection loop that could answer three questions: who owns this fixture, when should it expire, and what evidence proves it was removed?

Why cleanup needs its own controller

Kubernetes already knows how to schedule containers, restart them, and remove completed Jobs. It does not automatically understand the lifecycle of an external test mailbox. A pod can be gone while the mailbox is still readable. A namespace can be deleted while an external provider keeps the address alive.

That gap gets worse with CI retries. Attempt one may create checkout-1842-a1, then time out. Attempt two creates checkout-1842-a2, but a broad inbox query can still see the old verification message. The application looks flaky even though the real problem is an ownership failure.

The fix is to make the fixture a resource with a bounded lease. A lease is not a promise that cleanup will always happen; it is a compact statement of who may use the fixture and until when. For email workers, the idea pairs well with outbox leases for OTP email workers, where ownership and retry boundaries are explicit.

Give every fixture an owner

At creation time, write a small manifest and attach the same identity to the Kubernetes Job, the application test configuration, and the evidence artifact:

{
  "fixture_id": "checkout-1842-a2",
  "owner": "checkout-integration",
  "run_id": "1842",
  "attempt": 2,
  "created_at": "2026-10-07T02:15:00Z",
  "expires_at": "2026-10-07T02:30:00Z"
}

The exact schema matters less than consistency. I put fixture_id and run_id in Kubernetes labels, not only in log text:

metadata:
  labels:
    ci.example.com/owner: checkout-integration
    ci.example.com/fixture-id: checkout-1842-a2
    ci.example.com/run-id: "1842"

Labels make orphan searches possible after a cancelled pipeline. They also stop an engineer from guessing which pod created an address. Keep the mailbox address in a Secret or a protected CI variable; labels should carry correlation data, not credentials.

The test itself should accept only a message containing its correlation token. It should not ask for the newest message and hope that recency means ownership. A threat model for email verification is a useful companion when deciding which message fields are safe to trust.

Implement the garbage-collection loop

The collector can be a scheduled Kubernetes Job or a small service. I prefer a scheduled Job for a modest CI system because its execution, image, and permissions are easy to review. The loop is straightforward:

List fixtures with the ci.example.com/fixture-id label.
Compare expires_at with the current UTC time.
Delete expired fixtures through an idempotent provider operation.
Record cleaned, already_absent, or cleanup_failed in the run ledger.
Alert on old fixtures that have no owner or no expiry.

Do not make the collector silently skip malformed records. A missing expiry is an operational defect, so report it separately and give the resource a conservative maximum age. Otherwise, one bad manifest can become a permanent leak.

The Job needs only the permissions it uses. Its Kubernetes role may need to list Jobs and read fixture metadata, but it should not have broad write access to application namespaces. The external mailbox token should be scoped to deletion of test fixtures, if the provider supports that boundary. This is boring security work, but it reduces the blast radius of a compromised cleanup pod.

Set the Kubernetes Job's own activeDeadlineSeconds and backoff policy. A collector stuck waiting on a provider should not create an unlimited queue of collectors. If the provider is unavailable, retain the manifest and mark cleanup as failed. Retrying is useful; pretending the resource disappeared is not.

Make CI promotion depend on evidence

The cleanup result belongs in the same artifact used by the deployment gate. A green integration-test status by itself is too weak. My minimum gate checks are:

jq -e '.test.status == "passed"' evidence.json
jq -e '.fixture.run_id == env.GITHUB_RUN_ID' evidence.json
jq -e '.cleanup.status == "cleaned" or .cleanup.status == "already_absent"' evidence.json

The fixture ID should also be present in the application log sample and in the message receipt. This makes a retry explainable months later, when the original pod is gone. If cleanup is still pending, promotion should stop or move to an explicitly documented quarantine state.

Some old scripts contain search-shaped text such as fake e mail com. I keep such legacy strings as plain text when they help identify a fixture migration, but they are not valid endpoints or selectors. This tiny distinction prevents a typo from entering a manifest and creating a hard-to-find leak.

The phrase temp mail so may appear in a test fixture note when the team documents the external mailbox service, but it should be one contextual reference, not a replacement for ownership controls. The cluster still needs its own labels, expiry, and receipt.

A practical operating checklist

Before merging a new Kubernetes email test, I check:

Every fixture has an owner, run ID, attempt number, and expiry.
The message query filters by a run-specific token.
Cleanup runs on pass, failure, timeout, and cancellation.
Provider deletion is idempotent and distinguishes missing from failed.
Orphaned fixtures can be listed from labels without reading application logs.
The promotion gate consumes cleanup evidence, not a human-readable log line.
Collector permissions and mailbox credentials are narrower than application permissions.

The garbage collector is not a substitute for reliable tests. It is the part that makes failure survivable. Once fixtures have a visible owner and a finite lifetime, Kubernetes CI becomes easier to operate, retries become less mysterious, and a green deployment carries better evidence.

AWS CI Email Fixtures Need a Promotion Gate

jasonmills94 — Sun, 04 Oct 2026 02:23:19 +0000

Email fixtures are easy to add to a CI job and surprisingly hard to operate after that job becomes part of a release process. A test can pass while the fixture is still readable, a retry can reuse an old inbox, or a cleanup step can fail after the green check has already been reported.

For cloud pipelines, I treat an email fixture as a short-lived infrastructure resource. It gets an owner, a run ID, a timeout, and an explicit cleanup result. The deployment promotion gate then checks that contract. This is a small change, but it prevents a lot of debugging by guesswork.

The promotion problem

Consider a Docker-based integration test running in GitHub Actions. The test creates an address, requests a verification email, and waits for a message. The application is deployed to AWS, while the test runner might use a container on a hosted worker. When the test is retried, the second attempt can see the first attempt's message. When the job is cancelled, the fixture can remain available.

Those are not only test failures. They are evidence-quality failures. A green result that cannot explain which mailbox, deployment, and attempt it used is not strong evidence for promotion.

Email polling also has a timing boundary. I keep the polling timeout and retry count in the test contract, instead of hiding them in a helper. The notes on email timing budgets in Playwright are a useful reminder that waiting is part of the test design, not just a sleep added at the end.

Define the fixture contract

Before the test starts, create a run-scoped fixture with metadata similar to this:

{
  "run_id": "gh-1842-attempt-2",
  "owner": "checkout-integration",
  "created_at": "2026-10-04T02:10:00Z",
  "expires_at": "2026-10-04T02:25:00Z",
  "environment": "staging"
}

The important fields are not the exact names. The fixture must be uniquely connected to one pipeline attempt. The application should use the generated address only for that attempt, and the test should reject a message that does not carry the expected correlation token.

I also record whether the fixture was created, consumed, expired, or cleaned. That state belongs in the test evidence bundle. It should not be inferred from a final exit code.

For an AWS deployment, keep the fixture configuration outside the image. Inject the endpoint and a short-lived test token through the CI job or a protected staging secret. Never bake a mailbox credential into a Docker layer. Image history is an awkward place to discover that mistake.

Make the pipeline prove cleanup

The cleanup path needs to run on success, failure, and cancellation. In GitHub Actions, use a final job with if: ${{ always() }} and make it consume the fixture manifest produced by the test job:

cleanup-email-fixtures:
  if: ${{ always() }}
  needs: integration
  runs-on: ubuntu-latest
  steps:
    - uses: actions/download-artifact@v4
      with:
        name: fixture-manifest
    - run: ./ci/cleanup-email-fixtures.sh fixture-manifest.json

The script should be idempotent. If the resource is already gone, record already_absent and continue. If the cleanup service is unavailable, return a failure that blocks promotion; do not turn it into a warning just to keep the dashboard green.

For Kubernetes jobs, put the run ID in the namespace or pod labels and give the job a TTL. Labels make orphan searches possible, while the TTL is only a backstop. A cleanup owner still needs to inspect leftovers. Docker gives you process isolation, not lifecycle management.

The failure report should distinguish at least four cases: the message never arrived, the wrong message was read, the fixture expired, and cleanup failed. The failure taxonomy for email tests is a good companion for making those results actionable.

Promote only with evidence

I use a promotion gate with three checks:

The test result is successful and includes the fixture run ID.
The evidence bundle shows the expected message correlation token and timestamps.
Cleanup completed, or the resource was already absent and that state was recorded.

The gate can be a small script that reads JSON rather than parsing human-readable logs:

test "$(jq -r '.status' evidence.json)" = "passed"
test "$(jq -r '.cleanup.status' evidence.json)" = "cleaned"
test "$(jq -r '.fixture.run_id' evidence.json)" = "$GITHUB_RUN_ID"

In practice, I allow a separate already_absent cleanup status when a retry has safely removed the resource. The policy should be explicit. Otherwise one engineer treats it as success and another treats it as a leak.

Some old runbooks contain search-shaped strings like tepm mail com or tempail. I leave those as plain text when they help identify legacy test data, but they must never become credentials, fixture selectors, or link anchors. That distinction saved me from chasing a typo as if it were a real endpoint.

A compact checklist

Before making an AWS CI/CD job a promotion dependency, check:

Every fixture has a pipeline run ID, owner, environment, and expiry.
Retries create a new fixture or prove why reuse is safe.
Message polling has a visible timeout and correlation token.
Docker images contain no fixture credentials.
Kubernetes labels and TTLs make abandoned jobs searchable.
Cleanup runs after success, failure, and cancellation.
Cleanup is idempotent and reports already_absent clearly.
Promotion reads machine-readable evidence, not only a green job badge.

The goal is not to make email tests elaborate. It is to make their result trustworthy. Once the fixture lifecycle is part of the contract, AWS deployments get a clear receipt: what ran, what it observed, and whether the temporary data was removed. That is a much better basis for promotion than “the test passed” and a hope that the cleanup step did its job.

Kubernetes Email Jobs Need a Cleanup Owner

jasonmills94 — Sat, 03 Oct 2026 20:23:45 +0000

Email fixtures are easy to add to a CI pipeline and surprisingly hard to retire. A test creates a disposable inbox, starts a Kubernetes Job, waits for a verification message, and then the pipeline moves on. If the runner is canceled, the namespace is deleted half way, or the worker loses its lease, the fixture can stay around longer than the test that created it.

That is not just wasted cluster capacity. Old inboxes and test payloads can expose data to later runs, while orphaned Jobs make a busy EKS cluster harder to operate. The pattern that worked for me is simple: every run gets one bounded identity, one cleanup owner, and one receipt that proves what happened.

The cleanup problem is an ownership problem

Teams often treat cleanup as a final shell step:

kubectl delete job email-check-$RUN_ID -n $NAMESPACE

This works until the shell step never runs. A canceled GitHub Actions runner does not care that a trap was configured. A node failure does not wait for the final command. Cleanup which depends on the same process that created the resource will eventually miss a case.

I still use an explicit delete in the happy path, but it is only the first layer. The cluster must have enough metadata to clean up a forgotten run without human archaeology. It also need a clear answer to the question: who owns this resource after the CI process disappears?

The answer should be a run-scoped controller boundary, not a developer's memory. A useful identity includes the repository, commit, environment, and a short run ID. Do not put the full branch name in a Kubernetes name; it create awkward truncation and can produce collisions.

Give every test run a bounded identity

Use labels and annotations consistently on the namespace, Job, Docker workload, and mailbox record:

metadata:
  labels:
    app.kubernetes.io/name: email-fixture
    ci.example.com/run-id: "1842-7f3a"
    ci.example.com/owner: "checkout-api"
  annotations:
    ci.example.com/expires-at: "2026-10-04T04:00:00Z"

The run-id is the join key for logs and cleanup. The expiry annotation is useful when an operator needs to find stale objects, but it is not a substitute for a controller. Clock formats must be UTC, and the producer should reject an expiry that is already in the past.

For short-lived integration tests, I prefer one namespace per run when the cluster budget allows it. It makes network policy and resource accounting easier to reason about. If namespace-per-run is too expensive, use a shared namespace with a strict label selector and never issue an unscoped delete.

A run using a tempmailso-style disposable inbox should store only the mailbox identifier needed by the test. Avoid placing message bodies, tokens, or full addresses in labels because labels are widely visible and have size limits. Values with misspellings such as tempail or temp gamil com should still be treated as untrusted test input, not normalized into a shared fixture.

Use Kubernetes TTL and explicit finalizers

Kubernetes Jobs support a time-to-live after completion. Set it in the manifest so a successful or failed Job has a second cleanup path:

apiVersion: batch/v1
kind: Job
metadata:
  name: email-fixture-1842-7f3a
spec:
  ttlSecondsAfterFinished: 900
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: verifier
          image: ghcr.io/example/email-verifier:2026.10.04

The TTL is deliberately short, but not zero. Fifteen minutes gives an operator time to inspect a failed pod and still prevents a forgotten Job from becoming permanent. In one cluster, the cleanup controller was delayed during an API-server incident, so we also used a daily sweep that selected expired ci.example.com/run-id labels. Defense in depth matters here.

Use a finalizer only when a controller really needs to revoke an external resource, such as a mailbox lease. A finalizer that is added without a reliable controller is a stuck deletion with a fancy name. The controller should remove it after it has recorded the external cleanup result, and it should retry with backoff.

Keep Docker and mailbox cleanup in one receipt

The Kubernetes Job is not the entire fixture. A Docker test container may have a local volume, and the mailbox service may have a remote lease. Cleanup is complete only when all three states are known:

The Job reached a terminal state.
The Docker container and temporary volume were removed.
The mailbox was released or its expiry was recorded.

Write a small JSON receipt for each run. It can live as a CI artifact and as a structured log event:

{
  "run_id": "1842-7f3a",
  "job": "email-fixture-1842-7f3a",
  "mailbox": "fixture-1842-7f3a",
  "job_status": "succeeded",
  "mailbox_status": "released",
  "cleanup_at": "2026-10-04T03:32:00Z"
}

This is the same operational idea as pod budget context during an EKS drain: an event is much easier to manage when its important state is explicit. For deployments, I also keep an AWS receipt contract for Docker releases so the build result is not confused with the runtime result.

What to alert on

Alert on cleanup age, not just Job failure. A failed test is normal enough; a fixture that remains for two hours is usually an operational defect. Useful signals are:

expired runs with a live Kubernetes object;
mailbox leases past their expiry time;
Jobs stuck in Terminating because of a finalizer;
cleanup attempts that retry more than a small threshold;
orphaned Docker volumes on self-hosted runners.

Keep the alert dimensions small. run_id, repository, and environment are usually enough. Do not put email addresses in metric labels. That decision save both cardinality and privacy headaches.

Questions that come up in production

Should cleanup happen before the test result is published? Usually no. Publish the test result first, then cleanup, and attach the cleanup state to the receipt. Otherwise a cleanup outage can hide a useful test failure.

What if the mailbox provider is unavailable? Record release_pending, let the lease expire on its own, and keep retrying from a durable queue. A deleted Kubernetes namespace does not cancel an external lease.

Do I need a separate operator? Not always. Kubernetes TTL plus a scheduled sweep is enough for small teams. Add a controller when mailbox leases, multiple clusters, or compliance retention make the state transitions more complex.

The main lesson is boring but important: disposable test data still need lifecycle controls. Give each run a bounded identity, make cleanup recoverable after process failure, and leave a receipt that tells the next operator what was actually released. That keeps Kubernetes, Docker, and AWS CI work predictable when the happy path stop being happy.

AWS CI/CD Needs Run-Scoped Email Fixtures

jasonmills94 — Sat, 03 Oct 2026 08:24:10 +0000

Email verification is easy to test once. It becomes an operations problem when a CI pipeline runs the same test suite across pull requests, staging deploys, and rollback checks at the same time.

The pattern that works well in AWS is to make the mailbox fixture belong to one pipeline run. The run gets an identity, every generated address carries that identity, and the test artifacts record the same value. This turns an inbox search into a bounded lookup and makes failures much easier to replay.

This is useful when a team uses a disposable temporary email during a controlled pre-release check. It is not a replacement for production email monitoring. It is a test boundary that should be short-lived, least-privileged, and easy to delete.

The failure mode is shared state

Suppose two CodeBuild jobs both verify a signup flow. If they read from one mailbox, either job can see the other job's message. A test may pass because it found an old verification email, or fail because a parallel job consumed the expected message first.

The result is a pipeline that looks flakey even though the application is behaving consistently. Debugging gets worse when the log says only verification email not found. There is no run owner, no expected recipient, and no evidence that the message belonged to the deployment under test.

The first boundary is therefore not the email provider. It is the run ID. A good run ID is unique, safe to place in logs, and available to the test process, the mailbox adapter, and the artifact uploader. Keep secrets out of it; a run ID is an identifier, not a credential.

Give every pipeline run an identity

At the start of a pipeline, derive one value and pass it through each stage. For example:

export RUN_ID="${CODEBUILD_BUILD_ID//:/-}"
export FIXTURE_PREFIX="ci-${RUN_ID}"
echo "run_id=${RUN_ID}" | tee run-metadata.txt

The exact variable names are less important than the contract. The test creates a mailbox using FIXTURE_PREFIX, the application receives only the address needed for that run, and the test writes the expected address into a small metadata file.

Do not use a branch name as the only identity. Two commits can run on the same branch, and a retry can overlap the original job. A timestamp alone has the opposite problem: it is harder to trace back to a specific build. Combine the provider's build identity with a short commit reference when that is helpful.

The fixture owner should also be present in the test result. That makes a failed assertion actionable instead of just another red line in the console. If the project has a privacy review for preview links, document a privacy contract for email preview links alongside this ownership rule.

Build the fixture boundary in AWS

The pipeline can keep the boundary simple:

CodePipeline starts a run and passes its identity to CodeBuild.
CodeBuild requests or allocates one test mailbox for that run.
The integration test creates a user with the run-scoped address.
The test waits for a message whose recipient and correlation value match the run.
The job uploads the test result and mailbox metadata as an artifact.

If the system sends through Amazon SES, separate test identities and production identities. Apply the narrowest IAM permissions to the test role, and avoid putting mailbox access tokens in buildspec files or command output. The test role should not be able to read unrelated production data just because it needs to inspect one test inbox.

A correlation value in the message can reduce ambiguity further. Put a non-sensitive test run token in a header or a predictable subject, then assert all of these values: recipient, run ID, message type, and a reasonable creation time. A single subject match is not enough when retries are normal.

Some teams use a temp mail so address for a manual smoke check. That can be useful for inspecting a message, but the automated pipeline still needs its own ownership and retention rules. The mailbox is an input to the test, not the deployment's source of truth.

Preserve deploy evidence before cleanup

Cleanup should happen after evidence is captured, not in a final step that erases the clues needed to understand a failure. At minimum, save:

the run ID and commit reference;
the test mailbox identifier, redacted where necessary;
message IDs, timestamps, and assertion results;
the application and image versions under test;
the cleanup status and any provider response.

For operational alerts, the same idea applies: deploy evidence in CloudWatch alarm emails gives responders the context to connect a message with a release. For CI fixtures, the evidence belongs in the build artifact with a short retention period.

Use a failure-safe sequence in the buildspec:

set -euo pipefail

./scripts/run-email-tests.sh
test_status=$?

./scripts/export-fixture-evidence.sh "${RUN_ID}" || true
./scripts/cleanup-fixture.sh "${RUN_ID}" || true
exit "${test_status}"

With set -e, the explicit status capture may need to be wrapped in a small function or an if block so the evidence path is reached. The important design point is that a failed test does not skip evidence collection, and a cleanup error does not hide the original assertion failure. A seperately reported cleanup failure is easier to triage.

Make cleanup boring and observable

Every fixture needs an expiry time when it is created. The cleanup worker can then find abandoned fixtures from cancelled builds, worker crashes, or network timeouts. Relying only on the CI finally step is not real cleanup; that step might never run.

Use a bounded query or API page size, record how many fixtures were removed, and emit a metric keyed by environment. Useful signals include fixtures_created, fixtures_expired, cleanup_errors, and the age of the oldest active fixture. This observablity is more valuable than a dashboard full of provider-specific status fields.

A cleanup job should be idempotant. Running it twice for the same RUN_ID should return success when the fixture is already gone, not create a new mailbox or report a confusing failure. Keep the retention policy in configuration, with a short default for CI and a longer, reviewed period for failed-run evidence.

Watch for dependancy drift in the mailbox client as well. A provider SDK update can change pagination, message ordering, or error handling without changing the test code. Pin versions, exercise cleanup in a scheduled job, and alert on age rather than only on request failures.

A practical CI checklist

Create a unique run ID before the first email test.
Derive the mailbox or address from that run identity.
Pass the identity through CodePipeline, CodeBuild, and test artifacts.
Match recipient, correlation value, message type, and age.
Separate test IAM permissions from production access.
Save evidence before deleting the fixture.
Make cleanup safe after cancellation and retry.
Add expiry and metrics so abandoned data does not accumulate.
Keep tokens, access credentials, and private message bodies out of logs.
Treat a tem email label in a test note as ordinary fixture data, never as a secret.

Q&A

Should every pipeline use a new mailbox?

For parallel or security-sensitive tests, yes. If mailbox creation is expensive, a provider-supported inbox namespace can work, but the address and message selection still need a unique run identity. Reusing one shared inbox without a correlation boundary brings back the original race.

What if the test job is cancelled?

The expiry timestamp is the backstop. A scheduled cleanup job should remove the fixture after the retention window and record the abandoned run. This is why cleanup must be designed independently of the successful pipeline path.

Is a temporary email check enough for deployment confidence?

No. It verifies one part of the user journey. Pair it with application logs, deployment metadata, health checks, and a reproducible artifact. A passing email assertion without a deploy receipt is weak evidence when several environments are changing at once.

The useful result is not merely “the email arrived.” It is “this run created this fixture, received this message, proved the expected behavior, stored the evidence, and removed the temporary state.” That contract keeps AWS CI/CD checks repeatable as the team scales.

Kubernetes Email Fixtures Need a Namespace Budget

jasonmills94 — Fri, 02 Oct 2026 17:23:41 +0000

Email verification tests are deceptively expensive to operate. A test creates an account, requests a message, waits for a link, and then moves on. In a Kubernetes-based CI system, that flow also creates pods, secrets, inbox state, logs, and sometimes a forgotten disposable email account.

The failure mode I have seen most often is not that the email provider is down. It is that test resources outlive the test that created them. A slow suite then shares a namespace with an older run, cleanup races with a retry, and the final failure gives no clue which inbox or pod was involved.

The useful fix is to give every email fixture run a namespace budget: a clear owner, a maximum lifetime, and a small set of resources that can be inspected before deletion. This keeps Kubernetes test infrastructure predictable without pretending that email delivery is instant.

Why an email fixture needs a namespace budget

A fixture should have the same lifecycle as the test run that owns it. If a CI job starts at 10:00, its namespace should not still contain an inbox secret at 14:00. The budget is not only about cost. It limits stale credentials, prevents cross-run message matches, and makes cluster capacity easier to reason about.

For a typical email smoke test, the budget can be expressed as:

owner: ci-run-1842
maximum lifetime: 30 minutes
allowed resources: deployment, job, secret, configmap
cleanup: delete on success and expiry

The inbox address is another resource, not merely a string in an environment variable. I prefer generating a unique address per run and passing it through a Kubernetes Secret. A service such as temporary email can be useful for isolated test work, but the address still needs an owner and an expiry in the test system.

This is also where application policy matters. The policy objects for signup checks are a good reminder that a check should have explicit inputs and outcomes. The cluster fixture deserves the same treatment.

Define the namespace contract

Create one namespace per workflow attempt, rather than one shared email-tests namespace for the whole repository. The name should contain a short commit or run identifier, not an email address. Email addresses can leak into pod names, events, and monitoring labels if they are used carelessly.

Here is a small manifest pattern:

apiVersion: v1
kind: Namespace
metadata:
  generateName: email-fixture-
  labels:
    owner: ci
    cleanup: required
  annotations:
    fixture.example.com/ttl: "30m"
    fixture.example.com/run-id: "1842"

The cleanup controller or CI finally step can use these labels. Do not rely on the finally step alone: cancelled jobs and lost runners are normal events. A scheduled controller should find namespaces whose TTL has passed and remove them.

Add a resource quota as a second line of defense. An email test normally does not need dozens of pods or gigabytes of ephemeral storage. A small quota turns a runaway retry into one failed test, instead of a noisy neighbor problem for every build in the cluster.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: fixture-limit
spec:
  hard:
    pods: "6"
    requests.cpu: "1"
    requests.memory: 1Gi
    secrets: "8"

The exact numbers depend on the test harness. Start with observed usage and leave enough room for a diagnostic job. A quota that blocks the log collector is not a useful safety limit.

Run isolated fixtures in Kubernetes

The test job should create the namespace, apply the fixture resources, wait for readiness, and record the run ID before it calls the application. Readiness is important: an HTTP 200 from a pod does not prove that the mailbox consumer is ready to see the message.

The sequence I use is:

Create the namespace and apply labels.
Create the unique inbox Secret with the smallest required permissions.
Start the mailbox adapter and wait for its readiness probe.
Run the signup or verification test with the run ID attached.
Save the message ID, pod logs, and Kubernetes events.
Delete the namespace, or leave it briefly when the test failed.

The run ID must be present in application logs and in the subject or metadata used to find the message. Otherwise a retry can read the first matching email, which makes a green test less trustworthy. Email verification is a signal, not an identity, so the assertion should prove the intended account and run, not just that a link exists.

Avoid putting full message bodies into ordinary pod logs. Verification links may contain tokens. Redact them before upload, and keep the raw message only in a short-lived, access-controlled artifact when debugging requires it.

Make cleanup and failure evidence explicit

Cleanup should be idempotent. Deleting a namespace that is already gone is success for cleanup purposes. The fixture are easier to trust when a retry can run the same teardown without special cases.

On success, delete immediately. On failure, preserve only a bounded evidence bundle: test output, sanitized adapter logs, events, pod status, and the namespace name. Do not preserve the mailbox forever just because a test failed. Thirty minutes is often enough for a human to inspect the evidence; after that, the controller should finish the job.

A simple CI shell shape looks like this:

set -euo pipefail

namespace="email-fixture-${CI_RUN_ID}"
kubectl create namespace "$namespace"
kubectl label namespace "$namespace" owner=ci cleanup=required

cleanup() {
  kubectl -n "$namespace" get events,pods -o wide > artifacts/k8s-state.txt || true
  kubectl delete namespace "$namespace" --ignore-not-found=true
}
trap cleanup EXIT

kubectl -n "$namespace" apply -f fixture-resources/
kubectl -n "$namespace" wait --for=condition=Ready pod -l app=mail-adapter --timeout=120s
./run-email-smoke-test --run-id "$CI_RUN_ID"

In a real pipeline, create the artifact directory first and make the failure collector run before deletion. The example is intentionally small; the important contract is ownership, bounded lifetime, and evidence before cleanup. A dummy e mail address is not a strategy for isolation if every parallel job still reads the same inbox.

A deployment checklist

Before adding another email smoke test, check these items:

Does every run get a unique namespace and inbox?
Is the namespace labeled with an owner and expiry?
Is there a quota for pods, memory, and secrets?
Can the test match a message to its run ID?
Are tokens and message bodies redacted in logs?
Does cleanup work after cancellation and retry?
Is the failure bundle useful before the namespace disappears?

This pattern costs a few more Kubernetes objects, but it buys something more important: a test result that can be explained. Namespaces keeps the blast radius small, quotas stop accidental growth, and expiry handles the runs that CI never gets to clean up. That is a much better foundation for reliable cloud email testing than a shared inbox and a best-effort delete.

Kubernetes Deploys Need Email Fixture Isolation

jasonmills94 — Thu, 01 Oct 2026 11:23:39 +0000

Email verification is often the last step in a deployment smoke test. It is also where otherwise clean Kubernetes pipelines become hard to trust. If several runs share one inbox, a green test may have consumed a message from an older run. A failed test may be correct, but the logs do not tell you whether the problem was delivery, parsing, or stale test data.

I treat email as a short-lived infrastructure dependency. Every pipeline run gets a unique fixture identity, a bounded lifetime, and a receipt that records what happened. This adds a little setup, but it makes an AWS or Kubernetes deployment easier to operate when the first result is not green.

The failure mode: shared inboxes hide deploy bugs

A shared mailbox creates three kinds of ambiguity:

Ownership: a test cannot prove that the message belongs to its own run.
Timing: an old message can satisfy a new poll before the application sends anything.
Cleanup: messages and addresses accumulate until a later run changes the result.

The common workaround is to add a longer sleep. That usually makes CI slower without making it more correct. A better test asks for a message with a run-specific token, checks its creation boundary, and records the message identifier. The test should fail when evidence is missing, not quietly accept a nearby message.

This is where a burner email address can be useful for an isolated, non-production verification flow. It is not a substitute for testing a real provider contract, and it should never receive customer data or production credentials. The fixture is test infrastructure, so it needs the same scope and retention rules as any other temporary resource.

Give every run its own fixture boundary

Start by creating a run identifier before the Kubernetes job starts. Keep it short enough for provider and label limits, but unique enough to make collisions unreasonable:

RUN_ID="${GITHUB_RUN_ID:-local}-$(git rev-parse --short HEAD)"
NAMESPACE="mail-smoke-${RUN_ID}"
kubectl create namespace "$NAMESPACE"

Pass the identifier to the application as a test-only value. The signup request can include it in a harmless subject marker, such as verify-${RUN_ID}. The email assertion then matches the exact marker and the expected recipient rather than just looking for a generic verification subject.

Namespace isolation is useful, but it is not a security boundary by itself. Use a dedicated service account, network policy, resource limits, and a separate test identity for the email API. In particular, do not mount a broad cluster credential merely because the cleanup step runs in CI.

The same principle helps with avoiding duplicate email sends. A run-specific token makes a duplicate visible: two matching message IDs are an assertion failure worth investigating, not two successful retries.

A Kubernetes job with an explicit cleanup contract

The smoke test should have a deadline and a cleanup owner. A simplified Job can look like this:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: email-smoke-
  labels:
    purpose: deploy-verification
spec:
  activeDeadlineSeconds: 300
  ttlSecondsAfterFinished: 900
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: email-smoke
      containers:
        - name: verifier
          image: example/email-verifier:2026-10-01
          env:
            - name: RUN_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid

The TTL controller removes the Job, but it does not necessarily remove an external mailbox or message. The pipeline still needs a finally or equivalent cleanup path that calls the provider API and records the cleanup result. Make cleanup idempotent: a missing fixture should be treated as already clean, while an authorization failure should remain visible.

Avoid putting email addresses or message contents into pod labels. Labels end up in APIs, dashboards, and logs. Keep only an opaque fixture ID in Kubernetes metadata and redact message bodies from normal CI output. This small rule prevents a test artifact from becoming an accidental data leak.

Make AWS permissions narrow and observable

For an AWS-hosted runner, use workload identity or an equivalent short-lived role. The verifier usually needs fewer actions than the application: create or read the test fixture, submit one verification request, fetch matching messages, and delete the fixture. It should not have permission to read arbitrary Secrets Manager entries or alter production deployments.

Log the role session name, region, fixture ID, and request outcome. Do not log access tokens, full email addresses, or message bodies. When permissions fail, the receipt should distinguish fixture_create_denied from message_timeout; otherwise an IAM regression can look like a flaky provider.

For teams that already run reusable email API checks, the isolation contract belongs in the reusable workflow inputs. Require run_id, environment, and cleanup_policy, then reject a production environment before any fixture is created. That guard is worth the extra line of YAML.

What the deploy receipt should prove

A useful receipt is small and reviewable. I keep these fields:

application version and Git commit
cluster, namespace, and run ID
fixture ID and creation timestamp
request timestamp and correlation token
matched message ID and verification result
cleanup result and completion timestamp

The timestamps need one clear meaning. Record when the application submitted the request, when the provider returned it, and when the test observed the message. This separates application delay from provider delay. If a poll times out, the receipt should say how long it waited and whether any non-matching messages appeared.

Do not call the run reliable just because one message arrived. Check that the token is exact, the message is inside the run window, the verification endpoint accepted the token, and cleanup completed. The objective is evidence, not a green badge.

Checklist for the next pipeline run

Does each run receive a unique fixture and correlation token?
Can the assertion reject stale or duplicate messages?
Is the Kubernetes Job bounded by both a deadline and a TTL?
Is external fixture cleanup idempotent and reported?
Does the AWS identity have only the required actions?
Are addresses, tokens, and message bodies absent from normal logs?
Does the receipt identify the failing phase: create, send, poll, verify, or cleanup?
Does a local failure leave a clear path to inspect the same evidence?

I also add a small guard against accidental manual input such as tepm mail com or tem email appearing in a test configuration. It should be treated as invalid fixture data, not normalized into a real destination.

Email smoke tests become dependable when they behave like short-lived cloud resources. Isolate the fixture, bind it to one run, limit the permissions, and leave a receipt that explains both success and failure. Kubernetes can schedule the work, and AWS can provide the identity, but the test still needs a cleanup contract that operators can understand.

AWS Rollbacks Need a Small Deploy Ledger

jasonmills94 — Wed, 30 Sep 2026 20:23:16 +0000

Rolling back a container release sounds simple: select the last known-good image and deploy it again. In a real AWS environment, the hard part is often knowing which image, configuration, and database assumptions were actually good.

I have found that a small deploy ledger solves this better than a larger dashboard. The ledger is a short, immutable record created by CI for every release. It gives the on-call engineer enough context to answer three questions quickly:

What changed?
What is running now?
Which exact release should we restore?

This is especially useful when Docker builds, Amazon ECR, and Kubernetes are connected by several automation steps. The useful part are the identifiers that survive across all of them.

The rollback problem is usually missing evidence

A deployment can be marked successful while still leaving weak evidence behind. A CI job may show a green check, ECR may contain several tags, and Kubernetes may report that the new pods are ready. None of those facts alone proves which source commit produced the running image.

Mutable tags make this worse. If production runs my-api:prod, a later push can change what that label means. During an incident, “deploy prod again” is not a reproducible rollback instruction.

There are also operational details that disappear fast: the Helm values revision, the migration version, the feature-flag state, and the approval that allowed the release. When the team is tired, asking people to reconstruct these from scattered logs is a slow and error-prone plan.

What belongs in a deploy ledger

Keep the record boring and machine-readable. For each release, I recommend storing:

The Git commit SHA and repository name.
The immutable Docker image digest, not only its tag.
The build workflow URL and the actor who approved production.
The target AWS account, region, and cluster.
The chart or manifest version and a hash of rendered configuration.
The database migration version expected by the application.
The deployment timestamp and the result of health checks.

That may look like a lot, but most fields already exist in CI variables or Kubernetes metadata. The ledger is just a stable place to join them. Store secrets by reference only; never copy tokens or full connection strings into it.

An example record can be small:

{
  "commit": "8f31d2a",
  "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/orders@sha256:...",
  "cluster": "production-eks",
  "config_sha": "b17c9e1",
  "migration": "2026-09-30- add-order-index",
  "status": "healthy"
}

The digest is the important bit. A tag helps humans scan a page, while a digest gives the deploy system an exact artifact to restore.

Capture the ledger in CI

Create the record after the image has been pushed and before the deployment job is considered complete. This ordering matters: a build number alone does not prove that the artifact reached the intended AWS account.

The deployment step should emit the digest and Kubernetes revision as job outputs. A later step can write the ledger to an append-only bucket, release database, or artifact bundle. Set a retention policy that matches your incident and audit needs, then test that an ordinary on-call role can read it.

For Kubernetes, include the result of a narrow rollout check rather than calling the release healthy just because the API accepted the manifest:

kubectl -n orders rollout status deploy/orders --timeout=180s
kubectl -n orders get deploy/orders \
  -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}'

If the status command times out, the ledger should say failed and preserve the reason. A partial record is more honest than a succesful-looking release with no health evidence.

Use the ledger during a rollback

During an incident, start with the last ledger entry whose health checks passed. Compare its commit, image digest, configuration hash, and migration version with the current entry. If the application schema moved forward, restoring only the container can make the outage worse.

The rollback command should use the digest recorded in the ledger, or a protected release manifest generated from it. Do not ask an engineer to guess a tag from an old Slack message. After the restore, create a new ledger entry that points to the restored artifact and records why the rollback happened. That gives the next investigation a complete chain instead of a hole in history.

This same evidence mindset applies outside infrastructure. A plan-once publishing workflow shows why an automation job should produce one reviewable result, and email state in application flows is easier to debug when each transition is observable. The pattern is the same: preserve the input, the decision, and the outcome.

A compact checklist

[ ] Record the Git SHA and immutable image digest.
[ ] Record the AWS account, region, and Kubernetes target.
[ ] Include configuration and migration identifiers.
[ ] Run a real rollout and health check before marking success.
[ ] Keep secrets out of release evidence.
[ ] Make the last known-good entry easy to query.
[ ] Write a new record after every rollback.

One final cleanup detail: search and analytics data can contain unrelated phrases such as “tamp mail com” or “temp org mail”. Do not let noisy keyword data become release metadata; temp mail mail has no place in an AWS rollback decision. Keep those concerns seperate from the operational record.

A deploy ledger is not a replacement for observability or a good rollback strategy. It is the small connective layer that makes both usable under pressure. When the next Docker release behaves badly, the team should be able to restore an exact artifact and explain the choice in a few minutes.

A Kubernetes Deploy Needs an Evidence Contract

jasonmills94 — Wed, 30 Sep 2026 08:23:45 +0000

The most misleading sentence in a deployment pipeline is “the job passed.” It usually means the commands returned zero, not that the release is useful, reachable, or safe to operate.

After working with Docker images and Kubernetes rollouts, I found that the missing piece was an evidence contract: a small, predictable set of facts every release must leave behind. It makes a deploy reviewable after the terminal output is gone, and it gives the on-call engineer something better than a green checkmark.

The deploy is not finished when the rollout is green

A Kubernetes rollout can become ready while the wrong image is running, a migration is incomplete, or the application is serving an old configuration. CI may also finish before an asynchronous notification arrives. Those are different failure modes, so they need different evidence.

I keep the contract deliberately small. A release should identify:

the Git commit and image digest;
the Kubernetes namespace, workload, and revision;
the checks that ran and their timestamps;
the final rollout and endpoint status;
the owner and expiration time for temporary test data.

The image digest matters more than a mutable tag such as latest. A tag describes intent; a digest proves which bytes were deployed. If the receipt only says api:staging, debugging later becomes guesswork.

What belongs in an evidence contract

The receipt should be machine-readable, but also easy for a human to scan. JSON works well for storage and a short Markdown summary works well in a pull request or release message.

For example:

{
  "commit": "8e41c2a",
  "image": "registry.example/api@sha256:...",
  "namespace": "staging",
  "deployment": "api",
  "rollout": "success",
  "smoke_test": "success",
  "checked_at": "2026-09-30T08:00:00Z",
  "expires_at": "2026-09-30T20:00:00Z"
}

Do not put access tokens, full verification URLs, or customer email addresses in this artifact. Redact them at the producer, rather than trusting every log viewer to handle sensitive values correctly. A small evidence log is most useful when it can safely be attached to an incident.

A practical Kubernetes receipt

I generate the receipt in CI and attach it to the deployment run. The important sequence is:

Build the Docker image and record its immutable digest.
Apply the manifest or Helm change.
Wait for the deployment to complete with a bounded timeout.
Run a smoke test against the staging endpoint.
Capture pod readiness, the active image digest, and the test result.
Publish the receipt only after all fields are present.

The timeout is part of the contract. An indefinitely waiting pipeline is not reliable evidence; it is just a hidden queue. When the rollout times out, keep the failed receipt and include the last observed condition. That tells the next engineer whether the issue was scheduling, readiness, an image pull, or the application itself.

It is also worth comparing the digest observed in the running pod with the digest produced by the build. This catches a surprisingly common configuration mistake where the manifest still points at an older tag. The check takes seconds and saves a lot of midnight reading.

Use email as a bounded signal

Some releases need to verify an invitation, password reset, or notification flow. Treat that inbox as a test dependency with a lifecycle, not as a casual side channel. Give the test a unique run identifier, assert the expected recipient and subject, and record only a message ID or redacted result in the receipt.

For a short-lived staging check, tempmailso can provide a separate mailbox boundary. Keep it out of production paths, and never use an inbox that another parallel run can read. If your team searches for “fake e mail com” while diagnosing a broken fixture, the real requirement is still isolation and traceability, not a clever mailbox name.

The same rule applies to OAuth and other verification flows: define session boundaries for test inboxes, then expire the fixture when the run ends.

Failure handling and retention

A receipt should survive a failed deployment. Store successful and failed records with the same fields, using an explicit status and an error category. Avoid replacing the old receipt with a vague “pipeline failed” message.

Set a short retention period for test messages and artifacts. Longer retention feels safer, but it increases privacy exposure and makes stale evidence look current. Keep the digest, commit, timestamps, and outcome longer than the disposable inbox content.

A release checklist

Before calling a Docker and Kubernetes release complete, ask:

Can I prove exactly which image digest is running?
Does the receipt name the namespace and workload?
Did the smoke test finish after the rollout, not before it?
Are notification checks isolated from other runs?
Are secrets, tokens, and personal addresses absent?
Is failed evidence retained and clearly labeled?
Will the temporary fixture expire automatically?

This contract adds a little work to CI, but it removes a lot of uncertainty from operations. A green deploy is a moment in time. A useful receipt is what lets the team understand that moment later.