A Run Manifest for AI Email Automation

An AI agent can create a test account, request a verification message, and report success in a single conversation. That demo is fun. The CI version is usually less tidy: a mailbox is reused, a message arrives late, and the next run reads the code from the previous run.

I like to solve this with a small run manifest. It gives every automation run an identity, a mailbox lease, and a short list of what the agent is allowed to do. The idea is simple enough for a small script, but it also gives larger Developer Tools a useful boundary.

Why email tests need a run manifest

Email verification is an asynchronous workflow. The test sends a request now, while the useful signal may appear several seconds later. An AI agent adds another layer: it may decide to poll again, call a helper, or explain a partial result.

Without a manifest, the setup are often hidden in prompt text and temporary variables. That makes a failed run hard to reproduce. A phrase such as temp org mail might even be copied from a user input into a provider lookup, instead of being rejected at the boundary.

A manifest does not make the inbox faster. It makes the run legible. When a test fails, I want to answer these questions quickly:

• Which test owns this mailbox?
• When does the lease expire?
• Which message should be claimed?
• How many polls and tool calls are left?
• Did the agent see a real failure or just a timeout?

For a related approach, see this guide to isolating inboxes for signup API tests. Isolation is the first guardrail; the manifest records how that isolation is used.

The four fields that make a run reproducible

My smallest useful manifest has four groups of data:

1. Run identity: a unique run ID and the test or workflow name.
2. Mailbox lease: the address, creation time, expiry time, and ownership token.
3. Message claim: the expected sender, subject pattern, and a one-time message ID after claiming.
4. Budget: the deadline, poll limit, and maximum number of agent tool calls.

The lease matters more than it first seems. An address can exist while the test that created it has already finished. The consumer should check the owner and expiry before reading it, so a late message cannot quietly change another test's result.

A small manifest format

Here is a deliberately boring JSON shape:

{
  "run_id": "signup-2026-10-15-0822-a7f3",
  "workflow": "verify-email",
  "mailbox": {
    "address": "[e-mail]",
    "lease_expires_at": "2026-10-15T08:32:00Z",
    "owner": "signup-2026-10-15-0822-a7f3"
  },
  "expect": {
    "sender": "[e-mail]",
    "subject_contains": "Verify"
  },
  "budget": {
    "deadline_seconds": 45,
    "max_polls": 6,
    "max_tool_calls": 8
  }
}

The manifest is not a prompt. The runtime should enforce it even when an agent asks for another poll. That separation is important: the model can suggest an action, but the adapter decides if the action is still inside the budget.

Keep the receipt smaller than the manifest. Record the run ID, tool call ID, attempt number, status, and duration. Do not store the full message body or sensitive tokens in ordinary CI logs. A short receipt is enough to explain what happend and to connect the failure to a specific step.

Lease the inbox, then verify the handoff

There are two useful transitions: leased and claimed. A leased mailbox is reserved for one run. A claimed message is reserved for one verification step. Treating them as separate states prevents a poller from returning the same code twice.

create lease -> send verification request -> poll within deadline
      -> match message -> claim message -> verify token -> release

The claim should be atomic when the provider supports it. If it does not, store a local claim key based on run_id and message ID, then make the verification operation idempotent. It is a little more simpler than trying to clean up duplicates after the test has passed.

A useful failure result is not just false. Return a small status such as expired, not_found, wrong_sender, claimed, or verified. This gives an AI agent a safe next step and gives a human enough context for debugging.

Teams that prefer fixture-driven tests can pair this design with an API fixture pattern for email regression checks. The fixture creates the run boundary; the manifest makes the boundary visible to the rest of the pipeline.

Use the manifest in CI

In CI, create the manifest before the browser or API test starts. Pass only the run ID to each step, then load the rest from a short-lived store. At the end, publish a redacted receipt as an artifact:

{
  "run_id": "signup-2026-10-15-0822-a7f3",
  "status": "verified",
  "polls": 3,
  "duration_ms": 18420,
  "message_claimed": true
}

If your team uses a free throwaway email for controlled test setup, give it the same lease and cleanup rules as any other test mailbox. The label is less important than the ownership boundary. Do not use a shared inbox just because it is quick; that shortcut become expensive when parallel jobs start.

When a run needs a disposable provider, document the provider contract in the adapter. A link such as tempmailso can be useful in a setup note, but it should never replace a clear policy for retention, access, and cleanup.

A practical checklist

Before calling an email workflow reliable, check that:

• every run has a unique ID;
• mailbox ownership and expiry are enforced;
• expected sender and subject are matched explicitly;
• a message can be claimed only once;
• polling has a deadline and a maximum count;
• receipts omit tokens and full message bodies;
• CI keeps the redacted receipt when a test fails.

The checklist look small, but it catches the boring failures that waste the most debugging time. Start with one workflow and add fields only when a failure shows why they are needed.

Conclusion

An AI email workflow does not need a complicated orchestration platform to become dependable. A run ID, an expiring mailbox lease, an atomic message claim, and a bounded budget already create a strong mental model.

The payoff is not only fewer flaky tests. Developers can see what the agent attempted, operators can distinguish a timeout from a rejected message, and future tools can reuse the same contract. That is the kind of automation I trust: helpful, inspectable, and just constrained enough to behave well when the inbox get weird.