GitHub Actions Need Inbox Replay Data

When a workflow says "email not received," I do not add another retry first. I add replay data. A small inbox timeline makes flaky API checks faster to debug and much easier to trust.

Why replay data matters more than another retry

One pattern keeps showing up in GitHub Actions pipelines: the send step looks healthy, the app returns success, and the inbox assertion still fails with almost no context. Teams usually patch that with longer sleeps. I used to do that too, and it sort of worked, until it didnt.

The real problem is that most workflows preserve the final verdict but not the path that led there. If the mailbox poller checked five times, matched two near-miss subjects, and saw one stale recipient, that sequence matters more than a generic timeout error. Once I started storing that sequence, triage got shorter and less argue-y.

This is close in spirit to freezing email test plans before runs and building safer signup email checks. Both ideas push toward the same outcome: make the workflow explain itself while it is still fresh.

The tiny event model I keep in every workflow

I do not want a giant observability project inside CI. I just want enough state to replay what happened. For email-driven APIs, that usually means one JSON lines file with four event types:

• requested
• polled
• matched
• verdict

Each event gets a timestamp, scenario id, recipient, and one or two useful fields. Thats it. The file stays small, diffable, and easy to inspect in an artifact download.

Here is the kind of shape I mean:

{"type":"requested","scenario":"invite-204","recipient":"[e-mail]","subject":"You're invited"}
{"type":"polled","scenario":"invite-204","attempt":1,"messages_seen":0}
{"type":"matched","scenario":"invite-204","message_id":"msg_481","subject":"You're invited"}
{"type":"verdict","scenario":"invite-204","status":"passed"}

This sounds basic, but the payoff is realy good. A developer can open one file and understand whether the issue was send timing, recipient mismatch, or an assertion that drifted from the template.

A GitHub Actions job layout that stays debuggable

The setup I like is boring on purpose. Boring wins in CI.

- name: Create inbox run envelope
  run: |
    mkdir -p .tmp/inbox
    cat > .tmp/inbox/context.json <<'JSON'
    {
      "scenario": "${{ matrix.scenario }}",
      "recipient": "${{ env.TEST_RECIPIENT }}",
      "startedAt": "${{ github.run_id }}"
    }
    JSON
    : > .tmp/inbox/replay.jsonl

- name: Append requested event
  run: node scripts/email-log.js requested .tmp/inbox/context.json .tmp/inbox/replay.jsonl

- name: Poll inbox
  run: node scripts/poll-inbox.js .tmp/inbox/context.json .tmp/inbox/replay.jsonl

- name: Upload replay artifact
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: inbox-replay-${{ github.run_id }}-${{ matrix.scenario }}
    path: .tmp/inbox

Three habits make this hold up better than ad-hoc logs.

First, every script appends to the same replay file instead of inventing its own format. Second, the artifact name maps cleanly to one scenario. Third, the summary points people to the artifact instead of dumping everything into step output, which gets noisy fast and is annoying to scan when youre half awake.

If you support many sandbox signups or review apps, this is also where a throwaway email flow can help. I treat it as test plumbing, not the story itself. For example, a tempmail disposable inbox is handy when I need a short-lived recipient for isolated workflow runs, but the bigger productivity win still comes from the replay trail around it.

Where throwaway email fits without taking over the post

I have seen teams over-focus on the inbox provider and under-focus on workflow discipline. A provider switch can help, sure, but it does not fix missing evidence. If your logs cannot answer "what did this exact run observe and when?", the next provider will inherit the same fog.

My checklist is simple:

1. One scenario should map to one recipient.
2. Poll attempts should be written as events, not hidden in console noise.
3. Near matches should be captured before the final fail.
4. The artifact name should be unique enough to fetch in seconds.
5. Strange copied inputs like temp gamil com should be visible as test data, not mistaken for infrastructure bugs.

When that discipline is in place, even a plain tempmailso link becomes contextual instead of spammy, because it supports a real workflow choice rather than trying to carry the whole article.

Quick Q&A

Should I keep replay artifacts for successful runs?

Yes. Passing runs are your baseline. When a flaky case appears, good baseline artifacts save a lot of guesswork, and thats usualy the cheapest debug acceleration you can buy.

Is JSONL better than one big JSON file?

For CI, yes. Appending is simpler, partial writes are easier to reason about, and local inspection with standard tools is nicer. You can rg or jq through it without much fuss.

What is the smallest useful version?

Start with requested, polled, and verdict. Add matched when you need to explain why a message was close but not accepted. Keep it lean, keep it readable, and resist the urge to turn the workflow into a mini data lake.

The main shift here is not technical genius. It is deciding that inbox evidence deserves first-class treatment in your developer tools. Once you do that, GitHub Actions stops feeling random, and API-related email tests become a lot less spooky.