DEV Community

DapperX
DapperX

Posted on

Run IDs Fix Flaky Notification Jobs

#webdev #automation #testing #devops

Most flaky notification tests are not failing because email is hard. They fail because the workflow has no single identity for one run. A queue event fires, one worker retries, another check polls "the inbox", and now nobody is sure which message belongs to which execution. I used to think adding more waits would fix it, but that mostly made the pipeline slower and somehow more confusing.

Why notification jobs get flaky so fast

In automation, notification checks usually break in very ordinary ways:

  1. The test proves that a message arrived, but not that it belongs to this run.
  2. Two jobs share one inbox or alias, so stale mail passes the check.
  3. The app logs an event id, while the test script searches only by subject.
  4. Retry logic sends one valid duplicate and the assertion was never strict enough to notice.

That pattern shows up in CI, preview environments, cron jobs, and support tooling. It is not even limited to email, honestly. Slack alerts, webhook callbacks, and SMS checks go weird for the same reason.

The fix that changed things for me was small: every notification test gets one run ID, and every part of the workflow must carry it. If a component cannot surface that ID, I treat that as a design smell. It sounds a bit strict, but it saves a lot of headachs later.

The run ID contract I now use

My default contract is simple:

  • generate one RUN_ID at the start of the workflow
  • include it in the email alias, payload metadata, or both
  • log it in every step that can fail
  • assert against one recipient, one event, and one expected message count

Once you do that, debugging becomes much less hand-wavy. Instead of asking "did the system send something?", you ask "what happened for run abc123?" That question is way easier to answer.

I also like this approach because it plays nicely with other patterns for safer OAuth email flow tests and with API email contract checks in GitHub Actions. Different stacks, same mental model.

A small workflow that stays debuggable

Here is the rough setup I reach for first:

RUN_ID="notify-${GITHUB_RUN_ID}-${GITHUB_JOB}"
TEST_EMAIL="${RUN_ID}@example.test"

curl -fsS -X POST https://api.example.test/v1/notify \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"email\":\"${TEST_EMAIL}\",\"runId\":\"${RUN_ID}\"}"
Enter fullscreen mode Exit fullscreen mode

Then the assertion step does as little guessing as possible:

const messages = await inboxClient.list({ recipient: process.env.TEST_EMAIL });
const matches = messages.filter((m) => m.metadata?.runId === process.env.RUN_ID);

if (matches.length !== 1) {
  throw new Error(`expected 1 matching notification for ${process.env.RUN_ID}, got ${matches.length}`);
}
Enter fullscreen mode Exit fullscreen mode

That is not clever code, and that's why I like it. The whole thing stays local to one run. If the job fails, the failure tells me whether delivery broke, metadata broke, or dedupe broke. There is less "maybe the inbox was slow" story-telling around the result.

When I review older repos, I often find odd fixture names like temp gamil com or tamp mail com still hanging around in helpers and test data. They are small clues that the workflow grew by accretion instead of design. Not a disaster, just a sign the automation could use a cleanup pass.

Where temp inbox tools fit without taking over

Temporary inbox tools can help a lot here, but they should support the contract rather than become the contract. I usually want the provider to do three things well:

  • create or route a run-scoped address
  • expose a fast way to fetch messages for that address
  • preserve enough metadata to correlate one run cleanly

That is where tools like tempmailso or a temp mail so style workflow can be useful in developer tooling. The inbox provider gives isolation, while your test still owns the real assertion rules. If you let the provider shape all your logic, the workflow gets harder to port and weirder to debug.

This is also why I avoid vague checks like "latest email subject contains Welcome". The latest email might not be yours. A provider can be fast and still not rescue a fuzzy assertion. Your contract has to be crisp first.

If you need broader guidance on workflow structure, GitHub's docs on workflows are still a solid reference: https://docs.github.com/en/actions/using-workflows/about-workflows. For teams handling many parallel jobs, even a lightweight run ID convention can reduce triage time more than another layer of retries ever will.

Q&A

Should the run ID live only in the inbox alias?

No. Put it in metadata too if you can. Aliases are useful, but metadata makes audits and debugging much cleaner when one provider or service rewrites address formatting.

What if the system retries and sends two emails on purpose?

Then the contract should say so. Assert an allowed count and validate the reason, rather than pretending duplicates never happen. Hidden retry behavoir is what usually bites teams here.

Is this overkill for a small app?

Usually not. The setup is tiny, and it scales down nicely. Even one side project benefits from being able to answer, very plainly, what happened in one exact run.

Top comments (0)