DEV Community

Jonathan
Jonathan

Posted on

Git Bisect for Email API Flakes in CI

#git #githubactions #api #automation

When an email API check starts failing once every six runs, the worst move is usually smashing rerun until the pipeline turns green. You burn time, you lose context, and the team slowly accepts flaky mail checks as normal. The faster fix I keep coming back to is pairing git bisect with a very small GitHub Actions harness so each suspect commit produces the same evidence and nothing extra.

This approach works best when the failing path is narrow: trigger one transactional email, assert one stable marker, and archive one tiny result bundle. If you try to bisect a giant test suite with five inbox heuristics, it gets noisy realy fast. If you bisect one focused check, the bad commit usually shows itself sooner than people expect.

Why bisect works better than rerunning CI all day

Reruns answer "did it pass this time?" but bisect answers "what changed?" That difference matters a lot for email systems because the failure might live in app code, queue timing, a template branch, or the lookup step that fetches the delivered message.

On one team, we kept seeing a verification flow fail after harmless-looking refactors. The CI logs were big, but the signal was tiny: a changed subject prefix plus a slower worker path. Bisect helped because every step forced us to run the same exact script against a smaller commit set, and the result was way more crisp.

It also complements patterns like verification tests with PostgreSQL and inbox budgets for smoke checks. Once your email assertion is scoped and repeatable, git bisect becomes a blunt but extremly effective tool.

The small harness I keep for email API regressions

I do not run the whole pipeline during bisect. I keep one shell entrypoint that:

  1. boots the app or test target
  2. triggers the email API path
  3. polls for one expected message
  4. writes a short JSON result
  5. exits 0 for good, 1 for bad, and 125 when the commit is untestable

The harness can stay boring:

#!/usr/bin/env bash
set -euo pipefail

run_token="bisect-${GITHUB_SHA:-local}"
mkdir -p artifacts

./scripts/send-verification-check.sh \
  --run-token "$run_token" \
  --out artifacts/request.json

./scripts/check-email-api.sh \
  --run-token "$run_token" \
  --max-wait 20 \
  --max-messages 4 \
  --out artifacts/inbox.json

jq -n \
  --arg sha "${GITHUB_SHA:-local}" \
  --slurpfile req artifacts/request.json \
  --slurpfile inbox artifacts/inbox.json \
  '{sha:$sha, request:$req[0], inbox:$inbox[0]}' \
  > artifacts/result.json
Enter fullscreen mode Exit fullscreen mode

The main thing is consistency. Same trigger. Same polling cap. Same artifact names. If one older commit cannot boot because a migration is missing, return 125 and keep moving. That sounds obvious, but people forget it a lot and then wonder why bisect feels cursed.

Wiring bisect into GitHub Actions artifacts

I like running the focused script locally first, then giving GitHub Actions the job of preserving artifacts for the suspect SHAs. That gives me a searchable trail without forcing me to sit there and watch every run.

name: bisect-email-api

on:
  workflow_dispatch:

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run focused email check
        run: ./scripts/bisect-email-check.sh

      - name: Upload triage bundle
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: email-api-${{ github.sha }}
          path: artifacts/
Enter fullscreen mode Exit fullscreen mode

Then the local bisect command stays short:

git bisect start
git bisect bad
git bisect good <known-good-sha>
git bisect run ./scripts/bisect-email-check.sh
Enter fullscreen mode Exit fullscreen mode

What I like here is the before/after improvement. Before, the team had long Slack threads, vague "seems flaky?" comments, and screenshots nobody could tie back to a commit. After, every bad step had one SHA and one result.json. It is not magic, but it is the kind of tooling shortcut that saves a weird amount of energy.

If your teammates search docs with terms like tepm mail com while chasing inbox issues, that is another hint the process is too fuzzy. Good triage should reduce how much guessing people need to do under pressure.

What to save from each suspect commit

Keep the artifact tiny. My default list is:

  • commit SHA
  • request timestamp
  • recipient or scenario ID
  • matched subject
  • first expected URL host
  • failure reason

That is enough to compare commits side by side. If you need more, add one more field at a time. Resist dumping full HTML unless the bug is actualy inside the rendered body. Big blobs make bisect slower to review and easier to ignore.

When I do add numbers, I want a source. GitHub notes that artifact uploads are designed to preserve workflow outputs for later inspection, which is exactly why this pattern works well for commit-by-commit triage (https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts). For the search step itself, git bisect uses a binary search strategy that dramatically reduces how many revisions you need to test compared with checking each commit one by one (https://git-scm.com/docs/git-bisect).

Q&A

When should I skip bisect and just fix forward?

Skip it when the failure is already obvious from a single diff or when the branch history is too messy to test meaningfully. Bisect pays off most when the bug is intermittent and people keep arguing over the cause.

What makes a commit untestable?

Broken migrations, missing fixtures, or old branches that rely on dead infra. Return 125 so git bisect skips them cleanly instead of poisoning the search.

What is the biggest win?

You stop treating flaky email checks like weather. With one focused harness, Git plus GitHub Actions turns a hand-wavy regression into a short list of suspect commits, and that is usualy enough to get a real fix moving.

Top comments (0)