Your CI failed. Was it a flake or a real break?

Last week I watched a team re-run a red CI job three times, merge on the third green run, and ship the bug anyway. The test was not flaky. The retry just hid it for one run.

The opposite happens just as often: a genuinely flaky test blocks a good PR for an afternoon while someone scrolls a 40,000-line log looking for the first real error.

I built a small step that answers three questions before anyone re-runs anything:

1. Where did it actually break first?
2. Is each failing test flaky, or genuinely broken?
3. Has this exact failure happened before?

It is called CI Failure Detective, and it runs entirely inside your runner. No log ever leaves your repo. No signup, no server, no bill.

How it works

Add one step after your test command:

- name: Run tests
  run: npm test 2>&1 | tee ci.log

- name: Diagnose failures
  if: failure()
  uses: techdevfun/ci-failure-detective@v1
  with:
    log-file: ci.log

It posts a diagnosis to the job summary: the first error line with context, a verdict per failing test with a confidence score and a reason, a stable 12-character signature for the failure, and suggested next steps. Suggestions only — it never quarantines, retries, or edits anything by itself. Automatically silencing failures hides real bugs, so that part stays human.

As a CLI it works the same way, with zero dependencies (Node.js 18+):

npx ci-failure-detective analyze ci.log
cifd analyze ci.log --format md >> $GITHUB_STEP_SUMMARY

Flaky detection needs history, so you commit a small JSONL file and append each run:

cifd record --history .cifd-history.jsonl --commit $GITHUB_SHA < ci.log
cifd history   # show failure clusters

The rules are deliberately boring and deterministic:

• A test that fails and passes on the same commit is flaky, confidence 0.95.
• A test that flips frequently across recent runs is flaky, 0.70.
• A test that fails consistently on a commit where it used to pass is a real regression, 0.80.
• No history at all? The verdict is unknown, with a reason. It never guesses.

Same-commit verdicts need per-test pass records, so run tests verbose (jest --verbose, pytest -v). Plain logs only record failures, so same-commit evidence is unavailable from them — flip-rate and regression verdicts still work.

What I verified before publishing (2026-10-24)

• Test suite: 46/46 passing locally; CI green on Node 20.x and 22.x.
• Six fixture logs classify correctly: Jest, pytest, go test, Mocha, npm install failure, and tsc failure.
• A real failing node:test run gets the right category (test failure) and the right first error line.
• Same-commit flaky detection works end to end through the real CLI: record a passing run, then analyze a failing run on the same commit, and it reports flaky at 0.95.
• Live GitHub Action run on a real pull request: I ran the composite action on a private test repo's PR with a deliberately failing test. Both action invocations succeeded — one on the real failing log, one on a Jest fixture log — and a verify step asserted both produced a non-empty verdict and a 12-hex-character signature (real-log signature 41968f39ccc7, fixture signature 4f709b012d30). The run itself stayed red by design, because the test step failed. That is the point of a failure-diagnosis tool.
• Parsers today: Jest, pytest, go test, Mocha. Failure classification and first-error extraction work on any raw log, even without a test framework.

How this compares (checked 2026-10-24)

Tool	What it does	What it costs you
BuildPulse ($99/month Startup) / Trunk / Datadog / Mergify	Flaky detection, dashboards, trend, auto-quarantine, alerts	You upload all test results and logs to their cloud
gh-failmap (OSS)	Clusters failed runs by shared error line	CLI only, needs Python + gh login, GitHub only, no flaky verdicts
flakestat (OSS, zero-dependency)	Local-first flaky hunting	Only hunts flakes via re-runs or JUnit XML; no log reading, no clustering, no Action
CI Failure Detective	First error + flaky-vs-real + clustering, in the runner, zero dependencies	No dashboard, no alerting, no auto-apply

Where rivals beat us (honestly)

BuildPulse and friends have dashboards, long-term trend analytics, auto-quarantine, multi-channel alerting, and AI grouping — we have none of that in v1. gh-failmap's clustering is a focused, proven approach; we have not run a head-to-head cluster comparison against it yet, so we claim no advantage there. flakestat's JUnit XML input is something we do not have yet.

CI Failure Detective is not a replacement for those platforms. It is a specialized triage layer for teams that cannot or will not send logs out, or that want a deterministic answer inside the runner before deciding whether a retry is even sensible.

Honest limits

• No native node:test parser yet: a real node:test log gets the right category and first error, but its tests are not listed individually.
• No JUnit XML input yet, no dashboard, no alerting, no auto-quarantine apply, no SARIF output.
• History is a JSONL file you own and rotate yourself; there is no automatic rotation yet.
• If your log contains a secret, that secret is in the log you already have; the summary shows to the same audience that can see the log. Redaction is on the roadmap.

Roadmap (not in v1)

JUnit XML as an additional input, Slack webhook output, more frameworks (PHPUnit, RSpec, .NET), SARIF output. Quarantine stays suggest-only by design.

The repo is MIT, the Action is techdevfun/ci-failure-detective@v1, and the README has the full input/output reference. If your team re-runs red jobs on hope, try the step that tells you whether hope was justified.