A failed CI job gives you one fact:
the job failed.
It does not tell you whether running it again is justified.
A runner may have disappeared.
A dependency or external service may have returned a transient error.
Or the exact same failure may simply happen again on the next attempt.
Yet these very different situations often lead to the same operational response:
Rerun it.
That is the problem I've been exploring with CI Retry Gate.
Evidence before retry
Instead of treating failure itself as sufficient reason for another attempt, CI Retry Gate asks a different question:
What evidence do we have that a retry is justified?
The current decision model can use signals such as:
- failure-cause evidence when it is actually available
- recovery of the same job identity across attempts
- recurrence of that identity on the next attempt
- prior machine-verified recovery history
These signals are deliberately kept separate.
For example, a workflow eventually becoming green does not erase the fact that the same job failed again on an intermediate attempt.
Likewise, historical recovery is supporting evidence — not automatic authorization for another retry.
When the available evidence is insufficient or contradictory, the gate fails closed.
Testing against real CI failures
I didn't want the decision model to exist only as a synthetic test suite.
I tested it against public GitHub Actions failures and found both sides of the problem.
In some cases, failed job identities recovered on the next attempt.
In others, one job recovered while another repeated the same failure.
That distinction matters.
"Eventually succeeded" and "the retry was supported by evidence" are not equivalent statements.
Public GitHub logs can also be unavailable. When that happens, CI Retry Gate does not manufacture a failure cause from incomplete information.
The missing evidence remains missing.
External consumer verification
For v1.0.2, the release candidate was also exercised from a separate consumer repository rather than relying only on tests inside the project itself.
The external verification covered:
- the root GitHub Action
- pytest
- Jest
- Vitest
- Setup Doctor
- ownership/JUnit fixtures
The release candidate passed that external consumer gate before the stable v1 line was updated.
Try it without installing it
If you have a failed public GitHub Actions run, you can test the analysis before installing the Action.
Repository:
https://github.com/achirothmane/workflow-failure-lab
Current release:
https://github.com/achirothmane/workflow-failure-lab/releases/tag/v1.0.2
The question I'm trying to answer now is no longer whether the mechanism can be built.
It's whether this evidence actually improves the operational decision.
If you maintain CI workflows, I'd like to know:
What evidence do you currently require before retrying a failed job?
Top comments (3)
I retry when the log points at the runner or the network: the job vanished, a registry returned 5xx, the checkout connection reset. An assertion or a type error gets read, not clicked again. If the same job comes back red with the same assertion, I treat that as the result.
That distinction is exactly what I was hoping to compare against real operator behavior.
Runner/network failures vs. deterministic assertion/type errors is a useful first boundary. The harder case for me is what happens after that first classification: if a job that looked transient is retried and the same job identity fails again, I treat that recurrence as evidence against another retry rather than letting the original “transient” classification keep authorizing attempts.
Do you use any retry limit or historical failure/recovery pattern beyond that, or is the current log plus the next attempt usually enough for you?
The dimension I would add to the evidence model is whether the job is safe to re-run at all. A transient failure in a pure test job is one thing, but retrying a job that publishes artifacts, runs migrations, or deploys can double-apply side effects even when the retry itself is justified by the logs. Evidence that the failure was transient is not the same as evidence that a rerun is harmless. Do you model job safety or idempotency anywhere in the gate, or is that deliberately out of scope?