A failed CI job gives you one fact:
the job failed.
It does not tell you whether running it again is justified.
A runner may have disappeared.
A dependency or external service may have returned a transient error.
Or the exact same failure may simply happen again on the next attempt.
Yet these very different situations often lead to the same operational response:
Rerun it.
That is the problem I've been exploring with CI Retry Gate.
Evidence before retry
Instead of treating failure itself as sufficient reason for another attempt, CI Retry Gate asks a different question:
What evidence do we have that a retry is justified?
The current decision model can use signals such as:
- failure-cause evidence when it is actually available
- recovery of the same job identity across attempts
- recurrence of that identity on the next attempt
- prior machine-verified recovery history
These signals are deliberately kept separate.
For example, a workflow eventually becoming green does not erase the fact that the same job failed again on an intermediate attempt.
Likewise, historical recovery is supporting evidence — not automatic authorization for another retry.
When the available evidence is insufficient or contradictory, the gate fails closed.
Testing against real CI failures
I didn't want the decision model to exist only as a synthetic test suite.
I tested it against public GitHub Actions failures and found both sides of the problem.
In some cases, failed job identities recovered on the next attempt.
In others, one job recovered while another repeated the same failure.
That distinction matters.
"Eventually succeeded" and "the retry was supported by evidence" are not equivalent statements.
Public GitHub logs can also be unavailable. When that happens, CI Retry Gate does not manufacture a failure cause from incomplete information.
The missing evidence remains missing.
External consumer verification
For v1.0.2, the release candidate was also exercised from a separate consumer repository rather than relying only on tests inside the project itself.
The external verification covered:
- the root GitHub Action
- pytest
- Jest
- Vitest
- Setup Doctor
- ownership/JUnit fixtures
The release candidate passed that external consumer gate before the stable v1 line was updated.
Try it without installing it
If you have a failed public GitHub Actions run, you can test the analysis before installing the Action.
Repository:
https://github.com/achirothmane/workflow-failure-lab
Current release:
https://github.com/achirothmane/workflow-failure-lab/releases/tag/v1.0.2
The question I'm trying to answer now is no longer whether the mechanism can be built.
It's whether this evidence actually improves the operational decision.
If you maintain CI workflows, I'd like to know:
What evidence do you currently require before retrying a failed job?
Top comments (2)
I retry when the log points at the runner or the network: the job vanished, a registry returned 5xx, the checkout connection reset. An assertion or a type error gets read, not clicked again. If the same job comes back red with the same assertion, I treat that as the result.
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support