Quick take
Before you label a failing test flaky, write down the dates it failed. Flaky has no rhythm. Yours might.
"Flaky" is the most expensive word in CI. It closes the investigation. Nobody looks at a flaky test again until it fails on the day of a release.
Here is a cheap way to earn the label instead of assuming it. Pull the dates of the last failures and look at the gaps:
gh run list --workflow=ci.yml --status=failure \
--limit 50 --json createdAt \
--jq '.[].createdAt' | sort
Random failures scatter. If the gaps land on the same number over and over, something on a timer is doing this to you, and it is not the test.
Seven days is the number worth knowing by heart: GitHub Actions evicts a cache entry that has not been touched for a week. A dependency that is only ever restored, never rebuilt, disappears on a schedule — and the first run after that is slow, cold, and sometimes fails on a timeout that was always marginal.
That failure is not random. It has a calendar. Yours might be a certificate, a token, a log rotation or a nightly database refresh. Same shape, different clock.
The long version — how we found ours, and why the fix was not a longer timeout — is here: Your CI is not flaky. Your cache expires every seven days.
I build cachly — persistent memory for AI coding assistants, over MCP. Your assistant re-reads your codebase every morning. It does not have to.
Free tier, hosted in the EU: cachly.dev
Top comments (2)
Weekly failure patterns are a great example of why “flaky” is too vague. If the failure has a calendar rhythm, the system is probably telling you about token expiry, cache rotation, quota reset, or scheduled dependency drift, not randomness.
Your list is better than mine — token expiry, quota reset, scheduled dependency drift. I'd only had cache rotation and certificates.
One practical trap when people try this: gh run list won't take you back far enough. GitHub keeps workflow runs for 90 days by default, and a quarterly rhythm is invisible in that window. If you suspect something slower than a month, pull the dates out early and keep them somewhere — otherwise the evidence expires before the pattern does.
And the inverse deserves saying too: if the gaps really are scattered, "flaky" has been earned rather than assumed, and the word finally means something. My complaint isn't the label — it's reaching for it before looking at the dates.