CI timeouts
A pull request that changed one blog page turned the build red. The lint job it broke had found zero problems. Someone re-ran it, it passed, and everyone moved on.
Zero issues, and red anyway
The log said two things that should never appear together. First: 0 issues. Then: Timeout exceeded.
The linter had finished its work. It had read every file, found nothing wrong, and then died on the clock. It ran for 311 seconds against a budget of 300.
Eleven seconds over. On a pull request that touched one page of prose and no Go code at all.
Why re-running it made things worse
The obvious move is to press re-run. It passed. That felt like proof that the first failure was noise.
It was proof of the opposite. Two lines above the timeout, the log said: Cache not found for input keys.
The first run had a cold cache and had to do the full analysis from scratch. It also populated the cache. The second run read that cache and finished in a fraction of the time. Re-running did not clear a random glitch. It removed the only condition under which the job fails.
So the green run taught us nothing, and it hid the fact that the budget was too small.
Three questions that separate a flake from a costume
Ask them before you press re-run. They take a minute and they work for any job, in any language.
One: did the work finish? A job that produced its result and then failed is not flaky. Something outside the work killed it. Look for a completed output line above the error.
Two: what was different about this run? Cache hit or miss, cold container, a dependency fetched from the network, a first run after a weekend. Real flakes have no pattern. This one had a name printed in its own log.
Three: how close was it? A job that finishes in 40 percent of its budget is healthy. One that finishes at 90 percent is a scheduled outage. Print the duration and compare it to the limit, every run.
Find out how often the condition returns
This is the step that turns a shrug into a decision. Our cache configuration said the invalidation interval is seven days.
That means the first run after every weekly expiry is cold, and every one of those runs was over budget. Not a rare glitch. A weekly failure, hidden by the six warm days around it.
The rest of the week papers over it. That is exactly why it looked random: the failures are regular, the observations are not.
The cost of calling it a flake
A red check that says nothing about the code is not free. It teaches people that red sometimes means nothing.
After the third time, nobody reads the log. They re-run it. And the day a red check means something real, it gets the same treatment.
That is the actual damage, and it is done to your team, not to your pipeline.
What it took to fix
One line. The timeout went from 5 minutes to 15. The job still runs the same checks and still finds the same problems.
The comment above that line is longer than the change. It names the numbers: 311 seconds, 300 budget, cold cache, seven day expiry. The next person will not need to rediscover it.
That is the whole trade. One minute of reading the log instead of pressing re-run, and a class of failure disappears.
I build cachly — persistent memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your codebase: the bug you fixed, why you chose Postgres, the deploy step that always breaks — including what your teammates learned. And every assistant you use reads the same memory.
Stop re-running jobs and hoping — and give your assistant a memory that remembers which failures were real. Free tier, hosted in the EU: cachly.dev
Top comments (4)
Cache expiry bugs are hard because they create a calendar-shaped failure. I like making the cache age visible in the CI output, not just the hit/miss state. If the job depends on a seven-day cliff, the report should say how close you are to it.
"Calendar-shaped failure" is the right name for it — the bug is already there, it just isn't due yet.
What fixed this shape for us was making every check report three states instead of two: reached / not reached / stale. Binary hit-miss can't express "this is technically fine and will break on Tuesday", so it reads green right up until it doesn't.
The self-own that got us there: we built the staleness check because we believed a published package was four versions behind. It wasn't — we'd misread an unsorted API response and taken the first entry for the latest one. So the first thing our new staleness check caught was our own bad measurement. Worth it anyway, because the shape was real even though the instance wasn't.
A neighbouring one, same family: a build that pulled a font from a CDN on every run. Nothing expired, nothing was cached wrong — the dependency was simply invisible in the report. Three red runs in one afternoon before anyone asked "does this build fetch anything from the network?"
The rule I'd add to yours: any check that reports state but not age is one quiet refactor away from lying. And it generalises past caches — we ended up putting a decay clock on stored knowledge for the same reason, so an entry that hasn't been used in ten days is marked as aging rather than silently trusted.
“Reached / not reached / stale” is a strong upgrade because it gives the system vocabulary for future failure. A binary cache check cannot express “green now, broken on Tuesday,” which is exactly the shape teams miss.
That vocabulary point is the part I underestimated. The third state only helps if it has a name people say out loud — we called ours "stale" and the arguments about what counts as stale turned out to be more useful than the check itself.
One thing I got wrong since: I shipped a guard for exactly this shape, and two days later a different daily message had been missing for almost two days without anything going red. The guard read the repo; the failure was on the machine. Reading state is not the same as watching it run.