CI timeouts
A pull request that changed one blog page turned the build red. The lint job it broke had found zero problems. Someone re-ran it, it passed, and everyone moved on.
Zero issues, and red anyway
The log said two things that should never appear together. First: 0 issues. Then: Timeout exceeded.
The linter had finished its work. It had read every file, found nothing wrong, and then died on the clock. It ran for 311 seconds against a budget of 300.
Eleven seconds over. On a pull request that touched one page of prose and no Go code at all.
Why re-running it made things worse
The obvious move is to press re-run. It passed. That felt like proof that the first failure was noise.
It was proof of the opposite. Two lines above the timeout, the log said: Cache not found for input keys.
The first run had a cold cache and had to do the full analysis from scratch. It also populated the cache. The second run read that cache and finished in a fraction of the time. Re-running did not clear a random glitch. It removed the only condition under which the job fails.
So the green run taught us nothing, and it hid the fact that the budget was too small.
Three questions that separate a flake from a costume
Ask them before you press re-run. They take a minute and they work for any job, in any language.
One: did the work finish? A job that produced its result and then failed is not flaky. Something outside the work killed it. Look for a completed output line above the error.
Two: what was different about this run? Cache hit or miss, cold container, a dependency fetched from the network, a first run after a weekend. Real flakes have no pattern. This one had a name printed in its own log.
Three: how close was it? A job that finishes in 40 percent of its budget is healthy. One that finishes at 90 percent is a scheduled outage. Print the duration and compare it to the limit, every run.
Find out how often the condition returns
This is the step that turns a shrug into a decision. Our cache configuration said the invalidation interval is seven days.
That means the first run after every weekly expiry is cold, and every one of those runs was over budget. Not a rare glitch. A weekly failure, hidden by the six warm days around it.
The rest of the week papers over it. That is exactly why it looked random: the failures are regular, the observations are not.
The cost of calling it a flake
A red check that says nothing about the code is not free. It teaches people that red sometimes means nothing.
After the third time, nobody reads the log. They re-run it. And the day a red check means something real, it gets the same treatment.
That is the actual damage, and it is done to your team, not to your pipeline.
What it took to fix
One line. The timeout went from 5 minutes to 15. The job still runs the same checks and still finds the same problems.
The comment above that line is longer than the change. It names the numbers: 311 seconds, 300 budget, cold cache, seven day expiry. The next person will not need to rediscover it.
That is the whole trade. One minute of reading the log instead of pressing re-run, and a class of failure disappears.
I build cachly — persistent memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your codebase: the bug you fixed, why you chose Postgres, the deploy step that always breaks — including what your teammates learned. And every assistant you use reads the same memory.
Stop re-running jobs and hoping — and give your assistant a memory that remembers which failures were real. Free tier, hosted in the EU: cachly.dev
Top comments (1)
Cache expiry bugs are hard because they create a calendar-shaped failure. I like making the cache age visible in the CI output, not just the hit/miss state. If the job depends on a seven-day cliff, the report should say how close you are to it.