DEV Community: Heinrich Neb

Your CI is not flaky. Your cache expires every seven days.

Heinrich Neb — Wed, 12 Aug 2026 20:48:49 +0000

CI timeouts

A pull request that changed one blog page turned the build red. The lint job it broke had found zero problems. Someone re-ran it, it passed, and everyone moved on.

Zero issues, and red anyway

The log said two things that should never appear together. First: 0 issues. Then: Timeout exceeded.

The linter had finished its work. It had read every file, found nothing wrong, and then died on the clock. It ran for 311 seconds against a budget of 300.

Eleven seconds over. On a pull request that touched one page of prose and no Go code at all.

Why re-running it made things worse

The obvious move is to press re-run. It passed. That felt like proof that the first failure was noise.

It was proof of the opposite. Two lines above the timeout, the log said: Cache not found for input keys.

The first run had a cold cache and had to do the full analysis from scratch. It also populated the cache. The second run read that cache and finished in a fraction of the time. Re-running did not clear a random glitch. It removed the only condition under which the job fails.

So the green run taught us nothing, and it hid the fact that the budget was too small.

Three questions that separate a flake from a costume

Ask them before you press re-run. They take a minute and they work for any job, in any language.

One: did the work finish? A job that produced its result and then failed is not flaky. Something outside the work killed it. Look for a completed output line above the error.

Two: what was different about this run? Cache hit or miss, cold container, a dependency fetched from the network, a first run after a weekend. Real flakes have no pattern. This one had a name printed in its own log.

Three: how close was it? A job that finishes in 40 percent of its budget is healthy. One that finishes at 90 percent is a scheduled outage. Print the duration and compare it to the limit, every run.

Find out how often the condition returns

This is the step that turns a shrug into a decision. Our cache configuration said the invalidation interval is seven days.

That means the first run after every weekly expiry is cold, and every one of those runs was over budget. Not a rare glitch. A weekly failure, hidden by the six warm days around it.

The rest of the week papers over it. That is exactly why it looked random: the failures are regular, the observations are not.

The cost of calling it a flake

A red check that says nothing about the code is not free. It teaches people that red sometimes means nothing.

After the third time, nobody reads the log. They re-run it. And the day a red check means something real, it gets the same treatment.

That is the actual damage, and it is done to your team, not to your pipeline.

What it took to fix

One line. The timeout went from 5 minutes to 15. The job still runs the same checks and still finds the same problems.

The comment above that line is longer than the change. It names the numbers: 311 seconds, 300 budget, cold cache, seven day expiry. The next person will not need to rediscover it.

That is the whole trade. One minute of reading the log instead of pressing re-run, and a class of failure disappears.

I build cachly — persistent memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your codebase: the bug you fixed, why you chose Postgres, the deploy step that always breaks — including what your teammates learned. And every assistant you use reads the same memory.

Stop re-running jobs and hoping — and give your assistant a memory that remembers which failures were real. Free tier, hosted in the EU: cachly.dev

Your publish pipeline is green. Nobody can install your plugin.

Heinrich Neb — Wed, 12 Aug 2026 12:59:35 +0000

Silent failure

For three weeks nobody could install your plugin. The publish job was green every single day. You find out when a user asks why the version is so old.

Three weeks green, zero installs

This happened to us. The publish job for our JetBrains plugin reported success on every run since the middle of July. The plugin was not in the marketplace at all.

The registry API answered with a 404. A search for the product name returned nothing. Meanwhile the build was green, the release notes were written, and the changelog was up to date.

Nobody noticed. Not the pipeline, not the dashboard, not us. The gap between the last good release and the discovery was three weeks.

Your pipeline is not lying to you

This is the part worth understanding, because it is why the same thing is probably waiting in your repository too.

A release pipeline has one job: get the artifact somewhere a stranger can install it. Almost every pipeline checks something else. It checks that the upload command exited zero.

Those two questions agree nearly always. Our publish step went further and did the sensible thing. It caught the failure, compared the error text against a list of known-harmless cases, and exited zero for those.

One of those cases was pending moderation. A new version sits in review before it becomes visible. Failing the build for that would be noise, so it was allowed through.

Here is the trap. A harmless transient state and a permanent block produce the same message. Once the plugin was stuck, every later run matched the same friendly pattern and reported success. The pipeline answered its question correctly. It was the wrong question.

The check that catches it, in about two minutes

Ask the store, not the pipeline. That is the whole idea, and you can add it today without changing anything else.

One: after publishing, fetch the public listing the way a stranger would. No credentials, no internal API, no authenticated client. Seeing what an outsider sees is the entire point.

Two: read the version from that response and compare it to the version in your manifest. Not to the version you just built, and not to a variable in the job. To the file that is checked in.

Three: print one line per place you publish to, and say which one you could not reach. Reached, not reached, or stale. Never a bare zero.

That last part matters more than it looks. A source that errors out and gets counted as zero results looks exactly like a quiet day.

Make it fail on purpose, before you trust it

A watcher nobody has seen go red is not a watcher. It is a hope with a log file.

Give it a fixture where the listing is stale and assert that it exits non-zero. Run it once and look at the red. It takes a few minutes and it is the only thing that separates a real check from a comfortable one.

We skipped this step once on a different watcher. It sat there for weeks reporting nothing, and reporting nothing was indistinguishable from everything being fine.

Once you see this shape, you see it everywhere

A cache file written into a workspace that the next run wipes clean. A daily report whose delta is computed against a seeded example row. A source that returns an HTTP error and gets counted as zero.

Every one of them is green. Every one of them delivers nothing.

The missing ingredient is always the same: a third state. Systems model success and failure and forget not measured. Without it, an outage is indistinguishable from a quiet day, and people act on the number anyway.

What changes for you

Before: you find out from a user, three weeks late. After: you find out the next morning, from a check you wrote in two minutes.

It also catches the one case no exit code can reach. Sometimes the upload really did work and the store still shows the old version. Your pipeline has no way to see that. A stranger looking at the page does.

Ours is about forty lines. It opens three public pages, one for each store we publish to. It reads the version shown on each page and compares it to the version number in the file we ship. No login, no API key, no access to anything private.

The day we switched it on, it went red. It had been red for three weeks and nobody knew.

Stop finding out from your users — give your assistant a memory that survives the session. Free tier, hosted in the EU: cachly.dev