One repo, one day: 4 CI guards that were green and blind

Guards that stopped guarding

Your test suite proves your code works. Nothing in your repository proves your checks still work.

A green check and a blind check look identical

We had a bad day. Not a dramatic one — nothing went down, no customer noticed. Just a day where four separate checks in the same repository were green, and all four had quietly stopped being able to fail.

That combination is worse than a broken check. A broken check is loud: the pipeline turns red, someone looks. A blind check is silent, and silence is indistinguishable from success. You keep merging, the dashboard stays green, and the thing the check was built to catch walks straight past it.

Here are all four, with the numbers, because the pattern only becomes obvious when you see them next to each other.

1. The timeout that could never fire

A lint job in GitHub Actions, configured like this:

jobs:
  lint:
    timeout-minutes: 15          # budget for the whole job
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-go@v6
      - run: golangci-lint run --timeout=15m   # budget for the tool

Both numbers are 15. Checkout, toolchain setup and module download all come out of the job's budget, so the tool's own timeout is unreachable — the job dies first.

That distinction is the whole point. A tool timeout fails with a sentence: timeout after 15m, package x. A job timeout kills the runner, and GitHub writes cancelled with no reason at all. Our deploy path was blocked for a full day across three commits, including one that only changed text, and not one of those runs said why.

The rule is one line: the job's limit must be larger than the limit of any tool inside it. Otherwise you have traded a diagnosable failure for a silent kill.

2. The fix that never reached the machine it was for

Months earlier we fixed line endings properly: * text=auto eol=lf in .gitattributes, verified, committed, done. It cost two days to find and it worked.

It came back. When we measured, it had never left: 1230 of 1455 text files in the working tree still had CRLF.

.gitattributes applies at checkout. A working tree created before the rule keeps its CRLF files forever, and nothing tells you — because git normalises when it compares:

git status         # "nothing to commit, working tree clean"
git diff           # empty
file config.ts     # ASCII text, with CRLF line terminators
prettier --check . # 287 files broken

In CI the same command reported two. Neither number is usable: believe the local one and you reformat 285 files nobody touched; ignore it and you miss the two that are real.

A fix that does not announce itself is not a fix for existing machines. We now have a check that says so out loud, and a one-liner that repairs it — no content change, no commit:

git rm --cached -r -q . && git reset --hard

3. The watcher that flagged its own comment

After fixing the timeout, we wrote a guard so it could not come back: read the workflow, and fail if any job's limit is not larger than the tool limits inside it.

It failed immediately. Not on the workflow — on the comment we had written above the fix, which quotes the old value --timeout=15m as part of explaining what went wrong.

The guard was scanning text and had no idea which lines were code. Two lines fixed it, and they are the interesting part:

for line in workflow.splitlines():
    if line.lstrip().startswith("#"):
        continue   # prose is not configuration

It is a small bug with an uncomfortable implication. A guard that reads text will eventually read the wrong text, and the failure mode is not "it misses things" — it is "it reports things that are not there", which trains everyone to ignore it. Then it misses things.

4. The report that counted us as customers

A bot message: 4 new company signups — high purchase intent. Four business email domains, worth reaching out to today.

One of them was provably our own laptop: our editor extension had started at 10:47:10 UTC and the account appeared at 10:47. The other three could not be attributed at all.

Two causes, both structural. Anonymous trial accounts get a generated address at <uuid>@trial.example.dev — our own domain — and the code that decides "is this a company?" only knew a list of free mail providers. Anything not on that list counted as a business.

And the signup handler recorded no origin at all. The clients had been sending an honest User-Agent for weeks. The server read it for abuse detection and threw it away.

So the report was not wrong about what it counted. It was counting something that could never answer the question it was asked.

The question that finds all four

None of these were found by running the checks. All four were green. They were found by asking a different question:

When did this check last say no?

That question separates "nothing is wrong" from "I cannot see anything", and those are the two states every dashboard renders identically. For a new guard it is cheap to answer — break the thing on purpose, watch it turn red, put it back:

# 1. break what the guard protects
sed -i 's/EXPECTED/WRONG/' config.yaml

# 2. the guard MUST fail here
npm run guard && echo "this guard is decoration" && exit 1

# 3. restore
git checkout config.yaml

Thirty seconds, once. If step 2 prints that line, you did not ship a guard — you shipped a green light with nothing behind it.

For guards that already exist, the answer is harder and more interesting: most repositories have no idea. There is no field anywhere recording when a check last rejected something. A guard that has been green for eight months is either protecting a very stable codebase or has quietly gone blind, and nothing in your tooling distinguishes those two.

We have not solved that part. The shape of a fix is obvious — one timestamp per guard, written every time it rejects something — but we have not built it, and I would rather say so than describe it as if we had. What we did do is start asking the question by hand, which is how all four of these turned up in a single day.

What this is really about

Every one of these four checks was written by someone competent, for a real risk, and worked on the day it was merged. Nothing rotted in the code. What changed was the ground underneath: a runner moved, a machine was older than a rule, a comment was added, a domain was generated.

Tests answer "does the code do the right thing?". Nothing in a normal repository answers "does the check still do its thing?" — and that second question has no owner, no runner, and no red light.

If you look for one thing after reading this: find the check in your pipeline that has been green the longest, and try to make it fail. You will learn something either way, and it takes about a minute.

---

I build cachly — persistent memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your codebase: the bug you fixed, why you chose Postgres, the deploy step that always breaks — including what your teammates learned. And every assistant you use reads the same memory.

Free tier, hosted in the EU: cachly.dev