My CI gate reported "passing" without ever checking anything

A red badge sat on my repo's README. Every check passed on my machine.

The cause took a few minutes to find. The interesting part was what the red badge had been hiding. Fixing it took four rounds, and each round exposed something the previous failure had covered up.

Round 1: one missing tool stopped more than 70 checks

The failing step was a fixture test. The script under test needs ripgrep, and it exits with a "blocked" code when rg isn't installed:

command -v rg >/dev/null 2>&1 || { echo "BLOCKED: rg (ripgrep) is not installed — no axis can be computed" >&2; exit 4; }

rg is on my machine and isn't on GitHub's ubuntu-latest runner, so the test could only fail in CI. The fix is one workflow step:

- name: Install ripgrep
  run: sudo apt-get update -qq && sudo apt-get install -y -qq ripgrep

The part I had missed: GitHub Actions stops a job at the first failed step. That failure was early in the job, so most checks had been skipped on every run for two weeks. The run page showed one failure, and it read like one small problem.

Round 2: the check that skipped and called it success

Installing rg turned a different gate red. This one runs every search command in my docs to prove it parses. It had this line at the top:

command -v rg >/dev/null 2>&1 || { echo "rg-parses: SKIPPED — ripgrep not installed"; exit 0; }

Exit 0. The gate had been in CI for a month and had never checked anything there. The only sign was one word in a log nobody reads.

Its first real run found three commands that fail on ripgrep 14, the version Ubuntu's apt installs:

• A nested glob, {application*.{yml,yaml},...}, which ripgrep 15 accepts and 14 rejects.
• --type vue in two files, which ripgrep 14 doesn't know.

These were bad in a quiet way. The commands exit with an error, and an agent reading the result sees no matches. Anyone on Ubuntu would have got "nothing found" from a check that never searched.

The fix was small:

rg -g '**/{application*.yml,application*.yaml}' ...
rg "pattern" --type-add 'vue:*.vue' --type vue --type ts ...

Round 3: my own fix broke a count

Now the job got further and failed on a gate that checks the number of steps in the workflow against the number in my contributing guide. I had added a step, so the guide was wrong by one.

That gate was doing its job. It was also the third failure in a row that had been invisible until the one before it was fixed. So I stopped pushing and guessing, and ran the workflow's steps locally in one go.

Green, finally. Then, a few days later, a commit that only touched docs went red.

Round 4: a gate that depended on filesystem order

The failing command was one I hadn't changed, in a file nobody had touched:

rg -q 'pattern' "$f" "$B/$f"

The gate runs each command in an empty temp directory so that nothing real gets searched. It replaced arrays with . and left plain variables unset. Look at what that does here:

"$B/$f"   # with both unset, this is just /

So the "empty directory" check was searching the entire filesystem. It stopped when it hit a match, and if it didn't hit one in 20 seconds, the gate failed. Whether it passed came down to the order the walk happened to visit files.

The fix was to bind every variable to .:

names = set(re.findall(r"\$\{?([A-Za-z_][A-Za-z0-9_]*)", seg)) - {"PATH"}
env = {**os.environ, **{name: "." for name in names}}

PATH is excluded because bash still has to find rg.

What I changed in how I work

A skip should be visible. If a check can't run, that's a different result from passing. In CI, a missing tool should fail.

After fixing a red build, run everything. The first failure hides the rest. A local script that runs every step and collects all the failures is slower than one push, but it shows them all at once.

Unset variables are values. "$A/$B" is / when both are empty. That's easy to miss in any script that runs commands from somewhere else.

Test the check itself. I now keep a known-good and a known-bad fixture for these gates. The new good fixture contains a search that can't match anything, so if the variable bug returns, the probe walks the whole disk and times out.

This happened in Refract, an open-source repo of mine. This article was created with the help of AI, and I checked it against the actual commits.