My CI Was Red for 255 Commits and Nobody Looked

A CI job failed on every single push to my release branch. Including its very first run. For 255 commits.

Nobody filed it. Nobody bisected it. I merged to main with it red, because by then "detector-eval is red" was weather — a thing that was true, that had always been true, that didn't require an explanation.

It had never been green. Not once. And I didn't know that, because I'd never thought to ask.

This is from the agentwatch v0.2.1 release — the agent-security tool in agentic-suite. The receipts are in the v0.2.0 field test report and the release-readiness review.

The failure, verbatim

The workflow is small. It was added as part of the detector-credibility milestone — build trust in the anomaly detectors by running them over a versioned corpus in CI:

- name: Detector eval corpus v0 (DET-1)
  run: python scripts/detector_eval.py --json

Every push to the branch, and every PR, ran that. Every run ended like this:

FileNotFoundError: [Errno 2] No such file or directory:
'.../services/analytics/data/detector-corpus-v0.json'
Process completed with exit code 1.

The script has a default corpus path baked in: services/analytics/data/detector-corpus-v0.json. That file exists on my machine. It does not exist in a fresh clone.

One line of gitignore

The root .gitignore has this rule, added with the comment # Local dev data:

data/

Scratch files, local runs, throwaway output — a reasonable rule. And it matched services/analytics/data/ with the same two-line indifference it matched everything else. The corpus — the input to a CI gate — was excluded by a rule meant for junk.

The part that stings: the generator was sitting in the repo the entire time. scripts/generate_detector_corpus.py, one directory over from the workflow file. The corpus is a buildable artifact, not a hand-maintained one. Two fixes were available — generate it in the workflow, or commit it — and I had a third option, which was the one I actually took for 255 commits: nothing, because I never classified it as a problem.

Red on arrival

Here's the property that made this survivable for so long, and it's the property I want to name because I think it's the general failure:

The workflow's first-ever run was red.

Not red after something regressed. Red from the first commit it existed on, on every push, forever. Which means nobody ever saw it turn green, which means nobody ever saw it turn meaningfully red. There was no delta. No before-and-after. The check had no authority because a check's authority comes entirely from "this was green, and now it isn't."

I went back and counted. The branch carried 255 commits at merge, and the workflow existed for essentially all of them:

$ git rev-list --count 31100ba..a673eef
255

Every push in that window — the full field test, the detector hardening, the release prep — shipped with that red X on it.

It wasn't alone. security.yml was failing too, in zero seconds, with GitHub's terse "This run likely failed because of a workflow file issue." Two red checks on every push, both meaningless, both ignored. And this is the part I keep turning over: while both were red, a real regression in either would have been invisible. The noise didn't just cost attention. It cost the signal the checks existed to carry.

What noise like this actually trains

I ran the release-readiness review with a simple rule: enumerate every red on the branch and demand an explanation for each. Not "is it red," but "why is it red, and is that reason acceptable."

The answers were:

Check	Why red	Verdict
detector-eval	reads a gitignored corpus; never had the input	broken on arrival — fix it
security.yml	workflow file issue, fails in 0s	broken on arrival — fix it

No regressions. Both broken checks. Which is the quiet hazard of CI noise: I'd been reading both as "flaky, probably fine" for weeks, and the truth was "structurally incapable of passing."

tip: "It's always been red" is the most expensive sentence in CI. It means no one has ever seen the check do its job, so no one can tell when it stops doing it.

The fix

Two lines of .gitignore and one commit:

!services/analytics/data/
!services/analytics/data/*.json

Whitelist the corpora, then commit them — detector-corpus-v0.json and the LLM corpus beside it. They're synthetic, shape-authored test fixtures; they're secret-scanned like everything else; there was no reason they shouldn't ship with the repo. The alternative — calling the generator inside the workflow — is equally fine. What mattered was that the input became reviewable, versioned state instead of ambient local files.

The receipt, from main after the fix:

completed  success  detector-eval  main  52s
completed  success  detector-eval  main  49s
completed  success  detector-eval  main  51s
completed  success  detector-eval  main  48s

Green. Deterministic. And now — for the first time — capable of going red for a reason, which is the entire value of the check.

What I keep from this

1. A check that has never been green is not a check. It's a decorative X. It cannot catch a regression because it can't distinguish regression from background.
2. Every new workflow gets observed green once before it's trusted. A green baseline, witnessed, at least one time. If it can't get green on arrival, that's a P0 bug in the check — it corrupts the signal of every check around it.
3. At release readiness, enumerate reds and demand reasons. Not to fix them all — security.yml is honestly carried with an owner and a known cause now — but to classify them. "Always been like that" moved from an excuse to a finding the moment I asked for it in writing.

The security.yml one is still open on main as I write this. Still 0s, still failing. But it's not weather anymore — it's in the review log with a line number and a name, and that difference is the whole point.

All of this shipped with agentwatch v0.2.1 — on PyPI.

The question worth asking of your own repo tonight: which of your checks has the longest red streak, and does anyone know why it's red — or just that it is? gh run list will tell you in about a minute. That minute is the cheapest audit you'll run all week.

References

• Release: agentwatch v0.2.1 on GitHub · on PyPI
• v0.2.0 field test report
• The workflow: detector-eval.yml · the script: detector_eval.py · the generator that was there all along: generate_detector_corpus.py
• The fix: .gitignore corpus whitelist — the corpora now ship in services/analytics/data/
• Suite: agentic-suite — the full set of agent-security tools

Comments

[Arhan Canli]

The part I'd add to "a check that has never been green is not a check" is how to catch it on day one, before 255 commits. Two cheap guards:

1. Make the first run prove the gate can fail. Add one deliberately bad corpus entry (or a flag that corrupts the input) and assert the job goes red on it; the job has to be seen both green and red before it is allowed to block anything.
2. For the gitignore trap specifically, have the workflow check that its inputs are tracked, not just present: git ls-files --error-unmatch services/analytics/data/detector-corpus-v0.json fails in a fresh clone with a clear message, and git check-ignore -v <path> names the exact rule (data/) that swallowed it. That turns FileNotFoundError into "your input is ignored by line N of .gitignore".

A broad pattern like data/ without a leading slash matches at every depth, which is why it reached into services/analytics/; anchoring it as /data/ would have kept the junk rule local. Did the security.yml zero-second failure turn out to be a single syntax error, or something that had also never run?

[Kent Bodrov]

I'd add a separate question to the release review: can this exact check actually block the merge? A workflow can be observed green and red yet remain optional in the branch rules, or run against a different revision from the one being merged. The acceptance test for the gate is a failing change that cannot merge, followed by the corrected change on the required check. That tests its authority as well as its execution.