DEV Community

Mahiro Hirakawa
Mahiro Hirakawa

Posted on

My CI has been red for 18 days. It never started a single job.

#ci #testing #github #devops

My CI went red on 2026-08-15 and it is still red today. For most of the time in between I read that colour as "the build is broken." It wasn't. The jobs had never started.

The tell was sitting in the API the whole time, and it took me embarrassingly long to look at it. Here is a run from the middle of the window:

$ gh api repos/OWNER/REPO/actions/runs/33053599015/jobs \
    --jq '.jobs[] | [.name, .conclusion, .started_at, .completed_at] | @tsv'

gates        failure  2026-08-27T08:20:28Z  2026-08-27T08:20:31Z
build-macos  failure  2026-08-27T08:20:28Z  2026-08-27T08:20:30Z
lean-build   failure  2026-08-27T08:20:28Z  2026-08-27T08:20:30Z
difftest     failure  2026-08-27T08:20:28Z  2026-08-27T08:20:30Z
gates-arm    failure  2026-08-27T08:20:28Z  2026-08-27T08:20:32Z
Enter fullscreen mode Exit fullscreen mode

Two to four seconds. A Rust build that normally takes minutes does not fail in two seconds. It does not even finish checking out the repository in two seconds.

The decisive artifact is one field further in:

$ gh api repos/OWNER/REPO/actions/runs/33053599015/jobs --jq '.jobs[0].steps'
[]
Enter fullscreen mode Exit fullscreen mode

Zero steps. Not "the first step failed." Not "checkout succeeded and the build step failed." The job has no steps recorded at all, because no step was ever dispatched. The actual reason, once I stopped reading the colour and read the text: The job was not started because recent account payments have failed or your spending limit needs to be increased.

What "red" was actually standing for

conclusion: "failure" was doing double duty for two facts that have nothing to do with each other:

  1. We ran your code and it did not pass.
  2. We never ran your code.

Those have different fixes. One is "fix the code," the other is "fix the billing account." Conflating them costs you the time twice: once because you are debugging code that may have been fine, and once because the thing that is actually broken is not being looked at by anyone.

I can put a number on the second cost, because the repo kept moving the whole time. The last green run was f65aac2f at 2026-08-15T17:21:42Z. Between that commit and main as I write this, the compare endpoint reports 3,312 commits. Every one of them landed with no machine verdict of any kind behind it, while a dashboard full of red X marks made it look like the machine had an opinion.

(The repo is private, so the commands below are shown with the owner and name redacted — you can run the identical shapes against anything you own.)

And the workflows kept firing into the void the entire time:

$ gh api "repos/OWNER/REPO/actions/runs?created=2026-08-16..2026-08-28" --jq '.total_count'
2450
Enter fullscreen mode Exit fullscreen mode

Two thousand four hundred fifty runs in thirteen days. None of them ran anything.

I made the same mistake again, today, while writing this

This is the part I want to keep in, because it is the same bug and I walked straight into it an hour ago.

To confirm the failures were confined to the window, I ran this:

$ gh api "repos/OWNER/REPO/actions/runs?per_page=100&status=failure" \
    --jq '[.workflow_runs[] | select(.created_at >= "2026-08-15" and .created_at < "2026-08-30")] | length'
0
Enter fullscreen mode Exit fullscreen mode

Zero. For about thirty seconds I believed there were no failed runs in the window at all, which would have contradicted everything above.

There is nothing wrong with that command except the thing that ruins it. per_page=100 returns the newest hundred runs, and my select filter runs client-side, after the API has already decided what to hand me. Every one of the newest hundred runs was from 2026-08-30 or later. The filter matched nothing because nothing in scope could have matched. The correct query moves the filter server-side:

$ gh api "repos/OWNER/REPO/actions/runs?created=2026-08-16..2026-08-28" --jq '.total_count'
2450
Enter fullscreen mode Exit fullscreen mode

Same repository, same question, same minute. 0 and 2450.

A zero from a filter is not "there are none." It is "none of what I was given matched," and whether those are the same statement depends entirely on whether you were given everything. My command could not tell me which one it meant, so it told me the more comfortable one.

The rule I keep re-learning

Every one of these is the same shape: a two-valued output reporting on a three-valued world.

  • CI has success and failure, and no value for never attempted, so never-attempted was filed under failure.
  • My query had "matched" and "did not match," and no value for was never in scope, so out-of-scope was filed under did-not-match.

The fix is not a better threshold or a tighter filter. Tightening a filter to remove false positives does not delete the cases you cannot measure; it just changes which direction they get miscounted in. The fix is to keep the third value and refuse to let it collapse: hit, miss, and unable-to-tell, with unable-to-tell reported as its own thing rather than folded into whichever neighbour is closer.

Concretely, for CI, that means the useful alert is not "the build failed." It is "no job in this repository has recorded a single step in N days" — a condition that is true during a billing block, a runner outage, or a broken workflow trigger, and false during an ordinary honest failure. Red is loud and gets normalised within about two days. Silence never announces itself at all.

If a signal has two colours and reality has three states, the missing state does not disappear. It just goes and hides inside one of the colours you do have, and it picks the one you are least likely to question.

I work on TraceFold, a Rust project about not collapsing that third value: an effect is admitted, denied, or unknown — where unknown means the check could not be reached, which is deliberately not the same as a denial. Apache-2.0. It is honest about what it does not do; docs/LIMITS.md is the page I would read first.

Top comments (0)