Last week I found two checks in my own repository that could never have failed. Not broken, not misconfigured. Structurally incapable of going red, sitting in CI for months, costing the same as a real check and reporting the same colour.
So I went looking for how common that is. Here is the count, the method, and what it does not prove.
What I counted
GitHub code search, all public repositories, on 5 September 2026.
workflow files (containing runs-on in .github/workflows) 7,094,272
of those, containing "continue-on-error: true" 264,704
containing "npm run lint" 301,568
containing BOTH "npm run lint" and "continue-on-error" 27,232
That last line is where it gets interesting, and also where code search stops being able to help. It tells you both strings live in the same file. It cannot tell you the continue-on-error is attached to the lint step rather than to something unrelated three jobs down.
So I read them.
The sample
I pulled 500 matching files from pages spread across the result set, shuffled them, and inspected the first 120 by fetching each file and walking it step by step.
Two things I had to fix in my own method before the number meant anything.
The first pass gave 80 percent, and it was wrong. path:.github/workflows also matches README.md, TROUBLESHOOTING.md and ci.md sitting in that directory, plus .yml.disabled files that do not run. Filtering to actual .yml and .yaml removed that noise.
The second was a distinction I nearly missed. One of the early hits was a step called "Lint (non-blocking)". That is not the problem. That is somebody being explicit. Counting it alongside the silent ones would have been unfair and would have inflated the number.
With both corrections, on 120 real workflow files:
check step that silently cannot fail 89 (74%)
check step that says so in its name 3 (2.5%)
no toothless check step at all 28 (23%)
Security audits are the worst of it
workflow files running "npm audit" 71,296
of those, also containing "continue-on-error" 19,200 (27%)
workflow files with "npm test" and "continue-on-error" 22,592
More than a quarter of the workflows that run a dependency audit also contain something allowed to fail. Given the 74 percent rate in the sample, most of those are the audit itself.
Three shapes I verified by reading the file
I am not linking these. They are small public projects and the point is the pattern, not the people. Run the search yourself and read the first ten results; you will find the same thing.
The plain one. A step named Lint, running npm run lint, with continue-on-error: true directly under it. The name promises a check. Nothing in the Actions tab tells you it cannot fail.
The one with the padlock. A step named 🔐 Check for security vulnerabilities, running npm audit --audit-level=moderate, with continue-on-error: true. The emoji is doing more work than the check.
The belt and braces. A step named Run tests (if available) whose command is npm test || echo "No tests configured", with continue-on-error: true on top. The || echo already swallows the exit code. The flag catches whatever the first mechanism missed. That step cannot report a failure through two independent layers.
Why this is worse than having no check
A missing check leaves you appropriately nervous. You know you are not covered.
A check that cannot fail costs exactly the same to run, occupies the same line in the Actions tab, renders the same green, and buys you confidence it has not earned. It is not neutral. It actively replaces the nervousness that would have made you look.
There is a second cost that surprised me. On my own suite, the twelve failing tests were 8 percent of the tests and 36 percent of the runtime, because a failing test burns its full timeout plus a retry. A dead check does not just fail to inform you. It also does not fail fast, because it never fails at all.
Where the flag is right
continue-on-error is a good feature and this is not an argument against it.
An experimental matrix leg on a nightly toolchain, an optional platform, a step whose failure genuinely does not block the merge: all fine. Three of the files in my sample used it exactly that way and said so in the step name.
The distinction is not the flag. It is whether the name tells the truth. Lint (non-blocking) is honest. Lint is not.
The check I now run on myself
Reading a check does not tell you whether it works. The only way to know is to introduce the exact defect it claims to catch and confirm it goes red.
I ran that against ten checks in one repository. Two could never have fired. One of them is worth describing, because I would not have predicted it.
The check verified that a curl in a notification step carried the --fail flag, by searching the step for the flag name. Directly above that curl sat a comment explaining why the flag mattered. Remove the flag, leave the comment, and the check stayed green, because the string was still on the page.
The more carefully I had documented why the flag was necessary, the more reliably I had disabled the check that enforced it.
Three measurements per case, not one: green before the mutation, red during, green after. If any of the three is off, the result is unknown rather than pass. My own harness produced four unusable results before it produced a trustworthy one, every time because of the harness and not the checks. An exercise about detectors that cannot fire is a good place to remember that the detector you just wrote is also a detector.
What this count does not prove
Worth stating plainly, because a number without its limits is a claim dressed up as a fact.
GitHub's code search totals are approximate. My sample came from the first thousand results rather than the full 27,232, so it is indicative and not a proper random estimate. I only measured the JavaScript ecosystem, through npm commands; Python, Go and Rust workflows are not in the count. My step parser is a heuristic that walks lines between step boundaries, and I hand-verified three of its hits rather than all 89.
What I am confident saying: this is common, it is not rare, and security audits are disproportionately affected.
The two-minute version
Open your workflows. Search for continue-on-error: true. For each one, ask whether the job or step name promises something it cannot deliver.
Then pick your most important check and break the thing it watches for. If it stays green, you did not have that check. You had a line item.
Top comments (0)