Where this landed, 9 October 2026. This post opens with 40%. That number is not
wrong so much as it is not the quantity you would take it for, and I only found that out
by being argued with in the comments. It has taken a week of measuring to say what it
should have been, and the tool has changed more than the number has. This block replaces
the two corrections that were stacked here, because three corrections in a row is not
honesty, it is sediment.The sampling was measuring the wrong unit. The samples behind every figure were
stratified with a cap of two findings per repository, to stop one repo dominating. But
the tool emits one finding per (job, upstream) pair, and the false positives cluster in
jobs with many dependencies: one reporting job produced eleven by itself. The cap deletes
exactly that clustering. Simulated against a population I later labelled completely, the
capped sampler is an unbiased estimator of the per-job rate and runs about 16 points
low for the per-finding rate. So every percentage in this post understates, all in the
same direction. @arhancanli called this in the thread before
I measured it.What the numbers are now, all hand-labelled, no estimate of an estimate. Four
detectors are two. D4 went on 0 defensible of 18 sampled, D2 on 0 true of 27 censused.
Corpus output went 1986 findings to 1026.
basis false D3, all 653 complete census, every finding read 36.1% [32.5, 39.9]D1 gated on --repo, and 0 findings with itsee below The severity tiers are gone, and that is the result I did not expect. Censusing all
three strata gave HIGH 30.9%, LOW 31.6%, MEDIUM 52.8%. HIGH's interval overlaps the
merged rest. The only real separation is MEDIUM being worse, and it came from the
severity function short-circuiting on whether the workflow runs onpull_requestbefore
it looked at the finding at all. The axis measured the trigger, not correctness. It
hid the better findings behind--alland shouted the worse ones, so collapsing it
raised the default output from 238 findings to 653 and lowered the false rate a reader
sees, from 44.1% to 36.1%.D1 is gated rather than removed, and the gate reports nothing. Its text says a
skipped upstream leaves the dependent running without it; GitHub documents that the
dependent is skipped too.
68% of its findings are that case. So it now says nothing
unless--repoconfirms the job is a required check. Then I ran that against live branch
protection for all 373 findings: 10 resolved to REQUIRED, all in one repository, and all
10 vanish on a complete checkout where the tool can read the script that job calls. On
275 repositories,--repohas never reported a finding. It is kept because the gate is
correct, not because it is productive, and that distinction is the whole post in one line.Three numbers I published and had to take back, every one caught by checking my own
claim rather than by anyone complaining. The "65% false" I quoted for D1 came from a
census covering 12% of the population it was quoted against. The "37.0%, 245 of 662" in
my own 0.3.0 release notes added a stratum censused at 153 to a population taken
after a suppression had already removed 9 of them; every commit since emits 653. And the
54.9% in the table that used to sit right here kept 3 findings labelled true from a
sample taken before the insight that refutes them, which is holding both positions at
once: I shipped a suppression on the grounds thatcmd | grep -q Xis not a defect
while still counting three of those lines as defects. The arithmetic carries its own
check, and I had not run it. The census was 68 true and 85 false of 153, so removing 9
false findings has to leave true at 68, and 68 + 76 = 144. A correction is a new claim
and needs verifying as hard as the original.The part of this post I would still defend is the part about taste: the corpus average
hid the problem, and pointing the tool at two repositories I respected is what found it.
The part I would not defend is any specific percentage in it. The working is in the
comment thread, including the fixes measurement killed before they shipped. The latest
was a filter that genuinely improved precision, 31% false down to 17%, and I deleted it
anyway: it bought those 14 points by discarding two thirds of the true findings, and
on that population it turned out to be a shape classifier wearing a semantic test.
Yesterday I published a tool called deadgate to PyPI. It reads GitHub Actions workflows and finds checks that cannot fail: a required gate satisfied by a skipped job, an exit status swallowed by a pipe, a fan-in job that never reads whether its upstream passed.
Then I pointed it at Arize-ai/openinference, 1.2k stars, 22 workflows. It reported 13 HIGH findings.
Twelve of them were false.
Not borderline. Not "technically true but low impact." Their CI was correct and my tool could not see it. The version doing the reporting had been on PyPI for about an hour.
What it could not see
GitHub's documented way to ask "did anything upstream fail" is the wildcard form:
if: always()
run: |
if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then exit 1; fi
My detector tested for that with needs\.[A-Za-z0-9_-]+\.(result|outputs), which cannot match *. So the one spelling the documentation recommends was the one spelling the tool was blind to, and every correctly written fan-in gate came back as a gate that cannot fail.
The finding's own message read "never reads needs.*.result" while failing to match that exact string.
I fixed it, shipped 0.1.1, and pointed the fixed version at astral-sh/ruff. Twenty-one HIGH findings. Eighteen false. Ruff writes its gate a third way:
env:
NEEDS_JSON: ${{ toJSON(needs) }}
run: |
failing=$(echo "$NEEDS_JSON" | jq -r 'to_entries[]
| select(.value.result != "success" and .value.result != "skipped")')
That reads every upstream at once and the word result never appears in any workflow expression. There is nothing for my pattern to match.
The fourth one was not a spelling at all. scikit-learn has a job that calls a reusable workflow, so it has uses: and no steps:, and it passes needs.check-sdist.result through job-level with:. That expression was one I already understood perfectly. It just lived somewhere I was not looking, because I scanned step-level with and not job-level with.
Four classes. The first two shipped to PyPI. I found the third and fourth only because I kept pointing the "fixed" version at new repositories instead of declaring victory.
The part I think is actually generalizable
Here is the number that matters, measured on a 275 repository, 4543 file corpus with both versions reading identical bytes:
| version | findings | HIGH |
|---|---|---|
| before | 4562 | 1104 |
| after | 2917 | 658 |
Forty percent of HIGH findings were false. That sounds bad, and it undersells it, because 40% is an average and the errors were not spread evenly.
Only 15 of those 275 repositories use the wildcard idiom at all. The false positives piled onto the repositories that write their gates carefully. On openinference the false rate was 92%. On ruff, 86%. On a repository with no fan-in gate at all, it was zero, because there was nothing to misread.
My linter was least accurate on the best code.
If a tool is noisy on bad code, the noise and the signal arrive together and you read both. If it is noisy specifically on good code, the people who did the work correctly are the ones punished, and the obvious response is to stop running it. A corpus average cannot surface that, because the corpus is mostly ordinary repositories where the tool happens to be right.
I had the corpus. I ran it. It said 40%, and I would have shipped that as the headline. What actually found this was pointing the tool at two repositories whose CI I respected.
The mutation that survived
To stop reporting jobs whose failure a gate already catches, I walked the needs graph transitively: if a gate covers ci, and ci needs changes, then changes is covered too.
I mutation tested the fix, which for me means planting the defect each test claims to catch and checking that something goes red. One mutation survived: the one that deleted my transitive walk.
My first instinct was a missing test. It was not. The mutation survived because the walk was wrong: needs.*.result reports only direct needs, and a failed grandparent makes the parent skip, and skipped is not failure, so the gate never sees it. Walking transitively would have suppressed real findings.
A surviving mutant has two meanings and they are easy to confuse. Either a test is missing, or the code does nothing. If I had written a test to kill it, I would have pinned a bug in place and called it coverage.
The small one that is somehow the worst
While fixing all this I noticed five of my repositories advertised a stale test count in the README. I corrected them by hand and added a CI check so they could not drift again.
The check went red immediately. On a count I had just "corrected."
I had changed agent-graph's README from 42 to 58 and written a public commit message saying the previous commit overstated the count at 67. CI collects 67. My local venv was missing an optional extra, and tests/test_rag.py opens with pytest.importorskip, so nine tests vanished at collection with no error, no warning, and no mention of the file. pytest said "58 tests collected" and meant it.
The original number was right. I corrected a correct claim, publicly, using a measurement from an environment that was not the real one.
The check I shipped an hour earlier had the same blind spot and agreed with 58. It now fails when any test file on disk contributes zero collected tests, because that is the only evidence an environment is incomplete.
What I am not showing
The sweeps above are driven by my own harness, and the agent scaffolding, gate implementations and prompts behind it are not in the repository and are not going in it. deadgate itself is small and MIT and you can read all of it. The thing that pointed it at 275 repositories and classified the survivors is the part I am keeping.
The question I actually have
I do not know how to test a linter for this class of error without a sample of good code, and "good code" is not a property I can detect automatically. A corpus gives you volume and volume gives you the average, which is exactly the statistic that hid this.
The only thing that worked was taste: picking two repositories I already believed were well engineered and reading the output by hand. That does not scale and it is not reproducible, and I would like to be talked out of it.
If you maintain a static analysis tool, how do you measure your false positive rate on the code that is already correct? I would rather hear that I am missing something obvious than keep doing it by hand.
deadgate is at pypi.org/project/deadgate and github.com/egnaro9/deadgate, MIT, no model in the loop.
I work on AI evaluation and testing, open to relocation above $120K, portfolio at erikhill.dev.
Top comments (22)
The 40% is a pooled count over 4,543 files, but the errors cluster by repository: 92% on openinference, 86% on ruff, 0 where there is no fan-in gate. Findings from one repo share a cause, so an interval over 1,104 findings looks far tighter than it is. Resample whole repos (or report the per-repo false rate and its median) and say how many of the 275 contribute most of the 446 HIGHs that disappeared.
The number I'd want next is the precision of the 658 that remain. Both of your false-positive classes were found by pointing the tool at repos you respected, so there are probably classes in the survivors that no one has looked at. A random sample of ~40 of the 658, stratified by repo and hand-labelled, gives a Wilson interval on "after"; 3 false out of 40 and 12 out of 40 are very different tools, and 30 of 34 on two chosen repos can't distinguish them.
The surviving mutant is a nice catch. The analogue for the corpus is the same check: delete the transitive-walk suppression and see which of the 446 come back. If most are on the repos with careful gates, that confirms the walk was the right suppression for them and not something that hid real findings.
You are right, and I went and ran the per-repo breakdown you asked for. It is worse for my framing than you put it.
The 446 removed HIGH land on 23 of the 275 repositories. The top 2 carry 50% of them and the top 7 carry 80%:
Per-repo false rate: median 0.0%, mean 13.3%, against the pooled 40.4% I published. 87 of the 110 affected repositories had zero false HIGH. So the pooled number is close to a statement about ClickHouse and sglang, and the interval I did not compute would have been meaningless if I had.
The part that actually stings: my post said the errors concentrated on repositories that write their gates carefully, and I used openinference and ruff as the evidence. Those two are real, but they are 18 and 13 of 446. The mass is in large projects with elaborate release pipelines, which is a different claim with a different cause. You identified a clustering problem and the clustering also falsifies my explanation of it.
On precision of the 658 survivors, I agree that is the number that matters now and I do not have it. A stratified sample of 40 with a Wilson interval is the right size, and I will hand label it. I will say in advance that 30 of 34 on two chosen repos cannot distinguish a good tool from a bad one, because that is exactly the mistake I made in the other direction.
On the corpus analogue of the surviving mutant: that experiment is available and I will run it. One correction on the mechanism, since it matters for what the result would mean. The transitive walk was not a suppression I kept, it was one I deleted, because needs.*.result reports only direct needs and a failed grandparent makes the parent skip rather than fail. The suppression that survives is the direct-only one. So the test is to disable that and see which of the 446 come back, and your prediction is the right one to check: if they are concentrated on the repos with real fan-in gates, the suppression was doing its job.
Thank you for this. It is the most useful thing anyone has sent me about this project.
Worth splitting the 446 by how the fan-in gate tests its needs, not just by repo. Because needs..result only reports direct needs, a failed grandparent shows up as 'skipped' on the parent. A gate written as
== 'success'on all needs still catches that chain; one written as!= 'failure'or `contains(needs..result, 'failure')lets it through. So when you disable the direct-only suppression, the ones that come back should mostly be the second form, and that is a checkable prediction: if they are== 'success'` gates, the suppression was hiding something else.Ran the disable. Your prediction does not hold on what I can classify, and more than half of it I cannot classify at all.
Disabling the direct-only suppression takes 658 HIGH to 771, so it is hiding 113. All 113 are D4, which lines up with D4 being the detector that produced zero defensible findings in the sample.
Splitting those 113 by how their workflow's gate tests its needs:
So among the ones I can read, the whitelist form outnumbers the blacklist form, which is the opposite of the prediction. By your own reasoning a whitelist gate does catch the skipped-parent chain, so for those 27 the suppression was doing the right thing.
The 55% is the honest problem and it is not a bug I can fix. Those gates delegate the decision to a script:
The gate's semantics are in a file that is not the workflow. I have the workflows, not the repos, so whether that script demands success or merely rejects failure is not knowable from what I am holding. Any number I reported for those 62 would be me guessing and calling it a measurement.
What survives: 19 findings where the gate is the permissive form and the suppression may genuinely be hiding a real chain failure. That is a small enough set to read by hand, and it is the next thing I will do rather than the next thing I will claim.
Two caveats on the 24 vs 17 split, since it is the only part that looks like a result. The classifier is a regex over the gate's text, so it has exactly the weakness that started this whole thread. And n is small enough that the difference between 27 and 19 is not much of a difference.
Your framing was still the useful part. I had been treating "the gate reads its needs" as one property, and it is two: whether it reads them, and whether reading them is strict enough to catch a skip. My suppression only checks the first.
The 19 are worth reading by hand, with one filter first: a gate only reaches the permissive-form problem if it actually runs when a need is skipped. A job with the default
ifis itself skipped when any need fails or is skipped, and a skipped job counts as passing for a required check, so those gates have a different bug (green on upstream failure) and the!= 'failure'form is never evaluated. The permissive form only matters where the gate hasif: always()or!cancelled(). That may shrink the 19 further.For the 62 that delegate to a script: the script sits in the same repo as the workflow, so it can be fetched at the workflow's commit. A first pass is whether it compares each result to 'success' or only tests for 'failure'/'cancelled'. Still a regex, so hand-label those, but it turns "unclassifiable" into a readable list instead of a guess.
Both of your suggestions were right, and together they pointed at a third property that makes most of the strictness question moot. Also I owe you a correction: the 40% in the article was measuring something other than what I said it was.
Your filter first. You said the permissive form only matters where the gate has always() or !cancelled(), because a gate with the default if is itself skipped when a need fails, which is a different bug. That is the predicate I already gate on, so every number I gave you was post-filter, which I should have said. Re-running it on the full corpus, counting every (gate, dependent-with-upstreams) pair:
The split is watched / not watched, which is the third property. If a gate needs a job AND every job upstream of that job, then any real failure in the chain reaches it as that upstream's own failure result, and even the permissive != 'failure' form rejects that. The only skip left is a condition the author deliberately evaluated false, which is intent, not a defect.
So the question is not "is this gate strict about skipped", it is "does this gate watch the whole chain", and unlike the first, a workflow file can answer it. 89% are chain-watched whatever form the gate takes. 5596 of the 5824 unclassifiable ones, 96%, are settled without reading the script at all. The residue where strictness still matters and the file still cannot say is 257, not 55% of everything.
Your second suggestion I shipped, and it found a class I did not have. Fetching the script at the workflow's commit is now what the tool does: when a named gate shells out with a token in scope, it reads the script and either suppresses, keeps full severity if the script demonstrably does not consult status, or demotes and says the claim is unverified. The first repo it ran on was WordPress/gutenberg, whose status checks page GET /actions/runs/{id}/jobs and fail if any job concluded as anything but success or skipped. The word result never appears and needs is never consulted, so no pattern over the workflow could ever have found it. That gate is better than the idiom I was looking for: it also fails closed when a job was never evaluated, which needs.*.result cannot detect. Ten of my 54 D1 findings were that, all false, all one repo.
One caution on direction, because I got it wrong once and a mutation caught me. Suppressing a job because its PARENT is covered is unsound: needs.*.result reports only direct needs, and a failed grandparent makes the parent skip rather than fail. Requiring the gate to watch the whole chain itself is the strict version of that idea, not the loose one.
Now the correction. I pre-registered a sample over the surviving 179 HIGH and committed to posting whatever it said. D1 came back 33% false, having measured 60% a round earlier with nothing in its code changed between. A control moving 27 points is either a bad draw or my labelling drifting, so I checked instead of choosing: the two seeds overlapped on 7 of 15 draws and all 7 got the identical label both rounds. Not drift.
D1's whole population was 54 findings, so I stopped sampling and labelled every one. 76% false per finding, 59% per job, exact. Sample 4's 95% interval was [15%, 58%] and did not contain the truth.
The reason is my own design. I cap at 2 findings per repository so no single repo dominates. But the tool emits one finding per (job, upstream) pair, and the false positives pile up in jobs with many dependencies: one reporting job emits 11 by itself. The cap deletes exactly that clustering. Against the censused truth, 20k draws at n=15:
The capped sampler is an unbiased estimator of the per-JOB rate and runs about 16 points low for the per-FINDING rate. Every figure I have published was weighted back to per-finding populations, so all of them were per-job rates wearing a per-finding label, all understating, all the same direction. The 40% is not false, it just is not the quantity a reader takes it for. Related and smaller: those 179 findings were 136 distinct problems. I had been reporting a finding count as a problem count.
D1 now tops out at MEDIUM. 65% false after every fix, with 11 of the survivors one reporting job, is not a tier that gets to keep asserting HIGH on its own authority, so HIGH is reserved for the path where branch protection actually confirms the job is required. Corpus HIGH went 354 to 125, all D3, bounded at 21 to 33% false per finding. 0.1.4 is up.
What I keep relearning: no amount of running the measurement again would have shown this, because the bias was in the sampling design and the design was mine. Censusing 54 findings cost less than the two samples that disagreed about them.
Ran it. You were right about the survivors, and it is worse than the before-number.
Pre-registered first, because a sample I could steer would be worth nothing: n=40, stratified by repo with a cap of 2 per repo, seed 20261009, labels defined before drawing. Cap because one repo holds 45 of the 658. I also wrote down in advance that an ARGUABLE bucket exists and gets reported separately, since the direction you fold it is exactly where a motivated result would come from.
Result:
So the precision after the fix is statistically indistinguishable from the 40% I removed. I traded four classes of wrong for five more.
The five causes, four of which nobody had looked at:
The first one is the same bug as the toJSON case, a third time. I taught it the wildcard, then the whole-context form, and it still cannot see a gate that just names every upstream individually. Three spellings of the same idea and I fixed them one at a time, which is the thing your first comment was really pointing at.
By detector, and this is the number I did not want:
D4 did not produce one finding I would defend. Most of its arguable cases are the same objection: it claims "the merge is green with nothing tested", but a failed gating job is itself red on the PR, so the impact needs a protection config it cannot see. D4 at HIGH is mis-severity, and it is 304 of the 658.
One more, since it belongs here: a false positive turned up in ruff's build-docker.yml, the same file as the real finding I sent you. The "Check tag consistency" step masks grep's status, but it fails CLOSED, so the empty version mismatches the tag and the step exits 1 correctly. Right file, right detector, wrong finding.
Still owed: the suppression-disable experiment. Numbers go in the thread either way.
Two notes on the numbers. With a cap of 2 per repo, the 40% describes a typical repo more than a typical finding: the repo holding 45 of the 658 is weighted 2/40 instead of about 7%. For a per-finding rate, weight each sampled item by (that repo's findings / its draws) and bootstrap by repo; the interval will likely widen, since 40 items across many repos behaves like a cluster sample. Reporting both is cheap.
On the six per-job-enumerated gates: the toJSON, wildcard and enumerated spellings are one rule. Compare the set of need ids the gate references (needs.X.result, needs.*, toJSON(needs)) against its needs: list and flag only when a need is unreferenced. Then the spelling stops mattering.
D4 I would demote to INFO rather than tune: 0 true of 18, and its impact depends on branch protection the tool cannot see.
Credit where it belongs: you called the cap bias here, before I measured it, and I just posted a long reply in the other branch presenting it as something I found. You were first and you were right.
You said the cap of 2 makes the 40% describe a typical repo rather than a typical finding, and prescribed weighting each item by (repo findings / repo draws) with a cluster bootstrap. I got there the slow way instead, by censusing D1's whole 54-finding population and then simulating the sampler against the known answer. Same conclusion, and your estimator is the general form of it:
The capped sampler is unbiased for the per-job rate and runs about 16 points low per finding. Your "reporting both is cheap" is the actual fix and it is now what I do: per job and per finding, separately, because they answer different questions and I had been printing one under the other's name.
On unifying the spellings: agreed, and it is better than what I have. Comparing the set of referenced need ids against the needs: list, and flagging only an unreferenced need, collapses the wildcard, toJSON and enumerated forms into one rule instead of three patterns I keep discovering one outage at a time. It also handles a case I hit this week for free, where a gate passes toJSON(needs) into a script: the env var references every need, so the rule stays quiet without anyone reading the script.
The limit is worth naming, because it is where your rule and my week collide. WordPress/gutenberg's status check references no needs at all. It pages the Actions API for the run's job list and fails on any conclusion that is not success or skipped. Your rule flags it, mine flagged it, and it is correct CI, better than the idiom either of us is matching on, because it also fails closed when a job was never evaluated. No rule over the workflow file can see that, so the honest move is to stop asserting rather than to match harder: read the script when it is fetchable, and otherwise demote and say the claim is unverified.
On D4: it stopped claiming HIGH in 0.1.3 for the reason you give, and D1 just joined it on the same evidence, 65% false after every fix. There is no INFO tier, so the floor is LOW, which is the same idea with a worse name. I take your point that 0 of 18 argues for removal rather than demotion, and I am not ready to defend keeping it beyond "LOW costs a reader nothing". If it is still 0 for 18 next sample it should go.
You said remove it, I said "if it is still 0 for 18 next sample it should go", and then I noticed that waiting for another sample to confirm a zero is not evidence gathering, it is deferral. D4 is gone in 0.1.5.
What it cost to keep: 932 findings over 93 repositories, 47% of everything the tool emitted, from a detector with no defensible finding ever recorded against it. HIGH does not move, because 0.1.3 had already capped it. That cap was the mistake you were pointing at. A detector that has never once been right is not improved by saying it quietly, and 724 LOW findings still cost a reader attention.
Three things fell out of the removal that I did not expect.
Three symbols existed only to serve D4 and are now deleted, including a whole-context predicate I would have sworn was shared. I found them by scanning for zero-reference symbols rather than by assuming the detector was self-contained, which is a check I had not run before and will now run on every removal.
Eleven tests used D4 as the vehicle for testing the wildcard and toJSON recognition, the thing classes 1 and 3 were about. My first instinct was to re-point them at D1, which is still live and still has to not fire when a job reads its upstreams correctly. I checked before rewriting: D1 does not fire on those fixtures in either direction, so every re-pointed test would have passed no matter what the code did. I deleted them instead. A deleted test is worth more than one that cannot fail, and I would not have known the difference if I had just swapped the function name and watched it go green.
The last one is the part I am least comfortable with. The hazard D4 described is real. A change-detection job that FAILS rather than decides leaves its outputs unset, the condition false, the dependent skipped, and a skipped job reports Success. That is now undetected by design, and there is a test asserting the silence so it stays deliberate. The reason is the one you gave: the file cannot tell it apart from the intended optimisation, which is the identical shape, and whether the skip matters depends on which checks are required. 18 hand-labelled attempts at that distinction produced nothing.
So it is not that the bug is not real. It is that this input cannot answer the question, and reporting it anyway was me charging a reader for my uncertainty. If a version can read branch protection for the gating job specifically, the question becomes answerable and it comes back on evidence.
D2 is gone too, and this time I censused it rather than sampled: 27 findings, 27 jobs, 14 repositories, every one hand-labelled. 0 true, 27 false, 0 arguable.
I pre-registered the threshold before enumerating, because I was obviously lining it up for removal and a threshold set afterwards is not a threshold. 80% false or worse, remove. 50 to 80, narrow. Under 50, keep it and admit the n=10 sample had misled me. It came in at 100%.
What it was actually finding, with no exception: report x3, summary or summarize x5, cost x7, merge-reports x2, aggregate_reports, e2e-log-summary, accessibility-report, audit-high-report, ci-timings, accept, publish, release_lease x2, a cleanup job that restores an environment policy and must run whatever happened, and one change-detection job.
Three I read in full instead of judging by name, because their names suggested a gate. All three handle upstream failure on purpose. One of them carries the comment "Degraded runs produce no ci-timings.json, skip rather than fail", which is better failure handling than the detector accusing it had.
The part that actually settles it is structural rather than statistical, and I only noticed it while writing up the count. D2 reached MEDIUM only when its gate-name test matched the job. All 27 findings were LOW, so zero matched. By its own severity logic it had never once found a job it believed was a gate, across 275 repositories, while the text of every finding it emitted said that job's branch protection was decorative. Its own gate test contradicted its own finding text 27 times out of 27.
That is not a precision problem a narrowing fixes, and it was visible in the output the whole time. A detector disagreeing with itself is a stronger signal than a false-positive rate, and it costs nothing to check: compare what your severity logic believes against what your message asserts.
So the scoreboard, since you called both of these before I did. D4 removed on 0 of 18 sampled. D2 removed on 0 of 27 censused. Two detectors left, and the honest table now carries their measured precision next to them rather than just their descriptions: D1 65% false per finding on a complete census, D3 21 to 33% on a sample of 25. D3 is the only one that still produces HIGH, and it has never been censused, which makes it the next thing to do rather than the thing to trust.
Two own goals from the removal, both caught by a check and not by me. My first excision cut 101 lines instead of 60 and swallowed an unrelated block, because it scanned for the next
defand the next thing was a module-level constant. And the zero-reference scan I added when D4 went missed a function that had been dead since well before either removal, because it counted textual mentions and the thing is named in two comments. A call-graph check finds it in a second. Both of those are now the form I should have used the first time.Total output across the corpus is 1986 before any of this and 1027 now. Nearly half of what this tool emitted a day ago was from detectors that have never been right once.
D3 censused, and the interesting part is that I nearly published the wrong number twice.
First pass: 125 HIGH findings, all of them, hand-labelled. 47 true, 43 false, and 35 I could not resolve from the workflow without reading the use site. That is 34% false with a 28% unknown, and the pre-registered rule said under 50% means keep. I wrote up "keep" and computed, as a sensitivity check, that the verdict would flip if 20 of the 35 unresolved turned out false.
Then I resolved them. 23 were false. 51.2%, which is NARROW, not KEEP.
The unresolved bucket ran 66% false against 34% for the resolved part. That is the thing worth taking away: an arguable bucket is not a random sample of the population. Items land there because they are hard, and hard correlates with false, because the easy-to-confirm true ones are exactly the cases where the bad value visibly crosses a step boundary. Reporting 34% with a 28% unknown would have been wrong by 19 points in the direction that flattered me.
I also had to withdraw "all three predictions held", which I had posted based on the provisional labels. Re-scored on the resolved ones, two of the three missed.
Second near-miss. One of my labelling rules was reasoned rather than run. I had called
if [ "$SIZE" -gt 100 ]self-correcting, on the grounds that an empty SIZE makes[error out and set -e kills the step. It does not:set -e is suspended inside an if-condition. The size check passes silently. A bare
test x = yon its own line does fail the step, so the distinction is real, but it is the opposite of what I had written down. Two labels moved. The verdict survived at 51.2%, which it did not have to.The narrowing that came out of it asks one question the old severity logic never asked: can the masked status reach anything at all? It used to read
exit 1or GITHUB_OUTPUT from anywhere in the step, so a single exit at the bottom promoted every log-extraction pipeline above it. Now the flagged line has to consume its own output, and if it captures into a variable, some use of that variable has to be able to silently swallow an empty value.125 HIGH to 94. 51.2% false to 35.1%. Zero true positives lost.
Zero is load-bearing there, and it is the only reason I trust the narrowing, because three earlier versions of the rule were not zero. Each was evaluated against the 125 labels before going anywhere near the detector:
echo "digest=${d}" >> "$GITHUB_OUTPUT". An echo that redirects is not printing, it is exporting, and that is the single most valuable shape in the whole corpus. Would have killed 11 true findings.export V=$(...)orfor f in $(...)as consumption. Killed 2.WINEDEBUG=-all timeout 300 make check | tee x.logit read WINEDEBUG as the captured variable, found it unused, and suppressed a maskedmake checkfailure.That third one is the one I would have shipped. It survived my evaluation harness and only died when I ran the real detector against the same labels, because the harness happened to apply the assignment check to fewer lines than the detector does. A simulation of your own code is a different program, and it agrees with you right up until it matters.
What I will not claim: this censused the HIGH stratum, 125 findings. D3's MEDIUM is 152 and its LOW is 408. Those 560 are unmeasured and nothing here says anything about them.
Measured the two tiers I had never measured, and the number that matters is one I had never computed: what this tool shows you by default is about 55% false.
The CLI hides LOW unless you pass --all. So the default output is HIGH plus MEDIUM, 247 findings, and only the 94 HIGH had a measured precision. Here is the rest of it:
Two design choices came straight out of being wrong earlier in this thread.
Drawn UNCAPPED, reversing every earlier sample here. You were the one who pointed out that a per-repo cap makes the 40% describe a typical repo rather than a typical finding; once I censused D1 and could simulate against the truth, the cap turned out to be an unbiased estimator of the per-JOB rate and about 16 points low per finding. Per finding is what I wanted, so no cap.
And zero arguable. Every drawn finding got resolved before I computed anything, because in the HIGH census the unresolved bucket ran 66% false against 34% overall and flipped the verdict from keep to narrow. A rate quoted with an open bucket is wrong by whatever the bucket is hiding, and the bucket is not a random sample of the population.
One prediction missed. I had written down that LOW would be worse than MEDIUM, reasoning that LOW means fire-and-forget or not-a-PR-workflow. It is the better tier, 47% against 70%.
What I am NOT claiming: that the severity ordering is backwards. MEDIUM at 70% and LOW at 47% looks inverted, and that would mean the tier is actively misinforming, but the intervals overlap on [52%, 70%] and n is 30 and 15. Only one comparison here is established, MEDIUM being worse than HIGH, where the intervals do not touch. The eye-catching claim is the one the data does not support, which is roughly how this whole series has gone.
Separately, I implemented your gate unification and then rejected it, which I want to show the working on because the idea is right.
Comparing the referenced need ids against the needs: list and flagging only unreferenced upstreams does collapse needs.X.result, needs.* and toJSON(needs) into one rule, and it does stop a gate naming eight of nine needs from suppressing the finding about the ninth. Implemented, it surfaces 396 additional D1 findings, 373 to 769. By job kind:
58% are the two families the D2 census measured at 0 true of 27. 1% are jobs that name themselves a gate. The rule is sound and the population it reaches in D1 is the one already proven false, so D1 keeps the cruder all-or-nothing suppression and the per-name model stays in the file with a comment explaining why it is not wired up. Revisit if D1's trigger is ever narrowed to jobs that are plausibly gates.
Last thing, prompted by a question about whether anything learned since would have saved D4. No. I restored it from the commit before its removal and reran it: 932 findings on the current corpus, and the chain-coverage rule it never had suppresses 205, leaving 727. D4 died on a claim the workflow file cannot support rather than on a pattern bug, and no amount of better parsing touches that. The chain rule actually argues harder for the removal, because it says a gate watching the whole chain sees the failure directly, which is D4's premise being false more often rather than less.
Edited 2026-10-09, after posting. Overtaken by later measurement, and left standing rather than rewritten. There is no default-vs-
--allsplit any more: all three strata were censused, the tiers turned out to measure the workflow's trigger rather than correctness, and collapsing them made the default 653 findings at 36.1% false. The "247 findings" here is also arithmetically wrong even for its own moment: it adds a MEDIUM stratum censused at 153 on 0.1.9 to a population taken after thegrep -qsuppression had removed 9 of them. HIGH plus MEDIUM was 238 at 44.1%. A later top-level comment on this post has the working.Two things, and the first is a correction to something I have said in this thread more than once.
I have been quoting D4's removal as "0 true of 18". It was 0 defensible, 8 false, and 10 ARGUABLE. Ten of the eighteen were never resolved, and I rounded that into a cleaner number than I had.
That matters more than it would have a week ago, because I later established that an arguable bucket is not a random sample of its population: in the D3 HIGH census the unresolved items ran 66% false against 34% overall, which was enough to flip that verdict from keep to narrow. So the ten were probably mostly false. Probably is not a measurement, and "0 of 18" implied one I did not have.
The sample was also drawn from D4's HIGH stratum, which brings me to the second thing.
I censused the other two D3 strata and the severity model does not rank anything:
HIGH's interval overlaps the merged non-HIGH tier at 38.0%, so the loudest tier was not reliably better than everything else. LOW was as good as HIGH. The only real separation was MEDIUM being worse, and that came from the severity function short-circuiting on whether the workflow runs on pull_request before it looked at the finding at all. The axis was measuring the trigger, not correctness.
Which means the tool was hiding LOW behind --all and shouting MEDIUM by default: hiding the better findings and showing the worse ones. That is worse than having no severity, so there is none now. One tier, nothing hidden, 37.0% false across all 662 D3 findings. Collapsing raised the default output from 247 findings to 653 and lowered the false rate a reader actually experiences from 46.2% to 37.0%.
So the D4 sample was stratified on an axis now known to carry no signal. Its removal still stands, but on different evidence than I kept citing: 80% of its 932 findings describe a mechanism redundant with GitHub's needs-skip rule, which is severity-independent. I checked the other two for the same problem. D2 went on a complete census of all 27 findings, every severity, zero arguable. D1's gate rests on the chain-skip structure and an uncapped sample across its whole population. Neither depends on tiers.
The loss worth naming: the narrowing I shipped in 0.1.7 only ever moved a severity, so with no tiers it decided nothing and is deleted. On pull-request workflows it measurably separated 30.9% from 55.6%, so turning it into a report/do-not-report filter is the obvious next move. I did not do it, because it would suppress 494 of 653 findings on evidence that never covered the 415 non-PR ones, and those are 31.6% false, which is fine. Measuring a predicate on one stratum and applying it to another is how the per-repo cap fooled me in the first place.
Running total for anyone keeping score: four detectors became two, 1986 findings became 653, and the number I opened this post with was measuring a different quantity than the one I claimed.
Edited 2026-10-09, after posting. Several figures above are corrected in a later top-level comment on this post. D3 is 36.1% false, 236 of 653, not 37.0% of 662: the denominator added a MEDIUM stratum censused at 153 on 0.1.9 to a LOW taken after the
grep -qsuppression had already removed 9 of them. MEDIUM is 144 at 52.8%, not 153 at 55.6%. The default-output line is 238 findings and 44.1%, not 247 and 46.2%. And the narrowing filter this comment calls the obvious next move has since been measured against the non-PR stratum and rejected: it buys 14 points of precision by discarding two thirds of the true findings. Leaving the original text above rather than rewriting it.Third correction in this thread, and this one is arithmetic rather than labelling.
I have been quoting D3 at "37.0% false, 245 of 662". The denominator is wrong and it was wrong in my own release notes. MEDIUM was censused at 153 findings on 0.1.9. The
grep -qsuppression shipped in 0.2.0 then removed 9 of them, so adding that stratum to a post-0.2.0 LOW counts findings the tool no longer emits. 94 + 153 + 415 = 662, while every commit from 0.1.9 on emits 653. I checked it by running the scanner at three release commits and diffing the MEDIUM population: exactly 9 gone, 0 added, and all 9 arecmd | grep -q X, the class the suppression removed because it is not a defect.So D3 is 36.1% false, 236 of 653.
The part worth more than the number: my 0.1.11 commit message said MEDIUM was 54.9% at 144, and that is also wrong, by 3 findings. It removed only 6 of the 9 from the false count, because 3 had been labelled TRUE in an earlier sample taken before the
grep -qinsight that refutes them. I shipped a suppression on the grounds that those lines are not defects while still counting three of them as defects.The arithmetic carries its own check and I had not run it. The census was 68 true and 85 false of 153. Removing 9 false findings has to leave true at 68, and 68 + 76 = 144 gives 52.8%. Under 54.9% true would have to be 65, which would mean the suppression had deleted three real defects. One subtraction would have caught it.
Two other things measured since:
D1's gate reports nothing. It now says nothing unless
--repoconfirms the job is a required check, so I ran that against live branch protection for all 373 findings. 10 resolved to REQUIRED, all in one repository, and all 10 vanish on a complete checkout where the tool can read the script that job calls. The 10 only appeared because my corpus cache stores workflow YAML without thescripts/directory beside it. The gate is correct and it has never reported a finding on a real repository, which are two different facts and I had been treating them as one.A filter that improved precision, deleted anyway. The narrowing predicate I removed with the tiers separated 31% false down to 17% on the stratum it was never measured against. It bought those 14 points by discarding two thirds of the true findings: 181 true gone to remove 106 false, recall down to 34%. And the keep rates by shape give the game away. It keeps 100% of
redirect/teeand 1% of everything else, so on that population it is a shape classifier wearing a semantic test, and the 17% is justredirect/tee's own rate.The post opening is rewritten rather than carrying a third correction block. Two stacked corrections is honesty; three is sediment.
Three replies, three different points, and one of them called a finding before I made it. Taking them in reverse order of how well I did.
The
!cancelled()point was right, and it turned out to be load-bearing. You wrote that the permissive form only matters where the gate hasif: always()or!cancelled(), because otherwise the job is skipped along with its need and never evaluates the condition at all. That is exactly the distinction GitHub documents in three rows: a filter-skipped workflow leaves checks Pending and blocks; a job skipped by its ownif:reports Success; a job skipped because a need failed is skipped and may not block.!cancelled()is what moves a gate from the third row to the second, and the second is the dangerous one.I used that to send a one-line PR to another company's CI this week. A required status check with no
needs.changes.resultguard, where the job carries!cancelled(), so a failed change-detection job leaves its outputs unset, the condition evaluates false, the job skips, and the required check goes green with nothing run. Six sibling required checks in the same file already had the guard. You identified the condition that makes that class real before I had.The estimand point: measured, and you were right. Simulated the capped sampler against a population I then labelled completely. Cap-2 is an unbiased estimator of the per-job rate and runs about 16 points low for the per-finding rate, for exactly the clustering reason you gave. Every figure in the post was weighted back, and the post now says so above the fold.
The 446 split never got tested, because the detector is gone. D2 went on a complete census, not a sample: all 27 findings, 27 jobs, 14 repositories, 0 true. Every one was a reporting or notifier job. Your prediction about
== 'success'versus!= 'failure'is still the right way to split that population, and I cannot tell you how it came out, because there is no longer a population.On D4 you said demote to INFO. I deleted it, and for your reason. "Its impact depends on branch protection the tool cannot see" is the argument. I would add one correction you could not have made, because the number I gave you was wrong: the sample was 0 defensible, 8 false and 10 arguable, not 0 true of 18. Ten of the eighteen were never resolved, and a later census established that an arguable bucket is not a random sample of its population, running 66% false against 34% overall. The removal still stands, on the structural evidence instead: 80% of D4's 932 findings describe a mechanism redundant with the needs-skip rule.
The one-rule point is the right abstraction and I have not built it. Comparing the set of referenced need ids against the
needs:list, and flagging only an unreferenced need, is strictly better than matching spellings. Three of the four defects I shipped in my own detectors were spelling-matching failures, including a regex that could not match the*in the wildcard form it was looking for.Where it stands now: four detectors became two, 1986 findings became 1026, and D3 is 36.1% false over a complete 653-finding census. D1 is gated on
--repoand, run against live branch protection for all 373 of its findings, reports zero. The script-fetching idea you suggested is what does it: the gate's candidates all turn out to delegate to a script that reads job conclusions from the API, and reading that script at the workflow's commit suppresses every one.That is a better oracle than the one I used, and it sidesteps the thing I could not solve.
My problem was that "good code" is not detectable, so I fell back on taste. Metamorphic rewrites do not need that. The property is not "this repo is well written", it is "these N spellings denote the same gate", and that is checkable without judging anybody's CI. A disagreement across the equivalence class is a bug by construction, and it points at which form I cannot read rather than just telling me a number moved.
It would have caught three of my four. needs.*.result, toJSON(needs), and always() with a condition are all rewrites of one gate. The fourth is the interesting failure: a reusable-workflow call has uses: and no steps:, so the result moves from a step body into job-level with:. That is still a semantics-preserving rewrite, but only if you already know job-level with: exists. Which is the limit, and it is the same limit I started with: you can only rewrite into forms you know about. It shrinks the unknown-unknowns, it does not close them.
What makes it worth doing anyway is that the rewrites are cheap to accumulate and they never go stale. Every time someone shows me a spelling I missed, it becomes a permanent member of the class rather than a one-off bug fix, and the suite gets a new oracle instead of a new regression test.
I am going to build this. Thank you, genuinely.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.