Where this landed, 9 October 2026. This post opens with 40%. That number is not
wrong so much as it is not the quantity you would take it for, and...
For further actions, you may consider blocking this person and/or reporting abuse
The 40% is a pooled count over 4,543 files, but the errors cluster by repository: 92% on openinference, 86% on ruff, 0 where there is no fan-in gate. Findings from one repo share a cause, so an interval over 1,104 findings looks far tighter than it is. Resample whole repos (or report the per-repo false rate and its median) and say how many of the 275 contribute most of the 446 HIGHs that disappeared.
The number I'd want next is the precision of the 658 that remain. Both of your false-positive classes were found by pointing the tool at repos you respected, so there are probably classes in the survivors that no one has looked at. A random sample of ~40 of the 658, stratified by repo and hand-labelled, gives a Wilson interval on "after"; 3 false out of 40 and 12 out of 40 are very different tools, and 30 of 34 on two chosen repos can't distinguish them.
The surviving mutant is a nice catch. The analogue for the corpus is the same check: delete the transitive-walk suppression and see which of the 446 come back. If most are on the repos with careful gates, that confirms the walk was the right suppression for them and not something that hid real findings.
You are right, and I went and ran the per-repo breakdown you asked for. It is worse for my framing than you put it.
The 446 removed HIGH land on 23 of the 275 repositories. The top 2 carry 50% of them and the top 7 carry 80%:
Per-repo false rate: median 0.0%, mean 13.3%, against the pooled 40.4% I published. 87 of the 110 affected repositories had zero false HIGH. So the pooled number is close to a statement about ClickHouse and sglang, and the interval I did not compute would have been meaningless if I had.
The part that actually stings: my post said the errors concentrated on repositories that write their gates carefully, and I used openinference and ruff as the evidence. Those two are real, but they are 18 and 13 of 446. The mass is in large projects with elaborate release pipelines, which is a different claim with a different cause. You identified a clustering problem and the clustering also falsifies my explanation of it.
On precision of the 658 survivors, I agree that is the number that matters now and I do not have it. A stratified sample of 40 with a Wilson interval is the right size, and I will hand label it. I will say in advance that 30 of 34 on two chosen repos cannot distinguish a good tool from a bad one, because that is exactly the mistake I made in the other direction.
On the corpus analogue of the surviving mutant: that experiment is available and I will run it. One correction on the mechanism, since it matters for what the result would mean. The transitive walk was not a suppression I kept, it was one I deleted, because needs.*.result reports only direct needs and a failed grandparent makes the parent skip rather than fail. The suppression that survives is the direct-only one. So the test is to disable that and see which of the 446 come back, and your prediction is the right one to check: if they are concentrated on the repos with real fan-in gates, the suppression was doing its job.
Thank you for this. It is the most useful thing anyone has sent me about this project.
Worth splitting the 446 by how the fan-in gate tests its needs, not just by repo. Because needs..result only reports direct needs, a failed grandparent shows up as 'skipped' on the parent. A gate written as
== 'success'on all needs still catches that chain; one written as!= 'failure'or `contains(needs..result, 'failure')lets it through. So when you disable the direct-only suppression, the ones that come back should mostly be the second form, and that is a checkable prediction: if they are== 'success'` gates, the suppression was hiding something else.Ran the disable. Your prediction does not hold on what I can classify, and more than half of it I cannot classify at all.
Disabling the direct-only suppression takes 658 HIGH to 771, so it is hiding 113. All 113 are D4, which lines up with D4 being the detector that produced zero defensible findings in the sample.
Splitting those 113 by how their workflow's gate tests its needs:
So among the ones I can read, the whitelist form outnumbers the blacklist form, which is the opposite of the prediction. By your own reasoning a whitelist gate does catch the skipped-parent chain, so for those 27 the suppression was doing the right thing.
The 55% is the honest problem and it is not a bug I can fix. Those gates delegate the decision to a script:
The gate's semantics are in a file that is not the workflow. I have the workflows, not the repos, so whether that script demands success or merely rejects failure is not knowable from what I am holding. Any number I reported for those 62 would be me guessing and calling it a measurement.
What survives: 19 findings where the gate is the permissive form and the suppression may genuinely be hiding a real chain failure. That is a small enough set to read by hand, and it is the next thing I will do rather than the next thing I will claim.
Two caveats on the 24 vs 17 split, since it is the only part that looks like a result. The classifier is a regex over the gate's text, so it has exactly the weakness that started this whole thread. And n is small enough that the difference between 27 and 19 is not much of a difference.
Your framing was still the useful part. I had been treating "the gate reads its needs" as one property, and it is two: whether it reads them, and whether reading them is strict enough to catch a skip. My suppression only checks the first.
The 19 are worth reading by hand, with one filter first: a gate only reaches the permissive-form problem if it actually runs when a need is skipped. A job with the default
ifis itself skipped when any need fails or is skipped, and a skipped job counts as passing for a required check, so those gates have a different bug (green on upstream failure) and the!= 'failure'form is never evaluated. The permissive form only matters where the gate hasif: always()or!cancelled(). That may shrink the 19 further.For the 62 that delegate to a script: the script sits in the same repo as the workflow, so it can be fetched at the workflow's commit. A first pass is whether it compares each result to 'success' or only tests for 'failure'/'cancelled'. Still a regex, so hand-label those, but it turns "unclassifiable" into a readable list instead of a guess.
Both of your suggestions were right, and together they pointed at a third property that makes most of the strictness question moot. Also I owe you a correction: the 40% in the article was measuring something other than what I said it was.
Your filter first. You said the permissive form only matters where the gate has always() or !cancelled(), because a gate with the default if is itself skipped when a need fails, which is a different bug. That is the predicate I already gate on, so every number I gave you was post-filter, which I should have said. Re-running it on the full corpus, counting every (gate, dependent-with-upstreams) pair:
The split is watched / not watched, which is the third property. If a gate needs a job AND every job upstream of that job, then any real failure in the chain reaches it as that upstream's own failure result, and even the permissive != 'failure' form rejects that. The only skip left is a condition the author deliberately evaluated false, which is intent, not a defect.
So the question is not "is this gate strict about skipped", it is "does this gate watch the whole chain", and unlike the first, a workflow file can answer it. 89% are chain-watched whatever form the gate takes. 5596 of the 5824 unclassifiable ones, 96%, are settled without reading the script at all. The residue where strictness still matters and the file still cannot say is 257, not 55% of everything.
Your second suggestion I shipped, and it found a class I did not have. Fetching the script at the workflow's commit is now what the tool does: when a named gate shells out with a token in scope, it reads the script and either suppresses, keeps full severity if the script demonstrably does not consult status, or demotes and says the claim is unverified. The first repo it ran on was WordPress/gutenberg, whose status checks page GET /actions/runs/{id}/jobs and fail if any job concluded as anything but success or skipped. The word result never appears and needs is never consulted, so no pattern over the workflow could ever have found it. That gate is better than the idiom I was looking for: it also fails closed when a job was never evaluated, which needs.*.result cannot detect. Ten of my 54 D1 findings were that, all false, all one repo.
One caution on direction, because I got it wrong once and a mutation caught me. Suppressing a job because its PARENT is covered is unsound: needs.*.result reports only direct needs, and a failed grandparent makes the parent skip rather than fail. Requiring the gate to watch the whole chain itself is the strict version of that idea, not the loose one.
Now the correction. I pre-registered a sample over the surviving 179 HIGH and committed to posting whatever it said. D1 came back 33% false, having measured 60% a round earlier with nothing in its code changed between. A control moving 27 points is either a bad draw or my labelling drifting, so I checked instead of choosing: the two seeds overlapped on 7 of 15 draws and all 7 got the identical label both rounds. Not drift.
D1's whole population was 54 findings, so I stopped sampling and labelled every one. 76% false per finding, 59% per job, exact. Sample 4's 95% interval was [15%, 58%] and did not contain the truth.
The reason is my own design. I cap at 2 findings per repository so no single repo dominates. But the tool emits one finding per (job, upstream) pair, and the false positives pile up in jobs with many dependencies: one reporting job emits 11 by itself. The cap deletes exactly that clustering. Against the censused truth, 20k draws at n=15:
The capped sampler is an unbiased estimator of the per-JOB rate and runs about 16 points low for the per-FINDING rate. Every figure I have published was weighted back to per-finding populations, so all of them were per-job rates wearing a per-finding label, all understating, all the same direction. The 40% is not false, it just is not the quantity a reader takes it for. Related and smaller: those 179 findings were 136 distinct problems. I had been reporting a finding count as a problem count.
D1 now tops out at MEDIUM. 65% false after every fix, with 11 of the survivors one reporting job, is not a tier that gets to keep asserting HIGH on its own authority, so HIGH is reserved for the path where branch protection actually confirms the job is required. Corpus HIGH went 354 to 125, all D3, bounded at 21 to 33% false per finding. 0.1.4 is up.
What I keep relearning: no amount of running the measurement again would have shown this, because the bias was in the sampling design and the design was mine. Censusing 54 findings cost less than the two samples that disagreed about them.
Third correction in this thread, and this one is arithmetic rather than labelling.
I have been quoting D3 at "37.0% false, 245 of 662". The denominator is wrong and it was wrong in my own release notes. MEDIUM was censused at 153 findings on 0.1.9. The
grep -qsuppression shipped in 0.2.0 then removed 9 of them, so adding that stratum to a post-0.2.0 LOW counts findings the tool no longer emits. 94 + 153 + 415 = 662, while every commit from 0.1.9 on emits 653. I checked it by running the scanner at three release commits and diffing the MEDIUM population: exactly 9 gone, 0 added, and all 9 arecmd | grep -q X, the class the suppression removed because it is not a defect.So D3 is 36.1% false, 236 of 653.
The part worth more than the number: my 0.1.11 commit message said MEDIUM was 54.9% at 144, and that is also wrong, by 3 findings. It removed only 6 of the 9 from the false count, because 3 had been labelled TRUE in an earlier sample taken before the
grep -qinsight that refutes them. I shipped a suppression on the grounds that those lines are not defects while still counting three of them as defects.The arithmetic carries its own check and I had not run it. The census was 68 true and 85 false of 153. Removing 9 false findings has to leave true at 68, and 68 + 76 = 144 gives 52.8%. Under 54.9% true would have to be 65, which would mean the suppression had deleted three real defects. One subtraction would have caught it.
Two other things measured since:
D1's gate reports nothing. It now says nothing unless
--repoconfirms the job is a required check, so I ran that against live branch protection for all 373 findings. 10 resolved to REQUIRED, all in one repository, and all 10 vanish on a complete checkout where the tool can read the script that job calls. The 10 only appeared because my corpus cache stores workflow YAML without thescripts/directory beside it. The gate is correct and it has never reported a finding on a real repository, which are two different facts and I had been treating them as one.A filter that improved precision, deleted anyway. The narrowing predicate I removed with the tiers separated 31% false down to 17% on the stratum it was never measured against. It bought those 14 points by discarding two thirds of the true findings: 181 true gone to remove 106 false, recall down to 34%. And the keep rates by shape give the game away. It keeps 100% of
redirect/teeand 1% of everything else, so on that population it is a shape classifier wearing a semantic test, and the 17% is justredirect/tee's own rate.The post opening is rewritten rather than carrying a third correction block. Two stacked corrections is honesty; three is sediment.
Three replies, three different points, and one of them called a finding before I made it. Taking them in reverse order of how well I did.
The
!cancelled()point was right, and it turned out to be load-bearing. You wrote that the permissive form only matters where the gate hasif: always()or!cancelled(), because otherwise the job is skipped along with its need and never evaluates the condition at all. That is exactly the distinction GitHub documents in three rows: a filter-skipped workflow leaves checks Pending and blocks; a job skipped by its ownif:reports Success; a job skipped because a need failed is skipped and may not block.!cancelled()is what moves a gate from the third row to the second, and the second is the dangerous one.I used that to send a one-line PR to another company's CI this week. A required status check with no
needs.changes.resultguard, where the job carries!cancelled(), so a failed change-detection job leaves its outputs unset, the condition evaluates false, the job skips, and the required check goes green with nothing run. Six sibling required checks in the same file already had the guard. You identified the condition that makes that class real before I had.The estimand point: measured, and you were right. Simulated the capped sampler against a population I then labelled completely. Cap-2 is an unbiased estimator of the per-job rate and runs about 16 points low for the per-finding rate, for exactly the clustering reason you gave. Every figure in the post was weighted back, and the post now says so above the fold.
The 446 split never got tested, because the detector is gone. D2 went on a complete census, not a sample: all 27 findings, 27 jobs, 14 repositories, 0 true. Every one was a reporting or notifier job. Your prediction about
== 'success'versus!= 'failure'is still the right way to split that population, and I cannot tell you how it came out, because there is no longer a population.On D4 you said demote to INFO. I deleted it, and for your reason. "Its impact depends on branch protection the tool cannot see" is the argument. I would add one correction you could not have made, because the number I gave you was wrong: the sample was 0 defensible, 8 false and 10 arguable, not 0 true of 18. Ten of the eighteen were never resolved, and a later census established that an arguable bucket is not a random sample of its population, running 66% false against 34% overall. The removal still stands, on the structural evidence instead: 80% of D4's 932 findings describe a mechanism redundant with the needs-skip rule.
The one-rule point is the right abstraction and I have not built it. Comparing the set of referenced need ids against the
needs:list, and flagging only an unreferenced need, is strictly better than matching spellings. Three of the four defects I shipped in my own detectors were spelling-matching failures, including a regex that could not match the*in the wildcard form it was looking for.Where it stands now: four detectors became two, 1986 findings became 1026, and D3 is 36.1% false over a complete 653-finding census. D1 is gated on
--repoand, run against live branch protection for all 373 of its findings, reports zero. The script-fetching idea you suggested is what does it: the gate's candidates all turn out to delegate to a script that reads job conclusions from the API, and reading that script at the workflow's commit suppresses every one.That is a better oracle than the one I used, and it sidesteps the thing I could not solve.
My problem was that "good code" is not detectable, so I fell back on taste. Metamorphic rewrites do not need that. The property is not "this repo is well written", it is "these N spellings denote the same gate", and that is checkable without judging anybody's CI. A disagreement across the equivalence class is a bug by construction, and it points at which form I cannot read rather than just telling me a number moved.
It would have caught three of my four. needs.*.result, toJSON(needs), and always() with a condition are all rewrites of one gate. The fourth is the interesting failure: a reusable-workflow call has uses: and no steps:, so the result moves from a step body into job-level with:. That is still a semantics-preserving rewrite, but only if you already know job-level with: exists. Which is the limit, and it is the same limit I started with: you can only rewrite into forms you know about. It shrinks the unknown-unknowns, it does not close them.
What makes it worth doing anyway is that the rewrites are cheap to accumulate and they never go stale. Every time someone shows me a spelling I missed, it becomes a permanent member of the class rather than a one-off bug fix, and the suite gets a new oracle instead of a new regression test.
I am going to build this. Thank you, genuinely.