The four-question split is the useful bit, because separating subject from producer is what makes a rerun answerable at all.
On the no-op item in your "does not prove" list: I hit that in a shape the gate could never catch, because the skip decision lived inside the test process. Every live suite opened with const run = process.env.API_KEY ? describe : describe.skip, so with no secret vitest exits 0 and the only trace is "2 skipped" in a log nobody opens. Green meant "the contract holds" and "nothing was checked" identically.
Moving the guard out into its own job fixed it, because then the skip arm shows up in the job list, where a name-based required check can see it. Does that sit inside your producer-identity question, or is it a fifth one?
Producer identity — and I think the reason it feels like it might be a fifth question is worth naming.
Your original shape had a producing job with an identifiable name (test) and a green conclusion; what had no identity anywhere was the decision that the suite applied. So the receipt was complete about the producer it names and silent about the actor that scoped it. Same layer as "which workflow and concrete job produced this result", just with two roles in one process — runner and applicability judge — where only one of them reaches the check-run namespace. The article's own "does not prove" list parks this as "an evidence-producing job did meaningful work instead of a no-op", which reads like an evidence-layer limit; I'd argue it is a provenance one, because a no-op is only invisible while the actor that chose it cannot be named.
What your fix actually does is make the decision nameable, and the name is the currency here: required checks select by name, so a decision that never gets one can never be required, forbidden, or read as missing. Worth being precise about the residual hole, though — selection sees a job, not what the job did. A job that shells out and swallows a non-zero exit is green under an excellent name. The split upgrades a silent decision into a checkable one; it does not audit the shell.
If you want the whole family covered with one rule instead of one job per guard, evaluate the run's structured report in a separate required check: assert expected > 0 and skipped == 0 over the required set from --reporter=json. That catches describe.skip, a t.skip() inside a case, an early return in beforeAll, and a config-level ignore decision — none of which a job split necessarily catches — and it never asks the test process to grade itself.
One polarity note, because it is the cheapest version of the fix: an absent secret is evidence unavailable, not evidence not applicable. Written the other way round — skip only when the secret exists but is malformed, fail when it is missing — the trap does not exist and no new job is needed. The job split is what the genuinely inapplicable case needs, where the right output is a recorded not_applicable with a reason rather than a green check.
I'd keep it under producer identity, with one important boundary. Once the applicability decision becomes its own named check, the gate can require it. If describe.skip stays inside Vitest, CI Evidence Gate cannot see it today.
The clean bridge is a separate required check over structured test output, for example expected > 0 and skipped == 0. And yes, a missing secret is evidence unavailable, not "not applicable."
This also exposed a gap in FFA-001: the write-up mentions green no-op jobs, but the executable fixture only covers stale SHA binding. I'll split the no-op case into its own runnable pattern. Thanks for the sharp example.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The four-question split is the useful bit, because separating subject from producer is what makes a rerun answerable at all.
On the no-op item in your "does not prove" list: I hit that in a shape the gate could never catch, because the skip decision lived inside the test process. Every live suite opened with
const run = process.env.API_KEY ? describe : describe.skip, so with no secret vitest exits 0 and the only trace is "2 skipped" in a log nobody opens. Green meant "the contract holds" and "nothing was checked" identically.Moving the guard out into its own job fixed it, because then the skip arm shows up in the job list, where a name-based required check can see it. Does that sit inside your producer-identity question, or is it a fifth one?
Producer identity — and I think the reason it feels like it might be a fifth question is worth naming.
Your original shape had a producing job with an identifiable name (
test) and a green conclusion; what had no identity anywhere was the decision that the suite applied. So the receipt was complete about the producer it names and silent about the actor that scoped it. Same layer as "which workflow and concrete job produced this result", just with two roles in one process — runner and applicability judge — where only one of them reaches the check-run namespace. The article's own "does not prove" list parks this as "an evidence-producing job did meaningful work instead of a no-op", which reads like an evidence-layer limit; I'd argue it is a provenance one, because a no-op is only invisible while the actor that chose it cannot be named.What your fix actually does is make the decision nameable, and the name is the currency here: required checks select by name, so a decision that never gets one can never be required, forbidden, or read as missing. Worth being precise about the residual hole, though — selection sees a job, not what the job did. A job that shells out and swallows a non-zero exit is green under an excellent name. The split upgrades a silent decision into a checkable one; it does not audit the shell.
If you want the whole family covered with one rule instead of one job per guard, evaluate the run's structured report in a separate required check: assert
expected > 0andskipped == 0over the required set from--reporter=json. That catchesdescribe.skip, at.skip()inside a case, an earlyreturninbeforeAll, and a config-level ignore decision — none of which a job split necessarily catches — and it never asks the test process to grade itself.One polarity note, because it is the cheapest version of the fix: an absent secret is evidence unavailable, not evidence not applicable. Written the other way round — skip only when the secret exists but is malformed, fail when it is missing — the trap does not exist and no new job is needed. The job split is what the genuinely inapplicable case needs, where the right output is a recorded
not_applicablewith a reason rather than a green check.I'd keep it under producer identity, with one important boundary. Once the applicability decision becomes its own named check, the gate can require it. If
describe.skipstays inside Vitest, CI Evidence Gate cannot see it today.The clean bridge is a separate required check over structured test output, for example
expected > 0andskipped == 0. And yes, a missing secret is evidence unavailable, not "not applicable."This also exposed a gap in FFA-001: the write-up mentions green no-op jobs, but the executable fixture only covers stale SHA binding. I'll split the no-op case into its own runnable pattern. Thanks for the sharp example.