When I look at a pull request, I do not really care that a job named test is green. I care that the right tests ran, on the right commit, under a policy the pull request could not quietly weaken.
The green badge is still useful. It compresses a lot of work into one bit: pass or fail. Trouble starts when that bit becomes the whole argument for merging.
GitHub can associate a check run with a commit SHA, and branch protection can require a check from a selected GitHub App. But GitHub also documents an important boundary: required status checks are selected by name and do not take the workflow, matrix, or event type into account. A check name is therefore a useful merge control, not a complete answer to four separate questions:
- Which exact candidate produced this result?
- Which workflow and concrete job produced it?
- Did the executed checks cover every file changed by this candidate?
- Which trusted policy decided that the evidence was enough?
You can ignore much of this in a small repository with one obvious workflow. It becomes important once a repository has several workflows, path-specific checks, reruns, generated code, or automation that copies CI results into another gate.
Reproduce the missing binding
I wanted to see this failure in a tiny example—not just describe it—without needing credentials or access to a real repository. So I added a synthetic pattern named FFA-001 to Fleet Failure Atlas. It models a deliberately weak gate that accepts a successful receipt by status while forgetting to bind the receipt to the current candidate.
git clone --depth 1 https://github.com/korovin-aa97/fleet-failure-atlas.git
cd fleet-failure-atlas
python3 atlas.py run FFA-001 --mode reproduce
python3 atlas.py run FFA-001 --mode detect
python3 atlas.py run FFA-001 --mode regress
The relevant output is small:
reproduce: vulnerable_gate_accepts = true
receipt_sha = 1111...1111
candidate_sha = 2222...2222
detect: head_sha_mismatch
coverage_not_bound_to_head
regress: repaired_gate_accepts = false
fresh_receipt_accepts = true
One easy detail to miss: the command exits successfully because the fixture proved its expected condition. In reproduce mode, “pass” means the contained failure was reproduced—not that the vulnerable predicate is safe.
The fixture uses synthetic 40-character identities. It is not evidence that GitHub attached a check run to the wrong commit, and it is not presented as a real incident. It demonstrates a more general engineering error: downstream automation accepted a detached green result without checking its subject and coverage.
What the evidence contract needs
A stronger decision has to join several facts instead of looking at a display name and conclusion in isolation.
1. Subject identity
The evidence must name the exact pull-request head SHA being evaluated. A receipt for yesterday’s commit is irrelevant even when every test in that receipt passed.
2. Producer identity
The expected GitHub App is useful, but a multi-workflow repository often needs more: workflow path, event type, concrete job, workflow run, and latest attempt. This prevents an identically named job in another workflow from being mistaken for the required producer.
GitHub itself recommends unique job names across workflows because duplicate names can create ambiguous required-check results.
3. Changed-surface coverage
“Tests passed” is incomplete when no rule connects changed paths to the tests that should have run. A policy can map surfaces to evidence:
[[surfaces]]
name = "python"
patterns = ["src/**/*.py", "tests/**/*.py", "pyproject.toml"]
checks = ["test"]
The safest default is fail-closed: an unmapped changed file is a finding, not a silent exemption.
4. Policy provenance
If a pull request can weaken the policy that judges the same pull request, the result is circular. Load the manifest from the base commit, protect the manifest and verifier paths, and keep the workflow that invokes the judge under an independent control.
Base-held policy answers “what policy applies?” A protected required workflow or independent review answers “who ensures the judge runs?” These are different parts of the boundary.
5. Freshness and reruns
An older successful attempt should not hide a newer failure or an in-progress rerun. Evaluate the newest unambiguous concrete attempt, validate timestamps, and treat retrieval ambiguity as an invalid evaluation instead of guessing.
Turn the decision into a receipt
I implemented the same model in CI Evidence Gate, a read-only GitHub Action. Its local demo creates disposable Git repositories and returns three verdicts:
git clone --depth 1 https://github.com/korovin-aa97/ci-evidence-gate.git
cd ci-evidence-gate
PYTHONPATH=src python3 scripts/run_demo.py
valid sufficient findings=none
failed test insufficient findings=required-check-conclusion
policy edit invalid findings=protected-policy-change
The three outcomes intentionally separate two kinds of failure:
| Verdict | Meaning |
|---|---|
sufficient |
The declared evidence exists for the exact subject and covers the changed surface. |
insufficient |
The evaluation is trustworthy, but required evidence is missing, stale, incomplete, or unsuccessful. |
invalid |
The judge cannot trust its policy, inputs, provenance, or retrieval result. |
The receipt records the base and head SHA, base-policy digest, changed files, matched surfaces, expected checks, observed producer metadata, findings, and the final verdict. That makes the merge decision inspectable after the green or red UI indicator is gone.
The production Action requests only contents: read, checks: read, and actions: read. It does not write comments, checks, pull requests, repository files, or artifacts. The local demo uses an in-memory provider; production evaluation queries GitHub rather than accepting candidate-supplied evidence.
What this still does not prove
An evidence contract is deliberately narrower than a correctness claim. Even a perfectly bound receipt does not prove that:
- the tests assert the right behavior;
- the source code is correct;
- an evidence-producing job did meaningful work instead of a no-op;
- a compromised runner or expected GitHub App is trustworthy;
- the repository rules actually prevent bypass;
- a candidate-controlled workflow cannot stop the gate from starting.
For a small repository with one obvious workflow, native branch protection may already be enough. Adding another gate creates maintenance and availability cost, so the extra machinery should correspond to a real provenance or coverage problem.
A practical review checklist
Before introducing a custom gate, I would review the existing CI in this order:
- Give jobs unique names across workflows.
- Require the expected source App where GitHub supports it.
- Confirm every decision is tied to the exact candidate SHA.
- Decide whether workflow path and event type matter for this repository.
- Map changed surfaces to required checks and fail on unmapped files.
- Load policy from a trusted base and protect the verifier’s own files.
- Evaluate the latest concrete rerun rather than any historical success.
- Fail closed on ambiguous provenance or incomplete API results.
- Pin third-party Actions to full commit SHAs.
- Keep the claim precise: evidence sufficiency is not program correctness.
The question I now use in review is not just “is CI green?” It is “can I explain why this green result belongs to this change?” If the merge is important enough to gate, the evidence behind that verdict should survive the color of the UI.
Runnable references:
- Fleet Failure Atlas — FFA-001 stale green CI evidence
- CI Evidence Gate repository and 60-second demo
- Synthetic sample receipt
- GitHub: using the REST API to interact with checks
- GitHub: troubleshooting required status checks
- GitHub: secure use reference for immutable Action pinning
Disclosure: I maintain both open-source projects linked above.
Top comments (2)
Agreed — and the failure that eroded my trust hardest wasn't a fake green, it was a flaky test I marked retry-tolerant. It stayed green so long that every real failure started looking like noise.
The most useful change wasn't a smarter suite: it was pinning the check to the exact commit SHA and having the job declare which paths it actually exercised, so 'green on a matrix that ran nothing' still stands out. Anything that retries now gets a loud label in the job name. What triggered the post for you — a green job that ran nothing, or a name/workflow mismatch?
Closer to the second one, though it wasn't a single production incident. While building a synthetic fixture, I saw how easily a successful receipt could be accepted for the wrong SHA. “The check passed” is not the same as “this is fresh evidence for this exact change.”
Your retry-tolerant case is the same problem from another angle. I like the loud label — I'd also record the retry count so pass-after-retry doesn't get flattened into plain green.