What green CI doesn’t prove after a PR’s base moves

OHCAYGO — Tue, 01 Sep 2026 21:58:53 +0000

A pull request can be conflict-free, have green CI, and still lack evidence that the final merged state was ever checked.

That is the narrow problem behind merge-proof, a small open-source Git / GitHub Action tool I just published.

This is not a claim that AI-authored code is broken. It is not a claim that GitHub CI is broken. It is not a claim that a flagged PR contains a bug.

It is only about evidence.

The strongest objection first

If your repository uses strict required status checks correctly, and especially if it uses a properly configured GitHub merge queue, that is the stronger answer to base drift.

A merge queue tests the combined state before merge. For the specific problem of "did this PR get validated against the base it is actually merging into?", that is better than a diagnostic tool after the fact.

merge-proof is narrower. It is useful where that stronger enforcement is absent, incomplete, advisory, or not easy to inspect. It surfaces the evidence gap: which base commit, which candidate commit, which overlapping files, and which protected-path categories made the result unproven.

If your conclusion is "GitHub already solves the valuable part of this for my repo," that is useful feedback, and I would rather hear it than not.

The merge-state evidence problem

A common PR path looks like this:

A branch is cut from main at commit A.
CI runs on the candidate branch and passes.
main moves to commit B while the PR is open.
The PR still merges cleanly.
The final merged state is now B plus the PR.

The clean merge only tells you Git could combine the text. It does not prove that CI, review, or any other validation checked that exact combination.

This matters most when the base changed in files the PR also changed. There may be no conflict, but the combined behavior was not the state represented by the earlier green check.

What merge-proof checks today

merge-proof currently implements two blocking checks.

BASE_DRIFT_UNVERIFIED — the base advanced after the candidate diverged, and files changed by the candidate were also changed on the base during that interval.

PROTECTED_BOUNDARY — the candidate changes paths classified as expensive to undo if merged without explicit evidence: migrations, schemas, auth, secrets, billing, or access-policy / RLS / RBAC areas.

It also reports STALE_BASE as advisory only, and reports CI and deployment configuration changes as an advisory ci-deploy category. Neither advisory changes the verdict.

The tool runs on local Git state. It has zero npm dependencies. It makes no network calls, no model calls, no API calls, and has no telemetry. In shallow clones it fails closed instead of reporting confidence from history it could not see.

One disclosure: the GitHub Action wrapper's only third-party dependency is actions/github-script@v7, a GitHub-owned action, used solely to post the optional sticky PR comment. The verifier itself has none.

What merge-proof does not check

This release does not read CI results, so it cannot establish CI_RAN_ON_FINAL_HEAD.

It does not read reviews or approvals.

It does not verify that a candidate is durable on a remote.

It does not compare a PR against a declared scope.

It is not a bug detector, security scanner, code reviewer, human-approval verifier, or correctness proof. A VERIFIED result only means the implemented checks did not find their specific evidence gaps.

The study

We ran merge-proof against 2,037 agent-authored PRs across 32 public repositories. 24.1% returned NOT_PROVEN under the implemented checks.

Immediately important: NOT_PROVEN does not mean broken or buggy. It means the repository does not contain evidence that the candidate was validated against the state it merged into. No conclusion about code correctness follows from that verdict. This is a prompt to look, not an accusation.

The exact result:

2,037 usable PRs
42 unresolvable PRs excluded from all rates
490 NOT_PROVEN, or 24.1%
1,547 VERIFIED, or 75.9%
0 FAIL
410 base-drift findings, or 20.1%
109 protected-path findings, or 5.4%
29 PRs triggered both conditions

The study used deterministic Git data. Agent-authored PRs were identified from commit trailers, not model judgment. The scan walked squash merges on the default branch's first-parent history, used the squash commit's first parent as the base-at-merge, fetched the PR head directly, and ran the same analyzer that ships in the package — not a separate research implementation.

It is reproducible from the repository: