Your open-source project probably added an AI-contribution policy in the last few months. Apache has one. The Linux Foundation has one. The OpenSSF Technical Advisory Council is finalizing a foundation-wide one right now. Bitcoin Core, ripgrep, uv, and dozens of smaller projects have copied the pattern. If you maintain something popular, you've either written one or you're about to.
Almost all of them say some version of the same three things: disclose when you use AI; a human must review and remain responsible for every change; no fully autonomous agent PRs.
These are good norms. But I want to point at something uncomfortable: as written, almost none of them are enforceable. They're prose in a CONTRIBUTING.md, and prose is an honor system. So I went looking for how big the gap actually is.
What agent PRs actually do at scale
I scanned 2,204 recently merged, agent-authored pull requests across public GitHub repos (Devin, Copilot coding agent, Codex, Claude Code, Cursor). The scan is deterministic and checkout-free — it reads PR metadata and file contents through the GitHub API, never executes PR code, and never calls an LLM, so every finding replays. Three results are directly relevant to anyone writing one of these policies:
0 of 2,204 declared a machine-checkable scope for the change. Not a low number — zero. Your policy can say "a human must review and understand every change," but a reviewer opening an agent PR has exactly what a reviewer of a human PR has: the diff, and a description written in prose. The agent knew precisely what task it was given. None of that intent survives into the PR in a form anything can check against. So "review every change" is real, but "verify the change matched its intent" isn't available to you yet.
3.9% modified agent control-plane files — AGENTS.md, CLAUDE.md, .mcp.json, .cursor/rules/**, and similar. These are the files that steer future agent runs in your repo. An agent PR that edits its own instructions is a quiet privilege-escalation path, and it's exactly the kind of change that reads as a boring docs diff and sails through review. Most policies don't mention this file class at all.
Of the PRs that touched CI workflows, ~13% raised GitHub Actions permissions, and ~17% introduced unpinned actions. Your DCO and attribution language will never catch a contents: read -> write bump in a workflow file. That's not a licensing question; it's a supply-chain one, and it's invisible to the parts of your policy that are about authorship.
One more, because it decides where to aim: repositories with 10k+ stars had roughly half the finding rate of the long tail (4.3% vs 8.6%). Established projects have guardrails. The projects getting hit hardest are the small, beginner-friendly ones — the exact repos that just adopted a prose policy and have the least review bandwidth to back it up.
(Full methodology, every query, and the raw approach are public: https://github.com/sjh9714/mergewarden/blob/main/docs/study/methodology.md — findings are review evidence, not vulnerabilities or misconduct claims; I publish aggregates only and name no repositories.)
Which policy clauses are actually checkable
Here's the useful reframe. Split your policy into clauses you hope people follow and clauses a machine can check:
| Clause (typical wording) | Enforceable automatically? |
|---|---|
| "You could have written this yourself" | No — honor system |
| "Explain your changes in your own words" | No — honor system |
| "Disclose AI use" | Partly — require a checkbox/trailer, but you can't verify honesty |
| "No autonomous-agent PRs" |
Yes — they arrive on cursor/**, codex/**, copilot/** branches or *-swe-agent[bot] authors |
| "Don't touch files that steer future agents" | Yes — a path-glob diff check |
| "Don't quietly escalate CI permissions" |
Yes — diff the workflow permissions: block, base vs head |
| "Stay within the scope of the task" | Yes, but only if scope is declared — the missing primitive |
The last row is the interesting one. If an agent PR carried a tiny machine-readable declaration of what it intended to touch, "did it stay in scope" becomes a deterministic check instead of a reviewer's guess:
<!-- ai-contract
version: 1
agent: codex
task: update session expiry handling
allowed_paths:
- src/auth/**
- test/auth/**
-->
I'd honestly rather see something like this become a vendor-neutral convention than any one tool's feature. Agent vendors already have the task context at generation time; nothing carries it into the PR. That's the gap under every one of these policies.
The point
If you're writing an AI-contribution policy: keep the prose norms, they matter. But separate the honor-system clauses from the ones a cheap, deterministic check can actually enforce, and wire the second group into CI. The three checks worth adding first, in order of bang-for-effort: (1) flag control-plane edits (AGENTS.md / .mcp.json / rules files) for mandatory human review; (2) diff workflow permissions base-vs-head and flag any escalation; (3) detect autonomous-agent PRs by branch/author so they get the extra scrutiny your review bandwidth is already rationing.
None of that needs an LLM, none of it needs to check out PR code, and all of it is boring, replayable, and free. The policy tells people what's expected; a check tells you when it didn't happen.
I built an open-source tool (MergeWarden, MIT) that does exactly these checks as a GitHub Action, which is what produced the numbers above — but the argument stands whatever you use, including a few lines of your own CI. The takeaway isn't "adopt my thing." It's: your policy is prose, agent PRs are a firehose, and the enforceable half of your policy is smaller and cheaper to automate than you think.
Top comments (0)