AI coding agents — Devin, Copilot coding agent, Codex, Claude Code, Cursor — open and merge pull requests at scale now. Every agent vendor knows, at the moment of generation, exactly what the task was. My question: does any of that intent survive into the PR in a form a machine could check? And while I was looking: how often do agent PRs cross boundaries that deserve human eyes?
So I scanned 2,204 recently merged, agent-authored PRs on public GitHub repos with a checkout-free policy engine, using only its built-in default policy. It reads PR metadata and file contents through the GitHub API — it never executes PR code and never calls an LLM, so every finding is deterministic and replayable.
The one number that surprised me
0 of 2,204 PRs declared a machine-readable scope for the change. Not a low percentage — zero. Agent vendors emit rich task context at generation time, and none of it reaches the PR as something a machine could verify. If you want to know whether an agent PR stayed inside its intended task, there is currently nothing to check it against.
What else showed up
7.0% of complete analyses had at least one boundary finding (153 of 2,191). The structure underneath that number is the interesting part:
- Of the 349 PRs that touched GitHub Actions workflows or package manifests, 12.9% escalated workflow permissions and 17.5% introduced unpinned actions, reusable workflows, or containers. Workflow-touching agent PRs are where the risk concentrates.
- 3.9% changed agent control-plane files — AGENTS.md, CLAUDE.md, .mcp.json, and similar. These files steer every future agent PR in the repo, which makes them a quiet privilege-escalation path: an agent that edits its own instructions today shapes what the next agent does tomorrow.
- Repos with 10k+ stars showed a 4.3% finding rate — roughly half the long-tail rate (8.6%). Established projects have guardrails. The long tail of small repos, where most agent PRs actually land, is where agents run with the least oversight.
Honest denominators
Percentages hide choices, so here are mine: workflow-rule rates use PRs that actually touched workflow or manifest content; control-plane rates use all complete analyses; the contract statistic uses everything. Incomplete analyses fail closed and are reported as their own bucket, never silently dropped. A "finding" is not an accusation — most of the escalations I saw are probably benign. That is exactly the point: nobody declared them, nobody checked them, and benign-until-it-isn't is not a security posture.
The missing primitive
The zero is the story. Agent PRs today are reviewed the way human PRs are — by reading the diff — but agents differ from humans in one reviewable way: their intent is machine-generated and could be machine-checkable. A PR-body contract as small as this would close the loop:
<!-- mergewarden-contract
version: 1
agent: codex
task: update session expiry handling
allowed_paths:
- src/auth/**
- test/auth/**
-->
The contract is an untrusted declaration — the base-branch policy stays authoritative — but once it exists, "did the PR leave its declared scope" becomes a deterministic check instead of a reviewer's guess. I'd rather see this become a vendor-neutral convention than one tool's feature; the format above is one concrete proposal.
Reproduce it
Every query, date window, and aggregation script is published in the methodology writeup. The scanner (MergeWarden, MIT) runs against any public PR without installing anything:
npx mergewarden scan owner/repo#123
Full methodology and the tool: https://github.com/sjh9714/mergewarden
If you maintain a repo that receives agent PRs and want the scan results for your own recent PRs, open an issue — I'm looking for maintainers to help measure false-positive rates against real-world judgment. And if you've seen an agent PR quietly cross a line in your own repo, I'd love to hear about it in the comments.
Top comments (0)