The gap between "works" and "stayed in scope"
An AI coding agent can produce a pull request that builds, passes tests, and
implements the requested feature. That same PR can also edit a release
workflow, increase a GitHub token permission, change AGENTS.md, or touch a
file outside the declared task.
That does not make the PR malicious. Still, the reviewer should be able to see
the boundary change without reconstructing it from a large diff. Ordinary tests
answer whether the repository still works. A semantic reviewer asks whether the
code makes sense. Neither reliably records whether the PR stayed inside the
agreed scope.
I built Agent Gate for that narrower
question.
A checkout-free policy gate
Agent Gate is an open-source GitHub Action and CLI. It collects pull-request
metadata and selected file content through GitHub APIs, loads agent-gate.yml
from the exact base commit, and produces deterministic findings.
It intentionally does not:
- checkout or execute pull-request code;
- run target-repository package scripts;
- load policy from the PR head branch; or
- call an LLM at runtime.
That gives the policy a useful trust boundary. A pull request can change the
policy for future PRs, but it cannot weaken the policy evaluating itself.
If GitHub reports 42 changed files and Agent Gate can collect only 41, the
analysis is incomplete and fails closed. Authentication failures, rate limits,
and server errors are not silently treated as missing files or default policy.
Try it before installing it
Node 20 or newer is enough to scan a public PR:
npx --yes @jinhyuk9714/agent-gate@0.3.1 scan owner/repo#123
The CLI clones nothing and executes nothing from the target repository. Human,
JSON, and Markdown output are available, and the Action uses the same analysis
contract.
What the evidence looks like
The public composite proof PR
starts with a docs-only contract and then changes documentation, a workflow,
and AGENTS.md. Its Action run
reports:
- two
contract/out-of-scopefindings; - a high-risk workflow path;
-
agent-control-plane/driftforAGENTS.md; and - workflow- and job-level permission escalation.
The report also records the policy digest, exact base/head SHAs, expected and
analyzed file counts, and whether collection completed. Finding IDs are derived
from canonical evidence rather than display wording, so a waiver can target one
exact finding with a reason and expiry.
Differential workflow checks
Existing workflow risk should not be re-reported whenever an unrelated line
changes. For modified workflows, Agent Gate compares canonical base/head sets
and reports newly introduced or expanded permissions, unpinned references,
secret references, head checkout patterns, and other configured checks.
Added workflows are compared with an empty base. Deleted workflows are reported
explicitly. Malformed or unavailable content makes the analysis incomplete
instead of producing a partial pass.
Adopt it without turning CI red on day one
The intended rollout is:
- Observe: collect findings and identify repository-specific noise.
- Warn: require review while the policy and waivers are tuned.
- Block: enforce only the findings whose evidence the team trusts.
Checks can be configured independently as off, warn, or error. Exact,
expiring waivers live in base-branch policy and remain visible in reports.
Honest limits
Agent Gate is not a replacement for tests, a semantic code reviewer, or a tool
such as zizmor. Its agentic-workflow rule recognizes registered agent actions
and a deliberately narrow set of direct or one-hop prompt flows. Shell, files,
step outputs, and cross-job data flow are outside the v0.3 scope.
It also has no SaaS, GitHub App, product telemetry, or runtime model judgment.
The goal is reproducible change-control evidence, not an opaque risk score.
I am looking for policy and false-positive feedback from repositories where AI
agents create or review PRs. Which findings would you trust enough to block,
and which should remain warnings or be turned off?
Top comments (0)