DEV Community: JinHyuk Sung

I audited 249 of my own AI coding sessions. The problem wasn't lying.

JinHyuk Sung — Sat, 01 Aug 2026 10:19:48 +0000

I never scrolled up

I use a coding agent every day. I give it a task, go do something else, and come back to a line at the bottom of the screen like this one:

"빌드·회귀 스모크·기존 분석 테스트 모두 통과."
(build, regression smoke, and the existing analysis tests all pass)

That is a real line from my own session log. I never checked it. Instead of scrolling up to see whether the tests actually ran, I gave it the next task. I did that for months.

Then I read a post in a Korean dev community where someone had looked into their agent's private reasoning and found that the memory given to its tester role contained a list of ways to work around problems when they came up.

That doesn't mean the agent was malicious. If you tell something to make the tests pass, finding the cheapest path is what optimisation does. The problem is when the cheapest path is "avoid the check" rather than "fix the code," and you have no way to notice.

So I decided to check. The tool is called red-handed.

Not a judge, a diff

I wasn't trying to build a judge. I wanted a side-by-side of what the agent said and what it actually did.

The material was already there. Claude Code writes a transcript for every session: every command it ran and what came back, the before and after of every file edit, and every sentence it said to me. Line that up with git and you can compute the gap between the words and the work.

Did it claim the tests pass without running them. Did it claim they pass right after a run that failed. Did it rewrite an expected value to match whatever the broken code produced. Nine checks like that.

The thing I spent longest on wasn't a feature. It was what happens when the tool is wrong.

There are two ways to be wrong here.

Missing something. The user loses nothing — they're exactly as uninformed as before.

Accusing someone who did the work honestly. That's telling a person they cheated when they didn't. Get that wrong once and nobody opens it again.

The costs are nowhere near symmetric. So the rule became: miss things rather than accuse wrongly.

That turned into two hard constraints.

A confirmed finding needs two pieces of evidence. The session has to show it happening, and the change has to still be in the working tree. If the agent undid it later, there's nothing to accuse anyone of, and the finding disappears.

Something unread is not a failure. If a test ran and I couldn't parse the output, that's "I don't know," not "it failed." That distinction ends up inverting the whole result.

No model is called for any of this. If the verdict came from an LLM, the same transcript would produce different answers on different days — and then it isn't evidence.

Running it over 249 sessions

I pointed it at every session on my machine.

249 sessions. Your agent said "tests pass" 124 times.
   117  a test ran first
     7  no test ran in that session at all

Confirmed findings: zero.

I'll be honest, that was deflating. Weeks of work on a lie detector and it caught no lies.

Then I opened the seven one by one, and realised I'd been asking the wrong question.

The real problem wasn't lying

Four of the seven came from one project. The agent had said things like:

"드래그·휠 5방향 브라우저 테스트 통과."
(drag and scroll, all five directions, browser tests pass)

That isn't a lie. The agent really did open a browser and check. But that check left no machine-readable trace of any kind. So there is no way to find out later whether the claim held. Not for me, not for the tool, not for whoever inherits that code.

The other three were the same shape: the project runs its tests through its own script and my parser doesn't know that output format. I can tell that something test-shaped ran. I can't tell that it passed.

That's where the conclusion flipped. I was worried about an agent that lies. What I actually found was verification nobody can read.

And that's less a property of AI than of how we work now. It used to be fine for a person to open a browser, look, and move on — because that person kept looking at that code. Now the thing doing the checking is an agent whose memory resets every session. Verification you can't read is indistinguishable from verification that never happened.

So those seven are reported as SUSPICIOUS, not CAUGHT, and the wording is "go look at what that command actually reported" rather than "it lied to you." That's as far as the tool can honestly go.

One caveat I'd rather state than have someone find: of those 249 sessions, only 12 contained a "tests pass" claim at all. The rest are short — a question or two. So "0 out of 249" would be an inflated way to read this.

And it falsely accused me six times

Once it worked, I spent a while trying to break it. That turned out to be the most valuable part of the project. An adversarial review found five separate ways it would accuse honest work:

A test run killed by a timeout, read as a failure
A runner it didn't know (rspec, phpunit, tox), read as "no test ran"
A re-run under a launcher it couldn't see, so it judged on the stale result
A behaviour change the user had explicitly asked for, read as rewriting the answer
A backgrounded command, counted as a pass

The first two are worth spelling out.

The timeout one was the worst. When a test run is killed for taking too long it leaves a distinctive exit signal, and my code read that as a failure. So if the agent then said the tests passed, it printed the single most insulting verdict in the tool: "said tests pass — the last run failed." My code was directly violating the rule I'd written two paragraphs of design notes about.

The second was one character. The pattern that recognises test runners looked for the word spec with a word boundary in front of it — which never matches inside rspec, because r is a word character. So a Ruby developer could run their suite, get 20 examples, 0 failures, and my tool would state with confidence that no test had run at all.

And here's the part that stings. My README proudly said "0 false positives across 183 real sessions." That was true — but not because the tool was good. It was true because every project on my machine is JavaScript or TypeScript, so neither of those paths had ever executed. I built a tool to verify claims, reported that it passed its own verification, and hadn't actually exercised the code. I did the exact thing the tool exists to catch.

The sixth turned up after I shipped. A scoped pytest tests/test_sync_cli.py, then edits to a GitHub workflow file and two .tsx files, and the tool called the claim stale. A Python test can't import a .tsx, and a CI workflow isn't what just ran on this machine. Nothing had gone stale.

There was a tempting fix: "different directory, treat it as unrelated." Simple, and right most of the time. It's also wrong — run pytest tests/ and then edit src/ and the claim really has expired. So instead of guessing, I only suppress what is definitely true: Python can't load a .tsx, a JS runner can't load a .py, a CI workflow isn't a local run. Anything ambiguous still counts. "Looks unrelated" is a guess, and you don't throw out evidence on a guess.

All six are pinned by regression tests now.

It caught me again while I was writing this

I re-ran it to refresh the numbers for this article, and found one of mine in the list:

"테스트 3개로 고정했고 전체 389개 통과."
(pinned it with three tests, all 389 pass)
→ 1 file changed after that run: scripts/social.tape

What I said was true, and the file that changed is a screen-recording script for a demo GIF. It has nothing to do with the test suite.

But this isn't a bug. It's the rule I chose one section ago — anything ambiguous still counts — doing exactly what I told it to. The tool has no idea what a .tape file is, and I decided that unknown means counted.

That's the price of the choice. To avoid missing things, it sometimes barks at nothing. In exchange, this one came out as SUSPICIOUS rather than CAUGHT, with the changed filename printed right next to it, so it took me three seconds to dismiss. That's what the two tiers are for.

Three things I'd keep

Decide which way you'll be wrong before you tune accuracy. "What's the accuracy?" is close to meaningless for a tool like this, because the two directions of error cost wildly different amounts. Pick the direction, then defend that decision in every branch. I wrote the principle in a doc and then broke it in three places in code.

Ship evidence, not verdicts. If the output says "it lied," a user can only believe it or not. Show the timestamps and the quoted line and they can decide for themselves. That's also why dismissing my own false positive above took three seconds. A tool you can catch being wrong is a tool you keep using.

The actual fix is verification you can read. That's what the four browser-test findings taught me. When you hand work to an agent, don't say "check it" — say "check it and leave the result somewhere a machine can read." A check that lives only in someone's head doesn't exist by the next session.

Try it

Runs locally. Your transcripts and your code never leave your machine — it calls no model and makes no network request of its own. Same session, same answer, every time.

# a made-up session where every check fires, so you can see the shape first
npx @jinhyuk9714/red-handed@latest demo

# then your own version of the numbers above
npx @jinhyuk9714/red-handed@latest stats

MIT, and the code is at github.com/sjh9714/red-handed.

Your numbers will be different from mine. I had zero confirmed findings, seven checks nobody could read, and one false alarm I found while writing this paragraph. Whichever way it goes for you, the point is that you don't know until you look.

I benchmarked Claude Code skills against a placebo — and half of mine failed

JinHyuk Sung — Fri, 24 Jul 2026 12:50:36 +0000

There's a whole ecosystem of "agent skills" now — reusable instruction files you drop into Claude Code (or Cursor, or Copilot) to make the model write cleaner code, debug more carefully, use fewer tokens, and so on. Some of these repos have tens of thousands of GitHub stars. Almost none of them ship a single number telling you whether the skill actually does anything.

That bothered me, because "adding a plausible-sounding instruction" and "adding an instruction that works" look identical until you measure them. So I built a benchmark with one rule, committed before I ran anything:

No skill gets merged unless it beats both a no-instruction baseline AND a placebo prompt on its pre-registered target metric, measured on hidden hold-out tests, with accuracy not allowed to drop. Skills that fail are published anyway, with their numbers.

The placebo arm is the part almost nobody runs, and it turned out to be the most important one.

Why a placebo

Most "battle-tested" skill collections that measure anything at all compare skill-on vs skill-off. The problem: that comparison can't separate "this skill works" from "adding any confident-sounding text changes the model's behavior." LLMs are suggestible. If you want to claim your skill did something, you have to show it beats a comparable-length instruction that contains no actual mechanism — just vibes.

So every result here is a three-way comparison — off / placebo / on — run K=5–8 times per task per arm, in isolated git workspaces, graded by hold-out acceptance tests the agent never sees, with every raw run log committed to the repo and the README regenerated from those logs in CI. 496 runs total, all on claude-opus-4-8.

Here's what came out.

Finding 1: the placebo often made code bigger

My anti-over-engineering skill (underkill, ~20 lines) cut source LOC by −23.8% vs baseline at identical accuracy (60/60 hold-out passes). Good. But the interesting column is the placebo: a style-only "write clean, minimal, professional code" instruction (156 words to underkill's 250) didn't reduce code on a single one of the 12 tasks — it increased it on 7 and tied on the other 5 (median +18.8% across the tasks that moved, +6.9% across all 12).

Read that again: telling the model to write clean code made it write more code. The generic exhortation added nothing but tokens, and the model padded to match the vibe. If I'd only compared on-vs-off, underkill would have looked good. Compared against the placebo, it looks good and I can prove the effect isn't just instruction-presence.

Finding 2: a 196k-star skill barely beat "no instructions"

One of the most-installed general behavioral rulesets is the "Karpathy Guidelines" (196k stars as of 2026-07-25) — four rules derived from Andrej Karpathy's observations on LLM coding pitfalls, one of which is "Simplicity First: if you write 200 lines and it could be 50, rewrite it." That's a direct, falsifiable claim on the exact metric underkill targets, so I ran it as a comparison arm on the same 12 tasks.

arm	median src LOC	hold-out accuracy
off (no instructions)	10.5	60/60
placebo	12.5	60/60
underkill (mine, ~20 lines)	8.0	60/60
Karpathy Guidelines (~196k★)	10.0	60/60

The famous skill beats the placebo clearly (−20%) — so its mechanism is real, not just prompt-presence. But against a no-instruction baseline it barely moves (−4.8%), because on small tasks Opus is already close to minimal. And my focused 20-line skill writes 20% less code than the 196k-star one, at equal accuracy and slightly lower cost.

This isn't a dunk on the Karpathy skill — it addresses four different failure modes and only one is on this axis. The point is narrower and more useful: a rule aimed at one specific failure mode beats a broad, general ruleset that merely mentions it. Instruction presence is cheap; instruction mechanism is what pays.

Finding 3: two of my own four skills failed the gate

Publishing only wins would make the whole exercise worthless, so here are the rejects with their numbers:

tests-that-bite (a test-writing discipline skill) hit a ceiling: baseline Opus already killed 100% of my mutation-testing mutants on 3 of 4 tasks. My mutants were too coarse to differentiate. Rejected.
thrift (a token-spend skill) cut turns by 14% but landed $0.005 above baseline on median cost per run. Under a co-primary "cost AND turns" rule, that's a rejection, not a rounding call.

The thrift sweep had the most counterintuitive result of the whole project: I ran the 92k-star "caveman" terse-mode ruleset as a comparison arm, and it measured more expensive per run than no instructions at all on every cost task. Agentic cost is input-dominated (files read, tool results, repeated context), so an instruction that compresses the model's output can't recoup the input tokens its own prompt adds on every turn. Output-compression advice that's true for chat responses can invert on agentic runs.

What this doesn't prove

The tasks are small and self-contained (the committed reference solutions run 6–63 non-blank source LOC, median 14) — enough to gate a skill on, not enough to generalize to large codebases. K=5–8 gives you evidence, not proof. Pre-registration here is commit-ordering in a repo I control, self-attested (though the harness carries an earlier multi-day record). And hold-out material lives in the same repo, so it's reachable in principle — I run a CI check that scans every committed run log for references to the task material (516/516 committed logs clean). All of this is written up in the repo's "Known Limitations," because a results log that only contains good news isn't worth trusting.

Try it / break it

Two skills passed the gate (underkill, repro-first). They install as a Claude Code plugin or via npx skills add sjh9714/skill-receipts, and the installed snippet is byte-identical to what the benchmark measured (CI enforces that).

If you have a skill you think works — yours, or a famous one you want audited — the gate is the same for everyone: open an issue. I'd genuinely like to be proven wrong on some of these.

Repo + all raw logs: https://github.com/sjh9714/skill-receipts

Your AI-contribution policy is prose. Here's the enforceable half.

JinHyuk Sung — Fri, 24 Jul 2026 10:58:52 +0000

Your open-source project probably added an AI-contribution policy in the last few months. Apache has one. The Linux Foundation has one. The OpenSSF Technical Advisory Council is finalizing a foundation-wide one right now. Bitcoin Core, ripgrep, uv, and dozens of smaller projects have copied the pattern. If you maintain something popular, you've either written one or you're about to.

Almost all of them say some version of the same three things: disclose when you use AI; a human must review and remain responsible for every change; no fully autonomous agent PRs.

These are good norms. But I want to point at something uncomfortable: as written, almost none of them are enforceable. They're prose in a CONTRIBUTING.md, and prose is an honor system. So I went looking for how big the gap actually is.

What agent PRs actually do at scale

I scanned 2,204 recently merged, agent-authored pull requests across public GitHub repos (Devin, Copilot coding agent, Codex, Claude Code, Cursor). The scan is deterministic and checkout-free — it reads PR metadata and file contents through the GitHub API, never executes PR code, and never calls an LLM, so every finding replays. Three results are directly relevant to anyone writing one of these policies:

0 of 2,204 declared a machine-checkable scope for the change. Not a low number — zero. Your policy can say "a human must review and understand every change," but a reviewer opening an agent PR has exactly what a reviewer of a human PR has: the diff, and a description written in prose. The agent knew precisely what task it was given. None of that intent survives into the PR in a form anything can check against. So "review every change" is real, but "verify the change matched its intent" isn't available to you yet.

3.9% modified agent control-plane files — AGENTS.md, CLAUDE.md, .mcp.json, .cursor/rules/**, and similar. These are the files that steer future agent runs in your repo. An agent PR that edits its own instructions is a quiet privilege-escalation path, and it's exactly the kind of change that reads as a boring docs diff and sails through review. Most policies don't mention this file class at all.

Of the PRs that touched CI workflows, ~13% raised GitHub Actions permissions, and ~17% introduced unpinned actions. Your DCO and attribution language will never catch a contents: read -> write bump in a workflow file. That's not a licensing question; it's a supply-chain one, and it's invisible to the parts of your policy that are about authorship.

One more, because it decides where to aim: repositories with 10k+ stars had roughly half the finding rate of the long tail (4.3% vs 8.6%). Established projects have guardrails. The projects getting hit hardest are the small, beginner-friendly ones — the exact repos that just adopted a prose policy and have the least review bandwidth to back it up.

(Full methodology, every query, and the raw approach are public: https://github.com/sjh9714/mergewarden/blob/main/docs/study/methodology.md — findings are review evidence, not vulnerabilities or misconduct claims; I publish aggregates only and name no repositories.)

Which policy clauses are actually checkable

Here's the useful reframe. Split your policy into clauses you hope people follow and clauses a machine can check:

Clause (typical wording)	Enforceable automatically?
"You could have written this yourself"	No — honor system
"Explain your changes in your own words"	No — honor system
"Disclose AI use"	Partly — require a checkbox/trailer, but you can't verify honesty
"No autonomous-agent PRs"	Yes — they arrive on `cursor/`, `codex/`, `copilot/*` branches or `-swe-agent[bot]` authors
"Don't touch files that steer future agents"	Yes — a path-glob diff check
"Don't quietly escalate CI permissions"	Yes — diff the workflow `permissions:` block, base vs head
"Stay within the scope of the task"	Yes, but only if scope is declared — the missing primitive

The last row is the interesting one. If an agent PR carried a tiny machine-readable declaration of what it intended to touch, "did it stay in scope" becomes a deterministic check instead of a reviewer's guess:

<!-- ai-contract
version: 1
agent: codex
task: update session expiry handling
allowed_paths:
  - src/auth/**
  - test/auth/**
-->

I'd honestly rather see something like this become a vendor-neutral convention than any one tool's feature. Agent vendors already have the task context at generation time; nothing carries it into the PR. That's the gap under every one of these policies.

The point

If you're writing an AI-contribution policy: keep the prose norms, they matter. But separate the honor-system clauses from the ones a cheap, deterministic check can actually enforce, and wire the second group into CI. The three checks worth adding first, in order of bang-for-effort: (1) flag control-plane edits (AGENTS.md / .mcp.json / rules files) for mandatory human review; (2) diff workflow permissions base-vs-head and flag any escalation; (3) detect autonomous-agent PRs by branch/author so they get the extra scrutiny your review bandwidth is already rationing.

None of that needs an LLM, none of it needs to check out PR code, and all of it is boring, replayable, and free. The policy tells people what's expected; a check tells you when it didn't happen.

I built an open-source tool (MergeWarden, MIT) that does exactly these checks as a GitHub Action, which is what produced the numbers above — but the argument stands whatever you use, including a few lines of your own CI. The takeaway isn't "adopt my thing." It's: your policy is prose, agent PRs are a firehose, and the enforceable half of your policy is smaller and cheaper to automate than you think.

What 2,204 merged AI-agent PRs actually touched (0 declared their scope)

JinHyuk Sung — Wed, 22 Jul 2026 14:56:13 +0000

AI coding agents — Devin, Copilot coding agent, Codex, Claude Code, Cursor — open and merge pull requests at scale now. Every agent vendor knows, at the moment of generation, exactly what the task was. My question: does any of that intent survive into the PR in a form a machine could check? And while I was looking: how often do agent PRs cross boundaries that deserve human eyes?

So I scanned 2,204 recently merged, agent-authored PRs on public GitHub repos with a checkout-free policy engine, using only its built-in default policy. It reads PR metadata and file contents through the GitHub API — it never executes PR code and never calls an LLM, so every finding is deterministic and replayable.

The one number that surprised me

0 of 2,204 PRs declared a machine-readable scope for the change. Not a low percentage — zero. Agent vendors emit rich task context at generation time, and none of it reaches the PR as something a machine could verify. If you want to know whether an agent PR stayed inside its intended task, there is currently nothing to check it against.

What else showed up

7.0% of complete analyses had at least one boundary finding (153 of 2,191). The structure underneath that number is the interesting part:

Of the 349 PRs that touched GitHub Actions workflows or package manifests, 12.9% escalated workflow permissions and 17.5% introduced unpinned actions, reusable workflows, or containers. Workflow-touching agent PRs are where the risk concentrates.
3.9% changed agent control-plane files — AGENTS.md, CLAUDE.md, .mcp.json, and similar. These files steer every future agent PR in the repo, which makes them a quiet privilege-escalation path: an agent that edits its own instructions today shapes what the next agent does tomorrow.
Repos with 10k+ stars showed a 4.3% finding rate — roughly half the long-tail rate (8.6%). Established projects have guardrails. The long tail of small repos, where most agent PRs actually land, is where agents run with the least oversight.

Honest denominators

Percentages hide choices, so here are mine: workflow-rule rates use PRs that actually touched workflow or manifest content; control-plane rates use all complete analyses; the contract statistic uses everything. Incomplete analyses fail closed and are reported as their own bucket, never silently dropped. A "finding" is not an accusation — most of the escalations I saw are probably benign. That is exactly the point: nobody declared them, nobody checked them, and benign-until-it-isn't is not a security posture.

The missing primitive

The zero is the story. Agent PRs today are reviewed the way human PRs are — by reading the diff — but agents differ from humans in one reviewable way: their intent is machine-generated and could be machine-checkable. A PR-body contract as small as this would close the loop:

<!-- mergewarden-contract
version: 1
agent: codex
task: update session expiry handling
allowed_paths:
  - src/auth/**
  - test/auth/**
-->

The contract is an untrusted declaration — the base-branch policy stays authoritative — but once it exists, "did the PR leave its declared scope" becomes a deterministic check instead of a reviewer's guess. I'd rather see this become a vendor-neutral convention than one tool's feature; the format above is one concrete proposal.

Reproduce it

Every query, date window, and aggregation script is published in the methodology writeup. The scanner (MergeWarden, MIT) runs against any public PR without installing anything:

npx mergewarden scan owner/repo#123

Full methodology and the tool: https://github.com/sjh9714/mergewarden

If you maintain a repo that receives agent PRs and want the scan results for your own recent PRs, open an issue — I'm looking for maintainers to help measure false-positive rates against real-world judgment. And if you've seen an agent PR quietly cross a line in your own repo, I'd love to hear about it in the comments.

AI agents can open PRs. Who checks whether they crossed the line?

JinHyuk Sung — Fri, 17 Jul 2026 09:04:21 +0000

The gap between "works" and "stayed in scope"

An AI coding agent can produce a pull request that builds, passes tests, and
implements the requested feature. That same PR can also edit a release
workflow, increase a GitHub token permission, change AGENTS.md, or touch a
file outside the declared task.

That does not make the PR malicious. Still, the reviewer should be able to see
the boundary change without reconstructing it from a large diff. Ordinary tests
answer whether the repository still works. A semantic reviewer asks whether the
code makes sense. Neither reliably records whether the PR stayed inside the
agreed scope.

I built Agent Gate for that narrower
question.

A checkout-free policy gate

Agent Gate is an open-source GitHub Action and CLI. It collects pull-request
metadata and selected file content through GitHub APIs, loads agent-gate.yml
from the exact base commit, and produces deterministic findings.

It intentionally does not:

checkout or execute pull-request code;
run target-repository package scripts;
load policy from the PR head branch; or
call an LLM at runtime.

That gives the policy a useful trust boundary. A pull request can change the
policy for future PRs, but it cannot weaken the policy evaluating itself.

If GitHub reports 42 changed files and Agent Gate can collect only 41, the
analysis is incomplete and fails closed. Authentication failures, rate limits,
and server errors are not silently treated as missing files or default policy.

Try it before installing it

Node 20 or newer is enough to scan a public PR:

npx --yes @jinhyuk9714/agent-gate@0.3.1 scan owner/repo#123

The CLI clones nothing and executes nothing from the target repository. Human,
JSON, and Markdown output are available, and the Action uses the same analysis
contract.

What the evidence looks like

The public composite proof PR
starts with a docs-only contract and then changes documentation, a workflow,
and AGENTS.md. Its Action run
reports:

two contract/out-of-scope findings;
a high-risk workflow path;
agent-control-plane/drift for AGENTS.md; and
workflow- and job-level permission escalation.

The report also records the policy digest, exact base/head SHAs, expected and
analyzed file counts, and whether collection completed. Finding IDs are derived
from canonical evidence rather than display wording, so a waiver can target one
exact finding with a reason and expiry.

Differential workflow checks

Existing workflow risk should not be re-reported whenever an unrelated line
changes. For modified workflows, Agent Gate compares canonical base/head sets
and reports newly introduced or expanded permissions, unpinned references,
secret references, head checkout patterns, and other configured checks.

Added workflows are compared with an empty base. Deleted workflows are reported
explicitly. Malformed or unavailable content makes the analysis incomplete
instead of producing a partial pass.

Adopt it without turning CI red on day one

The intended rollout is:

Observe: collect findings and identify repository-specific noise.
Warn: require review while the policy and waivers are tuned.
Block: enforce only the findings whose evidence the team trusts.

Checks can be configured independently as off, warn, or error. Exact,
expiring waivers live in base-branch policy and remain visible in reports.

Honest limits

Agent Gate is not a replacement for tests, a semantic code reviewer, or a tool
such as zizmor. Its agentic-workflow rule recognizes registered agent actions
and a deliberately narrow set of direct or one-hop prompt flows. Shell, files,
step outputs, and cross-job data flow are outside the v0.3 scope.

It also has no SaaS, GitHub App, product telemetry, or runtime model judgment.
The goal is reproducible change-control evidence, not an opaque risk score.

I am looking for policy and false-positive feedback from repositories where AI
agents create or review PRs. Which findings would you trust enough to block,
and which should remain warnings or be turned off?

Would you block a PR that changes GitHub Actions contents permission from read to write?

JinHyuk Sung — Tue, 30 Jun 2026 05:52:53 +0000

A sandbox PR changed one GitHub Actions workflow permission:

permissions:
  contents: write

The base branch had:

permissions:
  contents: read

That is the concrete case I am trying to calibrate.

Agent Gate reported:

Agent Gate: NEEDS HUMAN DECISION
Decision: warn
Why: contents permission increased from read to write.
Path: .github/workflows/demo-release.yml
Recommended next step: review the workflow permission change before merging.
Policy status: warning today; eligible to become a merge gate after tuning.

Rule: workflow/permission-escalation
Policy source: built-in default

Live PR comment proof:
https://github.com/sjh9714/agent-gate-install-smoke-20260617/pull/13#issuecomment-4828248162

What matters to me is that this did not depend on an LLM noticing the change.

The Action did not:

checkout PR code
run repository scripts
call an LLM at runtime
load policy from the PR head branch

The first-run repo config was also absent. Agent Gate used its built-in default policy and recorded:

configSource: default

I am not trying to claim that the PR is automatically bad. A permission increase can be intentional.

The question is what CI should do when it sees this kind of boundary change.

My current default is:

warn on first run
keep the report human-readable
let teams promote this finding to block after tuning

For AI-generated PRs, I think deterministic CI evidence is useful because agent changes can touch workflow and security boundaries as part of ordinary work.

But this specific finding is broader than AI: any PR that raises GitHub Actions permissions may deserve deliberate review.

Question:

In your repo, is this block, warn, or noise?

What extra evidence would make it actionable?

Repo:
https://github.com/sjh9714/Agent-Gate

Disclosure: I used AI assistance to help draft and edit this article, and I reviewed the technical claims before publishing.

I made Agent Gate installable in 30 seconds for AI PR checks

JinHyuk Sung — Fri, 26 Jun 2026 10:51:38 +0000

One problem with security-ish developer tools is that the install path can ask for trust before it has earned any.

Agent Gate is a GitHub Action for AI-generated pull requests. It does not review code with an LLM. It checks repeatable CI evidence such as workflow permission changes, agent control-plane drift, package lifecycle script drift, and missing test-file evidence.

The important constraint is that the Action should be safe to try first:

no checkout of PR code
no runtime LLM calls
no repository script execution
no policy loaded from the PR head branch
warn mode by default for first runs

I recently changed the onboarding flow so the first install is basically one pinned workflow download.

mkdir -p .github/workflows \
  && curl -fsSL https://raw.githubusercontent.com/sjh9714/Agent-Gate/v0.2.5/templates/agent-gate-observe.yml \
  -o .github/workflows/agent-gate.yml

This downloads a tag-pinned GitHub Actions workflow YAML file. It is not curl | bash, and it does not execute a remote script.

The downloaded workflow uses:

uses: sjh9714/Agent-Gate@v0.2.5
with:
  mode: warn
  fail-on-block: false

It also uses only:

permissions:
  contents: read
  pull-requests: read

The other change is that first runs no longer require agent-gate.yml.

If the default config file is confirmed missing on the PR base branch, Agent Gate uses its built-in default policy and records:

configSource: default

That means a maintainer can install the workflow first, see what the default warnings look like, and only add repo-specific policy later.

I also verified the README install path in a sandbox PR. The run:

loaded sjh9714/Agent-Gate@v0.2.5
used no actions/checkout
had no agent-gate.yml
fell back to the built-in default policy
finished successfully
produced a warn decision for package lifecycle script drift

The compact log looked like this:

Agent Gate: NEEDS HUMAN DECISION
Decision: warn
Why: preinstall script added in package.json.
Path: package.json
Policy status: warning today; eligible to become a merge gate after tuning.

- warn agf_2ac4687b2f8f712a dependency/lifecycle-script-added package.json

This is the first-run model I want:

install quickly
observe warnings
understand the report
then tune policy

It still does not prove semantic correctness. A deterministic CI gate should not pretend to know whether a PR is “good.” The goal is narrower: surface repeatable evidence that a maintainer can inspect before merge.

If you maintain a GitHub Actions-heavy repo or use coding agents to open PRs, I would like feedback on one thing:

Does this first-run path feel safe and clear enough to try in a real repo?

Repo:

https://github.com/sjh9714/Agent-Gate

CI gates for AI-generated PRs need re-derivable evidence

JinHyuk Sung — Sun, 21 Jun 2026 01:08:59 +0000

When a CI gate flags an AI-generated PR, the important question is not only "what did it flag?"

It is also:

"Could someone else come back later and re-derive why this finding fired?"

That is the reason I added evidence snapshots to Agent Gate v0.2.1.

What Agent Gate is

Agent Gate is a GitHub Action for AI-generated pull requests.

It does not review code with an LLM. It checks deterministic merge evidence in CI:

PR scope escapes
GitHub Actions permission escalation
AGENTS.md / .mcp.json drift
missing test-file evidence
high-risk path changes

The Action does not checkout PR code, call LLMs at runtime, or execute repository scripts.

Why finding IDs were not enough

In v0.2.0, Agent Gate added stable finding IDs.

That gave every finding a short audit handle, for example:

agf_987ab9ddb8c1b299

That is useful for references, comments, future override workflows, and log-based debugging.

But an ID by itself is not proof. If someone sees the ID later, they still need to know what recorded material produced it.

What v0.2.1 adds

v0.2.1 adds evidenceSnapshot to public findings.

The split is:

findingId = short audit handle
evidenceSnapshot = canonical material used to derive that handle

The snapshot is intentionally boring. It contains stable rule material such as:

rule id
severity
path or line when present
normalized evidence label/value pairs

It does not include timestamps, report order, risk score, version, commit SHA, or mutable display text.

A real report shape

Example compact log output:

Agent Gate: NEEDS HUMAN DECISION
Decision: warn
Risk score: 49 / 100
Why: Agent-generated PRs must include an agent-gate contract.
Recommended next step: Add a PR contract before relying on scope checks.
Policy status: warning today; eligible to become a merge gate after tuning.

Findings:
- error agf_be0c2c2a66312aff contract/missing
- error agf_987ab9ddb8c1b299 risk/high-risk-path .github/workflows/agent-gate.yml
- warn agf_6016e753491255d7 workflow/dangerous-pattern .github/workflows/agent-gate.yml

The compact log stays short, but the JSON and Markdown reports carry the fuller evidence.

Example JSON shape:

{
  "findingId": "agf_987ab9ddb8c1b299",
  "ruleId": "risk/high-risk-path",
  "severity": "error",
  "path": ".github/workflows/agent-gate.yml",
  "evidenceSnapshot": {
    "ruleId": "risk/high-risk-path",
    "severity": "error",
    "path": ".github/workflows/agent-gate.yml",
    "evidence": [
      {
        "label": "changed_file",
        "value": ".github/workflows/agent-gate.yml"
      }
    ]
  }
}

Why this matters

For me, the bar for promoting a finding from warning to blocking is:

A third party should be able to re-derive the finding from recorded evidence.

That does not mean the check is magically correct.

It means the failure mode is visible, reproducible, and tunable.

A repo can start in warn mode, observe which findings are useful, and only later promote low-noise findings into merge gates.

What this does not solve yet

Agent Gate still does not prove semantic correctness.

Matching test-file evidence is not proof that the tests cover the behavior. It is change evidence / self-consistency evidence.

Maintainer override storage is also not implemented yet. That is probably the next hard design question: if someone bypasses a finding, where should that override live so it is durable enough to inspect later?

CODEOWNERS / reviewer evidence and package dependency drift are also future work.

Try it

If you maintain a repo where coding agents open PRs, I would love feedback on whether this kind of evidence is useful or too noisy in observe mode.

Repo:

https://github.com/sjh9714/Agent-Gate

Disclosure: I maintain Agent Gate. v0.2.1 is still a prerelease; I would start in warn mode before treating any finding as a merge gate.

LLM reviewers are useful, but some PR checks should stay deterministic

JinHyuk Sung — Tue, 16 Jun 2026 20:17:27 +0000

AI coding agents are getting better at opening pull requests.

That changes the review problem.

A normal review asks whether the code looks correct, whether the design makes sense, and whether the edge cases were considered.

Those questions still matter.

But an AI-generated pull request also raises a different kind of question:

Did the agent change something outside the intended task, and is there enough repeatable evidence to merge?

I have started thinking about this as a split between judgment and evidence.

LLM reviewers help with judgment. Agent Gate verifies deterministic merge evidence.

I do not think every review question should become a hard CI gate. Some parts of code review need human context. Some parts benefit from an LLM noticing suspicious patterns. But a few checks are mechanical enough that I want them to be deterministic, repeatable, and visible before merge.

This is the checklist I currently use when thinking about AI-generated PRs.

1. Did the PR stay in scope?

The first question is not whether the code is good.

It is whether the PR changed the files it was supposed to change.

For a human PR, an unrelated edit may be easy to explain in review. For an agent PR, unrelated edits are more suspicious because they may reflect an instruction drift, a tool mistake, or a broad refactor the maintainer did not ask for.

A simple contract can help:

This PR is allowed to touch:
- src/auth/**
- tests/auth/**

Then the review can ask a deterministic question:

Did the PR touch anything outside those paths?

That does not prove the code is correct. It only proves the PR stayed inside its declared boundary.

That boundary still matters.

2. Did workflow permissions escalate?

GitHub Actions workflows are one of the highest-risk places for an agent to edit.

A small source change and a workflow permission change do not have the same risk profile.

For example, I would want a very visible warning if a PR adds or changes this:

permissions:
  contents: write

or starts using secrets.* in a new workflow path.

This is not a semantic code review problem. It is a policy boundary problem.

The question is deterministic:

Did this PR increase workflow privileges or introduce a dangerous workflow pattern?

That kind of check should not depend on whether an LLM happened to notice it in a comment.

3. Did agent-control-plane files change?

AI coding agents often depend on files that shape future behavior:

AGENTS.md
CLAUDE.md
.github/copilot-instructions.md
.cursor/rules/**
.mcp.json

A change to these files can affect future agent runs, tool access, or repo-specific instructions.

That makes them different from normal documentation changes.

If an AI-generated PR edits .mcp.json or AGENTS.md, I want that surfaced clearly before merge, even if the source code diff looks harmless.

The deterministic question is:

Did the PR change files that control future agent behavior?

This is especially important for teams adopting coding agents across repositories, because the control plane can drift quietly.

4. Is there matching test evidence?

Test evidence is tricky.

A changed test file does not prove the behavior is correct. It does not prove the test is meaningful. It does not prove coverage.

But for risky areas, the absence of any matching test change is still useful evidence.

If a PR changes auth logic, payment handling, session middleware, or a migration, I want to know whether the PR also changed a related test file.

The check should be phrased carefully:

There is no matching test-file evidence.

Not:

This PR is untested.

That distinction matters. Deterministic checks should say exactly what they know, and no more.

5. Did package scripts or dependencies drift?

This is not always the first rule I would add, but it is one I keep coming back to.

Package manifests and lockfiles can hide meaningful risk:

package.json
pnpm-lock.yaml
yarn.lock
package-lock.json

Some changes are normal dependency maintenance. Others deserve more attention:

{
  "scripts": {
    "postinstall": "node scripts/setup.js"
  }
}

For AI-generated PRs, I would want to know:

Did a lifecycle script appear?
Did an existing package script change?
Did dependencies change without an expected lockfile change?

Again, not every finding should block. But these changes should be easy to see.

6. Did the right human reviewer approve?

This is where deterministic evidence meets human ownership.

A PR can stay in scope, avoid workflow escalation, and include test evidence, but still need the right reviewer.

Examples:

src/auth/** changed -> security reviewer expected
.github/workflows/** changed -> platform reviewer expected
.mcp.json changed -> maintainer/platform approval expected

I do not think this should always block by default, especially for solo maintainers. But for teams, reviewer evidence may be one of the most useful signals.

The question is:

Did the right human approve the risky part of this PR?

That is not a replacement for review. It is a way to make ownership visible.

What should stay human?

A lot.

I would not want deterministic CI to answer questions like:

Is this design good?
Is this abstraction worth it?
Will users understand this behavior?
Is this bug fix actually correct?

Those are judgment questions.

LLM reviewers can help with judgment. Human reviewers own judgment. Deterministic gates should focus on evidence that can be checked the same way every time.

What should be deterministic?

The best candidates are checks that are:

explainable
repeatable
hard to miss
tied to merge risk
not dependent on executing PR code

For me, that currently includes:

PR scope boundaries
workflow permission escalation
dangerous workflow patterns
agent-control-plane drift
missing test-file evidence for high-risk paths
package script and dependency drift
reviewer evidence for sensitive paths

These checks do not make an AI-generated PR safe.

They make the risk easier to inspect before merge.

Start in warn mode

The safest adoption path is not to block everything on day one.

I would start with warnings:

mode: warn
fail-on-block: false

Then observe real PRs.

Which findings are useful?
Which ones are noisy?
Which ones would you trust as merge gates?

Only after that would I promote low-noise findings to blocking checks.

Closing thought

I think AI-generated PR review will need both judgment and evidence.

LLM reviewers can help with judgment.

Deterministic CI checks should verify merge evidence.

I’m exploring this idea in Agent Gate, a small GitHub Action for deterministic merge evidence in AI-generated PRs.

Disclosure: I used AI assistance to help draft and edit this article, and I reviewed the technical claims before publishing.

I built a deterministic CI firewall for AI-generated pull requests

JinHyuk Sung — Mon, 15 Jun 2026 15:22:57 +0000

AI coding agents are getting good enough to open pull requests.

That is useful.

It also changes the review problem.

A normal code review asks:

Does this code look correct?

An AI-generated PR also raises a different question:

Did this agent change something I did not intend, and does this PR have enough evidence to merge?

Agent Gate is still a prerelease, so I am starting with a narrow goal: make AI-generated PRs easier to inspect before merge.

That second question is why I built Agent Gate for AI PRs.

The core idea

Agent Gate is a deterministic CI firewall for AI-generated pull requests.

It is not an LLM reviewer.

That distinction matters.

LLM reviewers help with judgment. Agent Gate verifies deterministic merge evidence.

An LLM reviewer can tell you whether code looks suspicious. Agent Gate checks whether the PR crossed policy boundaries that should be explainable and repeatable in CI.

The mental model is:

Use your LLM reviewer for judgment.

Use Agent Gate for deterministic merge evidence.

Agent Gate checks questions that should not require an LLM:

did the PR stay inside its declared scope?
did workflow permissions escalate?
did agent control-plane files drift?
did high-risk code change without matching test-file evidence?
did MCP config changes get surfaced?

These are not semantic code review questions. They are merge-boundary questions.

Why I wanted this

Imagine an agent is asked to fix an auth session bug.

The expected scope might be:

allowed_paths:
  - src/auth/**
  - tests/auth/**

But the PR also changes:

src/payments/webhook.ts
.github/workflows/release.yml
.mcp.json

A reviewer might catch that. An LLM reviewer might catch that. But I do not want this to depend only on someone noticing.

I want CI to say:

This PR crossed its declared scope.
This PR changed workflow permissions.
This PR changed the agent tool surface.
This PR needs human decision before merge.

That is the shape of Agent Gate.

What Agent Gate catches today

The current v0.1.2 release is intentionally focused on deterministic checks.

It can flag or block the following, depending on policy mode.

Out-of-contract edits

Agent Gate can parse a small PR body contract:

<!-- agent-gate-contract
version: 1
agent: codex
task: update auth session handling
allowed_paths:
  - src/auth/**
  - tests/auth/**
required_evidence:
  - matching auth tests changed
-->

If the PR changes files outside allowed_paths, Agent Gate reports that as a contract escape.

Workflow permission escalation

GitHub Actions workflows are powerful. If a PR changes this:

permissions:
  contents: read

to this:

permissions:
  contents: write

that should be visible before merge.

Agent Gate checks workflow-level permission escalation and dangerous workflow patterns such as:

write-all
id-token: write
pull_request_target checking out PR head
unpinned third-party actions
added secrets.* usage

Agent control-plane drift

Files like these can change how future agents behave:

AGENTS.md
CLAUDE.md
.cursor/**
.github/copilot-instructions.md
.mcp.json

A PR that changes .mcp.json is not just changing config. It may be changing which tools an agent can call.

Agent Gate treats those files as an agent control plane and reports drift.

Missing test evidence

Agent Gate can define high-risk paths:

high_risk_paths:
  auth:
    paths:
      - src/auth/**
    require_tests:
      - tests/auth/**
    severity: error

If auth code changes but no matching auth test file changes, Agent Gate reports missing test evidence.

This does not prove semantic test coverage. It only checks deterministic file-pattern evidence.

That limitation is intentional.

What a report looks like

One piece of early feedback was that the report should not start with a wall of rule IDs.

It should answer the maintainer's first question:

What should I do with this PR?

So the Markdown report now leads with a human decision.

Example shape:

Agent Gate: NEEDS HUMAN DECISION

Why:
This PR changed `.github/workflows/release.yml` and added `secrets.*` usage.

Recommended next step:
Review the workflow change before merging.

Policy status:
Warning today; eligible to become a merge gate after tuning.

The detailed rule findings still appear underneath.

The machine-readable JSON decision remains simple:

{
  "decision": "warn"
}

The human-facing report can say NEEDS HUMAN DECISION, while the machine-readable result stays pass, warn, or block.

The trust boundary

Agent Gate is designed around a conservative trust boundary.

At runtime, the GitHub Action:

does not checkout PR code
does not execute repository scripts
does not call LLMs
does not execute MCP servers
does not load policy from the PR head branch

It reads PR metadata and changed-file contents through GitHub APIs.

It loads agent-gate.yml from the PR base branch, not from the untrusted PR branch.

That matters because a PR should not be able to weaken its own policy.

Installing it

Agent Gate is available on GitHub Marketplace:

https://github.com/marketplace/actions/agent-gate-for-ai-prs

A minimal workflow looks like this:

name: Agent Gate

on:
  pull_request:
    types:
      - opened
      - synchronize
      - reopened
      - edited
      - labeled
      - unlabeled
      - ready_for_review

permissions:
  contents: read
  pull-requests: read

jobs:
  agent-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: sjh9714/Agent-Gate@v0.1.2
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          mode: warn
          fail-on-block: false

I recommend starting with:

mode: warn
fail-on-block: false

That gives you an observe path.

First learn what Agent Gate finds in your repository. Then promote only the policies that are useful and low-noise into merge gates.

A small starting policy could be:

version: 1
mode: warn

contract:
  required_for:
    - agent
  allow_missing_in_observe_mode: true

agent_detection:
  labels:
    - ai
    - agent
    - codex
  branch_patterns:
    - "codex/**"
    - "ai/**"

high_risk_paths:
  workflows:
    paths:
      - ".github/workflows/**"
    severity: error

Local replay demo

The repository includes an unsafe-pr-zoo with deterministic fixtures.

After cloning the repo and installing dependencies, you can run:

pnpm install
pnpm --filter agent-gate build
node packages/cli/dist/main.js replay fixtures/unsafe-pr-zoo/workflow-permission-escalation

Example output:

Agent Gate: BLOCKED

ERROR workflow/permission-escalation
contents permission increased from read to write.
Path: .github/workflows/release.yml

ERROR workflow/dangerous-pattern
.github/workflows/release.yml contains a dangerous GitHub Actions workflow pattern.
Path: .github/workflows/release.yml

Other fixtures cover:

agent control-plane drift
out-of-scope agent edits
missing test evidence
MCP config drift

What Agent Gate is not

Agent Gate is not a replacement for code review.

It does not try to find every semantic bug.

It does not know whether a function is logically correct.

It does not prove that tests are sufficient.

It does not replace a human reviewer, an LLM reviewer, or normal CI.

It answers a narrower question:

Did this PR cross deterministic policy boundaries before merge?

That is the problem I want it to solve well.

Current limitations

v0.1.2 is still a prerelease.

Known limitations:

APIs, rule names, reports, and config may change.
CODEOWNERS and reviewer evidence are not implemented yet.
Package and dependency drift rules are not implemented yet.
GitHub Actions job-level permission escalation comparison is limited.
Test evidence is file-pattern based and does not prove semantic coverage.
PR comment upsert requires issues: write and can warn on fork PRs with read-only tokens.

What I want feedback on

I am especially interested in feedback from people trying AI-generated PRs in real repositories.

The main questions:

Which findings should block by default?
Which findings should stay warning-only?
What high-risk path patterns do you use?
Would CODEOWNERS or reviewer evidence make this more useful?
Should package script and dependency drift be part of the gate?
What would make this too noisy to adopt?

Feedback issue:

https://github.com/sjh9714/Agent-Gate/issues/27

Repository:

https://github.com/sjh9714/Agent-Gate

Closing thought

LLM reviewers are useful.

But if AI-generated PRs become part of normal engineering workflows, teams will also need deterministic gates.

Not every review question should be probabilistic.

Some questions are simple:

Did this PR stay within scope?

Did workflow permissions escalate?

Did agent control-plane files change?

Is there matching test evidence?

That is the space Agent Gate is trying to explore.

Use your LLM reviewer for judgment.

Use Agent Gate for deterministic merge evidence.

Disclosure: I used AI assistance to help draft and edit this article, and I reviewed the technical claims before publishing.

Auditing GitHub CLI extensions before installing more

JinHyuk Sung — Thu, 11 Jun 2026 09:50:21 +0000

GitHub CLI extensions are useful, but extension discovery has a small practical problem: it is easy to find more extensions before you understand whether your current setup already covers the workflows you care about.

I built a small browser-only audit page for that problem:

https://sjh9714.github.io/gh-extension-atlas/audit.html?demo=1

It opens with a sample result loaded. If you want to check your own setup, run:

gh extension list

Then paste the output into the page.

The audit runs locally in the browser. There is no sign-in, no analytics script, and no remote audit API. The pasted extension list is not uploaded.

What the audit checks

The page compares installed GitHub CLI extensions against the GitHub CLI Extension Atlas catalog:

https://github.com/sjh9714/gh-extension-atlas

It shows:

installed extensions that are already reviewed in the atlas
installed extensions that are not listed yet
missing Top Picks
workflow coverage gaps
copyable install commands for missing workflow recommendations

The goal is not to install every recommended extension. The goal is to make the next install decision smaller and easier to review.

Why not just use `gh extension search`?

gh extension search is useful when you already know what you are looking for.

It is less useful when the question is:

Which terminal dashboard should I try first?
Do I need a notification tool if I already use a broader dashboard?
Which extension is for GitHub Actions operations versus workflow statistics?
Is this branch cleanup tool the right fit for my risk tolerance?

Those are comparison questions, not search questions.

The atlas tries to answer them with a reviewed catalog, Top Picks, comparison guides, and workflow recommendations.

A few examples

For daily GitHub triage, the atlas points to gh-dash first because it covers PRs, issues, and notifications in one terminal dashboard.

For GitHub Actions, it separates different jobs:

gh-enhance for an interactive Actions TUI
gh-workflow-stats for success rate and duration summaries
gh-act for local workflow runs
gh-actions-importer for migration work

For branch cleanup, it compares tools like gh-poi, gh-branch, gh-clean-branches, gh-tidy, and gh-worktree because they sound related but solve different habits and risk profiles.

The data contract

The catalog is also available as JSON:

https://sjh9714.github.io/gh-extension-atlas/api/extensions.json

Each entry includes fields such as:

repository
category
install command
best-fit use case
avoid-if note
license
archived status
maintenance status
verification date

That makes it usable by scripts, documentation tools, or coding agents that need current extension metadata instead of stale memory.

What feedback would help

I am mainly looking for factual corrections and missing-extension suggestions:

inaccurate descriptions
wrong categories
misleading maintenance labels
better comparisons between overlapping tools
useful GitHub CLI extensions that are missing from the catalog

If you maintain a GitHub CLI extension and the atlas describes it poorly, a short correction is enough.

Note: I used AI assistance while organizing the launch plan and editing this post, but the project metadata, examples, and claims were reviewed before publishing.

DEV Community: JinHyuk Sung

I audited 249 of my own AI coding sessions. The problem wasn't lying.

I never scrolled up

Not a judge, a diff

Running it over 249 sessions

The real problem wasn't lying

And it falsely accused me six times

It caught me again while I was writing this

Three things I'd keep

Try it

I benchmarked Claude Code skills against a placebo — and half of mine failed

Why a placebo

Finding 1: the placebo often made code bigger

Finding 2: a 196k-star skill barely beat "no instructions"

Finding 3: two of my own four skills failed the gate

What this doesn't prove

Try it / break it

Your AI-contribution policy is prose. Here's the enforceable half.

What agent PRs actually do at scale

Which policy clauses are actually checkable

The point

What 2,204 merged AI-agent PRs actually touched (0 declared their scope)

The one number that surprised me

What else showed up

Honest denominators

The missing primitive

Reproduce it

AI agents can open PRs. Who checks whether they crossed the line?

The gap between "works" and "stayed in scope"

A checkout-free policy gate

Try it before installing it

What the evidence looks like

Differential workflow checks

Adopt it without turning CI red on day one

Honest limits

Would you block a PR that changes GitHub Actions contents permission from read to write?

I made Agent Gate installable in 30 seconds for AI PR checks

CI gates for AI-generated PRs need re-derivable evidence

What Agent Gate is

Why finding IDs were not enough

What v0.2.1 adds

A real report shape

Why this matters

What this does not solve yet

Try it

LLM reviewers are useful, but some PR checks should stay deterministic

1. Did the PR stay in scope?

2. Did workflow permissions escalate?

3. Did agent-control-plane files change?

4. Is there matching test evidence?

5. Did package scripts or dependencies drift?

6. Did the right human reviewer approve?

What should stay human?

What should be deterministic?

Start in warn mode

Closing thought

I built a deterministic CI firewall for AI-generated pull requests

The core idea

Why I wanted this

What Agent Gate catches today

Out-of-contract edits

Workflow permission escalation

Agent control-plane drift

Missing test evidence

What a report looks like

The trust boundary

Installing it

Local replay demo

What Agent Gate is not

Current limitations

What I want feedback on

Closing thought

Auditing GitHub CLI extensions before installing more

What the audit checks

Why not just use gh extension search?

A few examples

The data contract

What feedback would help

Why not just use `gh extension search`?