DEV Community

Gaurav Sharma
Gaurav Sharma

Posted on Originally published at mergejudge.si

How to gate AI agent pull requests on GitHub (without reading every line)

#ai #github #devops #opensource

Coding agents open more pull requests than people can read line by line. The goal isn't to read less — it's to spend human attention on the PRs that actually need it, and to make sure no agent can merge, or loosen the review rules, on its own.

If "good enough to merge" means "whatever the model thinks today", you get noisy line comments, and nothing stops an agent PR from quietly editing the instructions or workflow files it's reviewed with.

This walkthrough sets up AI pull request review where the repository holds the rules:

  1. Deterministic rules put every PR in a lane: auto (low risk), review (a human must look) or block.
  2. An AI model reads the diff and adds line comments, but can only make the lane stricter, never looser.
  3. The rules and policy are read from the base branch, never from the PR branch.

I'll use MergeJudge, an AI pull request reviewer with an open-source (MIT) engine I built for this, as the example. Disclosure up front: it's my project. The core ideas — base-branch config, a deterministic rules floor, and human-merged rule changes — carry over to any setup.

Real examples on GitHub (no install needed)

Before setting anything up, you can see what the output looks like on three real PRs:

Option A: Two-click GitHub App (free AI reviews on public repos)

If you don't want to manage YAML or keys first, install the GitHub App from https://mergejudge.si/guide/cloud?utm_source=devto. Public repositories get up to 10 free AI reviews per month out of the box with no API key and no config file. For private repos or higher volume, bring your own key or run the local CLI runner (npx mergejudge-runner setup), which uses your existing Claude Code or Codex login.

Option B: Self-hosted GitHub Action (no MergeJudge account)

If you want everything to run inside your own GitHub Actions runner with no MergeJudge account:

Step 1: Add your provider key as a repo secret

Pick Anthropic, OpenAI, z.ai or any OpenAI-compatible endpoint:

gh secret set AI_API_KEY
Enter fullscreen mode Exit fullscreen mode

(Or set provider: none to run deterministic rules only with no key.)

Step 2: Add .github/workflows/mergejudge.yml 

name: MergeJudge
on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
permissions:
  contents: read
  pull-requests: write
  checks: write
jobs:
  lane:
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: gauravsharmaGG/mergejudge-runner@v1
        env:
          AI_API_KEY: ${{ secrets.AI_API_KEY }}
        with:
          provider: anthropic        # or openai, zai, openai-compatible, none
          model: claude-sonnet-4-5   # any model your provider supports
Enter fullscreen mode Exit fullscreen mode

Open a PR and you get one sticky summary comment, a mergejudge/lane check (success = auto, neutral = review, failure = block) and inline comments on the diff.

Step 3: Put non-negotiable paths in .mergejudge.yml on your default branch

Anything that always requires a human — like .github/**, auth, or database migrations — shouldn't depend on a prompt noticing it:

lanes:
  protected:
    - ".github/**"
    - "deploy/**"
    - "**/auth/**"
    - "**/migrations/**"
Enter fullscreen mode Exit fullscreen mode

Protected paths always land in at least review, with or without an AI provider. Because .mergejudge.yml and AGENTS.md are read from the base branch tip, a PR opened by an agent cannot weaken the rules for its own review.

Step 4: Put judgement rules in AGENTS.md

If you already keep an AGENTS.md or CLAUDE.md in your repo for coding agents, add a ## Code review section in plain English (for example, "Every new HTTP handler under src/api/ must check resource authorization"). MergeJudge reads it from the base branch and uses it to raise the lane when violated.

What this setup does not do

Top comments (3)

Collapse
 
launchgatecheck profile image
Launch Gate •

The property that makes this trustworthy is reading the policy from the base branch tip: the diff under review can never edit the rules that judge it. Combined with "AI can only raise the lane, never lower it", the model's failure mode degrades toward more human attention rather than less, which is the right direction for a gate.

One gap worth a sentence in the docs: labels are in the trigger list (labeled, unlabeled), so anyone with triage permission can re-trigger a review by flipping labels. If label state ever feeds the lane decision, that becomes a way to shop for a friendlier verdict. Worth pinning labels as inputs-only, or excluding them from lane logic entirely.

And the "what this setup does not do" section is the part more tool pages should copy. Auto as a low-risk signal, not a guarantee, keeps branch protection where it belongs.

Collapse
 
gauravsharmagg profile image
Gaurav Sharma •

Good catch on the label trigger — we actually designed around both parts of that:

  1. Labels are one-way upward in core/rules.ts: labelLane() only checks labels.block (default do-not-merge) and labels.review (default needs-review). There is no labels.auto, so adding or removing a label can raise a PR to review or block, or release a manual hold back down to what the file rules + AI decided, never lower a diff below its rule/AI lane.
  2. No verdict shopping: on labeled / unlabeled events, the Worker reuses the cached LLM result for that head SHA and only recomputes the lane, check run, and labels without re-calling the model.

Agreed that's worth calling out explicitly in the guide — I'll add a line next to the workflow snippet.

Collapse
 
launchgatecheck profile image
Launch Gate •

The per-head-SHA cache is the elegant half of that answer: label flips recompute the lane from the same verdict, so they are cheap and cannot re-roll the dice on the same code.

One residual worth the docs line you are adding: removing labels.review releases the hold regardless of who set it, so anyone with triage permission can clear another reviewer's manual hold. If holds should be personal, keying the hold label to the labeler (or documenting that triage role equals hold authority) settles it. Otherwise the design reads clean.