DEV Community: Gaurav Sharma

How to run AI security code review on GitHub pull requests without alert fatigue

Gaurav Sharma — Sat, 24 Oct 2026 21:44:36 +0000

To run AI security code review on GitHub pull requests without drowning in false alarms, run deterministic rules first for hard stops (secrets, removed CI safety guards, and protected paths), then use an evidence-gated security checklist that only raises a pull request's risk lane when the model quotes an exact line from the diff.

MergeJudge is an AI pull request reviewer with an open-source engine for GitHub. Every pull request is sorted into an auto, review or block lane: deterministic rules set the floor first, and the AI step can only raise a lane. auto means low risk, not a guarantee; branch protection and required checks still apply, and MergeJudge doesn't merge pull requests itself.

1. Deterministic rules run first (no model needed)

Before any model is called, MergeJudge runs deterministic checks on the pull request diff using the .mergejudge.yml file from the base branch:

Coarse secret screen (block): Added lines that match common secret formats (such as AWS keys, private key blocks, GitHub tokens, or hard-coded api_key assignments) put the pull request in block.
Removed safety guards (block): Changes to protected paths that drop --match-head-commit, persist-credentials: false, set -e, or --frozen-lockfile / npm ci put the pull request in block.
Protected paths (at least review): Files matching lanes.protected (by default .github/**, deploy/**, **/auth/**, **/migrations/**, wrangler.toml, _headers, **/_headers, and **/.env*) always require human review, even if no AI provider is configured.
Default security rubric (security pack, on by default): When an AI provider is active, every review already scores sec.no-secrets (block), sec.no-exfiltration (block), sec.no-unsafe-exec (review), and sec.input-handling.

Because .mergejudge.yml is read from the base branch tip, a pull request cannot edit .mergejudge.yml to remove a protected path or turn off a check for itself.

2. Turn on optional security review mode

For teams that want a closer security audit on every pull request—especially when reviewing AI-agent pull requests or backend, webhook, and auth changes—MergeJudge includes an optional Security review mode. It is off by default.

Turn it on in .mergejudge.yml on your default branch:

# .mergejudge.yml on the default branch
review:
  mode: context        # diff | context | deep
  security: true       # default false; only a real boolean true turns it on

On the hosted GitHub App, you can also flip the Security review switch in the Set up wizard or Edit settings in the panel; it opens a setup pull request on mergejudge/setup-config for you to merge. On the GitHub Action and runner, add review.security: true to .mergejudge.yml directly.

When review.security: true is on, MergeJudge scores five high-severity attack-class criteria alongside the normal rubric (and overrides rubrics.enabled: false if rubrics were turned off):

Criterion ID	What it checks	Raises to
`sec.audit.injection`	Untrusted input (request params, headers, webhook/queue payloads, file/env content, model output) reaching a sink unescaped: SQL, shell/exec, `eval`, HTML/templates, file paths, redirects, deserialization	`review` (when confirmed in diff)
`sec.audit.authz`	A new or changed route, handler, job or tool acting for a caller without verifying identity and resource ownership (missing auth, IDOR, dropped or moved role check, tenant/owner ID taken from request)	`review` (when confirmed in diff)
`sec.audit.secrets-crypto`	Credentials, tokens or keys logged, returned, embedded or committed; weak or home-made crypto (MD5/SHA-1 for passwords, ECB, static IV/salt, `Math.random` for tokens, `==` on secrets, TLS verification off)	`review` (when confirmed in diff)
`sec.audit.callbacks`	Inbound webhooks or callbacks skipping signature, token or origin verification; server-side fetches of an attacker-chosen URL, host or redirect target without an allowlist (SSRF, internal addresses, metadata endpoints)	`review` (when confirmed in diff)
`sec.audit.ai-tools`	Model output or untrusted text gaining shell, file, network or secret access, or overriding a system prompt, without a check or confirmation (scored only when the diff touches agents, tools, prompts or model calls; otherwise `n/a`)	`review` (when confirmed in diff)

3. How confirmed and needs-validation findings stop hallucinated blocks

A common failure mode of LLM security reviews is flagging theoretical hardening wishes or guessing about middleware that lives outside the diff. MergeJudge constrains sec.audit.* in two ways:

Trust-boundary prompt discipline: The prompt instructs the model to fail a criterion only for a real trust-boundary break that the diff introduces or exposes—where attacker-controlled data crosses a boundary without the check that boundary needs. Style, hardening wishes, theory, and code already behind a check are scored pass.
Quote verification against the diff:
- ❌ Confirmed (fail): The model copied an exact line from the diff into evidence.quote, and MergeJudge verified that quote against the diff hunk. A confirmed finding appears with ❌ in the Rubric section and raises the pull request lane to review.
- ❔ Needs validation (unverified): The model suspects a trust-boundary break that depends on code outside the diff (for example, whether a router mounts auth middleware upstream) and leaves the quote empty. MergeJudge lists the finding with ❔ (path:line and "Title: why") so a human can check it, leaves the lane unchanged, and keeps low-risk code auto off for that pull request.

4. Pair security mode with `context` or `deep` mode for larger PRs

The five sec.audit.* criteria add about 500 tokens to every review prompt, counted against budget.max_tokens_per_pr.

Free AI tier (public repositories with no key configured): Keeps its usual token, diff-size, and monthly review limits; nothing is raised for security mode.
Larger pull requests or cross-file checks: Because diff mode only shows changed hunks, an auth check or sanitizer defined in another file may look unverified (❔) in diff mode. For larger pull requests or deeper passes where the reviewer should inspect callers and imports, run with your own provider key in context mode or use a runner on your machine or the GitHub Action in deep mode.

FAQ

How do I turn on security review mode in MergeJudge?

Set review.security: true in .mergejudge.yml on your repository's default branch, or toggle the Security review switch in the MergeJudge panel under Set up or Edit settings and merge the resulting pull request. It is off by default, and only a real boolean true turns it on.

What does MergeJudge's security review mode check?

When review.security: true is set, MergeJudge scores five high-severity attack-class criteria alongside its normal rubric: sec.audit.injection (untrusted input reaching SQL, shell, eval, HTML, path, redirect or deserialization sinks), sec.audit.authz (missing authentication, IDOR or dropped role/tenant checks), sec.audit.secrets-crypto (leaked credentials, weak hashes or ciphers, == on secrets, disabled TLS verification), sec.audit.callbacks (unverified webhooks or SSRF), and sec.audit.ai-tools (untrusted text or model output gaining shell, file, network or secret access when the diff touches agents, tools or prompts).

How does MergeJudge avoid hallucinated security failures raising the lane?

Every high-severity rubric failure must include an exact quote that MergeJudge verifies against the pull request diff. A finding with a verified diff quote is marked confirmed (❌) and raises the lane to review. If the model suspects a trust-boundary break that depends on code outside the diff and cannot quote a changed line, MergeJudge lists it with a question-mark icon (❔) as needs validation: it leaves the lane unchanged so a human can check it, while still keeping opt-in lanes.code_auto off for that pull request.

Can a pull request turn off `review.security: true` in its own diff?

No. MergeJudge always reads .mergejudge.yml and policy files such as AGENTS.md and REVIEW_POLICY.md from the pull request's base branch tip, never from the pull request head. Any pull request that edits .mergejudge.yml is placed in the review lane.

Does security review mode increase the free AI tier limits?

No. The five sec.audit.* criteria add about 500 tokens to every review prompt and count against budget.max_tokens_per_pr. The free AI tier on public repositories keeps its usual token, diff-size and monthly review caps. For larger pull requests or deeper cross-file checks, use the open-source runner on your machine or your own provider API key.

Full guide & rubric reference: mergejudge.si/guide/ai-security-code-review-github
Security rubric criteria (sec.audit.*): mergejudge.si/guide/rubrics
Open-source runner & GitHub Action (MIT): github.com/gauravsharmaGG/mergejudge-runner

How to gate AI agent pull requests on GitHub (without reading every line)

Gaurav Sharma — Sat, 24 Oct 2026 09:26:11 +0000

Coding agents open more pull requests than people can read line by line. The goal isn't to read less — it's to spend human attention on the PRs that actually need it, and to make sure no agent can merge, or loosen the review rules, on its own.

If "good enough to merge" means "whatever the model thinks today", you get noisy line comments, and nothing stops an agent PR from quietly editing the instructions or workflow files it's reviewed with.

This walkthrough sets up AI pull request review where the repository holds the rules:

Deterministic rules put every PR in a lane: auto (low risk), review (a human must look) or block.
An AI model reads the diff and adds line comments, but can only make the lane stricter, never looser.
The rules and policy are read from the base branch, never from the PR branch.

I'll use MergeJudge, an AI pull request reviewer with an open-source (MIT) engine I built for this, as the example. Disclosure up front: it's my project. The core ideas — base-branch config, a deterministic rules floor, and human-merged rule changes — carry over to any setup.

Real examples on GitHub (no install needed)

Before setting anything up, you can see what the output looks like on three real PRs:

auto (small docs typo): https://github.com/gauravsharmaGG/mergejudge-demo/pull/2
review (workflow file change): https://github.com/gauravsharmaGG/mergejudge-demo/pull/3
block (hard-coded API key): https://github.com/gauravsharmaGG/mergejudge-demo/pull/4

Option A: Two-click GitHub App (free AI reviews on public repos)

If you don't want to manage YAML or keys first, install the GitHub App from https://mergejudge.si/guide/cloud?utm_source=devto. Public repositories get up to 10 free AI reviews per month out of the box with no API key and no config file. For private repos or higher volume, bring your own key or run the local CLI runner (npx mergejudge-runner setup), which uses your existing Claude Code or Codex login.

Option B: Self-hosted GitHub Action (no MergeJudge account)

If you want everything to run inside your own GitHub Actions runner with no MergeJudge account:

Step 1: Add your provider key as a repo secret

Pick Anthropic, OpenAI, z.ai or any OpenAI-compatible endpoint:

gh secret set AI_API_KEY

(Or set provider: none to run deterministic rules only with no key.)

Step 2: Add `.github/workflows/mergejudge.yml`

name: MergeJudge
on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
permissions:
  contents: read
  pull-requests: write
  checks: write
jobs:
  lane:
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: gauravsharmaGG/mergejudge-runner@v1
        env:
          AI_API_KEY: ${{ secrets.AI_API_KEY }}
        with:
          provider: anthropic        # or openai, zai, openai-compatible, none
          model: claude-sonnet-4-5   # any model your provider supports

Open a PR and you get one sticky summary comment, a mergejudge/lane check (success = auto, neutral = review, failure = block) and inline comments on the diff.

Hold a pull request with a label

The GitHub Action workflow includes labeled and unlabeled, so a hold applies without a new push:

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]

Labels can only raise a lane. Adding needs-review or do-not-merge holds a pull request, and removing it sends the pull request back to its rule and AI lane. Anyone with GitHub triage access can remove it, so treat triage access as hold authority. Label changes reuse the cached verdict for that commit, so they never re-run the model.

Step 3: Put non-negotiable paths in `.mergejudge.yml` on your default branch

Anything that always requires a human — like .github/**, auth, or database migrations — shouldn't depend on a prompt noticing it:

lanes:
  protected:
    - ".github/**"
    - "deploy/**"
    - "**/auth/**"
    - "**/migrations/**"

Protected paths always land in at least review, with or without an AI provider. Because .mergejudge.yml and AGENTS.md are read from the base branch tip, a PR opened by an agent cannot weaken the rules for its own review.

Step 4: Put judgement rules in `AGENTS.md`

If you already keep an AGENTS.md or CLAUDE.md in your repo for coding agents, add a ## Code review section in plain English (for example, "Every new HTTP handler under src/api/ must check resource authorization"). MergeJudge reads it from the base branch and uses it to raise the lane when violated.

What this setup does not do

It doesn't merge PRs automatically — auto is a low-risk signal, not a guarantee, and human branch protection stays in charge.
It doesn't replace a full secret scanner (the secret check on added lines is a fast screen).
When an AI provider is enabled, the diff and context snippets are sent to that provider under your key.
Source & Action: https://github.com/gauravsharmaGG/mergejudge-runner
Marketplace: https://github.com/marketplace/actions/mergejudge-ai-code-review
Docs & guides: https://mergejudge.si/?utm_source=devto