When Generation Outruns Review: Add an Explanation Gate to Every Pull Request

Riley Xu — Mon, 10 Aug 2026 08:34:40 +0000

The constraint in many teams is no longer how quickly code can be produced. It is how quickly a qualified person can reconstruct why a change exists and predict what it will do at runtime. AI assistants widened that gap: drafts arrive in minutes, while careful reading still takes attention, context, and calendar time.

A useful response is not to ask the model for a verdict. Verdicts create a false choice between trusting software you cannot audit and ignoring it entirely. A better pattern is an explanation gate: before merge, the machine drafts a plain-language account of the diff, and a named reviewer must either affirm that account or correct it in writing. The model is there to make misunderstanding harder to hide, not to decide whether code ships.

The important shift is social as much as technical. The reviewer is no longer asked, “Does this look fine?” They are asked, “Here is a concrete interpretation of each changed hunk; where is it wrong?” That second task is easier to start and harder to fake.

A gate that tests comprehension, not prose

The workflow has four moving parts:

Build the exact patch CI will merge, usually git diff <base>...<head>.
Send that patch to an LLM with a narrow prompt: summarize intent per changed region, list observable effects, flag uncertainty, and raise questions. Ban ratings, approvals, and rewrite suggestions.
Post the model’s account back to the pull request as a review artifact.
Require a human response that walks the account line by line: confirm, amend, or reject. Branch protection blocks merge until that response exists.

If the model description is wrong, that is not automatically a tool failure. It can be evidence that the diff is ambiguous to a fresh competent reader. If the human cannot tell whether the description is right, the change is not ready, regardless of how clean the patch looks.

This design only works if running it on every pull request is boring operationally. Anything that feels expensive, fragile, or quota-anxious will be disabled during the first busy week. Keep the endpoint interchangeable and treat cost as an availability detail, not an architecture.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is a plausible backend for this kind of job because it offers free model access and a free server option; the workflow below still uses a generic OpenAI-compatible HTTP interface, so a self-hosted model or another compatible provider should work the same way. Do not build policy around any free tier lasting forever, having specific limits, or fitting your compliance posture without checking your own requirements.

Reference implementation

This version reads a diff from standard input, asks for a structured account, and prints Markdown with explicit sign-off checkboxes. It is intentionally small and uses only the Python standard library.

#!/usr/bin/env python3
"""Draft a reviewable account of a diff; a human must attest before merge."""
import json
import os
import sys
import urllib.request

API_ROOT = os.environ["EXPLAIN_GATE_URL"].rstrip("/")
API_KEY = os.environ["EXPLAIN_GATE_KEY"]
MODEL = os.environ["EXPLAIN_GATE_MODEL"]
PATCH_LIMIT = int(os.environ.get("EXPLAIN_GATE_PATCH_LIMIT", "40000"))

PROMPT = """You are preparing review notes for a code change, not deciding approval.
For every changed region in the patch:
1. State the likely authorial intent in one sentence.
2. List runtime or operational behavior that may change.
3. Mark unsupported inference as [UNCERTAIN].
4. Ask only questions a reviewer should answer before merge.
Do not grade style, do not approve, do not reject, do not propose edits.
End with a short 'Needs human confirmation' list."""

def read_patch() -> str:
    patch = sys.stdin.read()
    if not patch.strip():
        raise SystemExit("no patch on stdin; skipping explanation gate")
    if len(patch) > PATCH_LIMIT:
        patch = patch[:PATCH_LIMIT] + "\n[PATCH CUT OFF: inspect remaining files manually]"
    return patch

def call_model(patch: str) -> str:
    payload = json.dumps({
        "model": MODEL,
        "temperature": 0,
        "messages": [
            {"role": "system", "content": PROMPT},
            {"role": "user", "content": patch},
        ],
    }).encode("utf-8")
    req = urllib.request.Request(
        API_ROOT + "/chat/completions",
        data=payload,
        headers={
            "authorization": "Bearer " + API_KEY,
            "content-type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=240) as resp:
        data = json.loads(resp.read().decode("utf-8"))
    return data["choices"][0]["message"]["content"]

def main() -> None:
    account = call_model(read_patch())
    print("## Explanation gate")
    print()
    print(account)
    print()
    print("### Reviewer attestation")
    print("- [ ] I checked each changed region against the account above.")
    print("- [ ] I corrected every wrong or unsupported claim.")
    print("- [ ] I can explain the externally visible effects of this patch.")

if __name__ == "__main__":
    main()

A minimal CI shape looks like this:

explain-gate:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
      with:
        fetch-depth: 0
    - name: Draft explanation artifact
      env:
        EXPLAIN_GATE_URL: ${{ secrets.EXPLAIN_GATE_URL }}
        EXPLAIN_GATE_KEY: ${{ secrets.EXPLAIN_GATE_KEY }}
        EXPLAIN_GATE_MODEL: ${{ vars.EXPLAIN_GATE_MODEL }}
      run: |
        git diff origin/${{ github.base_ref }}...HEAD | python3 tools/explain_gate.py > gate.md
    - name: Comment on pull request
      uses: actions/github-script@v7
      with:
        script: |
          const fs = require('fs');
          await github.rest.issues.createComment({
            owner: context.repo.owner,
            repo: context.repo.repo,
            issue_number: context.issue.number,
            body: fs.readFileSync('gate.md', 'utf8')
          });

Then enforce the human half with branch protection: required review, dismissal of stale approvals on new pushes, and a convention that approval text must reference the gate artifact. Automation can post the notes; only repository rules can make the attestation mean anything.

Prompt rules that keep the model in its lane

The prompt should be restrictive enough that a bad answer is obviously bad. Three rules do most of the work:

No verdict vocabulary. If the output says “looks good” or “safe to merge,” treat the run as failed.
Forced uncertainty markers. Require [UNCERTAIN] for claims not directly supported by the patch. Confidence without evidence is the main hazard.
Questions must be merge-blocking. “Any concerns?” is useless. “Which callers still assume the old retry count?” is useful.

Keep temperature low and the patch bounded. A deterministic-ish witness is easier to calibrate than a creative one.

Calibrate before you trust it

Run the gate against patches designed to expose familiar failure modes before it blocks real work.

Constructed patch	Healthy output	Failure signal
A function renamed to imply validation it no longer performs	Describes the removed check and names the rename as misleading risk	Repeats the identifier as though it proves behavior
A condition inverted inside whitespace-only churn	Separates formatting noise from the logic reversal	Mentions only cleanup and misses the inverted branch
A config flag whose purpose cannot be inferred from the diff	Uses `[UNCERTAIN]` and asks who owns the flag	Invents a confident product rationale
A patch larger than the character budget	States that later files were excluded from analysis	Behaves as though the truncated tail was reviewed
Empty diff or merge-only noise	Exits quietly with no PR comment	Posts a generic summary on nothing

The row to care about most is the unsupported-rationale case. A gate that occasionally says “I cannot tell” is annoying; a gate that fabricates purpose is dangerous because it trains reviewers to outsource judgment.

Track one metric for the first month: how often the reviewer edits the model’s account in a way that changes merge readiness. If that number is always zero, either your patches are trivial or the human step has become ceremony.

Where this breaks

This pattern is strong for small and medium pull requests where the main risk is unexamined local change. It is weak in several places:

Cross-service effects. A diff can be internally coherent and still break a consumer elsewhere; no patch-level reader sees deployment topology.
High-assurance code. Authentication, cryptography, money movement, and safety logic still need domain reviewers. Treat generated notes as preparation, never evidence.
Sensitive source. If code cannot leave your environment, point the same script at an internal model or skip the approach. “Only a diff” is not a data classification.
Giant refactors. Truncation creates an illusion of coverage. Split the change or read it manually.
Checkbox culture. If reviewers can approve without writing corrections or confirmations, you have added latency and learned nothing.

Free access is also not a reliability promise. Keep provider details behind environment variables, log enough metadata to reproduce odd outputs, and make swapping endpoints a config change rather than a rewrite.

Start smaller than your whole repo

Pick one active service with frequent low-risk PRs and run the gate in comment-only mode for two weeks. Compare the generated accounts against what reviewers actually caught, then decide whether human attestation should become required. If you want a low-friction place to try the pattern, MonkeyCode’s free model access and free server option are one way to avoid metering every experiment; the same gate should still run against any compatible endpoint you trust.

The goal is not to make AI explain your codebase perfectly. It is to make every merged change carry a short public record showing that a person understood it well enough to disagree with a machine.