A Probability Is Not a Gate: Reviewing Agent PRs That Put LLMs in CI

An LLM step in CI is not extra review coverage. It is a new production dependency that can veto a merge with a different answer on every run. Agent pull requests that add that step should be triaged like any other hidden scheduler: keep a testable client, revert the gate, and prove the contract against a replayable endpoint.

This walkthrough uses an illustrative agent PR. It does not measure model quality. It measures something narrower. Non-deterministic jobs do not belong on the critical path of main.

The PR under review

The agent titled the change "add AI review to pull requests." The diff is small. The blast radius is not.

Typical files:

• .github/workflows/ai-review.yml
• scripts/ai_review.py
• a prompt file, often prompts/pr_review.md
• a new secret name in the README, and sometimes in the workflow env block

The workflow usually looks like this.

# illustrative — do not copy into a required check
name: ai-review
on: [pull_request]
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install httpx
      - env:
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
          MODEL_BASE_URL: ${{ vars.MODEL_BASE_URL }}
        run: python scripts/ai_review.py

Three properties turn a comment helper into a merge gate:

1. The job is required, or it will be made required "after we see it work."
2. A non-zero exit fails the pull request.
3. Model output is parsed for words such as BLOCK, FAIL, or security issue.

If any of those three is present, the PR did not add coverage. It added a lottery.

Agents add this pattern because the task prompt asked for quality, not because CI gained a deterministic oracle. A linter fails on the same AST every time. A model does not. Treating the second like the first is the defect.

What the script usually does

The companion script is also small. That is why it survives a skim.

# illustrative agent output — scripts/ai_review.py
# Example only. Not a recommended implementation.
import os, re, subprocess, sys
from pathlib import Path
import httpx

diff = subprocess.check_output(
    ["git", "diff", "origin/main...HEAD"], text=True
)
prompt = Path("prompts/pr_review.md").read_text() + "\n\n" + diff
url = os.environ.get("MODEL_BASE_URL", "https://api.example.invalid/v1/chat")
r = httpx.post(
    url,
    headers={"Authorization": f"Bearer {os.environ['MODEL_API_KEY']}"},
    json={"messages": [{"role": "user", "content": prompt}]},
)
text = r.json()["choices"][0]["message"]["content"]
if re.search(r"BLOCK|FAIL|CRITICAL", text, re.I):
    print(text)
    sys.exit(1)
print(text)

Concrete problems in that block:

• No timeout, size cap, or retry budget. A hung vendor becomes a hung required check.
• The full diff is uploaded. .env, lockfile churn, and private keys ride along unless something else strips them.
• Free-form prose is mapped to sys.exit(1). That mapping is the gate.
• A hardcoded host appears as a default. Environment injection becomes optional, then forgotten.
• There is no denylist test. Green CI means "the model answered," not "secrets stayed local."

Trust stops at the first branch that interprets natural language as a boolean.

What to trust

Trust only what you can replay without the model.

• YAML shape: checkout, language setup, which files are sent
• Secret names (not values) and whether those names are required
• HTTP client construction: timeouts, headers, maximum body size
• Redaction: whether the script drops .env, key files, and Authorization lines before upload
• Exit-code policy: whether the process always exits 0 after posting a comment

A timeout value is evidence. A prompt file is not. One is a contract. The other is input to a function you do not control.

What to revert

Revert the merge gate. Keep the client if it is actually a client.

Revert immediately when the script:

• Fails the job on phrases in model output
• Uploads the full diff, including paths matching **/.env*, **/*secret*, or **/id_rsa
• Hard-codes a vendor URL, model identifier, or token budget
• Posts unreviewed comments as GITHUB_TOKEN with pull-requests: write
• Leaves continue-on-error at the default false on a model call
• Reuses the production inference secret in CI

A comment a human may ignore is a tool. A red X a human must override is an owner. Agents rarely add the owner. They add the X.

Hard-coded hosts deserve the same revert as hard-coded credentials. Both prevent the next reviewer from pointing the client at a fixture. Both also couple main to one vendor's incident calendar.

What to test

Test the client as a library. Do not test "the model is smart."

Minimum assertions:

1. Diff collection respects a path denylist.
2. Request bodies are capped (for example 64 KiB). A timeout is set. Retries cannot amplify into a storm.
3. The parser never maps unknown text to fail.
4. The CI job sets continue-on-error: true, or the job is not a required status check.
5. Replay against a fixture endpoint succeeds or fails deterministically for canned payloads.

Item 5 is where a free model endpoint is useful. It is a network fixture. It is not a judge. A 200 proves the client can speak HTTP. It does not prove the prompt should block a merge.

Artifact: a triage harness

The following is a proposed local tool. It is not production CI. Run it on the PR branch before arguing about prompt wording.

Decision table

Signal in the diff	Trust	Revert	Test
HTTP client with timeout and size cap	Client code	—	Unit test the cap
Required GitHub check on model output	—	The check	Assert the job is optional
Prompt file with no input bound	—	Unbounded prompt	Length / cap test
Secret used in a required job	Name only	Coupling to prod key	Secret scanning
continue-on-error: true plus comment only	Comment path	—	Mock GitHub API
Hard-coded vendor host	—	Host and model id	Inject MODEL_BASE_URL
Diff sent without a denylist	—	Upload path	Fixture repo that contains .env

The table is the review. The script below only automates the first pass so the comment can be short.

Commands

# illustrative review commands — run on the PR branch
git fetch origin pull/N/head:pr-n
git switch pr-n

# 1. Find CI jobs that invoke the new script
rg -n "ai_review|MODEL_API_KEY|openai|anthropic|complete\(" \
  .github/workflows scripts

# 2. Fail the review if the workflow can veto merge on model text
python3 tools/review_llm_ci.py --workflow .github/workflows/ai-review.yml

tools/review_llm_ci.py

#!/usr/bin/env python3
"""Classify an agent-added 'AI review' workflow. Proposed review aid."""
from __future__ import annotations

import argparse
import os
import re
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("pip install pyyaml", file=sys.stderr)
    sys.exit(2)

FAIL_WORDS = re.compile(r"\b(block|fail|error|vulnerable|critical)\b", re.I)


def load_workflow(path: Path) -> dict:
    data = yaml.safe_load(path.read_text())
    if not isinstance(data, dict):
        raise ValueError(f"not a mapping: {path}")
    return data


def jobs(doc: dict) -> dict:
    raw = doc.get("jobs") or {}
    return raw if isinstance(raw, dict) else {}


def step_runs(job: dict) -> str:
    chunks = []
    for step in job.get("steps") or []:
        if isinstance(step, dict) and isinstance(step.get("run"), str):
            chunks.append(step["run"])
    return "\n".join(chunks)


def is_required_model_gate(job: dict) -> bool:
    if job.get("continue-on-error") is True:
        return False
    blob = yaml.safe_dump(job)
    return bool(re.search(r"MODEL_|API_KEY|base_url|openai|anthropic", blob, re.I))


def classify(doc: dict) -> list[str]:
    findings = []
    for name, job in jobs(doc).items():
        if not isinstance(job, dict):
            continue
        dumped = yaml.safe_dump(job)
        if is_required_model_gate(job):
            findings.append(
                f"REVERT gate: job {name!r} calls a model without continue-on-error"
            )
        if "GITHUB_TOKEN" in dumped and "pull-request" in dumped.lower():
            findings.append(f"TEST comment auth: job {name!r} can post on the PR")
        if step_runs(job) and "MODEL_BASE_URL" not in dumped:
            findings.append(f"REVERT missing MODEL_BASE_URL injection in job {name!r}")
    return findings


def scan_script(path: Path) -> list[str]:
    if not path.exists():
        return [f"TEST missing script: {path}"]
    text = path.read_text(errors="replace")
    findings = []
    if "timeout" not in text.lower():
        findings.append(f"REVERT missing timeout in {path}")
    if FAIL_WORDS.search(text) and "sys.exit(1)" in text:
        findings.append(f"REVERT text-to-exit mapping in {path}")
    if not any(s in text for s in (".env", "denylist", "deny_list", "skip_files")):
        findings.append(f"REVERT no path denylist in {path}")
    return findings


def replay_healthcheck() -> list[str]:
    """Optional: prove client config can reach a fixture. Not a quality test."""
    base = os.environ.get("REVIEW_REPLAY_URL")
    if not base:
        return ["TRUST skip replay: set REVIEW_REPLAY_URL to exercise the client"]
    try:
        import urllib.request

        req = urllib.request.Request(base, method="GET")
        with urllib.request.urlopen(req, timeout=5) as resp:
            if 200 <= resp.status < 500:
                return [f"TEST replay reached {base} status={resp.status}"]
            return [f"REVERT replay unexpected status {resp.status} from {base}"]
    except Exception as exc:
        return [f"TEST replay error (fixture, not product): {exc!r}"]


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--workflow", type=Path, required=True)
    parser.add_argument("--script", type=Path, default=Path("scripts/ai_review.py"))
    args = parser.parse_args()
    findings = []
    findings.extend(classify(load_workflow(args.workflow)))
    findings.extend(scan_script(args.script))
    findings.extend(replay_healthcheck())
    for line in findings:
        print(line)
    revert = sum(1 for item in findings if item.startswith("REVERT"))
    print(f"summary: {revert} revert-class finding(s), {len(findings)} total")
    return 1 if revert else 0


if __name__ == "__main__":
    sys.exit(main())

Exit code 1 means "do not merge as-is." Exit code 0 means "no revert-class signal." It does not mean the model is correct. Phrase lists go stale. Agents rename BLOCK to needs_changes. The human still reads the parser.

Replay fixture, not a second reviewer

Reviewers still need somewhere to point MODEL_BASE_URL when the client is worth keeping. A dead vendor URL blocks that proof. The harness treats replay as optional on purpose: classification of the gate should work on an airplane. Construction of the client should not require a paid account.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. For this protocol, that pair is only a replay fixture. Export REVIEW_REPLAY_URL and MODEL_BASE_URL so the client can be constructed, timed out, and failed on purpose. Do not attach that fixture as a required GitHub check. Doing so would recreate the defect under review. If the client cannot accept a base URL from the environment, that absence is a revert by itself.

Limitations

• The harness does not evaluate prompt quality, jailbreak resistance, or suggestion accuracy.
• A 200 from a fixture is not a latency, quota, or availability SLO.
• Regex over workflow YAML misses composite actions and reusable workflows in other repositories.
• Exit-word lists (BLOCK, FAIL) rot. Re-read the parser on every PR.
• Free endpoints change. Pin nothing in CI that you cannot replace with one environment variable.
• Product features that are inference need contract tests and capacity planning. They are out of scope here. A merge veto on prose is in scope.

Who should not use this approach

• Teams whose CI has no egress. Skip replay. Keep the YAML classification.
• Regulated repositories that cannot send diffs or prompts to any third-party model, free or not. Revert the upload path regardless of fixture.
• Changes that only add a local, deterministic linter. Those are not this PR.
• Anyone hoping an LLM job will replace human review of auth, migrations, or lockfiles.

Review comment to paste

When the agent PR matches the table, a short review comment is enough:

This job makes merge depend on non-deterministic model text.
Please:
1. Keep the HTTP client if it has timeout, size cap, and MODEL_BASE_URL.
2. Revert required status / non-zero exit on model output.
3. Add a denylist test with a fixture .env in the diff.
4. Set continue-on-error: true if a comment bot is still desired.

That is the protocol. Trust the wiring you can replay. Revert the gate. Test the denylist and the exit policy. The model can wait until it is a product, not a coin flip on main.