AI PRs Outpace Human Reviews: Build a Free-Token Triage Gate First

#ai #testing #productivity #opensource

Thursday, 16:47. A teammate merges an AI-generated fix that passes CI, touches three files, and quietly removes one assertion. Monday morning the checkout flow returns a 500, and nobody can name the commit that broke it. The generation was not the problem; the review capacity was.

The current community discussion keeps landing on the same uncomfortable line: AI promoted every developer to reviewer, and the reviewer part is the unmanaged bottleneck. Agents produce more diffs per week than any human queue can absorb at the old reading speed. The realistic fix is not a smarter model, but a cheaper first pass that separates diffs a human must read from diffs a human can skim.

A Pre-Review Triage Gate

The practical pattern for this bottleneck is a pre-review gate: a small script that checks every PR diff for three failure syndromes and returns PASS, REVIEW, or REJECT. Humans read only the REVIEW lanes; REJECT items bounce back to the author with the reason attached. The gate makes no judgment about business logic, which is exactly why it stays cheap to run.

Syndrome 1 — silent exception swallowing. An empty except block or a bare pass after an exception hides the one failure a test would otherwise surface. The pattern is easy to detect and always worth a comment.

Syndrome 2 — test weakening. Deleted assertions, assert True, and commented-out cases turn a green suite into a lie. This is the most expensive syndrome because a passing CI run gives everyone false confidence until production disagrees.

Syndrome 3 — hallucinated identifiers. Generated code loves to call functions your repo does not export. A model cannot see your package table by itself, so the gate builds a manifest from the actual module and hands it to the model as part of the prompt.

The Script

# triage_pr.py — first-pass review gate for AI-generated PRs
import os
import re
import subprocess

import requests


def build_manifest(module_name: str) -> set[str]:
    """Extract real public symbols from the package, not from the model."""
    code = f"import {module_name}\nprint(' '.join(dir({module_name})))"
    out = subprocess.check_output(['python', '-c', code], text=True)
    return {name for name in out.split() if not name.startswith('_')}


SILENT_EXCEPT = re.compile(r"except[^:]*:\s*(pass|\.\.\.)\s*$", re.M)
TEST_WEAKENING = re.compile(r"(assert\s+True|\.skip\s*\(|pytest\.mark\.skip)", re.M)


def triage(diff: str, module_name: str) -> dict:
    manifest = build_manifest(module_name)
    prompt = (
        'You are a PR triage agent. The codebase exports these symbols:\n'
        + ', '.join(sorted(manifest)[:200])
        + '\nFlag any identifier called in this diff that is not in that list. '
        'Reply with one line per finding: SYMBOL,LINE.\n\n'
        + diff
    )
    response = requests.post(
        os.environ['MODEL_ENDPOINT'],
        headers={'Authorization': f"Bearer {os.environ['MODEL_KEY']}"},
        json={
            'model': os.environ['MODEL_NAME'],
            'messages': [{'role': 'user', 'content': prompt}],
            'max_tokens': 500,
        },
        timeout=90,
    )
    content = response.json()['choices'][0]['message']['content']
    findings = [line for line in content.splitlines() if ',' in line]
    return {
        'silent': SILENT_EXCEPT.findall(diff),
        'weakened': TEST_WEAKENING.findall(diff),
        'hallucinated': findings,
    }


def verdict(result: dict) -> str:
    if result['hallucinated']:
        return 'REJECT: probable hallucinated API calls in diff'
    if result['silent'] or result['weakened']:
        return 'REVIEW: human must see this diff before merge'
    return 'PASS: safe for the fast lane'

The script reads the endpoint, key, and model name from environment variables, so the same file works against any provider. The response parsing assumes an OpenAI-compatible chat shape; if your endpoint differs, only the parse function changes. It never merges anything; it only labels traffic.

Wiring It Into CI and a Server Job

The cheapest useful placement is a GitHub Action on pull_request that posts the verdict as a comment:

name: pr-triage
on: pull_request
jobs:
  triage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install requests
      - run: python triage_pr.py
        env:
          MODEL_ENDPOINT: ${{ secrets.MODEL_ENDPOINT }}
          MODEL_KEY: ${{ secrets.MODEL_KEY }}
          MODEL_NAME: ${{ secrets.MODEL_NAME }}

PRs also age badly: a diff that looks clean at 09:00 can mask a conflict with the base branch by 17:00. A nightly job on a small server reruns the same gate over every open PR and comments when the verdict flips. Free servers restart without warning, so wrap the job in a watchdog and store state on disk rather than in memory; the predictable failure modes are restarts, rate limits, and expired credentials, not the model itself.

The One-Page Runbook

Paste this into your team wiki and keep it to five steps:

Author submits the PR with a one-line test plan; the triage gate labels it automatically.
REJECT items go back to the author without a human reviewer; the comment names the syndrome.
PASS items enter the fast lane and wait for the nightly re-check.
REVIEW items are the only ones that get human eyes, plus any PR touching auth, payments, or data migration.
The maintainer merges, and incident post-mortems always quote the gate verdict as evidence.

Where the Token Budget Comes From

This workload is exactly the kind of recurring, low-stakes traffic that burns a paid quota surprisingly fast. MonkeyCode is an open-source project whose free model access (the published allowance is 10M tokens as of 2026-08-28) and free server option fit this gate without requiring a credit card. Your checks run identically against any compatible endpoint, so switching providers later costs nothing but a new environment variable. Free-tier numbers change, so verify the current allowance on the project page before you rely on it in a team runbook.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations

A model-based gate is a heuristic, not an approval. It cannot judge whether the business logic is correct, so it must never auto-merge and never replace required reviews in regulated or security-sensitive repos. Teams without an existing test culture get a blind test-weakening check, because the gate only understands tests that already exist. And if your team generates a handful of PRs per month, the setup cost is higher than the time it saves; the fast lane only pays off when the diff volume actually outruns the humans.

Try It With a Small Bet

Start with one repository, one week, and no policy change. Let the gate comment on every PR, count how many REJECT labels match what a reviewer would have said, and only then turn on the fast lane. If you want a low-risk place to test the token-budget side of the gate, MonkeyCode's free tier is a reasonable first stop; the script runs just as well against your own key.

DEV Community