It is Tuesday morning, and your review queue contains fourteen pull requests that an agent opened overnight. Three of them touch a migration, two rewrite a workflow file, and every single one has green checks. The scenario is composite rather than a specific incident, but the shape is familiar to most teams running an agent inside CI today.
The uncomfortable part is not the volume; it is that agent diffs look uniform and read as finished. A human diff usually carries a small hint of hesitation, while a generated one arrives confident, formatted, and internally consistent. When reviewers are tired, that consistency is exactly the signal they stop questioning.
This article describes a triage SOP you can paste into a wiki, with three roles, one gate script, and a resumable worker stage. The artifacts are dependency-free Python plus a GitHub Actions step, and you can reproduce all of it in a scratch repository before adopting anything.
Why review attention, not agent throughput, is the real constraint
Most teams measure agent output in merged pull requests, which rewards volume and hides the cost behind it. The scarce resource is the first twenty minutes of a domain owner's attention, and that budget does not scale with the number of open diffs. If you route every agent diff to the same reviewer, you have automated authoring and centralized the debt.
A triage layer does one thing well: it decides who reads a diff first and how much trust that diff receives. It does not judge correctness, and it should never merge anything on its own. Keep that boundary written down, because it is the part people quietly relax after a few good weeks.
Step 1: Name three roles, and rotate exactly one of them
Author of record is the human who asked for the change and whose name stays on the pull request, even when the diff came from an agent. They are responsible for the description, the linked issue, and the rollback note. If nobody will sign as author of record, the pull request does not enter triage at all.
Gatekeeper rotates weekly and owns the classification result for every incoming diff. The gatekeeper does not review code in depth; they confirm the tier, assign the reviewer of record, and escalate tier C work immediately. Rotating this role weekly keeps the blast-radius rules honest and spreads the boring work.
Reviewer of record is the domain owner who reads the diff first and holds the merge button. They must be a human, they must be named in the pull request, and they cannot be the gatekeeper for the same diff. Two roles, one person, one bad merge: that is the whole reason for the separation.
Step 2: Tag authorship before anyone reads a single line
Make authorship machine-readable, because reviewers unconsciously extend more trust to diffs they believe a colleague wrote. Ask agents to add an Author-Agent: <tool-name> trailer to the commit body and to open pull requests with an agent-authored label. The trailer survives squash merges and label edits, so the audit trail does not depend on someone remembering to click.
Then add one rule to the wiki: a diff without the trailer and without a named author of record is returned, not reviewed. That single rule removes the ambiguous middle ground where reviewers guess at provenance and then guess at risk.
Step 3: Classify by blast radius with a dependency-free script
The script below classifies a diff into three tiers using changed paths and diff size. Tier A is docs and tests, tier B is ordinary source code, and tier C is anything that can damage data, credentials, infrastructure, or workflows. Save it as tools/triage.py; it uses only the standard library and shells out to git.
#!/usr/bin/env python3
"""triage.py - classify a pull request diff into human-review tiers.
Usage:
python3 tools/triage.py --base origin/main --head HEAD --json-out triage.json
Exit codes: 0 = tier A/B (agent may continue), 3 = tier C (human first).
"""
from __future__ import annotations
import argparse
import fnmatch
import json
import subprocess
from datetime import datetime, timezone
# fnmatch treats "*" as spanning "/", so "**/" here is documentation, not syntax.
TIER_C_GLOBS = [
"**/migrations/*", "**/*.sql", "infra/**", ".github/workflows/*",
"**/auth/**", "**/billing/**", "**/*secret*", "Dockerfile",
]
TIER_B_GLOBS = ["src/**", "lib/**", "app/**", "scripts/**", "tools/**"]
TIER_C_MAX_LINES = 400 # any diff larger than this waits for a human
def changed_files(base: str, head: str) -> list[str]:
out = subprocess.run(
["git", "diff", "--name-only", f"{base}...{head}"],
check=True, capture_output=True, text=True,
).stdout
return [line.strip() for line in out.splitlines() if line.strip()]
def diff_size(base: str, head: str) -> int:
out = subprocess.run(
["git", "diff", "--numstat", f"{base}...{head}"],
check=True, capture_output=True, text=True,
).stdout
total = 0
for line in out.splitlines():
parts = line.split("\t")
if len(parts) >= 2 and parts[0].isdigit() and parts[1].isdigit():
total += int(parts[0]) + int(parts[1])
return total
def matched(path: str, globs: list[str]) -> list[str]:
return [g for g in globs if fnmatch.fnmatch(path, g) or fnmatch.fnmatch(path, g.lstrip("*"))]
def classify(paths: list[str], size: int) -> tuple[str, list[str]]:
if size > TIER_C_MAX_LINES:
return "C", [f"diff is {size} lines, over the {TIER_C_MAX_LINES}-line limit"]
reasons = [f"{p} matches {g}" for p in paths for g in matched(p, TIER_C_GLOBS)]
if reasons:
return "C", reasons[:3]
if any(matched(p, TIER_B_GLOBS) for p in paths):
return "B", ["source change within a known directory"]
return "A", ["docs or tests only"]
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--base", required=True)
ap.add_argument("--head", default="HEAD")
ap.add_argument("--json-out")
args = ap.parse_args()
paths = changed_files(args.base, args.head)
size = diff_size(args.base, args.head)
tier, reasons = classify(paths, size)
record = {
"tier": tier, "reasons": reasons, "files": len(paths), "lines": size,
"base": args.base, "head": args.head,
"checked_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
}
if args.json_out:
with open(args.json_out, "w", encoding="utf-8") as fh:
json.dump(record, fh, indent=2)
print(json.dumps(record, indent=2))
return 3 if tier == "C" else 0
if __name__ == "__main__":
raise SystemExit(main())
Run it locally against an open branch before you touch CI, and read the reasons array rather than only the tier. If a plain docs pull request comes back as tier C, your glob list is too broad and the gatekeeper will lose trust in the output within a day. Tune the globs to your own repository layout; the ones above are a starting point, not a universal risk model.
Step 4: Wire the gate into CI as a warning, not a wall
A gate that fails the build on tier C teaches people to rename files instead of reading them. Start with a warning plus a job summary, and promote it to a hard failure only after two weeks of clean history. The step below writes the tier to the job output so a later step can react to it.
name: pr-triage
on:
pull_request:
types: [opened, synchronize, reopened]
permissions:
contents: read
jobs:
triage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Classify blast radius
id: triage
run: |
status=0
python3 tools/triage.py \
--base "origin/${{ github.base_ref }}" \
--head "${{ github.event.pull_request.head.sha }}" \
--json-out triage.json || status=$?
tier=$(python3 -c 'import json;print(json.load(open("triage.json"))["tier"])')
echo "tier=${tier}" >> "$GITHUB_OUTPUT"
echo "Triage tier: ${tier}" >> "$GITHUB_STEP_SUMMARY"
exit 0
- name: Flag tier C for a human-first pass
if: steps.triage.outputs.tier == 'C'
run: echo "::warning::Tier C diff: assign a reviewer of record before the agent continues"
Keep the classification step non-blocking and let the label do the enforcement work. Add an agent-tier-c label from a follow-up action, and set branch protection so that label requires an extra approval. The mechanical parts live in CI, while the judgement stays with the gatekeeper.
Step 5: Make the worker resumable, because your host is temporary
The stage that runs after classification needs a model endpoint and somewhere to execute, and teams often start that on free infrastructure. MonkeyCode is an open-source project that advertises free model access and a free server option, which makes it a reasonable place to run this worker once before you ask for budget. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free-tier terms and token allowances change, so read the project page for the current numbers instead of trusting a screenshot in a blog post.
Shared free capacity has one design consequence: assume your process will be killed mid-run. Write a small state file keyed by pull request and head SHA, re-run idempotently, and keep batches tiny so a restart costs seconds rather than minutes. Treat a rate-limit response as a stop signal, not as an invitation to retry in a tighter loop.
# resume.py - keep the triage worker idempotent on a host that can disappear
import json
import os
import pathlib
STATE = pathlib.Path(".triage/state.json")
def load(pr_id: int, head_sha: str) -> dict:
fresh = {"pr": pr_id, "head_sha": head_sha, "stage": "classify", "attempts": 0}
if not STATE.exists():
return fresh
data = json.loads(STATE.read_text(encoding="utf-8"))
if data.get("pr") == pr_id and data.get("head_sha") == head_sha:
return data # same revision: continue where the previous run stopped
return fresh # new revision: never reuse a verdict from old code
def save(data: dict) -> None:
STATE.parent.mkdir(parents=True, exist_ok=True)
tmp = STATE.with_suffix(".tmp")
tmp.write_text(json.dumps(data, indent=2), encoding="utf-8")
os.replace(tmp, STATE) # atomic rename on POSIX
The head_sha comparison is the important line, because a stale tier is worse than no tier. When an agent pushes a new commit that adds a migration, the old verdict must evaporate immediately and the diff must be classified again.
Step 6: Paste the one-page run into your wiki
Copy the block below into a wiki page named agent-pr-triage, and link it from the pull request template. Keep it short enough that a gatekeeper can follow it while drinking coffee.
## Agent PR triage run (one page)
Roles
- Author of record: human who requested the change; owns description + rollback note
- Gatekeeper (rotates weekly): confirms tier, assigns reviewer of record
- Reviewer of record: domain owner; reads first, holds the merge button
Handoff
1. Confirm the commit has `Author-Agent:` and the PR has `agent-authored`
2. Run: python3 tools/triage.py --base origin/main --head HEAD --json-out triage.json
3. Read the `reasons` array, not only the tier; adjust globs if it is obviously wrong
4. Tier A: one reviewer of record, normal SLA
5. Tier B: reviewer of record reads before any agent revision pass
6. Tier C: label `agent-tier-c`, ping the gatekeeper, human-first review required
7. Record the tier in the PR description; do not edit or delete the recorded tier later
Stop conditions
- No author of record named
- Diff touches migrations, auth, billing, or workflows and no domain owner is online
- Worker returns a rate-limit response: stop, do not loop
Review the page monthly, and delete rules nobody followed for two consecutive months. A playbook that grows without pruning becomes a document people skim and ignore.
Limitations and who should skip this
Path globs are heuristics, and a determined agent can rename a migration file into src/ to dodge them. Add a cheap second check on the final file list after any rename, and accept that adversarial cases still need the gatekeeper's eyes. The script also says nothing about correctness, test quality, or whether the description matches the diff.
Skip this approach if you review fewer than roughly five pull requests a week, because the role overhead will exceed the queue it manages. Skip it too if you are a solo maintainer, or if your review policy is set by a compliance process you cannot change. In regulated codebases, tier C should mean human-first review and human merge, never a label that unlocks auto-merge.
What to measure, without inventing numbers
Track four things weekly: the count of tier C diffs, the time from open to first human review, the number of tier verdicts the gatekeeper overrode, and the number of agent revisions requested per pull request. Overrides are the most useful signal, because a rising count means your globs no longer match how your repository is actually organized. Publish the four numbers in the wiki run so the rotation inherits context instead of opinions.
If you want to try the worker stage without a budget line, the project's free model access and free server option are a low-risk place to run step 5 once. Bring your own state.json, keep the batch small, and let the numbers above decide whether it earns a permanent place in your pipeline.
Top comments (0)