Add a Second CI Lane for Agent-Written Pull Requests

A green unit job is not a merge decision when an agent wrote the diff. You need a second CI lane. That lane inventories the change, rejects silent assumption smells, and only then opens a path to merge.

Human patches usually fail on intent. Agent patches fail on invented APIs, skipped tests, and env vars nobody set. Your default pipeline does not see that difference. It sees exit code 0.

Why one lane is the wrong default

Most CI is a boolean stack. Lint, unit, maybe a coverage floor. That stack was built for people who get paged when they skip a test.

An agent does not get paged. It will mark a test skip to keep the job green. It will catch Exception and return None. It will assume REDIS_URL exists because the README mentioned Redis once.

You cannot fix that with a longer prompt. You fix it in the merge path. The prompt is gone after the PR opens. The workflow file is not.

Treat the agent like an untrusted contributor with a fast keyboard. You already do this for third-party patches. Do it for generated ones too.

What you are going to build

This is a template, not a production SLA. Copy it. Then tighten the denylist for your repo.

You will add six pieces:

1. A PR label that marks agent-written work.
2. A change inventory JSON emitted from the diff.
3. An assumption gate that fails the job on specific smells.
4. A job-scoped flake budget so retries do not launder failures.
5. An optional structured review step you can run on a free model endpoint.
6. A required-check map that is the actual green-to-merge path.

The extra lane stays useful if you never call a model. Keep the model step optional on purpose. Required checks should survive an outage of any vendor.

Step 1: Label the pull request

Do not detect "AI style." Detection is noisy and easy to game. You want an explicit contract the author cannot shrug off.

Require a label such as agent-pr on any PR where a model wrote more than a trivial hunk. Team rule: if the agent touched the files, the author applies the label before requesting review. Branch prefixes can backstop the rule when people forget.

gh pr edit "$PR_NUMBER" --add-label "agent-pr"
gh pr view "$PR_NUMBER" --json labels --jq '.labels[].name'

Enforce the split in CI. Human PRs keep the old lane. Agent PRs gain checks.

# .github/workflows/agent-lane.yml — template
name: agent-lane
on:
  pull_request:
    types: [opened, synchronize, reopened, labeled, unlabeled]

jobs:
  classify:
    runs-on: ubuntu-latest
    outputs:
      agent: ${{ steps.flag.outputs.agent }}
    steps:
      - id: flag
        env:
          LABELS: ${{ join(github.event.pull_request.labels.*.name, ',') }}
          HEAD_REF: ${{ github.head_ref }}
        run: |
          if [[ "$LABELS" == *agent-pr* || "$HEAD_REF" == agent/* ]]; then
            echo "agent=true" >> "$GITHUB_OUTPUT"
          else
            echo "agent=false" >> "$GITHUB_OUTPUT"
          fi

If the label is missing and the branch starts with agent/, fail closed on the agent lane. Do not guess from commit message tone.

Step 2: Emit a change inventory

You need a file CI can hash, not a chat summary. Run this on every agent PR. Save the artifact. The next jobs should read it, not re-parse git diff with slightly different flags.

#!/usr/bin/env bash
# scripts/inventory.sh — template
set -euo pipefail
BASE="${BASE_SHA:-origin/main}"
mkdir -p artifacts
git diff --name-status "$BASE"...HEAD > artifacts/name-status.txt
git diff --numstat "$BASE"...HEAD > artifacts/numstat.txt

python3 - <<'PY'
import json, pathlib
rows = []
for line in pathlib.Path("artifacts/numstat.txt").read_text().splitlines():
    added, deleted, path = line.split("\t", 2)
    rows.append({"path": path, "added": added, "deleted": deleted})
src = [r for r in rows if r["path"].startswith("src/") or r["path"].startswith("lib/")]
tests = [r for r in rows if "test" in r["path"].split("/")[0] or "/tests/" in f"/{r['path']}/"]
payload = {
    "file_count": len(rows),
    "src_files": [r["path"] for r in src],
    "test_files": [r["path"] for r in tests],
    "src_without_tests": bool(src) and not tests,
    "lockfile_only": all(
        r["path"].endswith(t) for r in rows for t in ()
    ),
}
pathlib.Path("artifacts/inventory.json").write_text(json.dumps(payload, indent=2) + "\n")
print(json.dumps(payload, indent=2))
PY

src_without_tests: true is not an automatic fail for a docs PR. It is an automatic fail for an agent-pr that touched application code. Put that rule in the gate. Do not leave it as a reviewer comment that the agent never reads.

Run it locally before you wait on GitHub:

export BASE_SHA=origin/main
bash scripts/inventory.sh
cat artifacts/inventory.json

Step 3: Run the assumption gate

This script is a denylist. It is not an LLM. It greps the incoming diff and exits non-zero when the agent tried to stay green by hiding work.

#!/usr/bin/env python3
# scripts/assumption_gate.py — template, fail agent PRs on merge-blocking smells
from __future__ import annotations

import json
import os
import pathlib
import re
import subprocess
import sys

BASE = os.environ.get("BASE_SHA", "origin/main")
DIFF = subprocess.check_output(["git", "diff", "-U0", f"{BASE}...HEAD"], text=True)
ADDED = "\n".join(
    line[1:] for line in DIFF.splitlines()
    if line.startswith("+") and not line.startswith("+++")
)
PATHS = subprocess.check_output(["git", "diff", "--name-only", f"{BASE}...HEAD"], text=True)

RULES = [
    (
        "skip_without_ticket",
        re.compile(r"pytest\.mark\.skip(if)?\("),
        re.compile(r"(TICKET|JIRA|GH)-\d+"),
        "skip/skipif added without a ticket id",
    ),
    (
        "bare_except",
        re.compile(r"except Exception:\s*(pass|return None)"),
        None,
        "bare Exception handler swallows failures",
    ),
    (
        "xfail_not_strict",
        re.compile(r"pytest\.mark\.xfail\([^)]*strict\s*=\s*False"),
        None,
        "xfail with strict=False launders failures",
    ),
    (
        "hidden_todo",
        re.compile(r"TODO|FIXME|HACK"),
        re.compile(r"(owner|@)\s*\w+"),
        "TODO/FIXME added without an owner",
    ),
]

failures = []
for key, pat, allow, msg in RULES:
    if not pat.search(ADDED):
        continue
    if allow and allow.search(ADDED):
        continue
    failures.append(f"{key}: {msg}")

if re.search(r"(https?://)?localhost:\d+", ADDED) and not re.search(
    r"^(tests/|test_|conftest)", PATHS, re.M
):
    failures.append("hardcoded_localhost: localhost baked into non-test code")

inv_path = os.environ.get("INVENTORY_JSON")
if inv_path:
    data = json.loads(pathlib.Path(inv_path).read_text())
    if data.get("src_without_tests"):
        failures.append("inventory: application code changed and tests did not")

if failures:
    print("assumption gate failed:")
    for item in failures:
        print(f" - {item}")
    sys.exit(1)
print("assumption gate passed")

Wire it as a required job on the agent lane only:

  assumption-gate:
    needs: classify
    if: needs.classify.outputs.agent == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Inventory
        env:
          BASE_SHA: origin/${{ github.base_ref }}
        run: bash scripts/inventory.sh
      - name: Assumption gate
        env:
          BASE_SHA: origin/${{ github.base_ref }}
          INVENTORY_JSON: artifacts/inventory.json
        run: python3 scripts/assumption_gate.py
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: inventory
          path: artifacts/

Tune the regexes. A Rails app will want different smells than a Go service. The point is not these exact patterns. The point is that the agent cannot merge until the denylist is empty.

Step 4: Give flakes a budget, not a retry hammer

Flakes still exist. Agent PRs make them worse, because the model will "fix" a flake by skipping it. Retries without a cap do the same thing in slower motion.

Keep retry policy on the job. Cap it. Pair it with the gate so a new skip is still a hard fail.

  unit:
    needs: classify
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Test
        run: pytest -q --maxfail=1 --junitxml=artifacts/junit.xml
      - name: Re-run once on failure
        if: failure()
        run: |
          echo "retry_budget=1" >> artifacts/retry.txt
          pytest -q --maxfail=1 --junitxml=artifacts/junit.retry.xml
      - name: Block new skips on agent PRs
        if: needs.classify.outputs.agent == 'true'
        env:
          BASE_SHA: origin/${{ github.base_ref }}
          INVENTORY_JSON: artifacts/inventory.json
        run: python3 scripts/assumption_gate.py

If the second run passes and the first failed, keep the logs. Do not delete the first JUnit file. You want to see whether the agent "fixed" a flake or whether the suite blinked. A merge path that cannot tell those apart will rot.

Step 5: Optional structured review, not a chat overlay

Prose reviews do not belong in a required check. A required check needs a schema. If the model cannot fill the schema, it does not get a vote.

If you already have somewhere to run a small review job without standing up extra hardware, use it here. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free-tier model access and a free server option, which can host this review step as a side job so the second lane does not wait on your laptop. Do not send secrets. Do not paste .env. Send only the files the inventory listed.

The model must answer JSON. Nothing else.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": ["verdict", "blocking_reasons", "assumptions"],
  "properties": {
    "verdict": {"enum": ["allow", "deny"]},
    "blocking_reasons": {
      "type": "array",
      "minItems": 1,
      "items": {
        "enum": [
          "missing_tests",
          "invented_api",
          "hidden_skip",
          "secret_risk",
          "unpinned_dep",
          "none"
        ]
      }
    },
    "assumptions": {
      "type": "array",
      "items": {"type": "string", "maxLength": 160},
      "maxItems": 8
    }
  }
}

Validate the payload in CI. Fail on schema errors first. Fail on deny second.

#!/usr/bin/env python3
# scripts/validate_verdict.py — template
import argparse, json, sys
from jsonschema import Draft202012Validator

p = argparse.ArgumentParser()
p.add_argument("--schema", required=True)
p.add_argument("--input", required=True)
args = p.parse_args()

schema = json.load(open(args.schema))
raw = json.load(open(args.input))
payload = raw if "verdict" in raw else raw.get("json") or raw.get("content")
if isinstance(payload, str):
    payload = json.loads(payload)

Draft202012Validator(schema).validate(payload)
reasons = payload["blocking_reasons"]
if payload["verdict"] == "deny" and reasons == ["none"]:
    sys.exit("deny without a blocking reason")
if payload["verdict"] == "allow" and reasons != ["none"]:
    sys.exit("allow with blocking reasons")
print("review schema ok:", payload["verdict"])
if payload["verdict"] == "deny":
    sys.exit(1)

Caller sketch. Point MODEL_ENDPOINT at an endpoint you control. This is a template, not a live integration.

# template — structured review against a schema
jq -n --rawfile diff artifacts/named.diff --argfile inv artifacts/inventory.json '{
  task: "Review this diff. Return JSON only. No prose.",
  inventory: $inv,
  diff: $diff
}' > artifacts/review_request.json

curl -sS "$MODEL_ENDPOINT" \
  -H "content-type: application/json" \
  --data @artifacts/review_request.json \
  > artifacts/review_raw.json

python3 scripts/validate_verdict.py \
  --schema schemas/review_verdict.json \
  --input artifacts/review_raw.json

Keep this check advisory until the schema fail rate is boring. Then promote it to required on agent-pr only. If you make it required too early, you will stall merges on parse errors instead of on bad patches.

Step 6: Encode the green-to-merge path

Write the merge rule as a table your branch protection can implement. Do not keep it in a wiki that nobody updates.

Check	Human PR	Agent PR (agent-pr)	Blocks merge?
Unit tests	required	required	yes
Assumption gate	off	required	yes
Inventory src_without_tests	warn	fail	yes, agent only
Structured review schema	off	advisory, then required	only after you promote it
Retry budget exceeded	fail	fail	yes
New skip without ticket	warn	fail	yes, agent only

Map that table onto branch protection. The merge button should see unit for everyone and assumption-gate for agent work. If your host cannot do per-label required checks, make assumption-gate a no-op success on human PRs so the same context name always exists.

  assumption-gate:
    needs: classify
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Skip on human PRs
        if: needs.classify.outputs.agent != 'true'
        run: echo "human lane; assumption gate not required"
      - name: Run gate
        if: needs.classify.outputs.agent == 'true'
        env:
          BASE_SHA: origin/${{ github.base_ref }}
          INVENTORY_JSON: artifacts/inventory.json
        run: |
          bash scripts/inventory.sh
          python3 scripts/assumption_gate.py

The merge button now has two lanes. Green on the human lane is not green on the agent lane. That is the outcome you wanted.

Limitations

The denylist will miss a wrong algorithm that still has tests. Schema review will miss a plausible lie. A free-tier endpoint can throttle. If you made the model job required too early, you will stall merges for reasons that are not about the code.

This workflow also does not replace CODEOWNERS. If the agent touched auth/ or billing, you still want a human who owns that path. A JSON verdict is not a substitute for that review.

Do not send proprietary diffs to an endpoint you do not control. If your policy forbids that, skip Step 5 entirely. Steps 1–4 and 6 still work, and they are the parts that catch skipped tests.

Heuristic gates drift. Re-open the denylist when your stack changes. A React repo and a Python service should not share the same smells forever.

Who should not use this

Skip the second lane if your repo is a personal sandbox with no merge button. Skip the model step if you cannot keep secrets out of the prompt. Skip the whole design if agents only draft locally and a human always retypes the patch.

If your tests are already non-deterministic, fix that first. An assumption gate on top of a flake storm just produces more YAML. Get a stable unit job. Then split the lanes.

Small teams with two reviewers on every PR may not need the schema review. They may still want the skip denylist. Start there.

Close the loop

Start with the label and the denylist. Promote the schema review only when it fails closed on garbage JSON, not on taste.

The agent can still write the patch. You decide whether it merges.