What 1,135 agent-written pull requests taught me about reviewing AI code

John Problems — Fri, 04 Sep 2026 08:25:20 +0000

For the last five months I've run an autonomous software team inside a GitHub
repo. 26 agent roles: a Discussion becomes a spec, a spec becomes a pull
request, and nothing merges until code review, security review and acceptance
all pass. It has merged 1,135 PRs.

This isn't a post about the tool. It's about the five things that turned out to
be true, most of which I got wrong first. They apply to anyone putting agents
anywhere near a codebase, whether you use my thing or not.

1. Your reviewer and your author share blind spots

A model reviewing another model's code catches slips, conventions, and missing
cases that were visible on the surface. It is much weaker on the defects that
actually hurt — a race, an off-by-one under load, a guard on the wrong side of
a branch. Those read fine. They read fine to the author and to the reviewer,
for the same reasons.

If your pipeline is generate → review → merge and both steps are the same
kind of mind, you have one opinion sampled twice, not two opinions.

2. The evidence can be broken, and then good reviews go bad

This is the one that changed how I think.

A cleanup script's --dry-run reported 116 removals. The real run, on an
identical population seconds later, removed zero. 190 directories before, 190
after.

The skip that protects git-tracked worktrees is gated on the run not being a
dry run:

if [[ "$dry_run" == false && "$enable_git_tracked_removal" == false ]]; then
  skip_git_tracked
fi
# a dry run never evaluates this, falls through, and reports "would remove"

Nobody lost data — the divergence runs the safe way. What broke was subtler.
Two code reviews had already cited that dry-run output as their
verification. A reviewer asked "did you check this against the live
population?" and got a transcript from a tool that could not tell the truth
about the live population.

The reviews weren't lazy. The evidence was lying.

The takeaway isn't "distrust reviewers." It's that a verdict is only as good as
the artifact it cites, and almost nobody records which mode produced the
artifact. dry-run output and real output should not be the same shape of
evidence, and if your agents attach proof to their claims, that proof needs
provenance.

3. Make roles data, not code

An agent role in this system is a Markdown behaviour spec plus a JSON policy
record — timeouts, retries, token ceilings, concurrency caps. Adding a role is
writing a file. Changing what a reviewer cares about is editing prose.

The moment roles became data, three things got easier: diffing a behaviour
change, capping concurrency per role instead of globally, and testing
orchestration without spawning anything.

4. Structure the output or you'll parse prose forever

Every agent ends its final message with a structured envelope the orchestrator
parses for routing. Not "read the last paragraph and infer whether it passed."

This single decision removed most of the flakiness in the system. If you are
regexing model prose to decide what happens next, that is where your
intermittent failures live.

Related: add an unknowns[] field. An agent that can say "I could not tell
whether this path is reachable" gets routed to a research step for a few
thousand tokens, instead of producing a confident PR that burns a full review
cycle discovering the same thing.

5. Automation should refuse to guess

Provisioning halts after its first phase. It installs labels, hooks, state and
dependencies — then stops and asks for a backlog.

I fought this for a while because a halt feels like an unfinished feature. It
isn't. The next step needs to know what you actually want built and what's
explicitly out of scope, and a system that guessed would produce a hundred
confident pull requests solving the wrong problem. Fast, thorough, and wrong is
worse than stopping.

If you're building agent tooling, find the place where yours is guessing at
intent and make it stop there instead.

The uncomfortable one

Most of those 1,135 PRs are the system's own tooling. Spawn queues, review
gates, the circuit breaker, the sandbox guard. For a long stretch its only
product was itself.

I've come round to thinking that's the interesting part rather than the
embarrassing one — a team that can't improve its own process needs a human for
every improvement, forever — but you should decide that for yourself, which is
why the queue is public.

If you want to look

Everything above is from a real system you can inspect without installing
anything:

The live work queue, pulled from the repo as the page loads, including the bugs it filed against itself: https://fulcrumaxe.dev/discussions.html
The codebase as a browsable graph, 4,811 nodes: https://fulcrumaxe.dev/understand.html
Three rival architecture designs I haven't chosen between, one published with an unsolved cost problem in it: https://fulcrumaxe.dev/proposals.html

It's AGPL-3.0 and self-hosted — your machine, your repo, your key, no service
in the middle. Install is three lines in Claude Code:

/plugin marketplace add fulcrumaxe/fulcrumaxe
/plugin install fulcrumaxe@fulcrumaxe
/coldstart --path /path/to/your/repo --name your-project

Repo: https://github.com/fulcrumaxe/fulcrumaxe

Happy to answer anything in the comments — including what it costs to run,
which is the question I'd ask first.

DEV Community: John Problems