One LLM reviewing another LLM's code is a coin flip. Use a jury instead.

Krish — Thu, 30 Jul 2026 22:41:00 +0000

CodeJury is a terminal-first multi-agent coding pipeline where every change is reviewed by a panel of independent, differently-modelled judges

Every agentic coding tool eventually needs a reviewer, and almost all of them use the same design: ask one model whether the diff is good.

That fails in a predictable shape. A single judge is confident, fast, and blind in exactly the places the author was blind — same training distribution, same intuition about what "looks right", same willingness to accept a plausible diff that never actually wires the feature through. It's a single point of judgement on the one step whose whole job is catching what the previous step got wrong.

So CodeJury sends every change to a panel of independent, differently-modelled judges, plus a foreperson who synthesizes one verdict.

The evidence

This isn't a hunch. SE-Jury (Zhou et al., 2025) measured how well automatic judges agree with human experts on whether generated code is correct:

Metric	CoNaLa	Card2Code	APR-Assess (repair)	Summary	Avg
Best traditional metric (METEOR)	41.2	72.5	45.0	19.9	44.7
Vanilla LLM (one judge)	45.6	70.4	43.5	33.9	48.4
ICE-Score (one judge, SOTA prompt)	55.2	66.5	43.5	33.1	49.6
SE-Jury (ensemble)	63.5	80.3	76.2	37.3	64.3

The best single judge in the literature scores 49.6. The same model, arranged as a panel, scores 64.3 — and the gap is biggest on program repair (+75.2%), which is the closest task in the paper to what a coding agent actually ships.

The ablation kills the obvious objection ("maybe one judge is just the good one"): every individual strategy averages 48–58, all below the panel's 64.3. And single judges are wildly unstable — the best one overall is the second worst on summarization. Whichever judge you pick, you're picking its blind spot too.

Those numbers are SE-Jury's, measured on SE-Jury — evidence for the architecture, not a benchmark of my repo. Full tables and the human-agreement results are in the README.

Six stages, any model on each

CodeJury isn't one agent with a reviewer bolted on — it's six stages, and you pick the provider and model for each, live, from the terminal:

Stage	What it does
Knowledge	Builds the repo's code graph + structured views
PM	Socratic clarify-loop on your request, then drafts tickets
Planner	Decides how the change is made; every symbol verified against the graph
Dev	Implements on an isolated `agent/<key>` branch
QA	Runs tests, reviews independently of Dev
Review	The jury — N judges in parallel + foreperson

/model dev anthropic claude-sonnet-5
/model planner gemini gemini-3.5-flash-lite
/jury                                     # seat, unseat, re-model, re-brief judges

Claude plans, Codex writes, GPT reviews. The reviewer running a different family than the author is the same decorrelation argument, one layer up.

You may not need an API key at all — stages run headless on coding CLIs you're already logged into (Claude Code, Codex, Cursor, Aider, Gemini CLI), on your existing subscription. Or bring any key; defaults target Groq's free tier, so the whole thing can run for $0.

A jury costs tokens. The pipeline earns them back.

A cold agent pays the localization tax on every task — grep, open, guess, backtrack, re-derive the architecture from scratch. CodeJury pays it once, into a persistent code graph (158 languages) with local semantic search fused with BM25.

Head-to-head vs. plain claude -p, same plain-English request, on Textualize/textual (~82.5k LOC) and rich (~35.6k):

Repo	Task	CodeJury	Cold `claude -p`	Δ
rich	cross-cutting bug	$0.456	$0.828	−45%
textual	feature	$0.380	$0.590	−36%
textual	extreme cross-cutting bug	$1.705	$6.830	−75%
textual	greppable bug	$0.374	$0.401	−7%

On the extreme case the baseline spent $6.83 over 207 turns and 14.3M tokens hunting one cache-decorator bug. Repo size isn't the driver — localizability is.

It also loses sometimes: on a very cheap task the fixed gate floor costs more than a cold run saves, and on the hardest bug the cheap win shipped a narrower fix. Both are documented in the full report.

Try it

Python 3.11+ and git. That's the hard list.

git clone https://github.com/krishagarwal314/CodeJury
cd codejury && ./run.sh

Then, start to finish:

/kb add https://github.com/pallets/click     # index a repo
add a --dry-run flag to the CLI runner       # plain English; PM agent scopes it
/approve all                                 # the human gate
/run                                         # Planner → Dev → QA → jury → PR
/review                                      # the verdict, every juror's findings

Demo mode is on by default, so the PR stage is a dry run until you opt in.

⭐ github.com/krishagarwal314/CodeJury — MIT, contributions welcome.

I'd genuinely like to hear where the jury gets it wrong. Run it on a repo, and if the panel misses something obvious — or blocks something it shouldn't — open an issue with the diff. That's the failure mode I most want data on.

DEV Community: Krish

One LLM reviewing another LLM's code is a coin flip. Use a jury instead.

The evidence

Six stages, any model on each

A jury costs tokens. The pipeline earns them back.

Try it