CodeJury is a terminal-first multi-agent coding pipeline where every change is reviewed by a panel of independent, differently-modelled judges
Every agentic coding tool eventually needs a reviewer, and almost all of them use the same design: ask one model whether the diff is good.
That fails in a predictable shape. A single judge is confident, fast, and blind in exactly the places the author was blind — same training distribution, same intuition about what "looks right", same willingness to accept a plausible diff that never actually wires the feature through. It's a single point of judgement on the one step whose whole job is catching what the previous step got wrong.
So CodeJury sends every change to a panel of independent, differently-modelled judges, plus a foreperson who synthesizes one verdict.
The evidence
This isn't a hunch. SE-Jury (Zhou et al., 2025) measured how well automatic judges agree with human experts on whether generated code is correct:
| Metric | CoNaLa | Card2Code | APR-Assess (repair) | Summary | Avg |
|---|---|---|---|---|---|
| Best traditional metric (METEOR) | 41.2 | 72.5 | 45.0 | 19.9 | 44.7 |
| Vanilla LLM (one judge) | 45.6 | 70.4 | 43.5 | 33.9 | 48.4 |
| ICE-Score (one judge, SOTA prompt) | 55.2 | 66.5 | 43.5 | 33.1 | 49.6 |
| SE-Jury (ensemble) | 63.5 | 80.3 | 76.2 | 37.3 | 64.3 |
The best single judge in the literature scores 49.6. The same model, arranged as a panel, scores 64.3 — and the gap is biggest on program repair (+75.2%), which is the closest task in the paper to what a coding agent actually ships.
The ablation kills the obvious objection ("maybe one judge is just the good one"): every individual strategy averages 48–58, all below the panel's 64.3. And single judges are wildly unstable — the best one overall is the second worst on summarization. Whichever judge you pick, you're picking its blind spot too.
Those numbers are SE-Jury's, measured on SE-Jury — evidence for the architecture, not a benchmark of my repo. Full tables and the human-agreement results are in the README.
Six stages, any model on each
CodeJury isn't one agent with a reviewer bolted on — it's six stages, and you pick the provider and model for each, live, from the terminal:
| Stage | What it does |
|---|---|
| Knowledge | Builds the repo's code graph + structured views |
| PM | Socratic clarify-loop on your request, then drafts tickets |
| Planner | Decides how the change is made; every symbol verified against the graph |
| Dev | Implements on an isolated agent/<key> branch |
| QA | Runs tests, reviews independently of Dev |
| Review | The jury — N judges in parallel + foreperson |
/model dev anthropic claude-sonnet-5
/model planner gemini gemini-3.5-flash-lite
/jury # seat, unseat, re-model, re-brief judges
Claude plans, Codex writes, GPT reviews. The reviewer running a different family than the author is the same decorrelation argument, one layer up.
You may not need an API key at all — stages run headless on coding CLIs you're already logged into (Claude Code, Codex, Cursor, Aider, Gemini CLI), on your existing subscription. Or bring any key; defaults target Groq's free tier, so the whole thing can run for $0.
A jury costs tokens. The pipeline earns them back.
A cold agent pays the localization tax on every task — grep, open, guess, backtrack, re-derive the architecture from scratch. CodeJury pays it once, into a persistent code graph (158 languages) with local semantic search fused with BM25.
Head-to-head vs. plain claude -p, same plain-English request, on Textualize/textual (~82.5k LOC) and rich (~35.6k):
| Repo | Task | CodeJury | Cold claude -p
|
Δ |
|---|---|---|---|---|
| rich | cross-cutting bug | $0.456 | $0.828 | −45% |
| textual | feature | $0.380 | $0.590 | −36% |
| textual | extreme cross-cutting bug | $1.705 | $6.830 | −75% |
| textual | greppable bug | $0.374 | $0.401 | −7% |
On the extreme case the baseline spent $6.83 over 207 turns and 14.3M tokens hunting one cache-decorator bug. Repo size isn't the driver — localizability is.
It also loses sometimes: on a very cheap task the fixed gate floor costs more than a cold run saves, and on the hardest bug the cheap win shipped a narrower fix. Both are documented in the full report.
Try it
Python 3.11+ and git. That's the hard list.
git clone https://github.com/krishagarwal314/CodeJury
cd codejury && ./run.sh
Then, start to finish:
/kb add https://github.com/pallets/click # index a repo
add a --dry-run flag to the CLI runner # plain English; PM agent scopes it
/approve all # the human gate
/run # Planner → Dev → QA → jury → PR
/review # the verdict, every juror's findings
Demo mode is on by default, so the PR stage is a dry run until you opt in.
⭐ github.com/krishagarwal314/CodeJury — MIT, contributions welcome.
I'd genuinely like to hear where the jury gets it wrong. Run it on a repo, and if the panel misses something obvious — or blocks something it shouldn't — open an issue with the diff. That's the failure mode I most want data on.
Top comments (0)