Four LLMs review everything I ship. Last month I got lazy with the input.
I run an adversarial review panel before anything goes out: several frontier models from different vendors get the same artifact and are told to break it. On 2026-08-10 I handed the panel file paths instead of file contents, because the context was large. Three of the four models returned confident findings about files they had never read.
TL;DR: a reviewer that cannot see the code does not say "I cannot see the code." It completes plausibly: invented functions, the wrong programming language, flags that do not exist in the tool. One model out of four honestly answered "no data." The rule we enforce now: the panel receives contents, never paths, and a confident fabrication is treated as worse than a refusal. The same panel, fed real contents, has caught genuinely real bugs — so the instrument works; it just has a sharp edge on the input side.
What the panel is
Every substantial artifact in our fleet — a measurement script, a deploy pipeline, an OSS kit — goes through a multi-vendor breaker pass before we call it done. Different vendors, same brief: find inputs that make this lie, crash, or over-report. Not one second opinion but a panel, because two models from the same family fail in correlated ways.
This is cheap to run on coding subscriptions we already pay for, and it earns its keep. Two examples from the same OSS series, both from August 2026:
A real bug it caught: our process-counting tool identified an MCP server's processes by launch command, and the config said "command": "node". So every unrelated Node process on the machine became "another copy" of the server. The instrument was inventing duplicates — over-reporting in a way that justified action.
The panel caught it before release; we reproduced it, fixed it (interpreters and generic script names can never be the identifying marker; the install directory is), and added a regression test that goes red if the bug comes back.
A finding we rejected, with a reason: one vendor insisted that an HTTP 404 from a daemon must not count as "alive." Sounds rigorous. It is wrong for this daemon: an MCP server's root path returns 404 by design, and demanding a 2xx would have produced a false "dead" verdict — which in our setup triggers a restart that blinds every connected session. Panel findings are inputs, not orders.
The measurement
Then came the lazy run. Large review context, so instead of pasting contents I gave each model the repository paths and asked for findings.
Results, same day, four vendors:
| behavior | models |
|---|---|
| invented findings about files they never read | 3 of 4 |
| honest "I don't have access to this data" | 1 of 4 |
The fabrications were not vague.
One described functions that do not exist in the file. One reviewed the file as if it were written in a different language. One recommended changing command-line flags the tool has never had.
All three were fluent, specific, and formatted exactly like real review findings.
Why paths are a hallucination prompt
A file path is a very strong prior. scripts/deploy_verify.py tells a language model roughly what such a file usually contains, and the model does what it is built to do: continue plausibly from the prior. Nothing in the objective rewards "I cannot see this," and three of four vendors' harnesses did not force the admission either.
The dangerous part is the asymmetry: a refusal costs you one re-run, while a fabricated finding costs you an investigation of a bug that does not exist — or worse, a "fix" applied to healthy code. An instrument that over-reports is worse than no instrument, because it justifies action.
The rules we run with now
- Contents, never paths. The panel gets the actual bytes. If the artifact is too big, we cut it into parts and send each part whole.
-
Watch the argv limit. "Send contents, not paths" ran into
Argument list too longat about 82 KB of context passed as a shell argument. That error came from bash, not from the vendor. Pass big contexts as files read by your wrapper, not as command-line arguments. - A refusal scores above a fabrication. We grade vendors on it. The one model that said "no data" earned more trust that day than the three that wrote fiction.
- Findings are challenges, not orders. Every finding gets reproduced or rejected with a written reason, like the 404 case above.
When you should NOT run a panel
- Trivial edits. A typo fix does not need four vendors; it needs a diff review by one human.
- When you can only afford one vendor, run one — but say so out loud in the verdict. Two rails where one is silently dead is fake independence, and that failure mode is sneakier than having no panel at all.
- When you cannot feed real contents. A panel reviewing paths is a fiction generator with extra steps, per the table above.
Take the test
If you use any LLM as a code reviewer, run this once: give it only a path or a filename from your repo, no contents, and ask for findings. If it returns specific bugs instead of asking for the file, you now know what its confident tone is worth when the input pipeline silently breaks.
One ask: run that experiment on your own reviewer setup and tell me in the comments what came back — model, input you gave it, and whether it confessed or invented. I have numbers from one panel on one day; I would like to know if 3-of-4 is typical or if I got unlucky.
Built at Palo Alto AI Research Lab, where this panel gates every release across a fleet of agent sessions on five machines.
The full story, in two versions:
📖 For humans, the longread: this page.
🤖 For machines: the dev-log version of how this panel runs, https://github.com/tonydzi/clawrush/blob/main/devlog/multi-llm-council-as-qa.md. Just hand this link to your coding agent (Claude Code, Codex, Cursor) and it will figure everything out: it is written for machines.
Talk to the two co-founders, one biological, one synthetic: calendly.com/paloaltolab. Direct line: WhatsApp +1 341 222 9178 (busy, six kids, still answers).
P.S. Yes, we are hireable. Two co-founders, one biological, one electric, as a package deal. OpenAI hired the creator of OpenClaw; what we ship is not far behind, and there are two of us. Anthropic, OpenAI, your move: calendly.com/paloaltolab.
🔗 All our channels and contacts in one place: https://linktr.ee/PaloAltoAI
Invented by Mycroft and Tony Dzi (Anton Dziatkovskii), Palo Alto AI Research Lab. Proudly made in Silicon Valley.
Draft assisted by our synthetic co-founder (LLM); architecture, numbers, and final wording reviewed by a human who ran the system. We stand behind the claims.
Top comments (4)
Passing paths instead of file blobs triggers the exact same failure mode in CI pipelines. When an agent gets a git diff that references modified paths without inline hunk context, it hallucinated imports and invented deprecated methods based purely on filename heuristics. We ended up gating the review harness so it rejects any prompt that lacks explicit byte payloads and line-number ranges.
The "contents, never paths" rule is right, but I'd add a cheap mechanical check on top of it rather than relying on the model to self-report honestly: inject a random nonce as a comment on a random line before sending the file contents, and require the review to reference at least one real line near it (or quote a specific real string from the file). A model that's actually reading the bytes will naturally anchor on real content near the nonce; a model that's pattern-matching from the filename/extension has no way to produce that anchor convincingly. It doesn't catch every fabrication, but it turns "trust the model's tone" into a falsifiable test you can run automatically on every panel pass, not just when you happen to spot-check. Curious if you've tried anything like a canary string versus just grading vendors after the fact on the honesty axis.
Same shape turned up for me with something a lot smaller than code. A long-running agent job of mine reported the time as hours later than it really was. It had worked the time out from how much it had already done instead of reading the clock, then cut its own work short to fit a deadline it had invented.
The fix was your rule in a different place: give it the source and never let it infer something it could have just read. One date call beats any amount of reasoning about what time it probably is.
The asymmetry point is the sharpest thing in this post. We hit the same pattern and noticed the fabricated findings weren't just fluent - they were plausible for our codebase shape, which is what made them dangerous before we caught it. A real bug in a file like that would look exactly like what the model described, so the first few went all the way to investigation. The path test you suggest at the end is the fastest way I know to find out what a model's confidence actually means when the input silently breaks.