Researchers rewrote real papers so that the science stayed identical and only the rhetoric changed, then fed them to five AI reviewers. The scores moved by up to 0.93 points on a standard overall-assessment scale, and the shift was largest exactly where it does the most damage: near the line between acceptance and rejection. The strongest single lever was not writing quality or technical polish, but how the evidence was framed.
Key facts
- The anchor number: changing only how the evidence was framed moved the probability of a weak-accept recommendation by 13 percentage points on average.
- The mechanism: six rhetorical dimensions were varied while the scientific content stayed fixed; five different language models acted as reviewers.
- When: posted August 10, 2026, as "How Can Rhetoric Reward-Hack AI Reviewers?".
- Primary source: arXiv:2608.08975, with the rewriting and evaluation code at Dissecting_AI_Reviews.
Academic peer review is drowning. Submission volumes at the big machine learning conferences have grown faster than the pool of qualified reviewers, and the obvious relief valve is to point a language model at the problem. Several venues now permit it in some form. This paper asks the question that should have come first: if a model is doing the reviewing, what is it actually responding to?
The experimental design is the reason to trust the result. Rather than comparing good papers to bad ones, which confounds everything, the authors take the same paper and produce variants that differ only in presentation. The rewriting system can restructure the entire narrative, including captions, transitions and the way results are presented, while keeping the structural anchors and the substantive claims intact. Six dimensions were manipulated: how strongly novelty is claimed, how broadly the scope is described, how quantitative evidence is framed, how contributions are structured, the technical register and level of formalism, and the lexical and syntactic complexity of the prose.
Then five language models review the variants, under both a standard protocol and a deliberately strict one.
The findings are specific enough to act on. Evidence framing is the biggest lever: presenting the same numbers in a positive frame raises overall assessment by up to 0.93 points, and shifts weak-accept probability by roughly 13 points. Novelty stance is close behind, with a negative framing dragging assessment down by up to 0.73 points. Scope framing forms a weaker second tier. Technical register and prose complexity, the things authors usually agonize over, matter least.
The distribution of the effect is what elevates this from curiosity to problem. The score movement is not uniform. Lower-scoring papers tend to rise, higher-scoring papers tend to fall, and the contrasts are clearest through the middle of the range. That middle is where accept-reject decisions actually get made. A bias that mostly perturbs papers already destined for acceptance or rejection would be harmless. A bias concentrated at the decision boundary is not.
An intuition for what is happening: imagine a hiring manager who reads a thousand resumes and, exhausted, starts responding to how confidently accomplishments are stated rather than what the accomplishments are. The candidate who writes "led a redesign that cut load times substantially" beats the candidate who writes "contributed to a redesign; load times may have improved," even when they did the same work. That is not stupidity. It is a reasonable heuristic applied where it does not belong. The AI reviewers here are doing the machine version of the same thing, which is a textbook case of reward hacking: the measurable proxy, confident framing, has come apart from the thing it was supposed to measure.
The authors also tested whether more effort defeats the problem. It does not. More elaborate rewriting does not reliably produce bigger gains, guidance to the reviewer does not consistently beat an unguided second pass, and repeated rewriting shows diminishing returns. Their summary of the division of labour is worth keeping: the rewriter controls how far apart the rhetorical variants are, while the reviewer controls the sign and magnitude of the resulting score shift. Strict review instructions lowered every score by about 1.36 points on average, but did not change the pattern. The vulnerability survives a more conservative setting.
Venues are not aligned on any of this. The ICLR 2026 reviewer guide permits language models as general-purpose writing assistance, requires disclosure, and keeps the human reviewer fully responsible for the content. CVPR 2026 and ECCV 2026 go the other way and prohibit language models or chatbots in any part of the reviewing process, with CVPR carving out only background research and short clarity checks. This paper is ammunition for the second camp.
The honest caveat is scale. Roughly one point on a ten-point scale is real but bounded, and the paper does not include a human-reviewer rerun on the manipulated rewrites. Its human comparison is descriptive only, contrasting the AI reviewers' scores on the unmodified originals against those papers' mean human scores from official conference reviews. So the defensible claim is not that AI reviewers are gullible. It is narrower and more useful: AI reviewers carry a measurable, structured, exploitable sensitivity to framing, and it concentrates precisely at the decision boundary.
Ground Truth has tracked the surrounding fight: ICML found hidden prompt injections in submitted papers, NeurIPS banned the practice outright, and NeurIPS is running a randomized experiment on AI-assisted review. Background on the underlying method: what it means for AI to grade AI.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)