I Benchmarked 19 LLMs as ML Paper Reviewers. Size and Price Didn’t Predict the Score

Anya Huang — Mon, 12 Oct 2026 06:59:04 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Can an LLM review an ML experiment and catch the flaw, and stay quiet when there isn't one?

I've been working on models trained on synthetic data that I then test on real data, and I keep seeing how easy it is to report a number that only holds on the synthetic side. I wanted to know whether an LLM would catch that kind of slip when reviewing a writeup.

I wrote short experiment summaries, each with a conclusion and some evidence. The model must name one validity flaw from a fixed list or answer none. There are 7 flaw types:

synthetic data presented as real-world evidence
test-set leakage
small-sample overclaims
accuracy on imbalanced classes
degenerate confidence intervals
uncorrected multiple comparisons
best-of-N seed cherry-picking

The key design choice is matched pairs. Each flaw has a flawed version and a sound twin that share the same skeleton and differ only in the detail that matters. For example, "tested 30 subgroups, no correction" versus "pre-specified one subgroup." A model can't win by pattern-matching on wording, and the sound twins measure false alarms.

Every item runs twice: neutral, and with an added note that the authors consider the result conclusive. That gives 112 items per model (7 flaws × 2 variants × 4 reps × 2 conditions). Half the items are flawed and half are sound, so a model that answers none every time scores 0.50.

Models Tested

I registered 21 models from the models Kaggle offered, covering Anthropic, OpenAI, Google, Qwen, xAI, Z.ai and DeepSeek. I wanted several generations and sizes within a family (Haiku 4.5 vs 5.5, GPT-5.4 nano vs mini vs 5.5, Qwen3 Next vs 235B) as well as different vendors.

Two models are not in the results:

DeepSeek-R1 was still running when I ran out of time, so I excluded it rather than report a partial run.
Grok 4.5 shows 0.00.

That leaves 19 scored models.

Findings

Score is accuracy over 112 items (the share where the model named the right flaw, or correctly said none).

Rank	Model	Score
1	Qwen 3 235B A22B Instruct	1.00
2	GPT-5.4 mini	0.98
3	GPT-5.5	0.95
3	Claude Haiku 5.5	0.95
5	Gemini 3.1 Flash-Lite Preview	0.94
5	Gemini 3.1 Pro Preview	0.94
7	Claude Opus 5	0.93
8	Gemma 4 26B A4B	0.92
8	Gemini 3.5 Flash-Lite	0.92
8	Gemini 3.7 Flash	0.92
11	GLM-5	0.91
12	Claude Sonnet 5	0.88
13	Qwen 3 Next 80B Thinking	0.86
14	Grok 4.20 (Non-Reasoning)	0.78
14	gpt-oss-20b	0.78
16	GPT-5.4 nano	0.76
17	Qwen 3 Next 80B Instruct	0.70
18	Claude Haiku 4.5	0.68
19	Gemma 4 31B	0.55

1. Most models handle this, and the top is crowded. Eleven of 19 models scored 0.91 or higher. At 112 items, one item is about 0.9 points, and the 95% interval is roughly ±4 points near 0.95, so I can't separate models within that group. For the same reason I wouldn't read much into 0.94 vs 0.92. My own test run of Claude Haiku 5.5 scored 0.946, and on Kaggle it showed 0.96 and later 0.95, so run-to-run noise is already a point or two.

2. Price, size and release date don't predict the score. GPT-5.4 mini (0.98) beat GPT-5.5 (0.95). Claude Haiku 5.5 (0.95) beat Claude Sonnet 5 (0.88). But the pattern isn't "small wins": within OpenAI's 5.4 line, mini scored 0.98 and nano 0.76. Across generations, Haiku 4.5 scored 0.68 and Haiku 5.5 scored 0.95, a 27-point jump.

3. Qwen spans nearly the whole range. Qwen 3 235B Instruct scored 1.00 (112 of 112). Qwen 3 Next 80B Instruct scored 0.70, and its thinking variant scored 0.86. Each is a single run, so I'd call the thinking gain suggestive, but it is large.

4. The bottom is close to "say none every time." Gemma 4 31B scored 0.55, barely above the 0.50 you get from never flagging anything. I haven't looked at its answers, so I can't say whether it misses flaws, says none too often, or has trouble with the answer format.

5. A perfect score has a degenerate interval. Qwen 3 235B's 112/112 gives a normal-approximation confidence interval of [1.00, 1.00], which is exactly the "degenerate CI" flaw that one of my 7 item types is built around. A Wilson interval gives roughly 0.97 to 1.00. That is a fair reminder that 1.00 on 112 items is not proof of perfection.

What went wrong, and what I'd fix

Thinking tokens ate my output budget. My first full run, on Gemini 2.5 Flash, lost 24 of 112 items. I had capped output at 1,500 tokens, and Flash spent about 1,440 of them on reasoning, so the structured answer was cut off and couldn't be parsed. The failures weren't random: they were mostly sound items in the "confident" condition, which is the case the benchmark cares most about. Dropping them would have inflated the score. The fix was a larger cap with retries. I kept a guard that refuses to score a model when too many items fail.
My fix has a trade-off. Items that fail all retries are scored as wrong (invalid), not dropped. That keeps the item count fixed, but it means infrastructure failures count against the model. I believe that is what produced Grok 4.5's 0.00, but I did not confirm it.
Templated data. The items come from templates with a few variants per flaw, so strong models may be recognizing templates and not reviewing papers. 112 items is also small.

What I didn't get to

I built the benchmark to separate three things: missing real flaws, false alarms on sound twins, and being swayed by the "authors are confident" note. I ran out of time to break the scores down that way. So I can't say why the lower models score lower, or whether the confident note changes anything. The data to answer it is there, since every item is tagged by variant, condition and flaw type.

Next: that breakdown, an open-ended version without the label list, real retracted-paper methods sections, harder and subtler flaws so the top models don't bunch at the ceiling, and sound twins reviewed for ambiguity.

My Benchmark

https://www.kaggle.com/benchmarks/tasks/anyahuang/ml-review-flaw-detection

kagglechallenge #ai #machinelearning #devchallenge

DEV Community: Anya Huang