LLM self-review vs. a blind re-run on 160 gold-labeled name-matching pairs — no measurable difference at ceiling

If you ship anything an LLM produced and calm your nerves by asking the same model to "please double-check it," this is a small, pre-registered measurement of whether that actually buys you anything.

Short version: on this task it bought nothing measurable — but mostly because the task turned out to be too easy, which is a lesson in itself about evaluation design.

TL;DR

Setup: gpt-5.4-mini, 160 gold-labeled "are these two records the same person?" pairs, pre-registered before the run.

1. The first pass was near-ceiling: 159/160 overall, 120/120 on the confirmation set. So there was almost nothing left to fix — I ended up measuring "does re-review preserve correct answers," not "does it fix errors."
2. Re-review (shown its own draft) tied a blind re-run. Across all 160 pairs both scored 156/160. On the pre-registered confirmation set, the single discordant pair gives exact McNemar p = 1.0 — no power to tell them apart, not proof of equivalence.
3. Reversals of correct answers happened in both conditions, equally (3 and 3). Not specific to self-review; consistent with plain run-to-run variation (single runs, so I can't causally isolate the extra pass itself).
4. Honest caveat: with the first pass this high, I could not test the thing I actually wanted to test. Treat this as the ceiling data point of a series, not a verdict on self-correction.

Reproducibility scope

This is one model, one prompt, reasoning_effort=low, single runs in a fixed order, on an 80/80 balanced diagnostic benchmark (60/60 confirmation + 20/20 hard). Nothing here generalizes to "LLM self-correction" in general. It's a concrete, reproducible measurement on one task, with all raw judgments and the pre-registration public.

Why measure this

"Self-correction" reports point both ways. Self-Refine (Madaan et al. 2023) reports large gains from iterative self-feedback; Huang et al. 2024 ("LLMs Cannot Self-Correct Reasoning Yet") and Kamoi et al. 2024 (a critical cross-task survey) find that without a reliable external signal, self-correction often doesn't help and can degrade. Those negative results are mostly on reasoning tasks, so I treat any agreement here as a cross-genre extrapolation, not a replication.

The practical version of the question: an LLM says "these two records are the same person / different people." Does asking the same model to re-check make it safer? Entity matching (deciding whether two given records refer to one entity) is easy to build gold labels for, so you can measure this without subjective judgment.

Setup

• Task: match a Japanese footballer's Japanese name (kanji/kana) against a romaji name — e.g. is 久保建英 the same person as Takefusa Kubo?
• Model: gpt-5.4-mini-2026-03-17, reasoning_effort=low. Measured: 2026-07.
• 160 pairs: a confirmation set of 120 (60 true-match / 60 non-match) plus 40 hard pairs.
• Surface forms come from different sources (Japanese side = Wikidata, romaji side = a Transfermarkt-derived dataset), so club-name spellings and coverage don't line up perfectly.
• The model never sees dates of birth, IDs, or URLs — only names and club history. Gold is verified against an independent attribute (DOB), not the linking ID.
• Cost: 480 calls (160 × A/B/C), ~$0.45 total.

How I measured (pre-registered)

Three conditions:

• A — first pass: judge the two records.
• B — blind re-run: the byte-identical prompt, fresh context, no draft shown. ("Independent" here means A not shown, not statistical independence.)
• C — two-stage review: the original data plus A's structured output as a draft, with a neutral "KEEP or CHANGE" instruction.

The pre-registered primary comparison is C vs B on the 120-pair confirmation set (balanced accuracy difference + a paired interval). C vs B measures the whole "hand the model its own output and ask it to re-review" workflow, not the anchoring effect in isolation.

Results

1. The first pass was already at the ceiling

A scored 120/120 on the confirmation set and 159/160 overall. That single fact reframes the whole experiment: there was essentially nothing to fix, so I measured maintenance near the ceiling, not error correction.

2. Re-review did not beat a blind re-run

Descriptively, across all 160 pairs B and C both scored 156. On the pre-registered confirmation set the 2×2 is: both correct 117, C-only-correct 1, B-only-correct 0, both-wrong 2. That single discordant pair (川辺駿 / Hayao Kawabe) gives exact McNemar p = 1.0 — which means "no power to distinguish," not "proven equivalent." The balanced-accuracy gap is +0.8 pt, but with one discordant pair the paired bootstrap lower bound is pinned at 0, so I don't lean on it. I don't claim equivalence either.

3. "Destruction" happened in both conditions, and it isn't self-review-specific

Reversals of A's correct answers, over all 160 pairs: A→B = 3, A→C = 3 (each 3/159 = 1.9%, Wilson 0.64–5.4%). On the confirmation set B breaks 3 and C breaks 2, but that flips on the hard set — subset-dependent. C's final verdict actually matches A on 157/160; the few it moved were all in the wrong direction, but with A at 159/160, "everything it moved was wrong" is close to arithmetically inevitable — not evidence C selectively destroys correct answers.

Deviations from the pre-registration (honest)

The pre-registered CI was a paired bootstrap (one of two registered options); it degenerated with a single discordant pair, so I report exact McNemar as an added, post-hoc auxiliary and don't lean on the interval. Wilson intervals were pre-registered for the secondary counts but missing from the first analysis script (added afterward). A pre-registered fame-stratified breakdown wasn't run (no clean popularity proxy). "Confirmatory" claims are limited to C-vs-B on the confirmation set.

Honest scope note: I did not measure human review, a different model, or any tool that checks the answer against ground truth. So I can't say external verification is better — only that handing the same model its own output didn't beat a plain re-run here.

What I actually do now: at this accuracy, I don't spend the extra pass. If I want a real second opinion, I reach for a different signal (a rule, a lookup, a second model), not the same model re-reading the same cues.

Everything I got wrong

1. My non-match pairs were too easy. I first built "same surname, different person" pairs, but the Japanese-side player was consistently older (mean 3.5-year gap), so the model could separate them on era alone. I rebuilt them to minimize the birth-year gap. Lesson: the benchmark author has to suspect their own benchmark is too easy.
2. My headline was inflated, and my own data killed it. I started to write "self-review destroys correct answers" — then noticed a blind re-run destroyed just as many. The mechanism was "a second pass at ceiling," not "self-review." Lesson: reconstruct the alternative baseline in your own data before you name a cause.
3. I almost claimed the hard cases needed club matching. 79 of 80 non-matches are actually separable by reading the given name. Only one truly needed an auxiliary attribute — and it's the one the model missed:

荒井悠汰 and a different person written Yuta Arai have identical romaji but different birth dates. Reading can't separate them; you need club history. That one homonym was A's only error (a single case, n=1) — and re-review didn't fix it. Lesson: the same model, re-reading the same cues, repeats the same miss.

Limitations

• Ceiling effect (central). With 159/160 correct up front, "can it fix errors" is essentially untested; I measured preservation.
• Memory contamination is unaddressed. These are public players; I can't separate "inferred the reading" from "recalled the person." The fame-stratified probe I pre-registered didn't run.
• Same-model self-loop only. No external verifier, human, or ground-truth check. Not a Self-Refine replication (no separate feedback stage; one shot).
• Benchmark representativeness. The confirmation set is a fixed 60/60 diagnostic split with hand-built non-matches; not a production match-rate distribution.
• Single runs, fixed order, one model, reasoning_effort=low. The ceiling and the null may be specific to this low-effort setting.

Reproduce it

• Repo: https://github.com/axiom-pro/llm-nayose-matching
• PREREGISTRATION.md (committed before the run, d7a3328) → src/run_experiment.py → src/analyze.py recomputes every number from run_results.json. Result commits: fd42585, analysis add-ons 74a00b5.
• Public: pair info, model judgments, analysis code, pre-registration. Withheld: DOB/verification attributes and non-redistributable raw data.

Takeaways

1. "Ask it again" isn't automatically safer. Whether re-review helps depends on how much room there is to improve and where the checking signal comes from.
2. The same model re-reading the same input isn't a second opinion. If the cues are identical, so are the blind spots.
3. Before measuring self-correction, make sure the first pass can actually fail. A ceiling task can't tell you whether re-review fixes errors — it can only show whether it preserves them.

Don't read this as "LLM self-review is useless." Read it as: on a near-ceiling task, an extra same-model pass showed no measurable value over a blind re-run — and the interesting failure was a homonym that re-reading can't catch. The next data point is a deliberately harder set where the first pass drops to 75–90%.

---

Repo and pre-registration above. I'm explicit about what I did **not* measure: no human, no second model, no ground-truth checker — so I make no claim about external verification here.*

This is an English adaptation of my Japanese article — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. Japanese original on Qiita (a Japanese dev-blogging platform): https://qiita.com/axiompro70/items/e186a0fe97a8318a2e9c

About me: I measure AI-assisted development practices and verify AI-produced code/data/judgments against ground truth — GitHub: axiom-pro.