DEV Community: tauridev

LLM self-review vs. a blind re-run on 160 gold-labeled name-matching pairs — no measurable difference at ceiling

tauridev — Sun, 12 Jul 2026 01:43:06 +0000

If you ship anything an LLM produced and calm your nerves by asking the same model to "please double-check it," this is a small, pre-registered measurement of whether that actually buys you anything.

Short version: on this task it bought nothing measurable — but mostly because the task turned out to be too easy, which is a lesson in itself about evaluation design.

TL;DR

Setup: gpt-5.4-mini, 160 gold-labeled "are these two records the same person?" pairs, pre-registered before the run.

The first pass was near-ceiling: 159/160 overall, 120/120 on the confirmation set. So there was almost nothing left to fix — I ended up measuring "does re-review preserve correct answers," not "does it fix errors."
Re-review (shown its own draft) tied a blind re-run. Across all 160 pairs both scored 156/160. On the pre-registered confirmation set, the single discordant pair gives exact McNemar p = 1.0 — no power to tell them apart, not proof of equivalence.
Reversals of correct answers happened in both conditions, equally (3 and 3). Not specific to self-review; consistent with plain run-to-run variation (single runs, so I can't causally isolate the extra pass itself).
Honest caveat: with the first pass this high, I could not test the thing I actually wanted to test. Treat this as the ceiling data point of a series, not a verdict on self-correction.

Reproducibility scope

This is one model, one prompt, reasoning_effort=low, single runs in a fixed order, on an 80/80 balanced diagnostic benchmark (60/60 confirmation + 20/20 hard). Nothing here generalizes to "LLM self-correction" in general. It's a concrete, reproducible measurement on one task, with all raw judgments and the pre-registration public.

Why measure this

"Self-correction" reports point both ways. Self-Refine (Madaan et al. 2023) reports large gains from iterative self-feedback; Huang et al. 2024 ("LLMs Cannot Self-Correct Reasoning Yet") and Kamoi et al. 2024 (a critical cross-task survey) find that without a reliable external signal, self-correction often doesn't help and can degrade. Those negative results are mostly on reasoning tasks, so I treat any agreement here as a cross-genre extrapolation, not a replication.

The practical version of the question: an LLM says "these two records are the same person / different people." Does asking the same model to re-check make it safer? Entity matching (deciding whether two given records refer to one entity) is easy to build gold labels for, so you can measure this without subjective judgment.

Setup

Task: match a Japanese footballer's Japanese name (kanji/kana) against a romaji name — e.g. is 久保建英 the same person as Takefusa Kubo?
Model: gpt-5.4-mini-2026-03-17, reasoning_effort=low. Measured: 2026-07.
160 pairs: a confirmation set of 120 (60 true-match / 60 non-match) plus 40 hard pairs.
Surface forms come from different sources (Japanese side = Wikidata, romaji side = a Transfermarkt-derived dataset), so club-name spellings and coverage don't line up perfectly.
The model never sees dates of birth, IDs, or URLs — only names and club history. Gold is verified against an independent attribute (DOB), not the linking ID.
Cost: 480 calls (160 × A/B/C), ~$0.45 total.

How I measured (pre-registered)

Three conditions:

A — first pass: judge the two records.
B — blind re-run: the byte-identical prompt, fresh context, no draft shown. ("Independent" here means A not shown, not statistical independence.)
C — two-stage review: the original data plus A's structured output as a draft, with a neutral "KEEP or CHANGE" instruction.

The pre-registered primary comparison is C vs B on the 120-pair confirmation set (balanced accuracy difference + a paired interval). C vs B measures the whole "hand the model its own output and ask it to re-review" workflow, not the anchoring effect in isolation.

Results

1. The first pass was already at the ceiling

A scored 120/120 on the confirmation set and 159/160 overall. That single fact reframes the whole experiment: there was essentially nothing to fix, so I measured maintenance near the ceiling, not error correction.

2. Re-review did not beat a blind re-run

Descriptively, across all 160 pairs B and C both scored 156. On the pre-registered confirmation set the 2×2 is: both correct 117, C-only-correct 1, B-only-correct 0, both-wrong 2. That single discordant pair (川辺駿 / Hayao Kawabe) gives exact McNemar p = 1.0 — which means "no power to distinguish," not "proven equivalent." The balanced-accuracy gap is +0.8 pt, but with one discordant pair the paired bootstrap lower bound is pinned at 0, so I don't lean on it. I don't claim equivalence either.

3. "Destruction" happened in both conditions, and it isn't self-review-specific

Reversals of A's correct answers, over all 160 pairs: A→B = 3, A→C = 3 (each 3/159 = 1.9%, Wilson 0.64–5.4%). On the confirmation set B breaks 3 and C breaks 2, but that flips on the hard set — subset-dependent. C's final verdict actually matches A on 157/160; the few it moved were all in the wrong direction, but with A at 159/160, "everything it moved was wrong" is close to arithmetically inevitable — not evidence C selectively destroys correct answers.

Deviations from the pre-registration (honest)

The pre-registered CI was a paired bootstrap (one of two registered options); it degenerated with a single discordant pair, so I report exact McNemar as an added, post-hoc auxiliary and don't lean on the interval. Wilson intervals were pre-registered for the secondary counts but missing from the first analysis script (added afterward). A pre-registered fame-stratified breakdown wasn't run (no clean popularity proxy). "Confirmatory" claims are limited to C-vs-B on the confirmation set.

Honest scope note: I did not measure human review, a different model, or any tool that checks the answer against ground truth. So I can't say external verification is better — only that handing the same model its own output didn't beat a plain re-run here.

What I actually do now: at this accuracy, I don't spend the extra pass. If I want a real second opinion, I reach for a different signal (a rule, a lookup, a second model), not the same model re-reading the same cues.

Everything I got wrong

My non-match pairs were too easy. I first built "same surname, different person" pairs, but the Japanese-side player was consistently older (mean 3.5-year gap), so the model could separate them on era alone. I rebuilt them to minimize the birth-year gap. Lesson: the benchmark author has to suspect their own benchmark is too easy.
My headline was inflated, and my own data killed it. I started to write "self-review destroys correct answers" — then noticed a blind re-run destroyed just as many. The mechanism was "a second pass at ceiling," not "self-review." Lesson: reconstruct the alternative baseline in your own data before you name a cause.
I almost claimed the hard cases needed club matching. 79 of 80 non-matches are actually separable by reading the given name. Only one truly needed an auxiliary attribute — and it's the one the model missed:

荒井悠汰 and a different person written Yuta Arai have identical romaji but different birth dates. Reading can't separate them; you need club history. That one homonym was A's only error (a single case, n=1) — and re-review didn't fix it. Lesson: the same model, re-reading the same cues, repeats the same miss.

Limitations

Ceiling effect (central). With 159/160 correct up front, "can it fix errors" is essentially untested; I measured preservation.
Memory contamination is unaddressed. These are public players; I can't separate "inferred the reading" from "recalled the person." The fame-stratified probe I pre-registered didn't run.
Same-model self-loop only. No external verifier, human, or ground-truth check. Not a Self-Refine replication (no separate feedback stage; one shot).
Benchmark representativeness. The confirmation set is a fixed 60/60 diagnostic split with hand-built non-matches; not a production match-rate distribution.
Single runs, fixed order, one model, reasoning_effort=low. The ceiling and the null may be specific to this low-effort setting.

Reproduce it

Repo: https://github.com/axiom-pro/llm-nayose-matching
PREREGISTRATION.md (committed before the run, d7a3328) → src/run_experiment.py → src/analyze.py recomputes every number from run_results.json. Result commits: fd42585, analysis add-ons 74a00b5.
Public: pair info, model judgments, analysis code, pre-registration. Withheld: DOB/verification attributes and non-redistributable raw data.

Takeaways

"Ask it again" isn't automatically safer. Whether re-review helps depends on how much room there is to improve and where the checking signal comes from.
The same model re-reading the same input isn't a second opinion. If the cues are identical, so are the blind spots.
Before measuring self-correction, make sure the first pass can actually fail. A ceiling task can't tell you whether re-review fixes errors — it can only show whether it preserves them.

Don't read this as "LLM self-review is useless." Read it as: on a near-ceiling task, an extra same-model pass showed no measurable value over a blind re-run — and the interesting failure was a homonym that re-reading can't catch. The next data point is a deliberately harder set where the first pass drops to 75–90%.

Repo and pre-registration above. I'm explicit about what I did **not* measure: no human, no second model, no ground-truth checker — so I make no claim about external verification here.*

This is an English adaptation of my Japanese article — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. Japanese original on Qiita (a Japanese dev-blogging platform): https://qiita.com/axiompro70/items/e186a0fe97a8318a2e9c

About me: I measure AI-assisted development practices and verify AI-produced code/data/judgments against ground truth — GitHub: axiom-pro.

What actually breaks when a free Cloudflare D1 database hits its 500 MB wall

tauridev — Thu, 09 Jul 2026 13:57:02 +0000

Every "free serverless DB" comparison ranks capacity, price, and cold-start. Almost none answer the question you actually hit in production: what happens at the free-tier wall? So I filled a Cloudflare D1 database to its limit over the REST API and poked it.

The honest short version: the wall is not a full read-only lockout, a plain DELETE clears it immediately (but that recovery is not quota-free), and — the part I got wrong twice — building an index on a blob column roughly doubles your storage, so it slams into the wall on its own. This post has been revised twice: my own size_after logs, and then a follow-up experiment, corrected me. I'll show both corrections instead of hiding them.

Scope: Workers Free, throwaway DBs, over the D1 REST API /query, 2026-07. The coarse wall was reproduced on a second, freshly-created database; the exact-ceiling pin and the index-doubling are each n=1 (flagged). This is a REST probe, not a Worker Binding probe — don't generalize endpoint-specific behavior (especially VACUUM).

"The free-tier wall" is not one wall

D1 has several walls, with different scopes and — per Cloudflare's own error list — different messages:

Wall	Scope	Value (Free)	Documented message
per-DB storage	per database	500 MB (Paid: 10 GB)	`Exceeded maximum DB size` — action: "delete rows / shard"
account storage	account	5 GB	`Your account has exceeded D1's maximum account storage limit…`
daily rows read	account	5,000,000/day	daily-limit-exceeded → "can't run queries"
daily rows written	account	100,000/day	same (INSERT, UPDATE, and DELETE all count as writes)

(Free daily counters reset at 00:00 UTC; storage is not daily-reset. Max row 2 MB; 100 columns; 10 DBs/account. All confirmed against the D1 docs on 2026-07-09.) I measured the first one.

Result 1 — the wall is 500 MB (decimal); pinned to one row on one DB, reproduced coarsely on a second

Filling with bulk inserts stalls around ~499.9 MB. To pin it, I then looped single-row inserts at the wall:

… 24 single-row inserts succeed (size_after climbs)
25th: HTTP 400  [{"code":7500,"message":"Exceeded maximum DB size"}]
last successful size_after = 499,994,624 bytes  (≈ 499.99 MB ≈ 476.83 MiB)

That ceiling — 499,994,624 — is 500,000,000 (decimal 500 MB) minus 5,376 bytes, enforced at SQLite's 4 KB page granularity (499,994,624 = 122,069 × 4096). I reproduced the coarse wall on a second, freshly-created DB: it stopped at the identical 499,884,032 and returned the same 7500. One caveat so you can hold me to my own logs: the redacted results/*.jsonl carry no account IDs, and served_by_colo is Anycast edge routing, not an account signal — so what's verifiable from the evidence is a second database, and the byte-exact 499,994,624 pin is from the single-row probe on one of them (n=1). The wall firing at all means these DBs are on Workers Free (Paid = 10 GB).

Result 2 — not read-only, but small writes aren't "exempt" either (measured)

My first draft claimed small writes slip past the wall while big ones don't. Wrong — and my own numbers show why. Those 24 single-row inserts above consumed only 499,994,624 − 499,884,032 = 110,592 bytes (~110 KB) of headroom before even a single-row insert was rejected. So:

Reads always pass — near-tautological for a storage cap; reads don't grow the DB.
Writes pass only while headroom remains. Small writes weren't bypassing the wall; they were filling the last sliver before it. Call it slack, not exemption. (One headroom-exhaustion sequence, but the 25th insert failing at zero headroom is a clean counterexample to "small writes are exempt" — one is enough.)

Result 3 — recovery is a plain DELETE, no upgrade (but it spends write quota)

DELETE WHERE k < 20000   → 19,998 rows; size_after 499.9 MB → 243,884,032 (~244 MB) immediately; rows_written = 19,998
bulk INSERT (after delete) → restored
VACUUM                    → 7500 "cannot VACUUM from within a transaction: SQLITE_ERROR" (maybe REST-specific)

DELETE drops D1's reported size immediately and writes resume — no VACUUM, no plan upgrade (recovery class R1). Two honest qualifiers: (1) it isn't quota-free — D1 counts DELETE as a write (docs: "Write operations include INSERT, UPDATE, and DELETE"), so this one 19,998-row delete spent ~20% of the daily 100k budget, and the cost scales with row count, not bytes freed (this delete freed ~256 MB ≈ 12.8 KB/row; whether it's linear per chunk I didn't separately measure); (2) size_after is D1's reported size — the meter dropping doesn't prove the physical file shrank (cf. workerd#1618, local).

Result 4 — CREATE INDEX doubles your storage (the thing I got wrong twice)

At the wall, CREATE INDEX fails with out of memory: SQLITE_NOMEM. My first draft lumped that in as "large allocation blocked"; after review I "corrected" it to "that's a memory limit, independent of the storage wall." Both were wrong.

On a control DB at 300 MB — well below the 500 MB wall — CREATE INDEX failed not with SQLITE_NOMEM but with 7500 "Exceeded maximum DB size" (storage!). Below the wall, a storage error? I measured the mechanism directly — deleted the control to 80 MB and indexed:

CREATE INDEX ON bench(v) at 84,029,440 (80.1 MiB) → OK
  size_after 84,029,440 → 167,923,712   =  ×1.998 growth (measured once, at ~80 MB)

An index on a blob column carries a copy of the column's data as its keys, so it roughly doubles storage — inferred from this one growth point, not proven in general. If that ×2 holds at other sizes:

storage before index	after (~×2)	result
80 MB	160 MB	✅ succeeds (under the wall)
300 MB	~600 MB	⛔ `Exceeded maximum DB size` (over the 500 MB wall)
500 MB (at the wall)	—	⛔ `SQLITE_NOMEM` (OOMs during the build)

So the index failure is storage-bound via the index's own footprint, not "memory, unrelated to the wall." A rough rule of thumb — if the ×2 holds — is that on D1 Free you can only index a column holding up to ~250 MB of data (2 × 250 = 500 MB), but that's an extrapolation, not measured near 250 MB. The SQLITE_NOMEM is just what you see when you're already at the wall.

The error code, and what I did not measure

Cloudflare's error list classifies these by message (Exceeded maximum DB size, …account storage limit…, …isolate memory limit…); the number 7500 appears nowhere in the docs. So 7500 is a generic REST wrapper — the message is the discriminator.

Not measured / n=1 / extrapolated (so not claimed): the index ×2 is a single measurement at one size/row-shape; the ~250 MB threshold is extrapolated (not measured near 250 MB); DELETE linearity across chunks is not measured (one 19,998-row delete); the account 5 GB wall, the daily read wall (5M/day — reads stop, account-scoped), and Worker Binding behavior are all untested; size_after behaves like a coherent logical meter (dropped on the one DELETE, ~doubled on the one index build) but the docs don't say physical vs logical. One full cycle (fill 39k + recover 20k + control 24k ≈ 84k rows_written) fits in one UTC day but not two.

Reproduce it

git clone https://github.com/axiom-pro/free-tier-quota-probe
cd free-tier-quota-probe
npm install && npm run selftest
# .env: CF_ACCOUNT_ID / CF_API_TOKEN (D1:Edit) / D1_DATABASE_ID / D1_CONTROL_DATABASE_ID / BILLING_SAFE=confirmed
node --env-file=.env bin/d1-admin.mjs create wallprobe   # create the DB via API (no dashboard needed)
npm run ping        -- --target d1
npm run fill        -- --target d1 --confirm-destructive
npm run probe-slack -- --target d1                        # pins the wall + measures the headroom
npm run fill        -- --target d1 --control --cap-mb 300 --confirm-destructive
npm run probe-index -- --target d1                        # the index-doubling test

Node ≥ 20, ESM, D1 over native fetch (60 s timeout + one retry). Redacted captures are frozen in results/*.jsonl; predictions (and the follow-up amendment) are in PREREGISTRATION.md. Repo: https://github.com/axiom-pro/free-tier-quota-probe

I write about measurement discipline — filling the wall instead of quoting the limit, and correcting my own assumptions on the record. This post's Results 2 and 4 are the live examples: I reasoned wrong twice and only the measurements got it right. Written with AI assistance; every number is from the captured `results/.jsonl`, and the doc facts were re-verified against Cloudflare's D1 docs on 2026-07-09. A Japanese version of this write-up also exists.*

I had a local LLM review the same code 51 times: majority voting picks persistence, not correctness

tauridev — Tue, 07 Jul 2026 11:48:35 +0000

If you use an LLM to review code — or run it several times and keep "whatever most runs agree on" — this experiment suggests the votes you're counting measure persistence, not correctness.

TL;DR

Setup: Qwen2.5-Coder 1.5B / 7B (Apache-2.0) via Ollama 0.31.1, single RTX 4060 Ti, Q4_K_M quantization. N=51 repeated audits of the same small Python files with predeclared seeded defects (temp-0 baseline: N=11 per condition). The per-run records (normalized findings plus a SHA-256 of each raw response), ground truth, and aggregation code are public — every number quoted in this article regenerates deterministically from the frozen records via an independent verification script.

Fully pinned = deterministic (on this rig). At temperature 0 with a fixed GPU and quantization, the finding sets were identical across runs — Jaccard similarity 1.00 in all four conditions (N=11 each). In a probe, six repeats within one seed and six different seeds all produced byte-identical output: our temp-0 configuration also forces top_k=1 (greedy decoding in Ollama/llama.cpp), so the seed had no observable effect. "LLMs are always nondeterministic" is folklore, at least under fully pinned local conditions.
Sampling at temperature 0.7 breaks agreement. On the defect-seeded file — 51 runs, one seed per run (0–50) — run-to-run Jaccard dropped to 0.52 (7B) and 0.07 (1.5B). For 7B, allowing ±2 lines of tolerance recovers 0.52 → 0.80: over half of the disagreement is consistent with the same bug pointed at a different line.
The core finding: majority voting selects persistence, not correctness. A real resource leak that the model detected in 45 of 51 runs split its votes across two adjacent lines (26 + 19 — no run reported both, so those really are 45 distinct runs). Under the strict line+category key, the majority threshold of 26 barely keeps one line; one vote higher, at k=27, a defect found 45 times is gone. Meanwhile, two "grey" pattern warnings (eval and shell=True on constants, from the decoy file) survived at 33/51 and 28/51, because the model points at exactly the same place every time.
Honest caveats up front: on the defect-seeded file, against predeclared ground truth with category + ±2-line matching, the 7B's per-run recall was only 0.31 — but precision was 0.98. Within this benchmark: when it speaks, it's right, and it misses a lot. The 1.5B was too noisy to carry any claim (it hit 25 of the file's 32 lines at least once across 51 runs), so it serves as a contrast group, not evidence.

⚠ Reproducibility scope: the existence and rough magnitude of the variance reproduces; exact numbers are rig-dependent (GPU / quantization / inference-engine version). The repo ships frozen per-run records — normalized findings plus a hash of each raw response; the raw response bodies themselves were not retained — and readers re-run only the deterministic aggregation layer.

Why measure this

"LLM-as-a-judge" is everywhere, and its known weakness is nondeterminism. Prior work measures score stability — Klishevich et al. (arXiv:2502.20747) measured scalar-verdict consistency over 70 commits at temp 0, and a Japanese industrial blog measured same-score rates on cloud APIs. What I couldn't find measured anywhere: whether the set of findings itself — which issues, on which lines — is stable across runs, on a local model, free to reproduce, with run records frozen and shipped. That's the gap this fills.

Setup

Models: Qwen2.5-Coder 1.5B and 7B (Apache-2.0), run locally via Ollama. Chosen because they're free, local, and require no API key — anyone can rerun generation.
Sampling design: temp-0 conditions use a fixed seed and force top_k=1, top_p=1.0; temp-0.7 conditions use one seed per run (0, 1, …, 50) with Ollama's default sampler otherwise (top_k=40, top_p=0.9; not swept). The temp-0.7 numbers therefore measure across-seed sampling variability at temperature 0.7 — not same-seed runtime nondeterminism (with a fixed seed, repeats would be trivially identical).
Output format: Ollama structured outputs (JSON-schema-enforced). Each finding is {line_number, category, severity, description}, category constrained to an enum.
Identity of a finding: the composite key (line_number, category). Paraphrases are ignored; two findings are "the same" iff they name the same place and problem type. I deliberately did not use embedding similarity — it happily merges "missing null check on line 12" with "missing null check on line 45".
Target files (4): target_a (7 seeded defects), clean (no defects — false-positive baseline), decoy (looks dangerous, low actual harm — this classification gets revised later, see pitfall #4), easy (6 obvious defects).
Ground truth: predeclared in a gt.csv outside the code before measurement. No hints in the code or the prompt — I learned this one the hard way (pitfall #1).
Two-layer reproducibility: layer 1 (generation, nondeterministic) ran once and its per-run records are frozen in the repo; layer 2 (aggregation, Python stdlib only) is fully deterministic and is what you reproduce.

Component	Choice	Notes
Inference	Ollama 0.31.1	local, free
Models	Qwen2.5-Coder 1.5B / 7B	Q4_K_M (verified via `ollama show`)
Hardware	NVIDIA RTX 4060 Ti (CUDA)	`ollama ps` reported 100% GPU offload during runs
Aggregation	Python 3.12, stdlib only	layer 2 fully deterministic

Measured: 2026-07. Per-cell provenance (model quantization/size/family, Ollama version, prompt/schema/GT hashes, backend) is recorded in the frozen records.

How I measured

Reliability = Jaccard between runs (no ground truth involved): matched findings / union, computed with optimal bipartite matching — naive greedy matching gives order-dependent values (a bug I actually shipped and had to fix). Reported as the mean over all run pairs within a condition, with a line-tolerance window w (±0 strict / ±2 / ±5) reported alongside — this doubles as the sensitivity analysis for the identity key.
Validity = precision / recall / false positives against GT. Unless noted, recall/precision are per-run means on target_a, category-strict, line ±2. Precision excludes empty runs by definition — and there were none on target_a (0/51); recall averages over all runs.
Majority voting = keep findings that appear in ≥ k of N runs. Examined on two axes: threshold k × key granularity w.
Uncertainty: 95% CIs from bootstrap resampling at the run level. One honest structural caveat: 51 audits of the same file are pseudo-replication — nothing here generalizes across codebases, and I don't claim it does.

Result 1: pin everything, and it's deterministic

All four temp-0 conditions (2 models × {defect-seeded, easy} files, N=11): Jaccard 1.00 at every tolerance level, zero variance across runs. In a separate probe on one condition, six repeats within a single seed and six different seeds (1, 2, 3, 101, 202, 303) produced byte-identical outputs — our temp-0 configuration also forces top_k=1, i.e. greedy decoding in Ollama/llama.cpp, so the seed had no observable effect there. Folklore says "LLMs always wobble"; under fully pinned local conditions on this rig, they didn't.

📐 The honest asterisk on "deterministic"

In a separate spike I did observe a single category flip at temp 0 — only ever on the first run after model load, which also had the longest latency (cold-cache/warm-up signature). With a warm-up run added, the residual disappeared. I can't pin the mechanism (GPU parallel-reduction ordering is the usual suspect; cold cache isn't ruled out), but it isn't the seed. This is consistent with Thinking Machines' "Defeating Nondeterminism in LLM Inference", which attributes serving-time nondeterminism primarily to batch-size variation — a local single-request setup is effectively fixed-batch, which is consistent with the 1.00 here. The same mechanism would also predict the temp-0 flutter Klishevich et al. saw on cloud APIs (variable batching), though they discuss other candidate causes as well — read this as consistency with one proposed mechanism, not a proven attribution.

Result 2: sample at temperature 0.7, and it falls apart

People raise temperature precisely to widen candidate coverage across runs. At temp 0.7 on the defect-seeded file (one seed per run, 0–50), run-to-run Jaccard was 0.52 [0.47, 0.59] for 7B and 0.07 [0.04, 0.11] for 1.5B.

Two distinct failure modes hide in those numbers:

Line drift (7B): 0.52 recovers to 0.80 when the identity key tolerates ±2 lines. Over half of the disagreement is consistent with the same bug pointed at a different line — though tolerance matching can also create accidental matches, so treat ±2 as an upper bound on "drift".
Category flutter (1.5B, and easy): on the easy file, 7B's line-only recall is 0.81 but category-strict recall is 0.42 — the model keeps finding the right place and labeling it a different problem.

File	Model	recall (category-strict)	recall (line-only)
target_a	7B	0.31	0.31
target_a	1.5B	0.05	0.18
easy	7B	0.42	0.81

(For 7B on target_a the two recalls coincide — those misses are genuine non-detections, not drift.)

And here's where I hit my first big mistake: my early recall numbers were great — because I had left # DEFECT: SQL injection annotations in the target code as notes to myself, and the whole file goes into the prompt. The model was reading my answer key. Deleting the labels cratered recall and deepened the run-to-run variance. Details in the pitfall log below.

Result 3 (the core): majority voting rewards persistence

Majority vote = "keep findings that appear in ≥26 of 51 runs." Here is what that actually selects.

Figure: votes per finding (blue = predeclared seeded defects, red = persistent findings outside GT, dashed = majority threshold 26/51). Counts are pooled from two target files run under identical conditions: the blue bars come from the defect-seeded file, the red bars from the decoy file.

The mechanism. With (line, category) as the identity key, the same bug pointed at a neighboring line becomes a different ballot. The model detected the ResourceLeak in 45/51 runs — but the votes landed 26 on line 16 and 19 on line 17 (no single run reported both lines, so 26+19 really is 45 distinct runs). Loosen the question to "did it say ResourceLeak at all, anywhere?" and it's 49/51 — scattered across six different lines (12, 15, 16, 17, 18, 21). The vote count never sees that. Watch the threshold:

Threshold k	line 16 (26 votes)	line 17 (19 votes)	Outcome
k=19	kept	kept	both survive (duplicated)
k=26 (majority)	kept	dropped	survives by one vote
k=27	dropped	dropped	a defect reported in 45/51 runs is gone

Under the exact majority rule I predeclared (k=26) the leak barely survives on one line; the point is not that majority voting killed it, but that whether it survives is governed by how the votes split and where you draw the line — not by the model's detection ability. And it's not one anecdote: 2 of the 5 defects that 7B detected at least once (~40%) showed line drift between runs.

Two contrasts sharpen the picture:

Unsplit real bugs sail through: the SQL injection on line 9 got 51/51 unanimous votes. Voting doesn't kill strong detections — it kills wobbly-keyed ones.
Grey findings survive: warnings against eval("2 + 2") (33/51) and subprocess.run(..., shell=True) with a constant string (28/51) — both low-harm on constants, both from the decoy file — persist because the model reacts to the pattern and points at exactly the same place every time. (Bonus wobble: it labeled the latter CodeInjection rather than CommandInjection — categories flutter too.)

The clean file adds a neat corollary: 7B stayed silent in 37/51 runs (72.5%), and when it did complain, it complained about something different almost every time — non-empty clean runs agree at just 0.12. (Counting silent–silent pairs as agreement 1.0 the unconditional mean is 0.53; I report the conditional number because silence agreeing with silence isn't informative here.) Inconsistent false alarms are exactly what majority voting removes. Consistent grey warnings are exactly what it cannot.

Threshold × key-granularity sensitivity (majority k=26):

File (7B)	w ±0	±2	±5	Reading
`clean`	FP 0	0	0	scattered false alarms die (as intended)
`decoy`	2 persistent	2	2	consistent grey findings survive
`target_a`	TP 2	2	3 *	* the +1 is a double-credit artifact (below)
`easy`	TP 2 / FP 1	3 / 2	3 / 1 †	looser keys rescue split votes — at a price

† The easy false-positive count is non-monotonic in w (1→2→1) because widening the window changes how findings cluster — adjacent clusters merge or get absorbed. This is a property of line-window clustering, disclosed rather than hidden.

📐 Why the ±5 "rescue" is double-counting, not detection At w ±5, the ResourceLeak cluster (mostly line 16) becomes wide enough to overlap **two distinct** ground-truth defects — a connection leak (lines 6–11) and a file-handle leak (lines 15–17) — so one cluster gets credited to both (the aggregation deliberately allows multi-credit; that design choice is exactly what this row exposes). To be precise: no run put a ResourceLeak vote inside the connection-leak lines themselves (6–11); the nearest was a single vote at line 12 — one line outside, and far below any threshold. So the "+1 true positive" at ±5 is a windowing artifact, not a detection. Loosening keys rescues split votes, but the rescue is not free: on `easy` it temporarily raises false positives (1→2 at ±2), and here it manufactures a detection that never cleared any threshold. No free lunch.

So: majority voting filters for salience of the pattern response, not truth. Persistent findings survive whether they're real bugs or grey smells; real bugs whose expression wobbles can fall. "Persistent" and "correct" are different properties, and the vote measures the first one.

Honest scope note: in this sample the core claim rests mainly on one drifting real bug plus two persistent grey findings — read it as an observed mechanism, not a general law. It is not an argument against aggregation per se — this is also not the self-consistency setting, which votes over normalized final answers; the failure mode here lives precisely in the un-normalized key. Change the key (category-only, AST node, defect region) and the behavior changes — which is itself the point: threshold × key granularity is a design decision with no free lunch.

What I actually do now: treat multi-run LLM review as a candidate pool, not an auto-verdict. Prioritize high-frequency findings for review, don't discard low-frequency findings in severe categories, and let tests, static analysis, and a human's understanding of the spec make the final call.

Everything I got wrong (measurement-instrument failures included)

The most useful part of this experiment was watching my own tooling and my own memory fail. Five mistakes, in the order I made them:

My answer key leaked into the prompt. I'd annotated defect lines with # DEFECT: ... comments as notes to myself — and the audit prompt includes the whole file. High recall was the model reading my answers. Removing the labels cratered recall and deepened variance. Never let the benchmark's answers be visible to the system under test.
I thought I was benchmarking CPU inference. It was 100% GPU. I had designed a thread-count variable (num_thread 1 vs 4) — then ollama ps showed 100% GPU offload, where num_thread is a no-op. My thread axis had measured nothing. Record your accelerator provenance instead of assuming it — the act of recording is what exposed the error.
I broke my own measuring instrument while "fixing" it. A one-line "optimization" to a clustering helper introduced a classic append-then-clear aliasing bug that silently corrupted the majority-vote true/false-positive counts. I caught it only because a hand count said 2 where the tool said 3. Every number quoted in this article's text is now recomputed by an independent script (verify_independent.py) that bypasses the aggregation code entirely; the full threshold×tolerance table and the bootstrap CIs still come from the aggregation pipeline. Fixes need verification most of all.
My "false positives" were real smells. I'd built decoy assuming its eval/shell=True-on-constants patterns were harmless, so surviving warnings counted as false positives. But Bandit and friends flag exactly these. The finding got stronger through the correction: the claim isn't "false positives survive voting" — it's "persistence survives voting, regardless of truth", and I renamed those findings "persistent findings outside GT".
My own memory fabricated a result. An early draft said "the md5 warning also survived the vote." The frozen records say md5 got 0 votes in 51 runs. My memory had synthesized it from the impression of pitfall #4. Caught in pre-publication log reconciliation. The experimenter's memory is not a primary source either — recheck the frozen records every time you write a number.

This is "don't trust LLM output" applied symmetrically: to the model, to my own instruments, and to my own claims.

Limitations

Small sample: N=51, two model sizes, one rig (RTX 4060 Ti / CUDA / Q4_K_M / Ollama 0.31.1). CPU, other GPUs, other quantizations may behave differently.
The temp-0.7 condition varies the seed per run (by design — that's what sampling the distribution means here), so it measures across-seed variability at that temperature, not a pure temperature effect with everything else held.
The core claim rests on few instances (one drifting bug, two grey findings). Mechanistically coherent, statistically thin.
"Smaller models wobble more" is a hypothesis from a 2-point comparison — capability and variance are confounded.
Category-strict recall is low; this is not a benchmark of absolute audit quality. Some ground-truth entries depend on runtime contracts (e.g., whether a config loader may return None) — they are predeclared seeded defects under this benchmark's stated assumptions, not universally adjudicated bugs.
Structured output (JSON-schema-constrained decoding) modifies the logit distribution; the variance measured here is variance under that constraint. Free-form review text may behave differently.
No causal claims. Scope: free, local, small models, this corpus.

Reproduce it

Generation is nondeterministic and not the reproduction target; the frozen per-run records are. The deterministic layer regenerates every table in this article:

git clone https://github.com/axiom-pro/llm-audit-nondeterminism
cd llm-audit-nondeterminism

# Layer 2 (deterministic, no LLM needed): regenerate all numbers from frozen records
python scripts/aggregate.py            # -> results/m2_summary.json
python scripts/verify_independent.py   # independent recomputation of every number quoted in the text — should print ALL PASS
python scripts/make_figures.py         # -> figures/nofreelunch.png (needs Matplotlib)

# Layer 1 (optional, needs Ollama + ~5GB VRAM for 7B Q4_K_M):
#   regenerating gives you a *different* distribution — that's the point
#   OLLAMA_EXE=/path/to/ollama bash scripts/run_m2.sh
#   python scripts/seed_probe.py       # confirm seeds have no observable effect at temp 0

Numerical aggregation and verification need Python 3.12 and only the standard library; figure regeneration additionally needs Matplotlib. A few seconds total. (On Windows, set PYTHONUTF8=1.)

Takeaways

Pin everything and local LLM review was deterministic on this rig — including deterministically repeating the same misses. Determinism ≠ correctness.
One review run is a sample, not a verdict. At practical temperatures the finding set moves between runs, mostly by line drift.
Majority voting measures agreement strength, not truth. Use multi-run review as a candidate generator; adjudicate with tests, static analysis, and humans.

If you remember one sentence, please don't make it "LLM code review is useless" — that's not what the data says. Make it: multi-run review is a decent candidate generator; what's unsafe is automatic adjudication over a brittle vote key.

All per-run records (normalized findings + response hashes), ground truth, aggregation code, and the independent verification script: github.com/axiom-pro/llm-audit-nondeterminism. Anything not measured (other GPUs, quantizations, larger models) is listed as not measured.

This is an English adaptation of my Japanese article on Zenn — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. If you spot an error, comments and issues are open; I'll verify against the frozen records and correct with a changelog.

About me: I build developer tools in Rust/Tauri and measure AI-assisted development practices with frozen records and independent recomputation — GitHub: axiom-pro.