I had a local LLM review the same code 51 times: majority voting picks persistence, not correctness

tauridev — Tue, 07 Jul 2026 11:48:35 +0000

If you use an LLM to review code — or run it several times and keep "whatever most runs agree on" — this experiment suggests the votes you're counting measure persistence, not correctness.

TL;DR

Setup: Qwen2.5-Coder 1.5B / 7B (Apache-2.0) via Ollama 0.31.1, single RTX 4060 Ti, Q4_K_M quantization. N=51 repeated audits of the same small Python files with predeclared seeded defects (temp-0 baseline: N=11 per condition). The per-run records (normalized findings plus a SHA-256 of each raw response), ground truth, and aggregation code are public — every number quoted in this article regenerates deterministically from the frozen records via an independent verification script.

Fully pinned = deterministic (on this rig). At temperature 0 with a fixed GPU and quantization, the finding sets were identical across runs — Jaccard similarity 1.00 in all four conditions (N=11 each). In a probe, six repeats within one seed and six different seeds all produced byte-identical output: our temp-0 configuration also forces top_k=1 (greedy decoding in Ollama/llama.cpp), so the seed had no observable effect. "LLMs are always nondeterministic" is folklore, at least under fully pinned local conditions.
Sampling at temperature 0.7 breaks agreement. On the defect-seeded file — 51 runs, one seed per run (0–50) — run-to-run Jaccard dropped to 0.52 (7B) and 0.07 (1.5B). For 7B, allowing ±2 lines of tolerance recovers 0.52 → 0.80: over half of the disagreement is consistent with the same bug pointed at a different line.
The core finding: majority voting selects persistence, not correctness. A real resource leak that the model detected in 45 of 51 runs split its votes across two adjacent lines (26 + 19 — no run reported both, so those really are 45 distinct runs). Under the strict line+category key, the majority threshold of 26 barely keeps one line; one vote higher, at k=27, a defect found 45 times is gone. Meanwhile, two "grey" pattern warnings (eval and shell=True on constants, from the decoy file) survived at 33/51 and 28/51, because the model points at exactly the same place every time.
Honest caveats up front: on the defect-seeded file, against predeclared ground truth with category + ±2-line matching, the 7B's per-run recall was only 0.31 — but precision was 0.98. Within this benchmark: when it speaks, it's right, and it misses a lot. The 1.5B was too noisy to carry any claim (it hit 25 of the file's 32 lines at least once across 51 runs), so it serves as a contrast group, not evidence.

⚠ Reproducibility scope: the existence and rough magnitude of the variance reproduces; exact numbers are rig-dependent (GPU / quantization / inference-engine version). The repo ships frozen per-run records — normalized findings plus a hash of each raw response; the raw response bodies themselves were not retained — and readers re-run only the deterministic aggregation layer.

Why measure this

"LLM-as-a-judge" is everywhere, and its known weakness is nondeterminism. Prior work measures score stability — Klishevich et al. (arXiv:2502.20747) measured scalar-verdict consistency over 70 commits at temp 0, and a Japanese industrial blog measured same-score rates on cloud APIs. What I couldn't find measured anywhere: whether the set of findings itself — which issues, on which lines — is stable across runs, on a local model, free to reproduce, with run records frozen and shipped. That's the gap this fills.

Setup

Models: Qwen2.5-Coder 1.5B and 7B (Apache-2.0), run locally via Ollama. Chosen because they're free, local, and require no API key — anyone can rerun generation.
Sampling design: temp-0 conditions use a fixed seed and force top_k=1, top_p=1.0; temp-0.7 conditions use one seed per run (0, 1, …, 50) with Ollama's default sampler otherwise (top_k=40, top_p=0.9; not swept). The temp-0.7 numbers therefore measure across-seed sampling variability at temperature 0.7 — not same-seed runtime nondeterminism (with a fixed seed, repeats would be trivially identical).
Output format: Ollama structured outputs (JSON-schema-enforced). Each finding is {line_number, category, severity, description}, category constrained to an enum.
Identity of a finding: the composite key (line_number, category). Paraphrases are ignored; two findings are "the same" iff they name the same place and problem type. I deliberately did not use embedding similarity — it happily merges "missing null check on line 12" with "missing null check on line 45".
Target files (4): target_a (7 seeded defects), clean (no defects — false-positive baseline), decoy (looks dangerous, low actual harm — this classification gets revised later, see pitfall #4), easy (6 obvious defects).
Ground truth: predeclared in a gt.csv outside the code before measurement. No hints in the code or the prompt — I learned this one the hard way (pitfall #1).
Two-layer reproducibility: layer 1 (generation, nondeterministic) ran once and its per-run records are frozen in the repo; layer 2 (aggregation, Python stdlib only) is fully deterministic and is what you reproduce.

Component	Choice	Notes
Inference	Ollama 0.31.1	local, free
Models	Qwen2.5-Coder 1.5B / 7B	Q4_K_M (verified via `ollama show`)
Hardware	NVIDIA RTX 4060 Ti (CUDA)	`ollama ps` reported 100% GPU offload during runs
Aggregation	Python 3.12, stdlib only	layer 2 fully deterministic

Measured: 2026-07. Per-cell provenance (model quantization/size/family, Ollama version, prompt/schema/GT hashes, backend) is recorded in the frozen records.

How I measured

Reliability = Jaccard between runs (no ground truth involved): matched findings / union, computed with optimal bipartite matching — naive greedy matching gives order-dependent values (a bug I actually shipped and had to fix). Reported as the mean over all run pairs within a condition, with a line-tolerance window w (±0 strict / ±2 / ±5) reported alongside — this doubles as the sensitivity analysis for the identity key.
Validity = precision / recall / false positives against GT. Unless noted, recall/precision are per-run means on target_a, category-strict, line ±2. Precision excludes empty runs by definition — and there were none on target_a (0/51); recall averages over all runs.
Majority voting = keep findings that appear in ≥ k of N runs. Examined on two axes: threshold k × key granularity w.
Uncertainty: 95% CIs from bootstrap resampling at the run level. One honest structural caveat: 51 audits of the same file are pseudo-replication — nothing here generalizes across codebases, and I don't claim it does.

Result 1: pin everything, and it's deterministic

All four temp-0 conditions (2 models × {defect-seeded, easy} files, N=11): Jaccard 1.00 at every tolerance level, zero variance across runs. In a separate probe on one condition, six repeats within a single seed and six different seeds (1, 2, 3, 101, 202, 303) produced byte-identical outputs — our temp-0 configuration also forces top_k=1, i.e. greedy decoding in Ollama/llama.cpp, so the seed had no observable effect there. Folklore says "LLMs always wobble"; under fully pinned local conditions on this rig, they didn't.

📐 The honest asterisk on "deterministic"

In a separate spike I did observe a single category flip at temp 0 — only ever on the first run after model load, which also had the longest latency (cold-cache/warm-up signature). With a warm-up run added, the residual disappeared. I can't pin the mechanism (GPU parallel-reduction ordering is the usual suspect; cold cache isn't ruled out), but it isn't the seed. This is consistent with Thinking Machines' "Defeating Nondeterminism in LLM Inference", which attributes serving-time nondeterminism primarily to batch-size variation — a local single-request setup is effectively fixed-batch, which is consistent with the 1.00 here. The same mechanism would also predict the temp-0 flutter Klishevich et al. saw on cloud APIs (variable batching), though they discuss other candidate causes as well — read this as consistency with one proposed mechanism, not a proven attribution.

Result 2: sample at temperature 0.7, and it falls apart

People raise temperature precisely to widen candidate coverage across runs. At temp 0.7 on the defect-seeded file (one seed per run, 0–50), run-to-run Jaccard was 0.52 [0.47, 0.59] for 7B and 0.07 [0.04, 0.11] for 1.5B.

Two distinct failure modes hide in those numbers:

Line drift (7B): 0.52 recovers to 0.80 when the identity key tolerates ±2 lines. Over half of the disagreement is consistent with the same bug pointed at a different line — though tolerance matching can also create accidental matches, so treat ±2 as an upper bound on "drift".
Category flutter (1.5B, and easy): on the easy file, 7B's line-only recall is 0.81 but category-strict recall is 0.42 — the model keeps finding the right place and labeling it a different problem.

File	Model	recall (category-strict)	recall (line-only)
target_a	7B	0.31	0.31
target_a	1.5B	0.05	0.18
easy	7B	0.42	0.81

(For 7B on target_a the two recalls coincide — those misses are genuine non-detections, not drift.)

And here's where I hit my first big mistake: my early recall numbers were great — because I had left # DEFECT: SQL injection annotations in the target code as notes to myself, and the whole file goes into the prompt. The model was reading my answer key. Deleting the labels cratered recall and deepened the run-to-run variance. Details in the pitfall log below.

Result 3 (the core): majority voting rewards persistence

Majority vote = "keep findings that appear in ≥26 of 51 runs." Here is what that actually selects.

Figure: votes per finding (blue = predeclared seeded defects, red = persistent findings outside GT, dashed = majority threshold 26/51). Counts are pooled from two target files run under identical conditions: the blue bars come from the defect-seeded file, the red bars from the decoy file.

The mechanism. With (line, category) as the identity key, the same bug pointed at a neighboring line becomes a different ballot. The model detected the ResourceLeak in 45/51 runs — but the votes landed 26 on line 16 and 19 on line 17 (no single run reported both lines, so 26+19 really is 45 distinct runs). Loosen the question to "did it say ResourceLeak at all, anywhere?" and it's 49/51 — scattered across six different lines (12, 15, 16, 17, 18, 21). The vote count never sees that. Watch the threshold:

Threshold k	line 16 (26 votes)	line 17 (19 votes)	Outcome
k=19	kept	kept	both survive (duplicated)
k=26 (majority)	kept	dropped	survives by one vote
k=27	dropped	dropped	a defect reported in 45/51 runs is gone

Under the exact majority rule I predeclared (k=26) the leak barely survives on one line; the point is not that majority voting killed it, but that whether it survives is governed by how the votes split and where you draw the line — not by the model's detection ability. And it's not one anecdote: 2 of the 5 defects that 7B detected at least once (~40%) showed line drift between runs.

Two contrasts sharpen the picture:

Unsplit real bugs sail through: the SQL injection on line 9 got 51/51 unanimous votes. Voting doesn't kill strong detections — it kills wobbly-keyed ones.
Grey findings survive: warnings against eval("2 + 2") (33/51) and subprocess.run(..., shell=True) with a constant string (28/51) — both low-harm on constants, both from the decoy file — persist because the model reacts to the pattern and points at exactly the same place every time. (Bonus wobble: it labeled the latter CodeInjection rather than CommandInjection — categories flutter too.)

The clean file adds a neat corollary: 7B stayed silent in 37/51 runs (72.5%), and when it did complain, it complained about something different almost every time — non-empty clean runs agree at just 0.12. (Counting silent–silent pairs as agreement 1.0 the unconditional mean is 0.53; I report the conditional number because silence agreeing with silence isn't informative here.) Inconsistent false alarms are exactly what majority voting removes. Consistent grey warnings are exactly what it cannot.

Threshold × key-granularity sensitivity (majority k=26):

File (7B)	w ±0	±2	±5	Reading
`clean`	FP 0	0	0	scattered false alarms die (as intended)
`decoy`	2 persistent	2	2	consistent grey findings survive
`target_a`	TP 2	2	3 *	* the +1 is a double-credit artifact (below)
`easy`	TP 2 / FP 1	3 / 2	3 / 1 †	looser keys rescue split votes — at a price

† The easy false-positive count is non-monotonic in w (1→2→1) because widening the window changes how findings cluster — adjacent clusters merge or get absorbed. This is a property of line-window clustering, disclosed rather than hidden.

📐 Why the ±5 "rescue" is double-counting, not detection At w ±5, the ResourceLeak cluster (mostly line 16) becomes wide enough to overlap **two distinct** ground-truth defects — a connection leak (lines 6–11) and a file-handle leak (lines 15–17) — so one cluster gets credited to both (the aggregation deliberately allows multi-credit; that design choice is exactly what this row exposes). To be precise: no run put a ResourceLeak vote inside the connection-leak lines themselves (6–11); the nearest was a single vote at line 12 — one line outside, and far below any threshold. So the "+1 true positive" at ±5 is a windowing artifact, not a detection. Loosening keys rescues split votes, but the rescue is not free: on `easy` it temporarily raises false positives (1→2 at ±2), and here it manufactures a detection that never cleared any threshold. No free lunch.

So: majority voting filters for salience of the pattern response, not truth. Persistent findings survive whether they're real bugs or grey smells; real bugs whose expression wobbles can fall. "Persistent" and "correct" are different properties, and the vote measures the first one.

Honest scope note: in this sample the core claim rests mainly on one drifting real bug plus two persistent grey findings — read it as an observed mechanism, not a general law. It is not an argument against aggregation per se — this is also not the self-consistency setting, which votes over normalized final answers; the failure mode here lives precisely in the un-normalized key. Change the key (category-only, AST node, defect region) and the behavior changes — which is itself the point: threshold × key granularity is a design decision with no free lunch.

What I actually do now: treat multi-run LLM review as a candidate pool, not an auto-verdict. Prioritize high-frequency findings for review, don't discard low-frequency findings in severe categories, and let tests, static analysis, and a human's understanding of the spec make the final call.

Everything I got wrong (measurement-instrument failures included)

The most useful part of this experiment was watching my own tooling and my own memory fail. Five mistakes, in the order I made them:

My answer key leaked into the prompt. I'd annotated defect lines with # DEFECT: ... comments as notes to myself — and the audit prompt includes the whole file. High recall was the model reading my answers. Removing the labels cratered recall and deepened variance. Never let the benchmark's answers be visible to the system under test.
I thought I was benchmarking CPU inference. It was 100% GPU. I had designed a thread-count variable (num_thread 1 vs 4) — then ollama ps showed 100% GPU offload, where num_thread is a no-op. My thread axis had measured nothing. Record your accelerator provenance instead of assuming it — the act of recording is what exposed the error.
I broke my own measuring instrument while "fixing" it. A one-line "optimization" to a clustering helper introduced a classic append-then-clear aliasing bug that silently corrupted the majority-vote true/false-positive counts. I caught it only because a hand count said 2 where the tool said 3. Every number quoted in this article's text is now recomputed by an independent script (verify_independent.py) that bypasses the aggregation code entirely; the full threshold×tolerance table and the bootstrap CIs still come from the aggregation pipeline. Fixes need verification most of all.
My "false positives" were real smells. I'd built decoy assuming its eval/shell=True-on-constants patterns were harmless, so surviving warnings counted as false positives. But Bandit and friends flag exactly these. The finding got stronger through the correction: the claim isn't "false positives survive voting" — it's "persistence survives voting, regardless of truth", and I renamed those findings "persistent findings outside GT".
My own memory fabricated a result. An early draft said "the md5 warning also survived the vote." The frozen records say md5 got 0 votes in 51 runs. My memory had synthesized it from the impression of pitfall #4. Caught in pre-publication log reconciliation. The experimenter's memory is not a primary source either — recheck the frozen records every time you write a number.

This is "don't trust LLM output" applied symmetrically: to the model, to my own instruments, and to my own claims.

Limitations

Small sample: N=51, two model sizes, one rig (RTX 4060 Ti / CUDA / Q4_K_M / Ollama 0.31.1). CPU, other GPUs, other quantizations may behave differently.
The temp-0.7 condition varies the seed per run (by design — that's what sampling the distribution means here), so it measures across-seed variability at that temperature, not a pure temperature effect with everything else held.
The core claim rests on few instances (one drifting bug, two grey findings). Mechanistically coherent, statistically thin.
"Smaller models wobble more" is a hypothesis from a 2-point comparison — capability and variance are confounded.
Category-strict recall is low; this is not a benchmark of absolute audit quality. Some ground-truth entries depend on runtime contracts (e.g., whether a config loader may return None) — they are predeclared seeded defects under this benchmark's stated assumptions, not universally adjudicated bugs.
Structured output (JSON-schema-constrained decoding) modifies the logit distribution; the variance measured here is variance under that constraint. Free-form review text may behave differently.
No causal claims. Scope: free, local, small models, this corpus.

Reproduce it

Generation is nondeterministic and not the reproduction target; the frozen per-run records are. The deterministic layer regenerates every table in this article:

git clone https://github.com/axiom-pro/llm-audit-nondeterminism
cd llm-audit-nondeterminism

# Layer 2 (deterministic, no LLM needed): regenerate all numbers from frozen records
python scripts/aggregate.py            # -> results/m2_summary.json
python scripts/verify_independent.py   # independent recomputation of every number quoted in the text — should print ALL PASS
python scripts/make_figures.py         # -> figures/nofreelunch.png (needs Matplotlib)

# Layer 1 (optional, needs Ollama + ~5GB VRAM for 7B Q4_K_M):
#   regenerating gives you a *different* distribution — that's the point
#   OLLAMA_EXE=/path/to/ollama bash scripts/run_m2.sh
#   python scripts/seed_probe.py       # confirm seeds have no observable effect at temp 0

Numerical aggregation and verification need Python 3.12 and only the standard library; figure regeneration additionally needs Matplotlib. A few seconds total. (On Windows, set PYTHONUTF8=1.)

Takeaways

Pin everything and local LLM review was deterministic on this rig — including deterministically repeating the same misses. Determinism ≠ correctness.
One review run is a sample, not a verdict. At practical temperatures the finding set moves between runs, mostly by line drift.
Majority voting measures agreement strength, not truth. Use multi-run review as a candidate generator; adjudicate with tests, static analysis, and humans.

If you remember one sentence, please don't make it "LLM code review is useless" — that's not what the data says. Make it: multi-run review is a decent candidate generator; what's unsafe is automatic adjudication over a brittle vote key.

All per-run records (normalized findings + response hashes), ground truth, aggregation code, and the independent verification script: github.com/axiom-pro/llm-audit-nondeterminism. Anything not measured (other GPUs, quantizations, larger models) is listed as not measured.

This is an English adaptation of my Japanese article on Zenn — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. If you spot an error, comments and issues are open; I'll verify against the frozen records and correct with a changelog.

About me: I build developer tools in Rust/Tauri and measure AI-assisted development practices with frozen records and independent recomputation — GitHub: axiom-pro.

DEV Community: tauridev