DEV Community: Sumitsuke

I cut transfer by 81–257 KiB. Did LCP actually improve? 30 Lighthouse runs

Sumitsuke — Mon, 21 Sep 2026 08:32:18 +0000

"We made it lighter" and "we made it faster" are different claims, and a single Lighthouse run can't tell them apart — the last time I checked, repeated runs under the same conditions had a spread of more than a second. This is the acceptance sheet I ended up with, and the 30 runs behind it.

Setup

old = commit 88b702d: stale font subset version, two bold characters missing from the core subset, PNG images, no width/height, regular weight not split.
new = commit 1ba941a: versioned font URLs (?v=sha1), a build gate for bold characters, WebP, image dimensions, the regular weight split into core/rest, plus three rounds of visual fixes.
Both built with Astro from git worktrees on the same Windows PC and served with astro preview on :4321 / :4322.
Lighthouse 13.5.0, --preset=perf --form-factor=mobile --throttling-method=simulate, headless Chrome 153.
Three pages (home, one Lab article with two figures, the pricing page) × 5 runs × old/new = 30 runs, alternating old→new→new→old→… so time drift doesn't land on one side.
Hypotheses fixed before running: H1 home LCP goes down. H2 the article's CLS goes down (it got image dimensions). H3 pricing LCP goes down. Stop rule: if a difference is smaller than the 5-run interquartile range, the verdict is HOLD.

const SERVERS = { old: 'http://localhost:4321', new: 'http://localhost:4322' };
const ORDER = ['old', 'new', 'new', 'old', 'old', 'new', 'new', 'old', 'old', 'new'];
// Windows: chrome-launcher fails to delete its temp dir (EPERM) and exits 1 although the JSON is written —
// judge by the file's existence, not the exit code

Numbers

Transfer is bytes ÷ 1,024 (KiB), summed from Lighthouse's network-requests.

Page	Transfer old→new (KiB)	fonts	images	LCP median old→new	5-run min–max (old / new)	Δ	CLS
`/`	425→344	388→305	1→1	3,610→3,010 ms	3,460–3,610 / 3,008–3,012	−600	0→0
Lab article	678→421	359→275	276→101 (WebP)	4,806→3,459 ms	4,805–4,807 / 3,309–3,461	−1,347	0→0
`/works/pricing/`	395→314	359→275	1→1	3,313→2,859 ms	3,309–3,616 / 2,859–2,860	−454	0.001→0.001

FCP medians: 3,010→2,260 / 2,708→1,959 / 2,709→1,959 ms. TBT 0 everywhere.
Interquartile ranges (2nd vs 4th of the 5 sorted values — the stop rule uses this, not the min–max above): old 0–145 ms, new 1 ms. Simulated throttling is almost deterministic, so the differences are far outside the noise.
CLS did not move. Both versions sit at 0.000–0.001. The figures on the article are below the fold, so width/height never had a chance to show up in this metric. H2 is simply not observable this way.
LCP element is text on all three pages (h1, p.sub). The lcp-breakdown-insight audit puts it on the render-delay side (183–258 ms); what it was waiting for, the breakdown alone doesn't say.
benchmarkIndex (Lighthouse's host-CPU estimate) — per-page medians old→new: home 2,985→3,246, article 3,239→3,304, pricing 2,193→2,242. Higher on the new side for all three pages. Lighthouse's CPU slowdown is relative to host performance, so this is a bias in new's favour, and I don't attribute the whole difference to the change set. One observation the other way: within old's five runs on the home page, benchmarkIndex moved from 2,417 to 3,329 while LCP stayed within 3,460–3,610.

What went wrong

First run: chrome-launcher's temp-dir cleanup threw EPERM and the process exited 1 while the report JSON had been written. Fixed the runner to check for the file.
Lighthouse 13's perf preset no longer has largest-contentful-paint-element; the phases live in lcp-breakdown-insight.
The frozen "new" commit and the measured one differ. The design froze 098df1a; HEAD moved four commits (four Lab articles added) before I ran. The diff touches none of the three pages' HTML or CSS; the font woff2 files grew by a total of 2.6 KiB (ten bold characters added) — under 1% of transfer. I did not re-run; this line is the deviation record.

Verdict, and what the verdict means

H1 supported (−600 ms, outside the range). H3 supported (−454 ms). H2 not supported (old was already 0).
"PASS" here means: the pre-registered stop rule did not fire, the difference was observable. It does not mean the change set has been proven to cause the speed-up — the CPU index leaned toward new, and WebP, fonts and dimensions were changed together, so no per-change attribution is possible.
Not measured: field data. Simulated throttling estimates other conditions from one observed trace; whether real users got faster is a separate question for Web Analytics / Search Console / CrUX after enough days.

The three-column sheet

Column	This time	How to judge
Proxy metric	transfer −81 to −257 KiB	went down? (necessary, not sufficient)
Outcome metric	simulated LCP −454 to −1,347 ms; CLS 0→0	outside the reproduction range (5-run IQR)?
Stop rule	difference < IQR ⇒ HOLD	written before running

Put "bytes went down" and "LCP went down" in separate columns and sign off on each. Building both versions on one machine and alternating five runs each was enough here to separate the observed difference from the run-to-run spread; it is not enough to say what caused what.

Part 1 of the series "Measure the Outcome, Not the Proxy" (part 2: shorter ≠ requirements preserved; part 3: screenshot success ≠ threshold discrimination — both on this account).

Verification record (environment, verdict, the 30-run CSV, the LCP breakdown CSV and the runner) and the canonical Japanese write-up. What changed between old and new: the font-subset story and the image/metadata generators.

Can an LLM measure UI thresholds from screenshots? 164 crops vs DOM gold labels

Sumitsuke — Mon, 21 Sep 2026 08:31:50 +0000

People paste screenshots into an LLM and ask "does this page follow the rules?". I wanted a number for how far that works, with the answer key coming from somewhere the model can't see: the DOM.

Gold first

Ten rules, frozen from my own site's style rules (the thresholds are mine, not a standard):

R1 body text smaller than 15px
R2 two different H2 sizes on the same page (judged per width)
R3 body line length over 38em
R4 h1 wrapping to 5+ lines at 375px
R5 a 4-option radio group taller than 100px at 375px (wrapped)
R6–R10 tap targets under 44px at 375px: TOC summary, TOC items, footer site-map links, the Works sub-nav, breadcrumbs

I built the site at the commit before the fixes (7bec67f), opened 12 pages at 1280 and 375 in headless Chrome via CDP, and measured every rule with getComputedStyle / getBoundingClientRect. That gives 12 × 11 = 132 judgments (R2 counts twice), of which 36 are "element not present on this page". 96 pairs remain.

From the same render, the script clips 1:1 crops of the elements in question — 164 PNGs, SHA-256 of each in a manifest. That matters: my first attempt measured positions in an iframe and shot with headless Chrome, and at 375px the two disagreed by 300–700px, so six footer crops were blank white.

// R1 = smallest font-size among text nodes inside <main>, excluding chrome
const EXCL = 'nav, .crumbs, form, button, .works-nav, .toc-block, footer, .site-header, .site-footer';
const textEls = [...document.querySelectorAll('main *')].filter((e) =>
  !['SCRIPT','STYLE','SVG','PATH','NOSCRIPT'].includes(e.tagName) && !e.closest(EXCL) && vis(e) &&
  [...e.childNodes].some((n) => n.nodeType === 3 && n.textContent.trim()));

Judges

Each judge got, per page, a frozen instruction ("for each rule answer YES / NO / unknown / not-applicable with one line of reasoning; a rule with no image is not-applicable; do not guess; at most three free remarks") plus that page's crops.

GPT: pasted by hand into the ChatGPT UI, one page per submission, 12–15 images at once. The UI labelled the model "GPT5.6SOL"; the version was not captured.
Gemini 2.5 Flash: via the REST API, temperature 0, one page per call (the web UI caps at 10 images).

Two different delivery paths, so this is not a model-vs-model comparison. Scoring is a regex over the four verbatim values against the gold — no human in the loop.

The mistake in my first tally

My first summary said "GPT 94% on 96 pairs, and it got all 35 of the 44px pairs right, telling 40px from 45px". An external reviewer looked at the gold and pointed out that in the before version, every breadcrumb is 40px and every footer link is 45px: within each 44px rule there are only violations or only non-violations. A judge that answers "small links violate, the footer doesn't" scores 35/35 without measuring anything. The same holds for R5.

So the main metric is now the 60 pairs whose rule contains both labels (R1–R4). The 44px rules were scored (GPT 36/36) but that score cannot test threshold discrimination, so it is reported and not counted.

Results (60 contrastive pairs)

Accuracy = correct YES/NO including true negatives, divided by all pairs (unknowns count against). GPT answered 60/60, so the two bases coincide; Gemini answered 57/60, so I also give the answered-only rate.

Judge	Group	Pairs	Answered	Correct	FP rate	FN rate
GPT	contrastive R1–R4	60	100%	54/60 = 90%	7%	21%
GPT	visual (R2, R4)	36	100%	36/36	0%	0%
GPT	numeric (R1, R3)	24	100%	18/24	16%	60%
Gemini	contrastive R1–R4	60	95%	41/60 = 68% (41/57 = 72% of answered)	26%	36%
Gemini	visual	36	97%	28/36	12%	44%
Gemini	numeric	24	92%	13/24	47%	20%

All six GPT misses are R1: three pages at 13.5px answered NO, three pages at 15px answered YES. A 1.5px difference. R3 (38em vs pages at 18–33em, i.e. a 5–20em gap) was 12/12.
Gemini's misses: six pages at 18–33em judged as "over 38em"; four false negatives and three false positives on mixed H2 sizes.
Non-contrastive R5–R10 (36 pairs), for the record: GPT 36/36; Gemini answered only 56%, called the folded footer a "nav" five times and answered "not applicable" for elements that were in the image ten times.
Reproducibility, one observation: the top page was submitted to GPT twice (pilot, then batch) and its R1 answer flipped YES → NO. The scored run is the second one.

Pre-registered hypotheses

H1 "visual rules are caught by both judges" — GPT yes (36/36), Gemini no (FN 44%). Not supported as stated.
H2 "numeric rules are unknown or wrong in more than half of cases" — on hold: only 24 contrastive numeric pairs exist (GPT 18/24, Gemini 13/24), the 44px rules are unmeasured, and the one discriminating rule flipped between two runs.
H3 "free remarks are mostly outside the rules" — GPT wrote almost none (one remark in 12 pages); supported for Gemini.

Three questions before handing a check to a screenshot

Does the gold for that rule contain both labels? If not, a perfect score means nothing.
How far is the case from the threshold — how many px or em? A 1.5px gap failed even at 1:1.
Does the answer survive a second submission of the same images?

What I'd hand to image review today: mixed heading sizes, heading line counts, line length with a multi-em gap. What stays in the DOM: anything within a couple of pixels of its threshold. What I can't say yet: whether image review discriminates the 44px tap-target threshold — the fixed version of the site needs to be shot to create contrast.

Part 3 of the series "Measure the Outcome, Not the Proxy" (part 1: transfer size ≠ speed; part 2: shorter ≠ requirements preserved — both on this account).

Verification record (environment, verdict, evidence: gold JSON, both score sheets, the 164 crops as a ZIP, all 24 verbatim answers) and the canonical Japanese write-up. The DOM measurer that fixed the pages. Reproduction kit with the images, the gold, the judge script for Gemini and the scorer (code MIT, data CC BY 4.0).

I asked an LLM to cut 14 texts by 20–30%. Median: only 7%

Sumitsuke — Mon, 21 Sep 2026 08:31:43 +0000

I run a small site whose service pages are 1,100–1,800 Japanese characters each. I wanted them shorter, so I asked a model to cut them by 20–30% — and then, because the pages contain prices, limits and exceptions, I added "keep the numbers and the sentences that state conditions". My assumption was that the "keep" clause is what stops the model from reaching the target.

So I measured it instead of assuming. Fourteen documents, four instruction conditions, 56 outputs, each scored three ways.

Setup (frozen before generation)

Material: 14 pages from my own site, 1,149–1,797 characters after stripping front-matter, HTML tags, the closing CTA and whitespace. SHA-256 of each file frozen.
Protected atoms: every number + unit string in the source (154 of them: 「33,000 円」, 「3 つ」, 「1 営業日」…), as a multiset.
Boundary claims: 29 sentences that state a condition, a limit or an exception (hand-picked from candidates, frozen).
Conditions (nested, verbatim in the kit):
- 0 — "Shorten the following text by 20–30%. Output only the shortened text."
- A — 0 + "Do not change the structure of headings, paragraphs and lists."
- B — A + "Keep the numbers and the sentences that state conditions, limits and exceptions."
- C — B + "Lists may be converted to bullet points."
Model: claude -p --model claude-opus-5, text on stdin, run from an empty directory. Note that claude -p still carries Claude Code's own system prompt, so this is not a bare API call.
Hypotheses: H1 B compresses less than A. H2 A reaches −20% but drops an atom or a claim in more than half of the documents. H3 C compresses more than B.
HOLD threshold: I first ran 3 documents × condition A × 3 times. The same input landed anywhere from −3.5% to −20.8%; the median spread was 6.6 points. Any difference between conditions smaller than that is "not distinguishable".

Condition 0 was added after an external review pointed out that A–C all contain the "keep the structure" clause, so the main result could not be separated from it. The other 42 outputs were generated in randomized condition order (seed 20260919), one run per cell.

Scoring

Character change (whitespace removed).
Atoms: an atom is "lost" when it appears fewer times in the output than in the source. This is a string check — 「3 つの確認」 ("three checks") rendered as a three-item list still counts as lost. No meaning adjudication.
Claims: normalized-string containment for automatic pass; everything else was shown to a human with the closest output sentence (difflib) and judged paraphrased-kept / lost / inverted.
Structure: heading count and bullet count.

ATOM = re.compile(r'([\d,]+(?:\.\d+)?)\s*(円|件|本|日|時間|営業日|%|％|項目|段階|か所|回|人|社|通|枚|行|字|ミリ秒|秒|分|か月|つ|層|型|種|点|倍)')
lost = {k: c - out.get(k, 0) for k, c in src.items() if out.get(k, 0) < c}

Results

Condition	n	Median change	Quartiles	Min / max	Reached −20%	Docs that lost ≥1 atom string	Claims kept
0 no constraint	14	−6.5%	−7.5 … −5.1	−9.6 / −4.0	0	7	28/29
A keep structure	14	−6.9%	−8.2 … −5.4	−11.0 / −2.7	0	8	28/29
B + keep numbers & conditions	14	−8.2%	−10.3 … −6.5	−21.7 / −2.4	2	5	28/29
C + bullets allowed	14	−8.4%	−10.8 … −6.6	−21.1 / −3.7	1	7	28/29

Quartiles are the 4th and 11th of 14 sorted values; statistics.quantiles moves them by about half a point.

Differences between conditions: A−0 = 0.4, B−A = 1.3, C−B = 0.2 points. All inside the 6.6-point reproduction width. H1 and H3: HOLD.
H2 failed at its premise: A never reached −20% (0/14), and neither did the unconstrained condition 0 (0/14). The second half happened anyway — 9 of 14 A-outputs lost an atom string or a claim.
The three outputs that did reach −20% were all B or C. The "keep" clause did not prevent compression either.
What disappeared: counts — 「3 つ」, 「5 つ」, 「1 通」, 「3 か月」, 「10 件」. Every price (18 occurrences of 「…円」 across 6 documents × 3 conditions) survived.
Boundary claims, 29 × 4 = 116: 74 exact matches, 38 judged paraphrased-kept, 4 lost, 0 inverted. The 4 lost are the same sentence in all four conditions: 「合否条件を通した検証の結果と、確かめていない範囲も一緒に渡します」 ("we hand over the results of verification that passed the acceptance criteria, together with what was not checked") became 「検証の結果と…」 every time — the acceptance-criteria wording was dropped whether or not the instruction said to keep such sentences. (Its condition-0 row was first batch-judged as a paraphrase; a second review aligned it with A–C. The change is recorded in the published sheet.)
Heading and bullet counts were identical to the source in 56/56 outputs, including condition 0 where structure was never mentioned. Condition C's permission to convert lists into bullets was never used.

What I can and cannot say

Can: under these run conditions, a 20–30% request stopped at a median of −6.5 to −8.4% with or without constraints, and the constraint I suspected did not explain the shortfall. All 18 price strings survived while some count strings disappeared; structure was kept without being asked; and the acceptance-criteria phrase was cut from the same sentence in every condition.

Cannot: what actually sets the ceiling. Other models, other temperatures, a single-point target ("exactly −25%"), documents over 3,000 characters, or a second run of the same cell (n = 1 each; the pilot shows −21% and −7% for the same input).

The acceptance sheet I use now

Column	How to write it
Compression	character change (whitespace removed) ≤ −X%
Protected atoms	every `number + unit` string appears at least as often as in the source (string check only)
Boundary claims	every condition / limit / exception sentence survives, paraphrase allowed

"Shorter" is a proxy. What I wanted to know was whether the numbers and the conditions survived. Counting the two separately showed both things at once: the target was not reached, and things were still being dropped on the way.

Part 2 of the series "Measure the Outcome, Not the Proxy" (part 1: transfer size ≠ speed; part 3: screenshot success ≠ threshold discrimination — both on this account).

Verification record (environment, verdict, evidence links) and the canonical write-up in Japanese. Reproduction kit — the 14 source texts, the three scripts, all 56 outputs and the two adjudication sheets (code MIT, data CC BY 4.0).

I let an AI write a technical article with zero human editing, then ran it through my normal verification pass

Sumitsuke — Thu, 17 Sep 2026 04:29:22 +0000

I gave a model a folder of material and one sentence of instruction, and let it write a technical article end to end — topic selection, structure, prose, figures, and its own pre-publication checklist. Nobody edited a word. Then I froze that output and ran it through the verification pass I normally run on my own writing.

What I was measuring is not "is AI writing any good." It's what one verification pass actually removes.

Numbers first

11 findings on the six-type checklist I fixed before generation. One of them could recur in production.
4 more on operational rules the model was never given (publish flag, information I don't publish, AI-use disclosure, internal linking). Two of those were the serious ones.
Three external AI review rounds: 3 / 4 / 4 true new findings, and 1 / 1 / 2 false ones.
The same false finding appeared four times across separate sessions — twice inside the three rounds, and twice in two later passes (one that re-read A, one that checked the fixed version). Twice it was phrased as "a clear, reproducible error."
My own pass, before any AI review, found 0 of the 11.
Production time: 12 minutes. Verification: about 105 minutes.

Sample size is one article. I'm not generalizing from it.

The setup, fixed before generation


Material	3 files, 5,085 bytes: a topic memo, a measurement log (raw command output, no interpretation), and 13 lines of the source being discussed. Hashed.
Instruction	One sentence: write a publishable technical article from this material only; do the topic selection, structure, figures and pre-publication check yourself; invent no numbers that aren't in the material.
Environment	Fresh working directory, no project instructions anywhere up the tree, no user-level instruction file, settings sources restricted. Memory directory verified empty after the run.
Checklist	Six types, plus the definition of major/minor, written before generation and not given to the writing model.
A	The output, frozen (hash + commit). Never edited.
B	A copy of A with only confirmed fixes applied.

The six types: (1) wrong numbers or facts, (2) citation doesn't support the claim, (3) command or code doesn't reproduce, (4) internal contradiction, (5) generalization beyond what was measured, (6) conflating spec, observation and inference.

What the model did on its own

Twelve minutes produced an article, three figures, the scripts that generated them, and a self-check page. In that self-check it had executed the article's own reproduction snippet and compared the output, and machine-verified that the quoted source matched the material line for line. Mid-run it caught one of its own claims being wrong and rewrote it — I re-ran that case afterwards and the rewritten version is correct.

It also narrowed the topic without being told to. The memo listed five items; only four had real artifacts behind them. For the fifth, the material itself said "no run log exists," so the article put it under "what I did not cover" and published no number for it.

Where it actually broke

Not where I expected.

Type	Found in A	Left in B
Wrong numbers or facts	1	0
Citation mismatch	2	0
Doesn't reproduce	0	0
Internal contradiction	2	0
Generalization beyond measurement	3	0
Spec / observation / inference conflated	3	0
Total	11	0

Every number carried over from the material was right — 16 of them. The reproduction snippet ran and matched down to the exception's character offset. The quoted source matched the real file 10 lines out of 10.

Exactly one finding was "a number is wrong," and even that was a breakdown whose parts summed to 971 against a stated total of 973 — the material had only listed the top four categories and the article presented them as the whole set.

The other ten were all absences: a citation that was never given, a condition dropped (a regex claim stated without the qualifier that makes it true), a cause asserted from a grep that returned nothing, a conclusion wider than the range actually checked.

Accurate inside the material, expansive outside it. That was this article's failure mode.

The one that could recur

The article's subject was a guard that escapes invisible characters so a JSON line can't be split by the reader. The guard handled two characters.

Nobody — not the model, not me — asked how many characters can do that. The class isn't "invisible characters": it's the intersection of what the reader splits on and what the encoder leaves raw. For str.splitlines() and json.dumps(ensure_ascii=False) that intersection has three members. The third had been unguarded for four months, in the real tool, while every check stayed green.

That is not a wrong number. It's "sealed, probably." And the only thing that finds it is counting the members of the class, which is work that lives outside the guard.

I fixed the tool, then required red on a poison case for each of the three characters plus all three at once, before calling it closed.

The rules it was never given

Four more findings sat outside the checklist: the publish flag was left on, an identifier I don't publish appeared twice, there was no AI-use disclosure, and no link back to the canonical version. Two of those are the most expensive items in the whole list — they're the ones that cost you something after publication, not before.

I don't count them as model failures. None of those rules were in the material or the instruction. Charging a system for rules you never gave it measures your own setup, not its capability. They're recorded, separately, as an operations gate.

Three review rounds, and the interesting failure

I ran the frozen A past an external model three times: new session each time, same artifact, no B, no previous round's output.

	#1	#2	#3
True new findings	3	4	4
Repeats of earlier rounds	—	3	4
False findings	1	1	2

New true findings never hit zero in three rounds, so this experiment did not measure a stopping point. That's the result: not "three is too few" and not "four would find more" — I don't know where it saturates, because I stopped before it did.

The part worth your time is the false ones. Two of the four inside these three rounds were the same claim — and it came back twice more in later passes (a re-read of A, and the check on the fixed version), so the same misreading arrived four times in total, twice with language like "this is a clear, reproducible error, fix before publishing." The claim was that an escape sequence in the article's code was malformed and would break at runtime.

I pulled the exact line out of the file as bytes and executed it:

backslash count: correct
raw separator characters in the line: zero
evaluate the expression as written: one line out, value round-trips

The claim described a real failure mode — for code that wasn't in the article. Most likely an unescaping step somewhere in how the file was read.

Agreement across separate sessions is not evidence. Same model, same prompt, different session is not an independent trial; the errors correlate too. Neither is confident phrasing. The only thing that settled it was pulling the bytes and running them.

My own pass missed all 11

The order is: external review → I confirm each finding against primary sources and by executing things → a human adjudicates.

My own pass, run before any of that, found zero of the eleven. I verified that all 16 numbers matched the material — and never checked whether a breakdown summed to its own stated total. I read "it escapes the two characters" and never asked how many there were.

The reason is structural: I was checking claims against material, and every one of these lives in the gap between the material and the world. Checking and searching are different jobs, and I'd been doing only the first.

The fix also broke something

When I re-ran the checklist on B, it produced a finding of its own: I had moved a section during the fixes and left two references to the old state, so the same article said both "sealed" and "not sealed."

So the honest version of the summary isn't "11 → 0." It's:

detected in A: 11 (plus 4 operational)
of those, left in B: 0
introduced by the fixing: 1
undetected in B: unknown, and I'm not going to pretend otherwise

If you only count what you removed, the ledger flatters you.

If you want to run this

Freeze the material and hash it. The writing model gets the material and one sentence.
Fix the checklist before generation — types and the major/minor definition both.
Freeze the output. Never edit it; work on a copy.
Use AI review as a candidate generator. Truth is decided by primary sources and execution, not by the reviewer's confidence.
Keep rules the model was never given in a separate column. Don't charge it for what you didn't tell it.
Count three things: found, left, and introduced by fixing. The third one is the one people skip.

One line connects the article's own failure and the reviewers': the absence of an alarm was treated as evidence of absence. The guard was silent because it had never been handed the case; my pass was silent because I was checking claims rather than searching for the class; the reviewers were confident because agreement felt like verification. Each time, nothing firing was read as nothing being there.

What I'd take away, with one sample and no generalization: the writing was accurate where it had material and loose where it didn't, the expensive mistakes were operational rather than factual, and the review that mattered was the one that executed the code rather than the one that read it.

Verification record (environment, verdict, last verified date, evidence) and the canonical write-up (Japanese) — the frozen article A, the six-type checklist and the defect ledger: https://sumitsuke.jp/via/devto/lab/cost-of-one-verification-pass/ .

A JSONL record split in two: U+2028, U+0085, and the separator I missed

Sumitsuke — Wed, 16 Sep 2026 13:29:36 +0000

I keep a ledger as JSONL — one JSON object per line, written by my own importer, read by half a dozen small checkers. One day the checkers started reporting a line as unparseable, while json.loads on the same line was perfectly happy.

The character in the middle was U+2028 LINE SEPARATOR.

Why one line became two

Two facts, both documented, that only bite when they meet:

str.splitlines() treats U+2028 as a line boundary — along with \n, \r, U+2029, U+0085 and a few C0 controls (str.splitlines).
json.dumps(..., ensure_ascii=False) emits non-ASCII characters as themselves. U+2028 is not in the set JSON requires you to escape, so it stays raw (json).

So the file has exactly one \n per record, json.loads round-trips fine, and any reader that iterates with splitlines() sees a broken fragment.

import json

rec = {"id": 1, "body": "before\u2028after"}
line = json.dumps(rec, ensure_ascii=False) + "\n"

len(line.split("\n")) - 1     # 1   … one newline
len(line.splitlines())        # 2   … splits here
json.loads(line) == rec       # True … valid JSON the whole time

json.loads(line.splitlines()[0])
# JSONDecodeError: Unterminated string starting at: line 1 column 19 (char 18)

The fix I shipped, and why it was on the writer

There were several readers and one writer, so I fixed the writer:

def dumps_line(rec):
    return json.dumps(rec, ensure_ascii=False).replace("\u2028", "\\u2028").replace("\u2029", "\\u2029")

The invariant I should have written down on day one, and eventually did:

The serializer must not emit a raw code point that any of its readers treats as a line boundary.

Stated that way, the fix isn't "escape U+2028" — it's "keep the emitted set and the readers' boundary set disjoint," and the list of characters becomes something you derive rather than remember.

Three deliberate choices:

Keep ensure_ascii=False. Setting it to True would also solve this, but then every Japanese character becomes \uXXXX and the ledger stops being readable by eye. Being readable is the point of this file, so I targeted the separators only.
The value doesn't change. \u2028 is a standard JSON escape, so json.loads gives back the original character. I changed the representation, not the data.
Put the incident in the docstring. The reason a replace() like this exists is exactly the thing a future me deletes during cleanup.

Existing rows still held the raw character at that moment. They got fixed sideways: the ledger tool's update path rewrites every row through dumps_line, so the next update swept them. Counted today across all 973 rows: 0 raw U+2028, 0 raw U+2029, 0 raw U+0085. That the sweep happened at all was luck, not design — worth saying out loud.

Four months later, I counted

I wrote "escaping the two separator characters" in the docstring and in the article. Neither the code nor I had ever asked the obvious question: how many characters can actually do this?

Of the characters splitlines() splits on, the ones that survive json.dumps(ensure_ascii=False) as raw text are:

Character	Raw in `dumps` output?	`splitlines()` splits?
U+2028 LINE SEPARATOR	yes	yes
U+2029 PARAGRAPH SEPARATOR	yes	yes
U+0085 NEXT LINE	yes	yes
U+000B, U+000C, U+001C–U+001E	no (escaped as `\u000b`, …)	no

Three, not two. The C0 controls are harmless here precisely because json.dumps is required to escape them; U+0085 is above 0x20, so nothing escapes it for you.

My guard had been green for four months on a hole it was never given a chance to see. The cases it covered worked; the scope was incomplete.

def dumps_line(rec):
    _s = json.dumps(rec, ensure_ascii=False)
    for _ch, _esc in (("\u2028", "\\u2028"),
                      ("\u2029", "\\u2029"),
                      ("\u0085", "\\u0085")):
        _s = _s.replace(_ch, _esc)
    return _s

The check that would have caught it on day one

Not more test data. A positive control: feed in a case that must be caught and require the failure.

Injected	`splitlines()`	value round-trips
U+2028	1 line	yes
U+2029	1 line	yes
U+0085	1 line	yes
all three at once	1 line	yes

Before the fix, the U+0085 row returned 2 lines. That single red is the whole value of the exercise: a test that only ever passes has told you nothing about the cases you didn't hand it.

There's a pattern here that generalizes past Unicode. When you write a guard against a class of input, the number that matters is not "did it pass" but how many members of the class you enumerated. I had enumerated two out of three, and the guard could not tell me that, because enumeration is the part outside the guard.

What this doesn't cover

This is about the writer. A reader that uses splitlines() is still free to split on anything; I chose one place to fix instead of six.
Other producers writing to the same file (a different script, a hand edit) bypass it entirely.
The shape isn't Python-specific: the gap exists wherever a line splitter treats some code point as a boundary and the serializer emits that same code point raw. Which characters those are depends on the pair you're using — enumerate the intersection for your own splitter and encoder rather than copying my three.

What I'd do differently

When writing a guard against "characters of type X", print the enumerated list in the code, not in prose. ("\u2028", "\u2029", "\u0085") next to the replace is checkable; "the separator characters" in a docstring is not.
Add one poison case per member, and require red before the fix.
When the guard's scope changes, fix the message and the docstring in the same commit. Mine still said "only two" while the array said three, one revision later — the code was right and the text was lying.

Counted, not estimated: 973 rows with 0 raw U+2028 / U+2029 / U+0085 today, 3 characters that can split a line through this encoder, 1 of which was unguarded for four months.

Does AI-generated code silently swallow errors? 120 measured generations: every flagged case was a false positive or a documented fallback

Sumitsuke — Tue, 15 Sep 2026 12:32:16 +0000

If you gate AI-generated code with linters or a CI rule that hunts for swallowed errors, this experiment suggests the part you actually care about — "is this return None a contract or a cover-up?" — is exactly the part the rule cannot decide.

I started this project convinced that small LLMs routinely swallow failures — catch an error, return an empty value, pretend nothing happened — and that I'd measure the contamination rate. The hypothesis fell apart in a more interesting way than any clean number would have been. This article is the record of a prior being refuted by measurement, with the collapse documented step by step.

TL;DR

Setup: Qwen2.5-Coder 1.5B (Apache-2.0) via Ollama, CPU-only, Windows 11. 12 frozen tasks × 10 generations (seeds 0–9, temp 0.7) = **120 samples: 100 failure-path functions (50 Python, 50 TypeScript) + 20 pure-computation controls. A 7B robustness footnote adds 30 more (150 generations total, all frozen unfiltered in the repo). Classification: Python by AST, TypeScript by regex; **all 120 labels eyeballed, every swallow candidate and boundary case hand-adjudicated* against docstrings, comments, and the function's contract, published as gt.csv.*

A naive Semgrep detector — the kind you might reasonably drop into CI — flagged 4 candidates. Human adjudication: 0 true positives. Correction (2026-09-16): that is only the precision half. The corpus contains 4 samples adjudicated problematic_fallback (py_fetch_json_s2/s3/s7/s9), all of them guard_default, and the shipped rule set reaches none of them — recall 0/4 next to precision 0/4. Correction (2026-09-17): those two fours are different populations. Precision 0/4 mixes languages (2 TypeScript false positives + 2 Python documented fallbacks); recall 0/4 is Python-only. Per language: Python 2 flagged / 0 TP / recall 0/4; TypeScript 2 flagged / 0 TP / recall 0/0 — the TypeScript rows contain no adjudicated problem, so TS recall is undefined, not zero. Read the pair together: "4 flagged, 0 true positives" alone can be read as nothing was there to find, and the corpus says otherwise. Two were false positives (an empty catch in a usage example the model appended outside the function under test), two were legitimate fallbacks, documented (docstring or comment) as the function's contract. But this 0 is not "AI doesn't swallow errors" — my detector targeted try/except and structurally never looked at if-guard default returns, which is where the suspicious shapes actually lived (see finding 3).
In this sample, "fails loudly" and "no handling at all" dominated — not swallowing. Of Python's 50 failure-path generations, 39 (78%) used no try/except: 14 raise explicitly via if/else (loud), 4 return None behind an if guard, 21 just let exceptions propagate. TypeScript went the other way: 33/50 (66%) wrote try/catch with log and/or re-throw — proper handling. The only belief that collapsed was my prior that swallowing would be dominant.
The core finding: whether code "swallows errors" is not decidable from syntax. The same return None shows up as a documented, role-appropriate contract (parse_int: None if it cannot be converted) and as a hazard (fetch_json: 404, 500 — every non-200 collapses into the same None, while a network error raises instead, a third behavior the caller has to know about — with a comment saying so). A comment doesn't make it sound; a bare guard doesn't make it a bug. The deciding information — the function's role, the caller's expectations, the spec — lives outside the pattern. Static rules can surface candidates; they cannot deliver the verdict.
Honest caveats up front: one small model family, 12 tasks, N=10 per task (pseudo-replication — the independent unit is the task, not the generation), single-rater adjudication by me, and a detector whose scope hole I only noticed because the data rubbed my nose in it. Scope: free, local, small models, this corpus. No causal claims about "AI code vs human code" — that comparison belongs to prior work (AIRA, below), not to this experiment.

⚠ Reproducibility scope: generation is nondeterministic and ran once; all 150 generations are frozen unfiltered in the repo. What you reproduce is the deterministic analysis layer — same 120 files in, same distributions and candidate counts out. The final legitimate-vs-swallow labels are human adjudication, published transparently in gt.csv — and the fact that you can't regenerate that layer mechanically is itself the thesis of this article.

Why measure this

AI-generated code quality has a well-cited data point now: AIRA (arXiv:2604.17587, preprint, 2026) ran a deterministic, parser-backed static analyzer over 955 AI-authored and 955 human-authored code samples and reported 1.80× more high-severity findings in AI-authored code (0.435 vs 0.242 per sample), with Broad Exception Suppression (C03) — the "swallowed error" family — as the most frequent check (263 vs 185 in Study 3).

That number gets quoted as "AI code swallows errors." But read the paper closely and it says something more careful, twice:

Detection is deterministic and works — the paper even warns that an LLM evaluator misses these suppressions at a rate of 44:1 compared to the deterministic scanner. Pattern-matching is the machine's strong suit.
"Flagged ≠ defect." AIRA's own text states that some fail-soft patterns are contextually intentional and that human review is required before remediation. Two of its checks are permanently human-review-only.

So the open question isn't "can we detect suppression patterns?" (yes, deterministically). It's the step everyone skips: can the verdict — legitimate fallback or bug-hiding swallow — be automated too? I couldn't find a published measurement of that gap (pointers welcome). This article measures it with 120 locally generated functions, a naive detector, and every candidate opened by hand.

Scope declaration up front: this is neither "AI is dangerous" nor "static analysis is useless." It's a measurement of where the machine's jurisdiction ends.

Terms, before the numbers

Three things that look identical in grep output and must not be conflated:

Default return — on failure, return None / 0 / [] / "" instead of raising. A syntactic shape. Detectable.
Silent swallow — a default return that erases information the caller needed: failure becomes indistinguishable from "empty but fine." A semantic judgment.
Legitimate fallback — a default return that is the function's contract ("returns None if unparseable"), ideally documented. Also a semantic judgment.

The whole article is about the gap between the first item and the other two. Classification-wise this sits in CWE-703 (Improper Check or Handling of Exceptional Conditions), with CWE-1069 (empty exception block) and CWE-390 nearby; the exception-antipattern literature (de Pádua & Shang, arXiv:1704.00778, Java/C#) maps the same territory.

Why standard linters don't cover this (and what I built instead)

Checked against current docs, not from memory:

Python / ruff: S110 (try-except-pass) and S112 exist but are disabled by default — you must opt in via select/extend-select. And a typed swallow like except Exception: return [] sails through anyway.
TypeScript / ESLint: no-empty only fires on empty blocks, so catch (e) { return null; } is invisible — it's not empty. no-empty even allows a catch containing only a comment. (Note the cultural assumption baked in there: "has a comment" is treated as "is intentional." Whether the comment describes sound error handling is nobody's department. Hold that thought for the counterexample below.)

So I wrote a Semgrep rule that goes after the meaning-shaped pattern — the naive "before" version, verbatim from the repo:

rules:
  - id: py-swallow-return-default
    languages: [python]
    severity: ERROR
    message: "Exception caught and a default value returned without logging/re-raise (silent fallback)."
    patterns:
      - pattern: |
          try:
            ...
          except $E:
            return $R
      - metavariable-pattern:
          metavariable: $R
          patterns:
            - pattern-either:
                - pattern: None
                - pattern: "[]"
                - pattern: "{}"
                - pattern: "0"
                - pattern: "False"
                - pattern: '""'

  - id: py-except-pass
    languages: [python]
    severity: ERROR
    message: "Exception swallowed with pass (silent failure)."
    patterns:
      - pattern: |
          try:
            ...
          except $E:
            pass

(The TypeScript rules are the same shape — catch { return null }, empty catch — and ship in the repo.)

My hypothesis at this point: "run this over N generations, get a contamination rate." Here is where the measurement starts disagreeing with me.

Setup

Tasks: 12, frozen in tasks/tasks.json before generation. Per language: 5 failure-path I/O tasks (config loading, HTTP fetch, env vars, numeric parsing, file head) + 1 pure-computation control (mean/sum — no try needed). Prompts are neutral: no "handle errors" nudging.
Task-design disclosure (I did not farm for swallows): pilot runs showed that on plain tasks ("read JSON, return it") the model writes no try/except and nothing interesting happens — which means any "swallow rate" is heavily task-dependent and can be manufactured. I chose failure-inevitable I/O not to inflate the count (the result is ~0 anyway) but to give failure a stage — and kept the pure-computation controls to verify that off-stage, nothing appears. They did: all 20 controls showed zero try/except and zero candidates.
Generation: Ollama (MIT), CPU-only, qwen2.5-coder:1.5b (Apache-2.0), temperature 0.7, top_k=40 / top_p=0.9, seeds 0–9 per task, threads fixed at 4. The 7B model appears only in the robustness footnote (N=3 per task).
Determinism probe (before trusting anything): at temperature 0 with a fixed thread count, the same seed generated twice was byte-identical (SHA match). Distributions require temp > 0; hence 0.7.
Classifier asymmetry, disclosed: Python is classified via ast (a real parse); TypeScript via regex (not a full parser). Cross-language absolute comparisons carry instrument bias — which is why every label was eyeballed and every candidate hand-adjudicated rather than trusting classifier output.

Component	Choice
Inference	Ollama, CPU (Windows 11)
Model (main)	Qwen2.5-Coder 1.5B, default GGUF quantization (Q4_K_M-class)
Model (footnote)	Qwen2.5-Coder 7B
Detector	Semgrep 1.168.0 (CE) + custom rules above
Analysis	Python 3.12, ruff 0.15.12 for the lint baseline

Measured: 2026-07 (corpus generated 2026-07-01). Full per-run metadata in the repo's PROVENANCE file.

Result 1: mostly not swallowing — passing through, or failing loudly

How the 50 failure-path generations per language handled failure:

Failure-path tasks (n=50 per language)	Python	TypeScript
try/except + log or re-raise (proper)	9	33 (66%)
try/except returning a default (swallow candidate)	2	0
no try/except at all	39 (78%)	17 (34%)

Reading "no try/except = no handling" would be wrong. AST-splitting Python's 39:

14 raise via if/else (e.g. missing env var → raise ValueError(...)) — that's failing loudly, the opposite of swallowing.
4 return None behind an if guard — all four in fetch_json (non-200 → return None). A default return! Which my try/except-scoped detector structurally never saw. Remember these four; they're the article's best specimen.
21 bare — no failure path written; exceptions propagate to the caller.
(Check: 14 + 4 + 21 = 39.)

TypeScript's 66% try/catch majority mostly did console.error(...) + throw — textbook handling, no swallowing.

The language difference (Python avoids try/except, TS writes it) is an observation, not a finding: prompt phrasing, language idiom (async/await + try/catch is TS boilerplate), and the AST-vs-regex classifier asymmetry all confound it. The spine of this article is the counterexample below, not this table.

So the original hypothesis — "AI swallows failures at some rate N% I can report" — collapsed in the first table: in this sample, swallowing wasn't the dominant behavior at all.

Result 2: every detector hit was a false positive or documented-legitimate

The naive Semgrep rules flagged 4 of 120 (2 Python, 2 TypeScript). Opening all four by hand:

TypeScript, 2 hits (ts_load_config) = false positives. The function under test was exemplary (code condensed and annotated from the corpus):

async function loadConfig(path: string): Promise<any> {
    try {
        const data = await fs.promises.readFile(path, 'utf8');
        return JSON.parse(data);
    } catch (error) {
        console.error(`Error reading or parsing the file at ${path}:`, error);
        throw error; // logged and re-thrown -- not swallowed (proper)
    }
}
// ...but the model appended a usage example after the function:
(async () => {
    try { const config = await loadConfig('./config.json'); }
    catch (error) { /* Handle any errors... <- comment-only catch */ }
})();

The rule fired on the demo block's catch — (a) outside the function under test, (b) "empty" only because Semgrep's AST ignores comments. Context makes it an obvious false positive; the pattern alone can't know that.

Python, 2 hits (py_parse_int) = documented, legitimate fallbacks. The docstring states the contract — None if it cannot be converted (the second one says the same in a comment).

Bottom line: undisclosed swallowing inside try/except = 0 — and detected problems in the corpus = 0 out of 4 (Python; the TypeScript rows contain no adjudicated problem, so TS recall is 0/0). The first number is precision; the second is recall, and the article originally printed only the first. And now the honest part: my detector's own scope hole. The rules target try/except — the if-guard default returns (the four fetch_jsons) were never in scope. So the corpus does contain default returns; the accurate claim is "zero undisclosed swallows within the detector's scope," not "zero problematic fallbacks in the corpus." Whether those four are problems is precisely the question syntax can't answer — next section.

Result 3 (the core): the same `return None`, and what separates them isn't in the code

Two functions from the corpus (condensed and annotated). Which one swallows errors?

# (A) numeric parsing: None if unconvertible (contract stated in the docstring)
def parse_int_field(data, key):
    """Returns the int, or None if it cannot be converted."""
    try:
        return int(data[key])
    except ValueError:
        return None            # role-appropriate fallback

# (B) HTTP fetch: None on non-200 (and yes, there's a comment saying so)
def fetch_json(url):
    response = requests.get(url)
    if response.status_code == 200:
        return response.json()
    else:
        return None            # 404 or 500 -- every non-200 flattened into None
                               # (a network error raises instead: a third behavior)

Syntactically, the interesting part is near-identical: on failure, return None. Semantically they're opposites. (A) matches the function's role — "tell me whether this converts" — so None is the answer. (B) permanently destroys the caller's ability to distinguish "no data" from "the fetch failed." And (B) has a comment. Documentation doesn't settle it: a documented return 0 or return None can still silently poison every computation downstream. ("Has a comment = intentional" is exactly the assumption ESLint's no-empty institutionalizes.)

(B) is the strongest specimen this experiment produced: a documented-but-hazardous default return that appeared as a 4-sample cluster (not a one-off), sitting squarely in the blind spot of a try/except-scoped detector — while being exactly the "quietly fails" shape the AIRA numbers gesture at, at population level.

For contrast, non-swallowing code carries its intent inside the syntax:

def get_api_token():
    if 'API_TOKEN' in os.environ:
        return os.getenv('API_TOKEN')
    raise ValueError("The API_TOKEN environment variable is not set.")  # fails loudly

The point, stated carefully: syntactic patterns can surface candidates. The information that separates contract from cover-up — the function's role, the caller's expectations, the spec, and whether the documentation is right — lives outside the pattern. I'm not claiming "undecidable in principle": smarter analysis (types, dataflow, call-site analysis) absolutely narrows the candidates. But "what should this function return, in this context?" is a spec question, and the spec comes from outside the analyzer. Sharper tools shrink the pile; the final reconciliation against intent remains.

Honest scope note: this rests on a small corpus and a handful of specimens — read it as a demonstrated boundary, not a general law about all static analysis. What it demonstrates survives the small N, though, because it's an existence proof: two same-shaped snippets with opposite verdicts, and the verdict-relevant information demonstrably outside the syntax.

What I actually do now: machines generate the candidate list (patterns, distributions, CI notifications — semgrep scan --error exits 1 on candidates, which is fine); a human adjudicates candidates against role, caller, and spec. A green scan is read as "zero candidates for a human to look at," never as "pass."

Where this lands relative to AIRA

Neatly inside its own fine print, it turns out. AIRA detects suppression patterns deterministically and reports the 1.80× population-level difference — candidate generation is the machine's win, and the paper warns that semantic (LLM) evaluation underperforms there, 44:1. But the same paper marks the boundary: flagged patterns aren't necessarily defects, some fail-softs are intentional, human review precedes remediation, and two checks are permanently human-only.

That division of labor is what this experiment probed at specimen level:

Machines: sensitivity. Surface default-return shapes, chart try/except usage, gate CI with "candidates exist."
Humans: adjudication. Contract or cover-up — read the role, the caller, the spec.

One misquote I need to preempt, because I nearly published it myself (see pitfall 5): AIRA is not a human-evaluation study, and I am not claiming AI code swallows more than human code — that comparison is AIRA's, made with its own methodology, at population level, in a single-author preprint. This experiment neither confirms nor contradicts it; it maps the adjudication residue the paper explicitly leaves to humans.

Everything I got wrong

In the order I got it wrong:

"AI swallows failures; my rule will measure the rate." → Measured: in this sample, swallowing wasn't dominant — pass-through, loud failure, and (in TS) proper handling were. The contamination-rate framing died on the first table.
"The detector's 4 hits = 4 swallows." → All four were false positives or documented-legitimate. The honest number is 0, so I report 0 — with its scope. Don't round adjudication up to detection.
"The false positives just mean my rule is crude; refinement will fix it." → Refinement did kill the TS false positives. It could not touch the real problem: whether a return None is legitimate isn't decided by any pattern, or even by the presence of documentation. This is where the thesis flipped from "measure the rate" to "the verdict doesn't live in syntax." And auditing my own instrument surfaced the scope hole (if-guard defaults) it had from the start. When your tool's errors persist under refinement, suspect the question, not the regex.
The Windows encoding landmine. Semgrep read my TS rule file under Windows' legacy default codepage (cp932) and crashed with UnicodeDecodeError (exit 2) — on an em dash I'd left in a rule message. Fix: ASCII-only rule files + PYTHONUTF8=1 for every run. If your pipeline runs on Windows, pin UTF-8 explicitly or non-ASCII bytes in innocent places will cost you an afternoon.
Writing this article, I did the thing the article warns about. My research notes — AI-assisted — said AIRA judged suppressions "by human evaluation." Pre-publication, I went back to the primary source: it's the opposite. AIRA is a deterministic scanner, and it warns against relying on semantic evaluation for this class. I had a plausible summary, I trusted it, and a "verification discipline" article nearly shipped misstating its key citation's methodology. Corrected against the paper itself. Candidates can come from machines — or from your own notes. The final check against the primary source is the human step, and I almost skipped it.

Limitations

One model family, small models (1.5B main / 7B footnote), 12 tasks, one prompt style. Behavior is strongly model- and task-dependent; no generalization claimed beyond this corpus.
Statistical independence: N=10 generations per prompt are correlated pseudo-replicates. The independent unit is the task (5 per language), not the generation; don't read the n=50 tables as population estimates. The thesis rides on the counterexample, not the ratios.
Detector scope: rules and classifiers target try/except; if-guard defaults are out of scope, so "0 swallows" means "0 undisclosed swallows in scope" — the corpus contains default returns whose adjudication is the whole point.
Promise handlers (added 2026-09-20): promise .catch handlers are outside every pattern in the TypeScript rule file, braces or not — every pattern is anchored on a try { ... } catch clause. In this corpus that is 3 of 48 catch-shaped sites (45 try { ... } catch clauses plus 3 promise .catch handlers): ts_fetch_json_s0 and ts_load_config_s7 block-form, ts_fetch_json_s8 expression-form. All three console.error with no return, so 0 of 3 swallow silently; in each file the function's own catch logs and rethrows while the handler logs and stops, so these three are also the corpus's report-without-propagate sites. The numbers above do not move. promise.catch(() => null) — the form the corpus does not contain — is what the rule would miss. Raised by a reader, 2026-09-18.
Artefact gap (added 2026-09-16): in results/gt.csv the shape columns subtype and returns_default are filled for the Python rows only — 0 of 60 TypeScript rows carry them (their human_verdict is filled: 31 proper / 27 not_adjudicated / 2 false_positive). Anyone recomputing the cross-tab per language hits 27 unlabelled rows. Raised by a reader who re-derived the analysis from the repo. Partly closed 2026-09-17: for Python rows that have a handler, subtype is no longer blank — it is now guard_default / handler_default / both by where the default return sits (commit 82e5130), so the guard set (subtype in {guard_default, both}, 17 rows) is computable from gt.csv alone. Later the same day (commit f839873) the TypeScript rows got the same three shape columns via tree-sitter, with handling left as the regex verdict the tables above were computed with (the AST agrees on 58/60; the other 2 are the Semgrep false positives). What that exposes rather than closes: the TypeScript guard set is 11 rows (ts_get_item_s1-s9, ts_parse_int_s0/s5) and none of them has been human-adjudicated — their proper comes from the catch logging or throwing, not from anyone reading the default return outside it. TS recall stays 0/0 for that reason, not because the shape is absent — and, more to the point (reader, 2026-09-17), because the TypeScript half has no positive class by construction: the two halves share only 2 of 6 task families, and on fetch_json, the one family that carries all four Python problems, TypeScript is 10/10 proper. Read the TS half as a control arm, not a recall arm. Filling has_raise also moved the four throw-only TS samples from not_adjudicated to loud_fail (same machine rule as Python), so the TS split above now reads 31 proper / 23 not_adjudicated / 4 loud_fail / 2 false_positive.
Classifier asymmetry: Python AST vs TypeScript regex (nested-brace catches can slip). Human-verified labels backstop it, but discount cross-language comparisons.
7B robustness footnote (N=3/task, 30 generations): Python 10/15 no-try, 3 proper, 2 candidates; TS 8/15 no-try, 7 proper, 0 candidates. Both candidates: parse_int again, documented — one returns 0 on failure, which is documented and still capable of silently corrupting downstream arithmetic, i.e., the "(B) problem" one more time. N=3, so no robustness claim — only "not refuted in this range."
The adjudicator's hole, disclosed next to the detector's: the legitimate-vs-swallow verdicts are single-rater (me), no inter-rater agreement measured. Rubric: (1) does failure yield a default? (2) is that behavior disclosed as intent? (3) does it erase failure information the caller needs? Boundary cases ship with reasons in gt.csv. A second rater is future work — "the last step is human" cuts both ways, so I'm flagging my own last step.
Conflict of interest: I do AI-code auditing professionally. "The final call needs a human" is a conclusion that favors my line of work. Mitigation is transparency: frozen tasks, unfiltered generations, published adjudications, and the detector's failures disclosed by me, above.

Reproduce it

git clone https://github.com/sumitsuke/ai-silent-defect-scanner && cd ai-silent-defect-scanner
# Deterministic layer: same 120 files -> same distributions & candidate counts
PYTHONUTF8=1 python scripts/classify_split.py      # failure-path distributions (the n=50 tables)
PYTHONUTF8=1 python scripts/scan_and_count.py      # naive Semgrep candidates (the 4 hits)
PYTHONUTF8=1 python scripts/build_gt.py            # regenerate the adjudication layer, results/gt.csv
make scan-mine DIR=/path/to/your/repo              # candidates for YOUR repo (verdicts are on you)

Mechanical reproduction covers distributions and candidate counts. The final labels are my adjudications, published in results/gt.csv with reasons — by design not machine-rederivable (that's the thesis).
raw/ ships all 150 generations unfiltered; PROVENANCE records model, quantization, seeds, temperature, thread count, OS, dates. Regenerating gives you a different distribution — that's LLM sampling for you.
CPU-only, no GPU needed, no paid APIs anywhere in the pipeline.

Takeaways

Pattern rules find shapes, and the corpus's actual risk shape (if-guard default returns) wasn't the shape my rule watched. Budget for your detector's scope hole, not just its false-positive rate. Correction (2026-09-16): this takeaway originally covered two different boundaries, and only one of them is a defect. (a) The try/except scope is a hole — it costs recall: 100% of the corpus's adjudicated problems sit outside it. (b) The one-statement anchor (handler body is exactly return <default>) is a filter that buys precision — loosening it adds five samples (py_fetch_json_s6, py_first_line_s2, py_load_json_s4, py_load_json_s8, py_parse_int_s2), and all five are has_log=Y and adjudicated proper, 5/5. Keep (b), close (a).
Documentation is not adjudication. A commented return None and a return 0 documented in its docstring both appeared here as disclosed and still hazardous. "Has a comment" is a lint convention, not a verdict.
Split the jurisdictions and staff them accordingly: machine = candidates and distributions in CI; human = contract-vs-cover-up against role, caller, and spec. Green means "no candidates today," not "correct."

If you remember one sentence, please don't make it "static analysis is useless" — the detector did its detection job fine. Make it: syntax can surface the suspects, but conviction requires reading intent — and intent isn't stored in the AST.

Detection code, all 150 generations, and the human adjudication layer (gt.csv): github.com/sumitsuke/ai-silent-defect-scanner. Every number above is measured; anything not measured is labeled as not measured.

This is an English adaptation of my Japanese article on Qiita (Qiita is a Japanese dev-blogging platform) — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. If you spot an error, comments and issues are open; I'll verify against the frozen corpus and correct with a changelog.

Verification record (environment, verdict, last verified date, evidence) and the canonical write-up: https://sumitsuke.jp/via/devto/lab/ai-code-silent-fallback/ — code, data and reproduction: https://github.com/sumitsuke/ai-silent-defect-scanner.

Corrections

2026-09-16 — After publication, a reader re-derived the deterministic layer from the repo and pointed out that the report card was missing its recall half. I recounted from results/gt.csv and confirmed every checkable claim: 4 adjudicated problematic_fallback samples, all guard_default, none reachable by the shipped rules (recall 0/4); 9 of the 20 controls carry a default return; the five samples added by loosening the one-statement anchor are 5/5 has_log=Y and proper; the TypeScript rows carry no shape labels. One number I could not reproduce from the labels alone is their precision 4/17 — from gt.csv I reach 13 (returns_default=Y AND handling=no_try_except); the other four would have to come from reading the source. Changes above: the recall row in the TL;DR and in Result 2, the split of scope-hole vs anchor in Takeaway 1, and the artefact gap in Limitations. Nothing in the measured data changed.

2026-09-17 — The same reader came back with two things. First, a correction of their own: "precision 0/4, recall 0/4" reads like one 2×2 but is not — the 4 flagged are 2 TypeScript + 2 Python, the 4 problems are all Python, and TypeScript has no adjudicated problem at all (recall 0/0). I had printed the pair the same way; language is now attached in the TL;DR and Result 2. Second, the 4/17 I said I could not reproduce from the labels: they posted a 30-line AST detector (Return of a default value outside any except handler), and I ran it against raw/python/ — it fires on exactly 17 files, the four "extra" ones are py_fetch_json_s6:13, py_parse_int_s3:22, py_parse_int_s6:14, py_parse_int_s9:12, line numbers as they stated, and the verdicts on the 17 are 4 problematic_fallback / 2 legit_fallback / 2 proper / 9 controls (not_adjudicated) — recall 4/4, precision 4/17. The reason the labels could not reach those four is that gt.csv recorded whether a default return exists but not where; their proposed column closes it, and build_gt.py now writes guard_default / handler_default / both for every Python row that has a handler (of the four: py_fetch_json_s6 and py_parse_int_s3/s9 are both, py_parse_int_s6 is guard_default — its handler raises, and its only default return is in the else branch). Nothing in the measured data changed; one column was added — and, by evening, extended to the TypeScript rows (f839873), which turned the TS artefact gap from "no shape columns" into "11 guard-default samples nobody has adjudicated" (Limitations).

2026-09-17, later — The reader re-ran both fixes against the source (Python 17/60 and the four line numbers reproduce; the TypeScript guard set of 11 reproduces exactly) and added two things I had not stated. First, the "4 flagged" row is not one detector's output: the two Python hits come from py-swallow-return-default, the two TypeScript hits from ts-empty-catch (I confirmed this from scan_and_count.py's hit list); the return-default rule fires on 0 TypeScript files, because its one-statement anchor excludes all 9 TypeScript catches that log first — every one of them is console.error(...) then return null/undefined, and every one is proper. Second, the TypeScript half is best read as a control arm: it shares only two tasks with Python, and the task that carries all four Python defects (fetch_json) is 10/10 proper in TypeScript. (I originally wrote here that the TypeScript half "has no positive class by construction of the task sample" — withdrawn on 2026-09-18, see below; whether it has a positive class is an adjudication outcome, not a property of the sample.) The two outstanding rulings on ts_parse_int_s0/s5 remained open at this point. The reader asked to stay unnamed.

2026-09-18 — The same reader's third round. Two things. (1) The sentence "no positive class by construction of the task sample" was too strong and is withdrawn: the guard filter fires on the same two shapes in both halves; TypeScript's guard rows are 9 proper (machine label: the catch logs or throws) and 2 that were open. (2) I ruled the two open rows, ts_parse_int_s0/s5: no catch in either file, and the docstrings state the default ("it returns null" / "otherwise undefined") — the same rule that made py_parse_int_s3/s9 legit_fallback, so both are legit_fallback (commit in the repo, review_note filled). The TypeScript recall line therefore still has no positive to count, but the honest reading is now "0 of 2 opened, both documented", not "0/0, nothing to open". The Python half is not fully adjudicated either: with subtype ∈ {guard_default, both} and returns_default = Y it is 17 rows — 4 problematic / 2 proper / 2 legit / 9 not_adjudicated. One more correction of my own number in the comments: "49 of 60 TypeScript files end with a demo block" was the wrong reading (49 = files with a console.* call outside any catch); files whose last top-level statement is a console.* call are 22/60.

2026-09-20 — The same reader's fourth round: a scope hole the brace-shaped census could not see. Promise .catch handlers match nothing in the TypeScript rule file, with or without braces; in this corpus that is 3 of 48 catch-shaped sites (45 try/catch clauses plus 3 promise handlers), and all three are console.error with no return (0 of 3 swallow silently; all three log and stop where the function's own catch rethrows), so no number above changes. Stated in Limitations now, next to the if-guard hole, with promise.catch(() => null) as the one-line form the rule would miss.

2026-09-21 — Same reader, fifth round: the 48 had no instrument behind it and reads as 45 to anyone recomputing from the rule's anchor. It is now stated as 45 try/catch clauses plus 3 promise handlers, in Limitations and here; and the three handlers are named as what they are in this corpus — the report-without-propagate sites, where the function's own catch rethrows and the call-site handler logs and stops. No number moves.

LLM self-review vs. a blind re-run on 160 gold-labeled name-matching pairs — no measurable difference at ceiling

Sumitsuke — Sun, 12 Jul 2026 01:43:06 +0000

If you ship anything an LLM produced and calm your nerves by asking the same model to "please double-check it," this is a small, pre-registered measurement of whether that actually buys you anything.

Short version: on this task it bought nothing measurable — but mostly because the task turned out to be too easy, which is a lesson in itself about evaluation design.

TL;DR

Setup: gpt-5.4-mini, 160 gold-labeled "are these two records the same person?" pairs, pre-registered before the run.

The first pass was near-ceiling: 159/160 overall, 120/120 on the confirmation set. So there was almost nothing left to fix — I ended up measuring "does re-review preserve correct answers," not "does it fix errors."
Re-review (shown its own draft) tied a blind re-run. Across all 160 pairs both scored 156/160. On the pre-registered confirmation set, the single discordant pair gives exact McNemar p = 1.0 — no power to tell them apart, not proof of equivalence.
Reversals of correct answers happened in both conditions, equally (3 and 3). Not specific to self-review; consistent with plain run-to-run variation (single runs, so I can't causally isolate the extra pass itself).
Honest caveat: with the first pass this high, I could not test the thing I actually wanted to test. Treat this as the ceiling data point of a series, not a verdict on self-correction.

Reproducibility scope

This is one model, one prompt, reasoning_effort=low, single runs in a fixed order, on an 80/80 balanced diagnostic benchmark (60/60 confirmation + 20/20 hard). Nothing here generalizes to "LLM self-correction" in general. It's a concrete, reproducible measurement on one task, with all raw judgments and the pre-registration public.

Why measure this

"Self-correction" reports point both ways. Self-Refine (Madaan et al. 2023) reports large gains from iterative self-feedback; Huang et al. 2024 ("LLMs Cannot Self-Correct Reasoning Yet") and Kamoi et al. 2024 (a critical cross-task survey) find that without a reliable external signal, self-correction often doesn't help and can degrade. Those negative results are mostly on reasoning tasks, so I treat any agreement here as a cross-genre extrapolation, not a replication.

The practical version of the question: an LLM says "these two records are the same person / different people." Does asking the same model to re-check make it safer? Entity matching (deciding whether two given records refer to one entity) is easy to build gold labels for, so you can measure this without subjective judgment.

Setup

Task: match a Japanese footballer's Japanese name (kanji/kana) against a romaji name — e.g. is 久保建英 the same person as Takefusa Kubo?
Model: gpt-5.4-mini-2026-03-17, reasoning_effort=low. Measured: 2026-07.
160 pairs: a confirmation set of 120 (60 true-match / 60 non-match) plus 40 hard pairs.
Surface forms come from different sources (Japanese side = Wikidata, romaji side = a Transfermarkt-derived dataset), so club-name spellings and coverage don't line up perfectly.
The model never sees dates of birth, IDs, or URLs — only names and club history. Gold is verified against an independent attribute (DOB), not the linking ID.
Cost: 480 calls (160 × A/B/C), ~$0.45 total.

How I measured (pre-registered)

Three conditions:

A — first pass: judge the two records.
B — blind re-run: the byte-identical prompt, fresh context, no draft shown. ("Independent" here means A not shown, not statistical independence.)
C — two-stage review: the original data plus A's structured output as a draft, with a neutral "KEEP or CHANGE" instruction.

The pre-registered primary comparison is C vs B on the 120-pair confirmation set (balanced accuracy difference + a paired interval). C vs B measures the whole "hand the model its own output and ask it to re-review" workflow, not the anchoring effect in isolation.

Results

1. The first pass was already at the ceiling

A scored 120/120 on the confirmation set and 159/160 overall. That single fact reframes the whole experiment: there was essentially nothing to fix, so I measured maintenance near the ceiling, not error correction.

2. Re-review did not beat a blind re-run

Descriptively, across all 160 pairs B and C both scored 156. On the pre-registered confirmation set the 2×2 is: both correct 117, C-only-correct 1, B-only-correct 0, both-wrong 2. That single discordant pair (川辺駿 / Hayao Kawabe) gives exact McNemar p = 1.0 — which means "no power to distinguish," not "proven equivalent." The balanced-accuracy gap is +0.8 pt, but with one discordant pair the paired bootstrap lower bound is pinned at 0, so I don't lean on it. I don't claim equivalence either.

3. "Destruction" happened in both conditions, and it isn't self-review-specific

Reversals of A's correct answers, over all 160 pairs: A→B = 3, A→C = 3 (each 3/159 = 1.9%, Wilson 0.64–5.4%). On the confirmation set B breaks 3 and C breaks 2, but that flips on the hard set — subset-dependent. C's final verdict actually matches A on 157/160; the few it moved were all in the wrong direction, but with A at 159/160, "everything it moved was wrong" is close to arithmetically inevitable — not evidence C selectively destroys correct answers.

Deviations from the pre-registration (honest)

The pre-registered CI was a paired bootstrap (one of two registered options); it degenerated with a single discordant pair, so I report exact McNemar as an added, post-hoc auxiliary and don't lean on the interval. Wilson intervals were pre-registered for the secondary counts but missing from the first analysis script (added afterward). A pre-registered fame-stratified breakdown wasn't run (no clean popularity proxy). "Confirmatory" claims are limited to C-vs-B on the confirmation set.

Honest scope note: I did not measure human review, a different model, or any tool that checks the answer against ground truth. So I can't say external verification is better — only that handing the same model its own output didn't beat a plain re-run here.

What I actually do now: at this accuracy, I don't spend the extra pass. If I want a real second opinion, I reach for a different signal (a rule, a lookup, a second model), not the same model re-reading the same cues.

Everything I got wrong

My non-match pairs were too easy. I first built "same surname, different person" pairs, but the Japanese-side player was consistently older (mean 3.5-year gap), so the model could separate them on era alone. I rebuilt them to minimize the birth-year gap. Lesson: the benchmark author has to suspect their own benchmark is too easy.
My headline was inflated, and my own data killed it. I started to write "self-review destroys correct answers" — then noticed a blind re-run destroyed just as many. The mechanism was "a second pass at ceiling," not "self-review." Lesson: reconstruct the alternative baseline in your own data before you name a cause.
I almost claimed the hard cases needed club matching. 79 of 80 non-matches are actually separable by reading the given name. Only one truly needed an auxiliary attribute — and it's the one the model missed:

荒井悠汰 and a different person written Yuta Arai have identical romaji but different birth dates. Reading can't separate them; you need club history. That one homonym was A's only error (a single case, n=1) — and re-review didn't fix it. Lesson: the same model, re-reading the same cues, repeats the same miss.

Limitations

Ceiling effect (central). With 159/160 correct up front, "can it fix errors" is essentially untested; I measured preservation.
Memory contamination is unaddressed. These are public players; I can't separate "inferred the reading" from "recalled the person." The fame-stratified probe I pre-registered didn't run.
Same-model self-loop only. No external verifier, human, or ground-truth check. Not a Self-Refine replication (no separate feedback stage; one shot).
Benchmark representativeness. The confirmation set is a fixed 60/60 diagnostic split with hand-built non-matches; not a production match-rate distribution.
Single runs, fixed order, one model, reasoning_effort=low. The ceiling and the null may be specific to this low-effort setting.

Reproduce it

Repo: https://github.com/sumitsuke/llm-nayose-matching
PREREGISTRATION.md (committed before the run, d7a3328) → src/run_experiment.py → src/analyze.py recomputes every number from run_results.json. Result commits: fd42585, analysis add-ons 74a00b5.
Public: pair info, model judgments, analysis code, pre-registration. Withheld: DOB/verification attributes and non-redistributable raw data.

Takeaways

"Ask it again" isn't automatically safer. Whether re-review helps depends on how much room there is to improve and where the checking signal comes from.
The same model re-reading the same input isn't a second opinion. If the cues are identical, so are the blind spots.
Before measuring self-correction, make sure the first pass can actually fail. A ceiling task can't tell you whether re-review fixes errors — it can only show whether it preserves them.

Don't read this as "LLM self-review is useless." Read it as: on a near-ceiling task, an extra same-model pass showed no measurable value over a blind re-run — and the interesting failure was a homonym that re-reading can't catch. The next data point is a deliberately harder set where the first pass drops to 75–90%.

Repo and pre-registration above. I'm explicit about what I did **not* measure: no human, no second model, no ground-truth checker — so I make no claim about external verification here.*

This is an English adaptation of my Japanese article — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. Japanese original on Qiita (a Japanese dev-blogging platform): https://qiita.com/sumitsuke/items/e186a0fe97a8318a2e9c

Verification record (environment, verdict, last verified date, evidence) and the canonical write-up: https://sumitsuke.jp/via/devto/lab/llm-self-review-ceiling/ — code, data and reproduction: https://github.com/sumitsuke/llm-nayose-matching.

What actually breaks when a free Cloudflare D1 database hits its 500 MB wall

Sumitsuke — Thu, 09 Jul 2026 13:57:02 +0000

Every "free serverless DB" comparison ranks capacity, price, and cold-start. Almost none answer the question you actually hit in production: what happens at the free-tier wall? So I filled a Cloudflare D1 database to its limit over the REST API and poked it.

The honest short version: the wall is not a full read-only lockout, a plain DELETE clears it immediately (but that recovery is not quota-free), and — the part I got wrong twice — building an index on a blob column roughly doubles your storage, so it slams into the wall on its own. This post has been revised twice: my own size_after logs, and then a follow-up experiment, corrected me. I'll show both corrections instead of hiding them.

Scope: Workers Free, throwaway DBs, over the D1 REST API /query, 2026-07. The coarse wall was reproduced on a second, freshly-created database; the exact-ceiling pin and the index-doubling are each n=1 (flagged). This is a REST probe, not a Worker Binding probe — don't generalize endpoint-specific behavior (especially VACUUM).

"The free-tier wall" is not one wall

D1 has several walls, with different scopes and — per Cloudflare's own error list — different messages:

Wall	Scope	Value (Free)	Documented message
per-DB storage	per database	500 MB (Paid: 10 GB)	`Exceeded maximum DB size` — action: "delete rows / shard"
account storage	account	5 GB	`Your account has exceeded D1's maximum account storage limit…`
daily rows read	account	5,000,000/day	daily-limit-exceeded → "can't run queries"
daily rows written	account	100,000/day	same (INSERT, UPDATE, and DELETE all count as writes)

(Free daily counters reset at 00:00 UTC; storage is not daily-reset. Max row 2 MB; 100 columns; 10 DBs/account. All confirmed against the D1 docs on 2026-07-09.) I measured the first one.

Result 1 — the wall is 500 MB (decimal); pinned to one row on one DB, reproduced coarsely on a second

Filling with bulk inserts stalls around ~499.9 MB. To pin it, I then looped single-row inserts at the wall:

… 24 single-row inserts succeed (size_after climbs)
25th: HTTP 400  [{"code":7500,"message":"Exceeded maximum DB size"}]
last successful size_after = 499,994,624 bytes  (≈ 499.99 MB ≈ 476.83 MiB)

That ceiling — 499,994,624 — is 500,000,000 (decimal 500 MB) minus 5,376 bytes, enforced at SQLite's 4 KB page granularity (499,994,624 = 122,069 × 4096). I reproduced the coarse wall on a second, freshly-created DB: it stopped at the identical 499,884,032 and returned the same 7500. One caveat so you can hold me to my own logs: the redacted results/*.jsonl carry no account IDs, and served_by_colo is Anycast edge routing, not an account signal — so what's verifiable from the evidence is a second database, and the byte-exact 499,994,624 pin is from the single-row probe on one of them (n=1). The wall firing at all means these DBs are on Workers Free (Paid = 10 GB).

Result 2 — not read-only, but small writes aren't "exempt" either (measured)

My first draft claimed small writes slip past the wall while big ones don't. Wrong — and my own numbers show why. Those 24 single-row inserts above consumed only 499,994,624 − 499,884,032 = 110,592 bytes (~110 KB) of headroom before even a single-row insert was rejected. So:

Reads always pass — near-tautological for a storage cap; reads don't grow the DB.
Writes pass only while headroom remains. Small writes weren't bypassing the wall; they were filling the last sliver before it. Call it slack, not exemption. (One headroom-exhaustion sequence, but the 25th insert failing at zero headroom is a clean counterexample to "small writes are exempt" — one is enough.)

Result 3 — recovery is a plain DELETE, no upgrade (but it spends write quota)

DELETE WHERE k < 20000   → 19,998 rows; size_after 499.9 MB → 243,884,032 (~244 MB) immediately; rows_written = 19,998
bulk INSERT (after delete) → restored
VACUUM                    → 7500 "cannot VACUUM from within a transaction: SQLITE_ERROR" (maybe REST-specific)

DELETE drops D1's reported size immediately and writes resume — no VACUUM, no plan upgrade (recovery class R1). Two honest qualifiers: (1) it isn't quota-free — D1 counts DELETE as a write (docs: "Write operations include INSERT, UPDATE, and DELETE"), so this one 19,998-row delete spent ~20% of the daily 100k budget, and the cost scales with row count, not bytes freed (this delete freed ~256 MB ≈ 12.8 KB/row; whether it's linear per chunk I didn't separately measure); (2) size_after is D1's reported size — the meter dropping doesn't prove the physical file shrank (cf. workerd#1618, local).

Result 4 — CREATE INDEX doubles your storage (the thing I got wrong twice)

At the wall, CREATE INDEX fails with out of memory: SQLITE_NOMEM. My first draft lumped that in as "large allocation blocked"; after review I "corrected" it to "that's a memory limit, independent of the storage wall." Both were wrong.

On a control DB at 300 MB — well below the 500 MB wall — CREATE INDEX failed not with SQLITE_NOMEM but with 7500 "Exceeded maximum DB size" (storage!). Below the wall, a storage error? I measured the mechanism directly — deleted the control to 80 MB and indexed:

CREATE INDEX ON bench(v) at 84,029,440 (80.1 MiB) → OK
  size_after 84,029,440 → 167,923,712   =  ×1.998 growth (measured once, at ~80 MB)

An index on a blob column carries a copy of the column's data as its keys, so it roughly doubles storage — inferred from this one growth point, not proven in general. If that ×2 holds at other sizes:

storage before index	after (~×2)	result
80 MB	160 MB	✅ succeeds (under the wall)
300 MB	~600 MB	⛔ `Exceeded maximum DB size` (over the 500 MB wall)
500 MB (at the wall)	—	⛔ `SQLITE_NOMEM` (OOMs during the build)

So the index failure is storage-bound via the index's own footprint, not "memory, unrelated to the wall." A rough rule of thumb — if the ×2 holds — is that on D1 Free you can only index a column holding up to ~250 MB of data (2 × 250 = 500 MB), but that's an extrapolation, not measured near 250 MB. The SQLITE_NOMEM is just what you see when you're already at the wall.

The error code, and what I did not measure

Cloudflare's error list classifies these by message (Exceeded maximum DB size, …account storage limit…, …isolate memory limit…); the number 7500 appears nowhere in the docs. So 7500 is a generic REST wrapper — the message is the discriminator.

Not measured / n=1 / extrapolated (so not claimed): the index ×2 is a single measurement at one size/row-shape; the ~250 MB threshold is extrapolated (not measured near 250 MB); DELETE linearity across chunks is not measured (one 19,998-row delete); the account 5 GB wall, the daily read wall (5M/day — reads stop, account-scoped), and Worker Binding behavior are all untested; size_after behaves like a coherent logical meter (dropped on the one DELETE, ~doubled on the one index build) but the docs don't say physical vs logical. One full cycle (fill 39k + recover 20k + control 24k ≈ 84k rows_written) fits in one UTC day but not two.

Reproduce it

git clone https://github.com/sumitsuke/free-tier-quota-probe
cd free-tier-quota-probe
npm install && npm run selftest
# .env: CF_ACCOUNT_ID / CF_API_TOKEN (D1:Edit) / D1_DATABASE_ID / D1_CONTROL_DATABASE_ID / BILLING_SAFE=confirmed
node --env-file=.env bin/d1-admin.mjs create wallprobe   # create the DB via API (no dashboard needed)
npm run ping        -- --target d1
npm run fill        -- --target d1 --confirm-destructive
npm run probe-slack -- --target d1                        # pins the wall + measures the headroom
npm run fill        -- --target d1 --control --cap-mb 300 --confirm-destructive
npm run probe-index -- --target d1                        # the index-doubling test

Node ≥ 20, ESM, D1 over native fetch (60 s timeout + one retry). Redacted captures are frozen in results/*.jsonl; predictions (and the follow-up amendment) are in PREREGISTRATION.md. Repo: https://github.com/sumitsuke/free-tier-quota-probe

I write about measurement discipline — filling the wall instead of quoting the limit, and correcting my own assumptions on the record. This post's Results 2 and 4 are the live examples: I reasoned wrong twice and only the measurements got it right. Written with AI assistance; every number is from the captured `results/.jsonl`, and the doc facts were re-verified against Cloudflare's D1 docs on 2026-07-09. A Japanese version of this write-up also exists.*

Verification record (environment, verdict, last verified date, evidence) and the canonical write-up: https://sumitsuke.jp/lab/ — code, data and reproduction: https://github.com/sumitsuke/free-tier-quota-probe.

I had a local LLM review the same code 51 times: majority voting picks persistence, not correctness

Sumitsuke — Tue, 07 Jul 2026 11:48:35 +0000

If you use an LLM to review code — or run it several times and keep "whatever most runs agree on" — this experiment suggests the votes you're counting measure persistence, not correctness.

TL;DR

Setup: Qwen2.5-Coder 1.5B / 7B (Apache-2.0) via Ollama 0.31.1, single RTX 4060 Ti, Q4_K_M quantization. N=51 repeated audits of the same small Python files with predeclared seeded defects (temp-0 baseline: N=11 per condition). The per-run records (normalized findings plus a SHA-256 of each raw response), ground truth, and aggregation code are public — every number quoted in this article regenerates deterministically from the frozen records via an independent verification script.

Fully pinned = deterministic (on this rig). At temperature 0 with a fixed GPU and quantization, the finding sets were identical across runs — Jaccard similarity 1.00 in all four conditions (N=11 each). In a probe, six repeats within one seed and six different seeds all produced byte-identical output: our temp-0 configuration also forces top_k=1 (greedy decoding in Ollama/llama.cpp), so the seed had no observable effect. "LLMs are always nondeterministic" is folklore, at least under fully pinned local conditions.
Sampling at temperature 0.7 breaks agreement. On the defect-seeded file — 51 runs, one seed per run (0–50) — run-to-run Jaccard dropped to 0.52 (7B) and 0.07 (1.5B). For 7B, allowing ±2 lines of tolerance recovers 0.52 → 0.80: over half of the disagreement is consistent with the same bug pointed at a different line.
The core finding: majority voting selects persistence, not correctness. A real resource leak that the model detected in 45 of 51 runs split its votes across two adjacent lines (26 + 19 — no run reported both, so those really are 45 distinct runs). Under the strict line+category key, the majority threshold of 26 barely keeps one line; one vote higher, at k=27, a defect found 45 times is gone. Meanwhile, two "grey" pattern warnings (eval and shell=True on constants, from the decoy file) survived at 33/51 and 28/51, because the model points at exactly the same place every time.
Honest caveats up front: on the defect-seeded file, against predeclared ground truth with category + ±2-line matching, the 7B's per-run recall was only 0.31 — but precision was 0.98. Within this benchmark: when it speaks, it's right, and it misses a lot. The 1.5B was too noisy to carry any claim (it hit 25 of the file's 32 lines at least once across 51 runs), so it serves as a contrast group, not evidence.

⚠ Reproducibility scope: the existence and rough magnitude of the variance reproduces; exact numbers are rig-dependent (GPU / quantization / inference-engine version). The repo ships frozen per-run records — normalized findings plus a hash of each raw response; the raw response bodies themselves were not retained — and readers re-run only the deterministic aggregation layer.

Why measure this

"LLM-as-a-judge" is everywhere, and its known weakness is nondeterminism. Prior work measures score stability — Klishevich et al. (arXiv:2502.20747) measured scalar-verdict consistency over 70 commits at temp 0, and a Japanese industrial blog measured same-score rates on cloud APIs. What I couldn't find measured anywhere: whether the set of findings itself — which issues, on which lines — is stable across runs, on a local model, free to reproduce, with run records frozen and shipped. That's the gap this fills.

Setup

Models: Qwen2.5-Coder 1.5B and 7B (Apache-2.0), run locally via Ollama. Chosen because they're free, local, and require no API key — anyone can rerun generation.
Sampling design: temp-0 conditions use a fixed seed and force top_k=1, top_p=1.0; temp-0.7 conditions use one seed per run (0, 1, …, 50) with Ollama's default sampler otherwise (top_k=40, top_p=0.9; not swept). The temp-0.7 numbers therefore measure across-seed sampling variability at temperature 0.7 — not same-seed runtime nondeterminism (with a fixed seed, repeats would be trivially identical).
Output format: Ollama structured outputs (JSON-schema-enforced). Each finding is {line_number, category, severity, description}, category constrained to an enum.
Identity of a finding: the composite key (line_number, category). Paraphrases are ignored; two findings are "the same" iff they name the same place and problem type. I deliberately did not use embedding similarity — it happily merges "missing null check on line 12" with "missing null check on line 45".
Target files (4): target_a (7 seeded defects), clean (no defects — false-positive baseline), decoy (looks dangerous, low actual harm — this classification gets revised later, see pitfall #4), easy (6 obvious defects).
Ground truth: predeclared in a gt.csv outside the code before measurement. No hints in the code or the prompt — I learned this one the hard way (pitfall #1).
Two-layer reproducibility: layer 1 (generation, nondeterministic) ran once and its per-run records are frozen in the repo; layer 2 (aggregation, Python stdlib only) is fully deterministic and is what you reproduce.

Component	Choice	Notes
Inference	Ollama 0.31.1	local, free
Models	Qwen2.5-Coder 1.5B / 7B	Q4_K_M (verified via `ollama show`)
Hardware	NVIDIA RTX 4060 Ti (CUDA)	`ollama ps` reported 100% GPU offload during runs
Aggregation	Python 3.12, stdlib only	layer 2 fully deterministic

Measured: 2026-07. Per-cell provenance (model quantization/size/family, Ollama version, prompt/schema/GT hashes, backend) is recorded in the frozen records.

How I measured

Reliability = Jaccard between runs (no ground truth involved): matched findings / union, computed with optimal bipartite matching — naive greedy matching gives order-dependent values (a bug I actually shipped and had to fix). Reported as the mean over all run pairs within a condition, with a line-tolerance window w (±0 strict / ±2 / ±5) reported alongside — this doubles as the sensitivity analysis for the identity key.
Validity = precision / recall / false positives against GT. Unless noted, recall/precision are per-run means on target_a, category-strict, line ±2. Precision excludes empty runs by definition — and there were none on target_a (0/51); recall averages over all runs.
Majority voting = keep findings that appear in ≥ k of N runs. Examined on two axes: threshold k × key granularity w.
Uncertainty: 95% CIs from bootstrap resampling at the run level. One honest structural caveat: 51 audits of the same file are pseudo-replication — nothing here generalizes across codebases, and I don't claim it does.

Result 1: pin everything, and it's deterministic

All four temp-0 conditions (2 models × {defect-seeded, easy} files, N=11): Jaccard 1.00 at every tolerance level, zero variance across runs. In a separate probe on one condition, six repeats within a single seed and six different seeds (1, 2, 3, 101, 202, 303) produced byte-identical outputs — our temp-0 configuration also forces top_k=1, i.e. greedy decoding in Ollama/llama.cpp, so the seed had no observable effect there. Folklore says "LLMs always wobble"; under fully pinned local conditions on this rig, they didn't.

📐 The honest asterisk on "deterministic"

In a separate spike I did observe a single category flip at temp 0 — only ever on the first run after model load, which also had the longest latency (cold-cache/warm-up signature). With a warm-up run added, the residual disappeared. I can't pin the mechanism (GPU parallel-reduction ordering is the usual suspect; cold cache isn't ruled out), but it isn't the seed. This is consistent with Thinking Machines' "Defeating Nondeterminism in LLM Inference", which attributes serving-time nondeterminism primarily to batch-size variation — a local single-request setup is effectively fixed-batch, which is consistent with the 1.00 here. The same mechanism would also predict the temp-0 flutter Klishevich et al. saw on cloud APIs (variable batching), though they discuss other candidate causes as well — read this as consistency with one proposed mechanism, not a proven attribution.

Result 2: sample at temperature 0.7, and it falls apart

People raise temperature precisely to widen candidate coverage across runs. At temp 0.7 on the defect-seeded file (one seed per run, 0–50), run-to-run Jaccard was 0.52 [0.47, 0.59] for 7B and 0.07 [0.04, 0.11] for 1.5B.

Two distinct failure modes hide in those numbers:

Line drift (7B): 0.52 recovers to 0.80 when the identity key tolerates ±2 lines. Over half of the disagreement is consistent with the same bug pointed at a different line — though tolerance matching can also create accidental matches, so treat ±2 as an upper bound on "drift".
Category flutter (1.5B, and easy): on the easy file, 7B's line-only recall is 0.81 but category-strict recall is 0.42 — the model keeps finding the right place and labeling it a different problem.

File	Model	recall (category-strict)	recall (line-only)
target_a	7B	0.31	0.31
target_a	1.5B	0.05	0.18
easy	7B	0.42	0.81

(For 7B on target_a the two recalls coincide — those misses are genuine non-detections, not drift.)

And here's where I hit my first big mistake: my early recall numbers were great — because I had left # DEFECT: SQL injection annotations in the target code as notes to myself, and the whole file goes into the prompt. The model was reading my answer key. Deleting the labels cratered recall and deepened the run-to-run variance. Details in the pitfall log below.

Result 3 (the core): majority voting rewards persistence

Majority vote = "keep findings that appear in ≥26 of 51 runs." Here is what that actually selects.

Figure: votes per finding (blue = predeclared seeded defects, red = persistent findings outside GT, dashed = majority threshold 26/51). Counts are pooled from two target files run under identical conditions: the blue bars come from the defect-seeded file, the red bars from the decoy file.

The mechanism. With (line, category) as the identity key, the same bug pointed at a neighboring line becomes a different ballot. The model detected the ResourceLeak in 45/51 runs — but the votes landed 26 on line 16 and 19 on line 17 (no single run reported both lines, so 26+19 really is 45 distinct runs). Loosen the question to "did it say ResourceLeak at all, anywhere?" and it's 49/51 — scattered across six different lines (12, 15, 16, 17, 18, 21). The vote count never sees that. Watch the threshold:

Threshold k	line 16 (26 votes)	line 17 (19 votes)	Outcome
k=19	kept	kept	both survive (duplicated)
k=26 (majority)	kept	dropped	survives by one vote
k=27	dropped	dropped	a defect reported in 45/51 runs is gone

Under the exact majority rule I predeclared (k=26) the leak barely survives on one line; the point is not that majority voting killed it, but that whether it survives is governed by how the votes split and where you draw the line — not by the model's detection ability. And it's not one anecdote: 2 of the 5 defects that 7B detected at least once (~40%) showed line drift between runs.

Two contrasts sharpen the picture:

Unsplit real bugs sail through: the SQL injection on line 9 got 51/51 unanimous votes. Voting doesn't kill strong detections — it kills wobbly-keyed ones.
Grey findings survive: warnings against eval("2 + 2") (33/51) and subprocess.run(..., shell=True) with a constant string (28/51) — both low-harm on constants, both from the decoy file — persist because the model reacts to the pattern and points at exactly the same place every time. (Bonus wobble: it labeled the latter CodeInjection rather than CommandInjection — categories flutter too.)

The clean file adds a neat corollary: 7B stayed silent in 37/51 runs (72.5%), and when it did complain, it complained about something different almost every time — non-empty clean runs agree at just 0.12. (Counting silent–silent pairs as agreement 1.0 the unconditional mean is 0.53; I report the conditional number because silence agreeing with silence isn't informative here.) Inconsistent false alarms are exactly what majority voting removes. Consistent grey warnings are exactly what it cannot.

Threshold × key-granularity sensitivity (majority k=26):

File (7B)	w ±0	±2	±5	Reading
`clean`	FP 0	0	0	scattered false alarms die (as intended)
`decoy`	2 persistent	2	2	consistent grey findings survive
`target_a`	TP 2	2	3 *	* the +1 is a double-credit artifact (below)
`easy`	TP 2 / FP 1	3 / 2	3 / 1 †	looser keys rescue split votes — at a price

† The easy false-positive count is non-monotonic in w (1→2→1) because widening the window changes how findings cluster — adjacent clusters merge or get absorbed. This is a property of line-window clustering, disclosed rather than hidden.

📐 Why the ±5 "rescue" is double-counting, not detection At w ±5, the ResourceLeak cluster (mostly line 16) becomes wide enough to overlap **two distinct** ground-truth defects — a connection leak (lines 6–11) and a file-handle leak (lines 15–17) — so one cluster gets credited to both (the aggregation deliberately allows multi-credit; that design choice is exactly what this row exposes). To be precise: no run put a ResourceLeak vote inside the connection-leak lines themselves (6–11); the nearest was a single vote at line 12 — one line outside, and far below any threshold. So the "+1 true positive" at ±5 is a windowing artifact, not a detection. Loosening keys rescues split votes, but the rescue is not free: on `easy` it temporarily raises false positives (1→2 at ±2), and here it manufactures a detection that never cleared any threshold. No free lunch.

So: majority voting filters for salience of the pattern response, not truth. Persistent findings survive whether they're real bugs or grey smells; real bugs whose expression wobbles can fall. "Persistent" and "correct" are different properties, and the vote measures the first one.

Honest scope note: in this sample the core claim rests mainly on one drifting real bug plus two persistent grey findings — read it as an observed mechanism, not a general law. It is not an argument against aggregation per se — this is also not the self-consistency setting, which votes over normalized final answers; the failure mode here lives precisely in the un-normalized key. Change the key (category-only, AST node, defect region) and the behavior changes — which is itself the point: threshold × key granularity is a design decision with no free lunch.

What I actually do now: treat multi-run LLM review as a candidate pool, not an auto-verdict. Prioritize high-frequency findings for review, don't discard low-frequency findings in severe categories, and let tests, static analysis, and a human's understanding of the spec make the final call.

Everything I got wrong (measurement-instrument failures included)

The most useful part of this experiment was watching my own tooling and my own memory fail. Five mistakes, in the order I made them:

My answer key leaked into the prompt. I'd annotated defect lines with # DEFECT: ... comments as notes to myself — and the audit prompt includes the whole file. High recall was the model reading my answers. Removing the labels cratered recall and deepened variance. Never let the benchmark's answers be visible to the system under test.
I thought I was benchmarking CPU inference. It was 100% GPU. I had designed a thread-count variable (num_thread 1 vs 4) — then ollama ps showed 100% GPU offload, where num_thread is a no-op. My thread axis had measured nothing. Record your accelerator provenance instead of assuming it — the act of recording is what exposed the error.
I broke my own measuring instrument while "fixing" it. A one-line "optimization" to a clustering helper introduced a classic append-then-clear aliasing bug that silently corrupted the majority-vote true/false-positive counts. I caught it only because a hand count said 2 where the tool said 3. Every number quoted in this article's text is now recomputed by an independent script (verify_independent.py) that bypasses the aggregation code entirely; the full threshold×tolerance table and the bootstrap CIs still come from the aggregation pipeline. Fixes need verification most of all.
My "false positives" were real smells. I'd built decoy assuming its eval/shell=True-on-constants patterns were harmless, so surviving warnings counted as false positives. But Bandit and friends flag exactly these. The finding got stronger through the correction: the claim isn't "false positives survive voting" — it's "persistence survives voting, regardless of truth", and I renamed those findings "persistent findings outside GT".
My own memory fabricated a result. An early draft said "the md5 warning also survived the vote." The frozen records say md5 got 0 votes in 51 runs. My memory had synthesized it from the impression of pitfall #4. Caught in pre-publication log reconciliation. The experimenter's memory is not a primary source either — recheck the frozen records every time you write a number.

This is "don't trust LLM output" applied symmetrically: to the model, to my own instruments, and to my own claims.

Limitations

Small sample: N=51, two model sizes, one rig (RTX 4060 Ti / CUDA / Q4_K_M / Ollama 0.31.1). CPU, other GPUs, other quantizations may behave differently.
The temp-0.7 condition varies the seed per run (by design — that's what sampling the distribution means here), so it measures across-seed variability at that temperature, not a pure temperature effect with everything else held.
The core claim rests on few instances (one drifting bug, two grey findings). Mechanistically coherent, statistically thin.
"Smaller models wobble more" is a hypothesis from a 2-point comparison — capability and variance are confounded.
Category-strict recall is low; this is not a benchmark of absolute audit quality. Some ground-truth entries depend on runtime contracts (e.g., whether a config loader may return None) — they are predeclared seeded defects under this benchmark's stated assumptions, not universally adjudicated bugs.
Structured output (JSON-schema-constrained decoding) modifies the logit distribution; the variance measured here is variance under that constraint. Free-form review text may behave differently.
No causal claims. Scope: free, local, small models, this corpus.

Reproduce it

Generation is nondeterministic and not the reproduction target; the frozen per-run records are. The deterministic layer regenerates every table in this article:

git clone https://github.com/sumitsuke/llm-audit-nondeterminism
cd llm-audit-nondeterminism

# Layer 2 (deterministic, no LLM needed): regenerate all numbers from frozen records
python scripts/aggregate.py            # -> results/m2_summary.json
python scripts/verify_independent.py   # independent recomputation of every number quoted in the text — should print ALL PASS
python scripts/make_figures.py         # -> figures/nofreelunch.png (needs Matplotlib)

# Layer 1 (optional, needs Ollama + ~5GB VRAM for 7B Q4_K_M):
#   regenerating gives you a *different* distribution — that's the point
#   OLLAMA_EXE=/path/to/ollama bash scripts/run_m2.sh
#   python scripts/seed_probe.py       # confirm seeds have no observable effect at temp 0

Numerical aggregation and verification need Python 3.12 and only the standard library; figure regeneration additionally needs Matplotlib. A few seconds total. (On Windows, set PYTHONUTF8=1.)

Takeaways

Pin everything and local LLM review was deterministic on this rig — including deterministically repeating the same misses. Determinism ≠ correctness.
One review run is a sample, not a verdict. At practical temperatures the finding set moves between runs, mostly by line drift.
Majority voting measures agreement strength, not truth. Use multi-run review as a candidate generator; adjudicate with tests, static analysis, and humans.

If you remember one sentence, please don't make it "LLM code review is useless" — that's not what the data says. Make it: multi-run review is a decent candidate generator; what's unsafe is automatic adjudication over a brittle vote key.

All per-run records (normalized findings + response hashes), ground truth, aggregation code, and the independent verification script: github.com/sumitsuke/llm-audit-nondeterminism. Anything not measured (other GPUs, quantizations, larger models) is listed as not measured.

This is an English adaptation of my Japanese article on Zenn — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. If you spot an error, comments and issues are open; I'll verify against the frozen records and correct with a changelog.

Verification record (environment, verdict, last verified date, evidence) and the canonical write-up: https://sumitsuke.jp/via/devto/lab/llm-audit-51-runs/ — code, data and reproduction: https://github.com/sumitsuke/llm-audit-nondeterminism.

DEV Community: Sumitsuke

I cut transfer by 81–257 KiB. Did LCP actually improve? 30 Lighthouse runs

Setup

Numbers

What went wrong

Verdict, and what the verdict means

The three-column sheet

Can an LLM measure UI thresholds from screenshots? 164 crops vs DOM gold labels

Gold first

Judges

The mistake in my first tally

Results (60 contrastive pairs)

Pre-registered hypotheses

Three questions before handing a check to a screenshot

I asked an LLM to cut 14 texts by 20–30%. Median: only 7%

Setup (frozen before generation)

Scoring

Results

What I can and cannot say

The acceptance sheet I use now

I let an AI write a technical article with zero human editing, then ran it through my normal verification pass

Numbers first

The setup, fixed before generation

What the model did on its own

Where it actually broke

The one that could recur

The rules it was never given

Three review rounds, and the interesting failure

My own pass missed all 11

The fix also broke something

If you want to run this

A JSONL record split in two: U+2028, U+0085, and the separator I missed

Why one line became two

The fix I shipped, and why it was on the writer

Four months later, I counted

The check that would have caught it on day one

What this doesn't cover

What I'd do differently

Does AI-generated code silently swallow errors? 120 measured generations: every flagged case was a false positive or a documented fallback

TL;DR

Why measure this

Terms, before the numbers

Why standard linters don't cover this (and what I built instead)

Setup

Result 1: mostly not swallowing — passing through, or failing loudly

Result 2: every detector hit was a false positive or documented-legitimate

Result 3 (the core): the same return None, and what separates them isn't in the code

Where this lands relative to AIRA

Everything I got wrong

Limitations

Reproduce it

Takeaways

Corrections

LLM self-review vs. a blind re-run on 160 gold-labeled name-matching pairs — no measurable difference at ceiling

TL;DR

Reproducibility scope

Why measure this

Setup

How I measured (pre-registered)

Results

1. The first pass was already at the ceiling

2. Re-review did not beat a blind re-run

3. "Destruction" happened in both conditions, and it isn't self-review-specific

Everything I got wrong

Limitations

Reproduce it

Takeaways

What actually breaks when a free Cloudflare D1 database hits its 500 MB wall

"The free-tier wall" is not one wall

Result 1 — the wall is 500 MB (decimal); pinned to one row on one DB, reproduced coarsely on a second

Result 2 — not read-only, but small writes aren't "exempt" either (measured)

Result 3 — recovery is a plain DELETE, no upgrade (but it spends write quota)

Result 4 — CREATE INDEX doubles your storage (the thing I got wrong twice)

The error code, and what I did not measure

Reproduce it

I had a local LLM review the same code 51 times: majority voting picks persistence, not correctness

TL;DR

Why measure this

Setup

How I measured

Result 3 (the core): the same `return None`, and what separates them isn't in the code