People paste screenshots into an LLM and ask "does this page follow the rules?". I wanted a number for how far that works, with the answer key coming from somewhere the model can't see: the DOM.
Gold first
Ten rules, frozen from my own site's style rules (the thresholds are mine, not a standard):
- R1 body text smaller than 15px
- R2 two different H2 sizes on the same page (judged per width)
- R3 body line length over 38em
- R4 h1 wrapping to 5+ lines at 375px
- R5 a 4-option radio group taller than 100px at 375px (wrapped)
- R6–R10 tap targets under 44px at 375px: TOC summary, TOC items, footer site-map links, the Works sub-nav, breadcrumbs
I built the site at the commit before the fixes (7bec67f), opened 12 pages at 1280 and 375 in headless Chrome via CDP, and measured every rule with getComputedStyle / getBoundingClientRect. That gives 12 × 11 = 132 judgments (R2 counts twice), of which 36 are "element not present on this page". 96 pairs remain.
From the same render, the script clips 1:1 crops of the elements in question — 164 PNGs, SHA-256 of each in a manifest. That matters: my first attempt measured positions in an iframe and shot with headless Chrome, and at 375px the two disagreed by 300–700px, so six footer crops were blank white.
// R1 = smallest font-size among text nodes inside <main>, excluding chrome
const EXCL = 'nav, .crumbs, form, button, .works-nav, .toc-block, footer, .site-header, .site-footer';
const textEls = [...document.querySelectorAll('main *')].filter((e) =>
!['SCRIPT','STYLE','SVG','PATH','NOSCRIPT'].includes(e.tagName) && !e.closest(EXCL) && vis(e) &&
[...e.childNodes].some((n) => n.nodeType === 3 && n.textContent.trim()));
Judges
Each judge got, per page, a frozen instruction ("for each rule answer YES / NO / unknown / not-applicable with one line of reasoning; a rule with no image is not-applicable; do not guess; at most three free remarks") plus that page's crops.
- GPT: pasted by hand into the ChatGPT UI, one page per submission, 12–15 images at once. The UI labelled the model "GPT5.6SOL"; the version was not captured.
- Gemini 2.5 Flash: via the REST API, temperature 0, one page per call (the web UI caps at 10 images).
Two different delivery paths, so this is not a model-vs-model comparison. Scoring is a regex over the four verbatim values against the gold — no human in the loop.
The mistake in my first tally
My first summary said "GPT 94% on 96 pairs, and it got all 35 of the 44px pairs right, telling 40px from 45px". An external reviewer looked at the gold and pointed out that in the before version, every breadcrumb is 40px and every footer link is 45px: within each 44px rule there are only violations or only non-violations. A judge that answers "small links violate, the footer doesn't" scores 35/35 without measuring anything. The same holds for R5.
So the main metric is now the 60 pairs whose rule contains both labels (R1–R4). The 44px rules were scored (GPT 36/36) but that score cannot test threshold discrimination, so it is reported and not counted.
Results (60 contrastive pairs)
Accuracy = correct YES/NO including true negatives, divided by all pairs (unknowns count against). GPT answered 60/60, so the two bases coincide; Gemini answered 57/60, so I also give the answered-only rate.
| Judge | Group | Pairs | Answered | Correct | FP rate | FN rate |
|---|---|---|---|---|---|---|
| GPT | contrastive R1–R4 | 60 | 100% | 54/60 = 90% | 7% | 21% |
| GPT | visual (R2, R4) | 36 | 100% | 36/36 | 0% | 0% |
| GPT | numeric (R1, R3) | 24 | 100% | 18/24 | 16% | 60% |
| Gemini | contrastive R1–R4 | 60 | 95% | 41/60 = 68% (41/57 = 72% of answered) | 26% | 36% |
| Gemini | visual | 36 | 97% | 28/36 | 12% | 44% |
| Gemini | numeric | 24 | 92% | 13/24 | 47% | 20% |
- All six GPT misses are R1: three pages at 13.5px answered NO, three pages at 15px answered YES. A 1.5px difference. R3 (38em vs pages at 18–33em, i.e. a 5–20em gap) was 12/12.
- Gemini's misses: six pages at 18–33em judged as "over 38em"; four false negatives and three false positives on mixed H2 sizes.
- Non-contrastive R5–R10 (36 pairs), for the record: GPT 36/36; Gemini answered only 56%, called the folded footer a "nav" five times and answered "not applicable" for elements that were in the image ten times.
- Reproducibility, one observation: the top page was submitted to GPT twice (pilot, then batch) and its R1 answer flipped YES → NO. The scored run is the second one.
Pre-registered hypotheses
- H1 "visual rules are caught by both judges" — GPT yes (36/36), Gemini no (FN 44%). Not supported as stated.
- H2 "numeric rules are unknown or wrong in more than half of cases" — on hold: only 24 contrastive numeric pairs exist (GPT 18/24, Gemini 13/24), the 44px rules are unmeasured, and the one discriminating rule flipped between two runs.
- H3 "free remarks are mostly outside the rules" — GPT wrote almost none (one remark in 12 pages); supported for Gemini.
Three questions before handing a check to a screenshot
- Does the gold for that rule contain both labels? If not, a perfect score means nothing.
- How far is the case from the threshold — how many px or em? A 1.5px gap failed even at 1:1.
- Does the answer survive a second submission of the same images?
What I'd hand to image review today: mixed heading sizes, heading line counts, line length with a multi-em gap. What stays in the DOM: anything within a couple of pixels of its threshold. What I can't say yet: whether image review discriminates the 44px tap-target threshold — the fixed version of the site needs to be shot to create contrast.
Part 3 of the series "Measure the Outcome, Not the Proxy" (part 1: transfer size ≠ speed; part 2: shorter ≠ requirements preserved — both on this account).
Verification record (environment, verdict, evidence: gold JSON, both score sheets, the 164 crops as a ZIP, all 24 verbatim answers) and the canonical Japanese write-up. The DOM measurer that fixed the pages. Reproduction kit with the images, the gold, the judge script for Gemini and the scorer (code MIT, data CC BY 4.0). I verify AI-generated and outsourced work against pre-registered pass/fail criteria — text-only, no calls: Sumitsuke / verification.
Top comments (0)