A ruler that cannot fail is not measuring anything

Numbers come from one directory site, anclum.com, in the 24 hours to 2026-11-11. Every figure has a command or file next to it at the end.

---

A companion post covers what went wrong: 307 pages judged by a database column instead of the text a reader sees, and 58 pages whose description the template never printed. Both bugs were in the same place — the ruler, the function that maps a row to thin or not thin. That verdict becomes a noindex tag, a line missing from the sitemap, a page Google never sees.

This post is about the part we had been skipping: how to know the ruler works before trusting its verdict.

The test has to be a case that must fail

A ruler that returns "not thin" for everything looks perfect on a site where most pages are healthy. A ruler that returns "thin" for everything also looks consistent. Neither throws. A happy-path test passes on both.

So we run negative controls. Four of them, on every change:

• Take a real page whose body prose is under 120 characters. The ruler must say thin.
• Inject a 400-character paragraph into the same page. The ruler must say not thin.
• Give it a long body made mostly of code and tables. It must not count navigation, badges, or boilerplate as prose.
• Ask it to judge a page that does not exist. It must fail loudly, not return a verdict.

The last one is not hypothetical. A missing file, an empty field, or a broken fetch can all produce an empty string. A ruler that treats empty as "short but valid" hides the page; one that treats empty as "fine" indexes nothing useful. An empty input that produces a verdict is the most dangerous case, because the verdict is stable and meaningless. Either way the output looks like a decision.

A rule that cannot change a verdict is worse than no rule

When the 58 pages flipped, the sitemap grew by 58 and the noindex set shrank by 58 — the same pages seen from two directions. We made that pair a rule: if one side moves without the other, stop and investigate.

Then we added three more pages to the allowlist and expected the pair to move by three. It moved by zero: all three pages rendered empty body text, so their verdict never changed. The rule had done its job — it turned a silent non-event into a number we had to explain. But the three entries themselves did nothing except make the next reader believe those pages were covered. We removed them (6a1ab14) and left the attempt in a comment, so the next reader would know it had already been tried and had not worked.

A rule that cannot change a single verdict is worse than no rule at all, because its only effect is to make a gap look handled.

The decision we did not make

After both fixes, 591 English detail pages are still excluded from search. That is not a number we can honestly drive to zero with one change. A 40-URL random sample of those 591 splits four ways: 14 whose English lead comes out empty because the row's text is not English, 25 that render less than 120 characters of body prose, 1 that renders 684 characters today but still carries the noindex tag written by the 05:10 build, and 0 that failed to resolve to a row — all 40 map to an items row. That is a sample, not a census of the 591; what it shows is a mix, and each part would need its own evidence.

The tempting move is to write a description for each one. It is an afternoon's work, and the result would be 591 pages that look fuller and say nothing — indexed pages whose only content is padding we invented.

We left them thin. If those entries get real material, it should come from a data source: an upstream README, a vendor description, a real submission. Not from us filling an empty shell to move a number.

What to copy if you run a generated site

1. Write down what each hiding rule actually reads. If the answer is a column name and not "rendered text," you have a batch of pages you have not looked at.
2. Keep one case that must fail, and run it on every change. A ruler that cannot fail is not measuring anything; it is only producing output.
3. Feed it an empty input and make it refuse to answer. A stable verdict on empty input is worse than an error.
4. Treat a rule that changes no verdict as a bug in the rule. If it made you add pages that still do not move, remove them and leave the attempt where the next reader will find it.

None of these would have been caught by a build log, a linter, or a monitoring alert. The site was green throughout. The pages were fine. The only broken thing was the question we asked about them.

---

Numbers and where they came from

#	Number	Command / file	Output
1	307	2026-11-08 rebuild after changing the thin rule's input — recorded at the time; not re-measurable now	noindex 10,613 → 10,306 (−307); sitemap 9,340 → 9,647 (+307)
2	58	prose58 allowlist render, 2026-11-11 00:18	sitemap 9,784 → 9,842 (+58); noindex 10,596 → 10,538 (−58)
3	365	derived: 307 + 58	365
4	120	threshold in _body_prose, build-frontend.py	len(_body_prose.strip()) >= 120
5	591	python3 -c "import json;ni=json.load(open('/opt/dsh-app/var/noindex-urls.json'));print(len([u for u in ni if '/en/s/' in u]))" — run 2026-11-11 17:07 CST; file mtime 2026-11-11 05:10:16	591 (591 unique URLs; 591/591 resolve to an items row via slug_of())
6	40	random.seed(11); random.sample(sorted(EN /en/s/ noindex URLs), 40) in work/publish/20261112/sample591.py — run 2026-11-11 17:07 CST against noindex-urls.json (mtime 05:10:16) + aggregator.db (mtime 17:02:10)	40 sampled, 40 unique, 40/40 resolve to rows
7	14 / 25 / 1 / 0	same script, one bucket per sampled URL, in this order: EN lead empty + non-Latin row text / rendered _body_prose < 120 / rendered ≥ 120 but still tagged / no items row	{"B_non_english":14,"C_under_120":25,"D_other":1,"A_no_item":0} — sum 40; D_other is one page with _body_prose=684, page mtime 05:09:37
8	3	three entries added to the allowlist in 0069426, removed in 6a1ab14 — 3 attempted, 0 took effect	58 → 61, then 61 → 58; each of the three measured PROSE=0
9	9,842 / 10,538	disk counts after the 58-page flip	grep -c '<loc>' /opt/site/sitemap-skills.xml → 9842; page-level content="noindex" count → 10538