Exclusion Lists Managed by Name Always Rot. Mark the Population Instead.

Originally published on hexisteme notes.

I've written before about checks that pass for reasons that have nothing to do with what they claim to verify: a dead-code detector that flagged six constants that turned out to be unenforced specifications, not dead code, a detector whose zero false positives were indistinguishable from a stub's until a positive control showed it could actually fire, which it did eight times, a rule that evaluated cleanly because the field it read had no producer anywhere in the codebase and was always null. This one is a different shape from all three. The check itself was correct on the day I wrote it — it told articles apart from non-articles without any ambiguity. What broke wasn't the logic. It was the definition of the population the logic ran over, and it broke the same day.

The list that broke the same day it was written

I was building a check for a publishing pipeline: does every article on a hub site actually carry a conversion surface, or did one slip through empty? The check needed to walk every page and decide which ones counted as "articles" versus which ones were something else — an index page, a lead magnet, a policy page — that shouldn't be held to the same rule. My first pass did what most first passes do: a denylist of the slugs I already knew weren't articles.

_EXCLUDED_PAGE_SLUGS = frozenset({"privacy", "terms", "disclosure"})
...
return not (_EXCLUDED_PAGE_SLUGS & set(relative.parts))   # exact match

Three reasonable-looking strings. It was also already wrong, and the reason has nothing to do with which three strings they were — it would have been just as wrong with thirty.

The same day, a fourth static page

Later that same day I published one more static page: privacy-policy. The set comparison is an exact match, and privacy-policy is not privacy. The page sailed straight through the denylist, got counted as an article, and the check reported [PARTIAL] 13/14 — a false positive claiming a real article was missing a conversion surface it never needed, because it wasn't an article in the first place.

That's the more forgiving of the two directions a stale exclusion list can fail in. Under-exclusion produces false positives: a legitimate page gets permanently flagged, the check turns into background noise, and the next time it fires about something real, nobody's watching anymore because it already cried wolf. The other direction is worse. Over-exclusion produces false negatives — something that should have been checked quietly stops being checked, and nothing in the output says so. A list that's too narrow announces itself with noise. A list that's too broad doesn't announce anything at all.

The same bug, independently, in a different language

The part that made me stop treating this as a one-off typo: the identical defect already existed, independently, in a deploy script written in a different language against a different file:

case "$slug" in lead-magnets|privacy) continue ;; esac   # counts privacy-policy as an article, deploy halts

Nobody copied this bug from the Python check into the shell script, or the other way around. Two separate implementations of "which pages don't count as articles" reached for the same shape of answer: write down the names you currently know about. Both were correct on the day they were written. Both were wrong within that same day, for the same reason — a name list has no way to know that a name it hasn't seen yet belongs to the same category as the names it does know.

Asking what to include, instead of what to exclude

The fix wasn't a longer list, or a fuzzy match that would eventually hit the same wall with a different string. It was to stop defining the population by name at all. The site's renderer already leaves a mark on every real article and on nothing else: an application/ld+json structured-data block. I checked this empirically before trusting it — every one of the 13 real articles carried the block, and every one of the 9 non-articles (the index page, the lead magnets, both policy pages) didn't. Thirteen for thirteen, nine for nine. That split is the population definition. It doesn't need to know that privacy-policy exists as a string; it only needs to know whether a given page rendered the block, which is a fact about the page rather than a fact about whoever last updated the list.

Once the check reads the marker instead of the name, adding a tenth non-article page the next morning requires touching neither the Python file nor the shell script. The page either renders the block or it doesn't, and the check answers correctly either way — for a page that didn't exist when the check was written.

What this generalizes to

The move is to flip the question. "What should this check skip?" invites a list, and every list has a day it stops matching the world. "What does a real member of this population look like, from the object's own output?" invites a marker, and a marker doesn't go stale just because the set of instances grows.

Four things make that move stick instead of just sounding nicer:

• Define inclusion first, not exclusion. If there's any signal the object produces on its own — a class, a meta tag, a header line, a schema block — that a real member of the population always has and a non-member never has, that signal can replace the entire list.
• If no such signal exists yet, add one. A one-line marker the generator emits is cheaper to maintain forever than a name list is to keep updated even once.
• If a list is genuinely unavoidable, print how many items it excluded in every run's output. A silently stale list is invisible; a run that reports "excluded: 3" next to a population size that changed since last time is not. This applies to the marker approach too — its own failure mode is an item whose marker didn't render, dropping silently out of the population — so the excluded count needs to stay visible even after a marker has "solved" the problem.
• If the same rule is implemented in more than one place — a detector and a deploy check, in this case, in two different languages — use the identical marker in both, and leave a comment in each pointing at the other. Fixing one side without the other just lets the pair drift apart again, which is exactly what had already happened once.

Where this doesn't apply

This isn't a blanket argument against ever writing a list. It fails as advice in a domain where a marker can't attach reliably to every instance — where some real members of the population simply don't carry any distinguishing signal. There, switching to inclusion-by-marker just trades a visible false-positive problem for an invisible false-negative one, which is the worse trade. In that situation the right move is to keep the list and still hold onto the third rule above: expose the excluded count, so staleness is at least visible as a number instead of a silent gap.

It's also not worth the trouble when the excluded population is small and provably fixed — five or fewer items that can't grow because of something structural about the domain, not just "I don't expect more right now." A short, closed list is more readable than a marker nobody else on the team knows to look for. The argument here is about lists that grow by nature — page types, file categories, route names — not about enumerations of a genuinely finite, closed set.

A second shape, same day

The same day produced a second version of this failure in a completely different outfit. A regression test pinned only the sha256 hash of a build artifact — no copy of the artifact itself, just its checksum. The artifact it was supposed to compare against had quietly stopped existing, through three unrelated paths at once: the build directory was gitignored, the deployment process used an orphan commit plus a force push, and the live copy got overwritten on every redeploy. Each of those is individually a reasonable choice. Together they meant that the one time I actually needed to ask "is this difference the one I intended, or a regression," there was nothing left to diff against — only a hash with no file behind it.

It isn't the same mechanism as the exclusion list — nothing here is a name versus a marker — but it's the same family of mistake: a check that quietly stopped verifying anything, because the thing it depended on could disappear without the check noticing. The fix was to keep the actual golden file in the repository, so a failure produces a real, readable diff, and to keep the hash only as a secondary anchor for noticing if the golden file itself gets silently edited.

How you'd know this is wrong

The claim is specific enough to falsify. If a codebase's candidate markers turn out not to attach stably to every instance of the intended population — if genuine members sometimes render without the marker for legitimate reasons — then converting a name list into a marker check trades a visible failure mode for an invisible one, and that's a worse trade, not a better one. And if the excluded population genuinely can't grow past a handful of fixed names, the marker is solving a problem that was never going to occur. The 13/13 and 9/9 split is what made the marker trustworthy in this case; without a split that clean, the right fix is a list with its excluded count printed on every run, not a marker adopted on faith.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.