Your boilerplate filter is a share threshold, and that quietly protects the majority

A similarity guard's background frame is not monotone in its population size. The experiment that refuted that assumption also produced something more useful: a one-line law that says why a filter like this misclassifies exactly the people who write least.

Every number below is read from the package's committed result files, not from the manuscript prose or from notes.

The guard

Many duplicate detectors, plagiarism filters and near-duplicate moderators share a two-stage shape. First a similarity gate, then a "how much of the overlap is actually meaningful" gate. The second one needs a list of units that are too common to count — boilerplate, licence headers, template clauses — and that list is a background frame computed over some population:

fire(a, b)  iff  dice(a, b) >= tau   AND   |shared(a, b) - F_l| >= r

F_l = { u in U : df_l(u) >= theta }
df_l(u) = (# documents in population l containing u) / |population l|

Set tau = 0.5, r = 10, and l is the level the frame is computed at — a whole corpus, or one community inside it.

Two properties of this construction are worth writing down, because neither is usually stated.

The frame acts on the residue gate only. The similarity gate is computed on the raw unit sets, so it is level-independent; two levels differ only in which shared units get removed. If you attribute a difference in flags to "similarity", you have skipped a decomposition. (The first version of my instrument did exactly that, and it took a per-gate decomposition to see it.)

The frame is a threshold on a share, not on a count. A unit appearing in 90% of one community's documents has a corpus document-frequency of 0.9 * s, where s is that community's share of the corpus. If that product sits below theta, the unit is not in the corpus frame at all — the community's own boilerplate is invisible to the global filter.

The claim I registered, and what killed it

I registered a set-containment identity: F_corpus ⊆ F_pool, with residue monotonicity as its consequence. Before building anything on it I measured it, on 14,684 documents drawn from eight SHA256-pinned books, recomputing the frame at two levels (whole corpus, one book as the pool) for every book and both statistics.

It fails, and it fails in both directions at once:

cells where F_corpus ⊄ F_pool : 24 of 24
cells where F_pool   ⊄ F_corpus : 24 of 24

Not marginally, either: a corpus-generic unit absent from the pool occurs 21 to 125 times per cell, and a pool-generic unit absent from the corpus frame 9 to 82 times. Containment was simply the wrong object — the two frames are not nested, they are two different samples of the same share quantity.

The replacement is exact

What does hold is a dilution law. Take a population P of share s = |P|/N, and a unit u confined to P — every corpus document containing u lies in P. Then the document counts coincide, c_corpus(u) = c_P(u), and only the denominators differ:

df_corpus(u) = s * df_pool(u)      (exactly)

Measured, restricting to confined units:

statistic	confined pairs	worst |df_corpus − s·df_pool|	pool-generic but corpus-invisible
char bigrams, θ = 0.01	318	4.3e-19	7
word-3 shingles, θ = 0.01	685,502	8.7e-19	35

That is floating-point noise on 14.7k real documents — the relation is an identity, not a fit. The worst word-3 case is instructive: the unit is the white whale, generic inside one book and invisible to the corpus frame.

And it is a regression on s, not a claim about a particular corpus. Pooling the first k books and sweeping k, the fraction of pool-frame units the corpus frame misses:

pool share	word-3 hidden	char-bigram hidden
0.1345	88.6% (39 of 44)	3.9%
0.2652	76.0%	2.6%
0.5452	68.2%	4.9%
1.0000	0.0%	0.0%

The last row is the control: when the pool is the corpus, both frames are the same set by construction and the hidden fraction is zero. The effect is also statistic-dependent — 7 bigram units against 35 word-3 units at the same threshold — which becomes the whole story two sections down.

The misclassification is a function of who is writing

Put the guard on a five-language corpus (7,521 documents: 3,200 English, 1,354 Italian, 1,767 Spanish, 800 French, 400 German) and plant a register: a recurring block a community shares — a greeting, a header, a template clause — carried by a fraction phi of that community's documents. Its corpus prevalence is s * phi, so the prevalence at which a corpus-wide frame becomes able to remove it is

phi* = theta / s

At word3, tau = 0.5, theta = 0.05, with the Italian community as the smaller stratum (share 0.180) and English as the larger (0.425):

register prevalence φ	Italian: corpus-frame fires	English: corpus-frame fires
0.00	0 / 120	0 / 120
0.05	120 / 120	16 / 120
0.20	107 / 120	0 / 120
0.50	0 / 120	0 / 120
1.00	0 / 120	0 / 120

The community-level frame fires 0 of 120 in every one of those cells. So the same register, at the same prevalence, is missed entirely in the minority community and caught entirely in the majority — and the reason is arithmetic, not tuning. At phi = 0.20 the register's corpus prevalence is 0.036 in the Italian community (hidden, 107 pairs of fires induced) and 0.085 in English (visible, 0 induced).

The critical prevalence is measured by bracketing, not assumed: the Italian community flips between a prevalence at which the register is hidden and one at which it is visible, bracketing the predicted 0.278; English brackets 0.118. The prediction is checked exactly against theta / s in all twelve (statistic × stratum × threshold) rows of the artefact, which is what you would expect from an identity and would not from a fit.

That is what makes a mis-set frame a fairness defect rather than a tuning knob: the band of blindness (0, phi*) is wider for the smaller community, so the filter silently protects the majority's boilerplate and quietly flags the minority's.

The statistic decides whether your community has a stratum at all

The certificate s * phi < theta <=> register hidden from the corpus frame holds in 48 of 48 word-3 cells. On character bigrams it fails in all 48 — 0 of 48 predicted-hidden cells are actually hidden.

That failure is not a defect, it is the premise: character bigrams recur across languages, so they are not confined to a community, so their corpus frequency does not dilute, so they stay inside the corpus frame at any share. An analyst who picks the statistic is picking whether their community is visible at all — and nothing in the API of a dedup pipeline tells them they made that choice.

Two controls keep this from being "everything is a level effect". A verbatim repost and a 1%-edited repost (Dice 0.990) must fire under every level including the corpus frame; both fire in all three threshold cells. The level cannot remove a genuine copy — which is exactly what separates a register effect from a similarity effect.

The residue threshold has a boundary, and the zero above it is derived

Widen the grid to 1,980 cells (2 strata × 2 statistics × 6 prevalences × 3 frame thresholds × 3 similarity thresholds × 5 residue thresholds × 2 sampling rules) and sweep r.

The level-induced count (corpus-frame fires minus community-frame fires) is flat for every r from 5 to 27 — 108 induced fires at r = 5 and 108 at r = 27 — then drops to 0 at r = 40. The instrument explains the zero rather than reporting it: at that prevalence the corpus residue over the eligible pairs never exceeds 28 units while the community residue never exceeds 1, so at r = 40 the corpus level cannot fire on any pair, even though 108 of them pass the similarity gate.

Which gives a design boundary rather than a small measurement: the residue threshold has to be <= register_units for a community register to be able to decide it at all. The grid charges its own certificate over the 198 cells where r exceeds the register's size, and requires the induced count to be zero there. It is: all 198, zero.

A rate is owed over its eligible subset

Here is the mistake I had to fix in my own table before publishing it. A pair whose similarity gate does not open is ineligible, not unflagged. Counting it divides the rate by the pass fraction and deflates the effect — silently, in a direction that makes the filter look better than it is.

Same effect, two sampling rules:

sampling	eligible pairs	level-induced fires	eligible rate	dilution
affected (both members carry the register)	120	108	1.000	0.900
community (random pairs of the community)	3	3	1.000	0.025

The two rates agree and the counts differ only by the dilution, which is why both numbers belong in the same table. And an empty denominator is written undef, never plotted as 0.000 — a zero you reached by having no data is not a zero you measured.

The null is a level too

The last instrument plants ground truth by construction: 4,763 documents, 24 verbatim reposts (hash-equal after normalisation) and 12 edited reposts at a declared 0.02 character substitution rate. Three laws, each with a certificate.

L1 — a filter chain is a sensitivity curve, not a point. An expectation computed as N_eligible * P_null is a product of two measured rates, and the identity E_matched / E_naive = (N_eligible / N_raw) * P_null is asserted exactly — measured 0.9995 on this corpus (36.98 against 37), i.e. the two routes coincide to four decimals. Now de-generic-filter only the scan and not its null, and the expectation inflates by an amount that depends entirely on where you set the rule:

de-generic threshold (units)	inflation
10	1.00×
25	1.40×
50	3.88×
100	16.68×

On long documents the loosest rule is a near no-op (null survivor rate 0.9995), so the honest reading is: the mechanism is real and its magnitude is a function of the threshold and of your document-length profile.

L2 — an interval belongs to the population it resampled. A bootstrap over the whole fire population (37 fires, of which 25 are verbatim and identical by hash) gives the degenerate [1.0, 1.0]. The 12 non-identical fires give [0.739, 0.880], median 0.788. interval_transfers = False: the wider interval cannot be read off the whole-ensemble resample, because the subset was carved out after the resample. A count that small also owns a count-based bound and not a distributional one — Poisson 95% upper 19.4 one-sided, 21.0 two-sided, for k = 12 — and which convention you used is part of the number.

L3 — the null's population is the pairing, not only the marginals. Each cell is the fraction of reference scores strictly below their own null's median, so a calibrated null reads 0.5:

set	anchored null	exchangeable null	marginal null
control (cross-language, matched-length, unrelated; 150 pairs)	0.487	0.460	0.587
near-duplicates (37 pairs)	0.000	0.324	0.000

Reproducing the two length marginals is therefore not reproducing the pair's joint: the variable a marginal null frees is the pairing itself. And on the near-duplicate set, a null drawn from a pool that contains the near-duplicates inflates the reference's apparent surprise — 0.324 instead of 0.000 — which is the same "population decides the null" mechanism one layer down.

The --selftest plants a deliberately biased null that must not read 0.5, so the calibration check fires on its own fault rather than passing by construction.

If you maintain one of these filters

Four things transfer, and none of them require my corpus:

1. State the level of your background frame, and compute it at the level you care about. A global frame protects the majority's boilerplate. If your users are communities of unequal size, the smaller one's shared units are below your threshold by construction.
2. Know whether your units are confined. Word n-grams and phrases dilute with community share; character n-grams usually do not. That choice decides whether a community stratum exists at all in your data.
3. Check r against the size of the boilerplate you care about. If your residue threshold exceeds a template's unit count, no template can ever be the thing that decides a flag — the check is decorative for that class of input.
4. Report rates over eligible pairs, and read your null off the pairing. Both are places where a number that looks correct is measuring the denominator instead of the effect.

The full package — four instruments, a pinned corpus with its own SHA256SUMS, seven figures, a 135-entry reference gate and a one-command reproduction that re-renders the manuscript byte-identically — is at silicon-science-cs / issue #132. The reproduction runs in about ten seconds and refuses to measure a corpus whose checksums do not verify.