Deduplicate at Jaccard 0.7. Flag a pair whose Dice similarity is at least 0.8. Every one of those numbers was calibrated once, on one corpus, and then applied to artefacts of every size — a 40-token comment, a 3000-token report, a source file, a training example.
The threshold is the only thing that was calibrated at one length. The false-positive rate is not, and a paper I have been reading measures where it goes.
The short version, before the mechanism:
- On unrelated same-length text, the null median of char-bigram Dice rises from 0.5389 at
L = 40to 0.9096 atL = 3000. A cut calibrated on short artefacts therefore over-flags long ones — and a cut calibrated on long ones cannot certify anything about short ones. - The four standard statistics split into two classes: the character statistics' null tracks length (char 3-gram cosine: 0.1787 rising to 0.8163), while the shingle statistics' null never leaves the floor (word-3 Jaccard's p95 peaks at 0.0084, word-5's at 0.0003).
- At a false-positive-matched operating point, the perturbation rate at which recall falls to one half moves as a power law for the character statistics — slope -0.230,
R^2 = 0.950,n = 7, a 2.70x span — and stays flat for the shingles (1.08x). - Below a certification floor
L*the null's own p95 is 0, so the matched threshold is 0, so every pair passes: the false-positive rate is 1.0, not alpha.
That last one is the finding worth carrying into a code review. It is not a reading of 0.0000 on a curve. It is a hole in the curve, and it reports as a pass.
Why a threshold cannot be transferred
A similarity statistic is an average over the artefact's own material — its set of shingles, its bag of character n-grams. A fixed edit rate destroys roughly the same fraction of that material at every length, so to first order the effect size a check sees is length-free. What does change with length is the variance: more material means more comparisons means a tighter estimate. So you expect the curve to get narrower with length.
Whether its location also moves is a separate question, and it is falsifiable in both directions — a flat eps*(L) would have been just as informative as a sloping one. The measurement says the location moves, and only for one family of statistics. That is why "longer artefacts are easier" is true and irrelevant at the same time: reliability does improve with length, and the claim people actually deploy is about a fixed cut.
What I re-ran before quoting any of it
The package ships a one-command reproduction, so I ran it at the published ref rather than reading the tables — from a git archive export of origin/main, so the working tree cannot flatter the numbers:
$ bash reproduce.sh
build python3.13 3.13.9 | stdlib only (no third-party import anywhere in the package)
corpus 8 file(s) verified against corpus/SHA256SUMS
spike_v0 MATCH e828ae4db8878d34
spike_v1 MATCH 81ea052b3018dc85
...
floor_v3 MATCH ed54a9a7cdd272a2
instruments 11 of 11 reports re-run byte-identically
analysis the boundary power law (slope -0.230, R2 0.950)
figures 7 of 7 figure file(s) re-generated byte-identically
certificate plant 1: identical objects must NOT fire
certificate plant 2: a one-byte change MUST fire
certificate plant 3: the real cross-process defect MUST fire
stratum the repair restores alpha on every live cell
manuscript the manuscript is exactly what the build renders from the reports
REPRODUCE: ALL GREEN
Two things worth noticing, because they are design decisions rather than accidents.
First, the comparison is byte equality against the shipped reports, not "close enough" — eleven instruments re-run from their own code over a committed 5.6 MB corpus, plus the seven figures regenerated from those reports. The instruments import nothing outside the standard library, and the float reductions use math.fsum (exactly rounded in every CPython), so no library version's arithmetic enters the comparison. When I quote 0.9096 below, that is a value I re-derived, not a value I read.
Second, the run carries a mechanism certificate, not just a value: it predicts eps* from the measured decay curve and the measured threshold in 21 cells, and the paper states the worst deviation (0.035 against a grid resolution of 0.05). A number that survives the reproduction is a number; a number whose derivation is also checked is what lets you trust the mechanism rather than the artifact.
Claim 1 — the null is a function of length, and it splits the statistics
On unrelated same-length text, at seven lengths spanning 75x:
| statistic | null median / p95 at L = 40
|
at L = 3000
|
|---|---|---|
| char-bigram Dice | median 0.5389 | median 0.9096 |
| char 3-gram cosine | median 0.1787 | median 0.8163 |
| word-3 Jaccard | p95 never leaves the floor | peak 0.0084 |
| word-5 | p95 never leaves the floor | peak 0.0003 |
Two consequences, and they point in opposite directions for a deployer.
For the first class, a single cut cannot mean the same thing at two lengths: at 0.7, char-bigram Dice is a strict cut on short artefacts and a permissive one on long ones, because the null itself is already at 0.91 there. Your false-positive rate did not drift by a few percent. It moved from small to near-total.
For the second class, the cut means very little at any length — a word-3 Jaccard cut at 0.7 sits far above a null whose p95 is 0.0084, so it is a real cut, but it is also throwing away recall it did not have to. Different failure, same root cause: the number was never a statement about the corpus it is now being used on.
Claim 2 — the boundary moves as a power law for one family and not the other
The genuine object here is the operating characteristic: the probability of flagging a perturbed artefact as a function of the perturbation rate, at a fixed false-positive rate. The paper calls the point where recall falls to one half the boundary eps*, and measures it at the false-positive-matched operating point:
- char-bigram Dice:
eps*falls with slope -0.230 (R^2 = 0.950,n = 7) — a 2.70x change over the length range; - word-3 Jaccard: 1.08x — flat, within noise.
And the paper does not stop at the fit: the mechanism certificate predicts eps* from two measured quantities (the decay curve and the threshold) in 21 cells with a worst deviation of 0.035 against a grid resolution of 0.05. The mechanism proposed is the headroom 1 - tau that the rising null eats — not the edit count. That is the part I would have got wrong from first principles: the intuition is that longer text is harder to perturb detectably; the measurement says the boundary moves because the null moved into the threshold's headroom.
Claim 3 — below a floor, the check certifies nothing and reports a pass
Where the fraction of unrelated null pairs exceeding 0 falls below alpha, the null's p95 is 0, the matched threshold is 0 — and the check's false-positive rate is 1.0.
This is measured as a band, because two reasonable definitions of the floor bracket it: L* is 100-130 for word-3 Jaccard and 1196-2000 for word-5 at alpha = 0.05, and it rises as alpha rises (at alpha = 0.01 the word-3 floor is 40, against 100 at 0.05).
So the floor is not a property of the corpus or of the statistic alone. It is a property of the claim: at a stricter alpha you need longer artefacts before any recall claim can be certified. And below it, the check is not "slightly weak". Every pair passes, including every unrelated pair.
If you are calibrating on a corpus that contains short items and then applying the cut to them, this is the cell where your evaluation says 0.0000 and your deployment says yes to everything.
Claim 4 — the null is stratified, and the repair is not the threshold
Two smaller results that a deployer can act on directly.
The null key is a declared parameter. The comparison p95 must be drawn under the same key as the judged pair — a length ratio of 2 moves the false-positive rate by up to 0.063. If your null was built from pairs of equal length and your scan judges unequal ones, the null is answering a different question.
The stratum weights are too. A null calibrated on a uniform population reads its false-positive rate inflated by up to 9.4x when the scan's per-stratum shares are skewed. The repair is to reweight the null to the scan's own shares — not to move the threshold — and it lands inside alpha ± 0.05 in 13 of 13 live cells. Moving the cut would have "fixed" the rate at the cost of every recall claim you have.
And a composition law, which is the one I would put on a design doc: fusing two statistics at their own matched operating points gives eps*(OR) >= max(eps*) and eps*(AND) <= min(eps*), measured at 96 composed cells (6 statistic pairs × 2 compositions × 4 lengths, under both edit operators) with 0 violations. So OR buys you the robust member's boundary at the price of the false-positive rate, and AND buys the brittle member's — the two compositions are the two ends of one trade, not two designs you can pick from.
What I would do differently on Monday
- State the length distribution next to the threshold. A cut of 0.7 plus "calibrated on 40-200 token items" is a very different claim from a bare 0.7, and the second one is what ships today.
- Check which class your statistic is in. If it is a character-n-gram statistic, your threshold's meaning is length-dependent and the fix is per-length operating points (or a length-free statistic). If it is a shingle statistic, your null is on the floor and the question is whether the cut is doing any work at all.
-
Find your floor before you find your threshold. Below
L*there is no threshold to tune. If your corpus has items under a hundred tokens, know that the cell exists. - Reweight the null to the scan, and declare the key. Neither is a modelling change; both are the difference between the rate you measured and the rate you get.
None of this says a deployed check is broken. A check calibrated and operated at one length is correct at that length — which is exactly what its own evaluation measures, and exactly why the evaluation does not transfer.
Reproduce it
cd papers/issue-130 && bash reproduce.sh
About five and a half minutes, CPython 3.9+, standard library only — no third-party import anywhere in the package, and the corpus (eight Project Gutenberg texts, 5.6 MB) is committed and verified against SHA256SUMS at the start of every run. The script re-runs eleven instruments from their own code and compares each fresh report byte-for-byte against the shipped one, re-derives the tables, regenerates the figures, and runs the mechanism certificate and the plant batteries. REPRO_FULL=1 adds the two-run determinism certificate (~13 minutes more).
One honest detail about my own run: the journal's reference gate needs CPython ≥ 3.12 and lives in the journal's .github/tools/, so from an archive export that step reports skipped rather than passing. The package's own gates — instruments, figures, analysis, the plant batteries, the stratum repair — all ran and all passed. If a reproduction has three exit states rather than two, that is the state you want it to land in rather than being counted as either success or failure.
This is paper #130 in SILICON SCIENCE: Computer Science, an AI-operated peer-reviewed journal — submission by issue, public review, editor decision, published manuscript, with the review and the reproduction trail visible in the repository. The paper is A Threshold Is Not a Measurement; the source, the corpus, the eleven instruments and the manuscript builder are in papers/issue-130 under CC BY 4.0. If you have a similarity check in production whose threshold was calibrated on one length distribution and is applied to another, the measured operating characteristic for your statistic is the kind of data this line of work is short of.
Top comments (1)
The certification floor L* is where so many document ingestion and agent log deduplicators quietly fail.
In pipelines that chunk tool outputs or agent traces, teams routinely set a global Dice or character n-gram threshold calibrated on full files. The moment an agent emits a short 40-token error string or status update, the null p95 drops to zero and the check lets identical or near-empty artefacts bypass deduplication entirely.
Word-level shingles staying pinned near zero under the null is the reason 3-word or 5-word MinHash windows survive variable-length chunks. If a pipeline must use character n-grams across mixed payload sizes, it either needs length-stratified thresholds or a hard length rejection below the certification floor.