A decontamination report prints a number: how much of the benchmark overlaps the training corpus, by n-gram. The number is real. It is also the overlap it happened to look for, and there is a whole band of rewrites that keeps every point of the score inflation while taking that number to zero.
Run it: https://dev48.infy.uk/ai/days/day74-benchmark-contamination.html
The setup, with nothing simulated
- A four-choice benchmark whose honest score is exactly 1/4.
- A training corpus containing copies of some items.
- A model that answers from memory when it has seen a long enough contiguous stretch.
Nothing here simulates language. The engine simulates the overlap arithmetic: which stretches of an item survive a rewrite, which n-gram lengths a filter can look for without flagging a clean benchmark, and what the gap between those two numbers does to a published score.
The gap is structural
There are two lengths in play, and they are not the same length:
| m | the contiguous stretch a model needs to answer from memory |
| n | the n-gram length your filter can search for without false-flagging clean items |
A filter that searches for shorter n-grams flags legitimate overlap — common phrasing, shared boilerplate, the question format itself — so in practice n is pushed up. A model needs less than that to recall an item. Every rewrite that breaks n-grams of length n while preserving stretches of length m is invisible to the filter and fully effective for the model.
That band is not a corner case. It is where paraphrase-based augmentation naturally lands.
What to report instead
The filter's number answers "did I find copies of this exact string". The question you care about is "could the model have seen enough of this item to recall it". Those come apart, and the page shows exactly where — so report the assumption about m alongside the overlap, or the overlap alone will read as safety it cannot provide.
Independently verified, 0 failures.
Top comments (0)