TL;DR. Our ticket-routing eval scored 0.94 for five weeks. The number was manufactured. We had built a dynamic few-shot selector that retrieved the eight nearest labeled examples for each input, and we built its index out of the same labeled_tickets.jsonl the eval set was sampled from. So for every eval case, the nearest neighbor in that index was the eval case itself, gold label attached, pasted into the prompt directly above the question we were about to ask. The model was not answering. It was copying. Measured against tickets the index had never seen, real accuracy was about 0.79. The reframing that stuck for me: contamination is not just a training-time problem you inherit from a model vendor. Any pool you draw prompt content from is part of your eval's input. If your eval set and your few-shot pool share a parent file, the leak is yours, in your repo, shipped by someone on your team last Tuesday. Split the pools by content hash at the source, then check the prompt and not only the training data.
The trace that ruined a good number
I was not looking for this. I was chasing p99 latency on the routing endpoint, which had crept past two seconds and was making the queue back up. So I pulled a slow trace and started reading the prompt we actually send, top to bottom, the way you do when you suspect somebody stuffed too much into the context window.
The prompt had a system block, then eight few-shot examples, then the ticket to classify. We rendered them nearest-last, so the closest match sat directly above the question.
Example eight was the ticket to classify. Same text. Same customer. And sitting under it, formatted as the demonstration answer, was the label we were about to grade the model on.
I read it three times. Then I checked four more traces. Same shape every time: the last example before the question was the question.
Our eval had been reporting 0.94 since the selector shipped. Nobody questioned it because 0.94 is a believable number. Five weeks. Nobody asked once. If the suite had printed 1.00, someone would have opened it within the hour, because a perfect score reads as a bug. A 0.94 reads as a good quarter. A total leak still did not mean a perfect score: the other seven demonstrations pulled against the exact match often enough to cost a few points, which is precisely what made 0.94 look earned. It went in a deck. Somebody put it on a slide with an arrow pointing up. The score was high enough to celebrate and low enough to trust, which is the worst place a wrong number can sit.
select_examples()
The mechanism is boring, which is why it survived review.
We started with static few-shot: eight hand-picked examples, hardcoded, same eight for every request. It worked fine and it was obviously fine, because you could read the eight in the diff.
Then someone (me, partly, in a design review I do not get to distance myself from) pointed out that a fixed eight cannot cover billing and abuse reports and integration bugs at once. Retrieve the examples instead. For each incoming ticket, embed it, pull the eight nearest labeled tickets out of the index, put those in the prompt. Better coverage per token. This is standard practice and I still think it is right.
The index got built from labeled_tickets.jsonl, which was where every labeled ticket lived. Roughly 1,900 rows at the time.
The eval set was 600 rows sampled from labeled_tickets.jsonl.
Those two sentences are the whole bug. Both artifacts were correct in isolation. Both were reviewed. Nobody put them side by side, because they lived in different files, owned by different people, merged six weeks apart.
That gap is the part worth generalizing. Neither diff was wrong. A reviewer on the selector PR sees a sensible retrieval change and asks about latency and index freshness. A reviewer on the eval PR sees a defensible sample size and asks whether 600 cases is enough. Both are good questions. Neither reviewer gets asked the only one that mattered, which is whether these two things read the same rows, because that question is not visible in either diff. It lives in the space between them. We had no review process for the space between two files, and I am not sure most teams do.
Here is what the selector did, reduced to the part that matters:
def select_examples(query, index, k=8):
# index was built from every row in labeled_tickets.jsonl
return index.search(embed(query), k=k)
Then the eval harness:
cases = random.sample(load("labeled_tickets.jsonl"), 600)
Read them together and the failure is arithmetic, not machine learning. The eval case is in the index. The eval case's own embedding is its nearest neighbor, at distance zero. Every single one of the 600 eval cases retrieved itself as its own nearest neighbor and carried its gold label into the prompt. Not some of them. All of them. A retriever asked to find the most similar labeled example to a text that is sitting in its own index will return that text, because nothing is more similar to a string than the string.
We also had about 40 exact duplicate tickets in the pool, because customers paste the same complaint twice and support macros generate identical bodies. Those would have leaked across a naive random split even without a retriever. The retriever just made the leak total instead of partial.
Why the retriever hands over the answer
Worth being precise about what broke, because "contamination" gets used loosely enough to stop meaning anything.
The model did not memorize our tickets during pre-training. The vendor did not train on our data. Our fine-tune was clean. Every version of contamination I had read about was about the training corpus, and every one of those was genuinely not our problem.
The leak happened at prompt construction time, in our code, at inference, on every request. The model saw the answer in the line directly above the question, in a block explicitly labeled as examples of correct behavior. It did what any competent few-shot learner does with a demonstration that exactly matches the query. It copied the label.
So the eval was measuring copy fidelity. That is a real capability. It is not the capability we were shipping, and it is not the one the number claimed.
The published work here is almost entirely about training data, and it is worth knowing even though it does not describe this exact bug. Zhou et al., in Don't Make Your LLM an Evaluation Benchmark Cheater, work through how benchmark leakage into training "can dramatically boost the evaluation results," producing an unreliable read on what a model can do. Their setting is pre-training and fine-tuning corpora. Mine was a JSONL file and a vector index.
The mechanism ports anyway. The model cannot tell where in its input a leaked answer came from, and neither can your score. Weights or context window, the arithmetic is the same: if the answer reached the model before the question, the number measures retrieval, not reasoning. The difference is that training contamination is mostly somebody else's to fix, and prompt contamination is entirely yours. I find that clarifying rather than comforting.
contamination_check.py
The check we should have had. Standard library only, so it runs anywhere Python does and there is nothing to install in CI.
Two passes. A normalized hash catches verbatim and cosmetically edited duplicates. An n-gram containment score catches the case where an eval item sits inside a longer example. Containment rather than Jaccard on purpose: a short eval case buried in a long few-shot example still scores 1.0, and that is exactly the leak worth failing on.
"""contamination_check.py: does the few-shot pool already contain the eval answer?"""
import hashlib
import re
import unicodedata
_PUNCT = re.compile(r"[^\w\s]", flags=re.UNICODE)
_WS = re.compile(r"\s+")
def normalize(text):
"""Casefold, strip punctuation, collapse whitespace. Defeats cosmetic edits."""
text = unicodedata.normalize("NFKC", text).casefold()
return _WS.sub(" ", _PUNCT.sub(" ", text)).strip()
def fingerprint(text):
return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()
def ngrams(text, n):
toks = normalize(text).split()
if len(toks) < n:
return {" ".join(toks)} if toks else set()
return {" ".join(toks[i:i + n]) for i in range(len(toks) - n + 1)}
def containment(case, example, n):
"""Fraction of the eval case's n-grams that also appear in the example."""
gc = ngrams(case, n)
if not gc:
return 0.0
return len(gc & ngrams(example, n)) / len(gc)
def audit(eval_set, fewshot_pool, n=5, threshold=0.6):
"""Return (eval_idx, pool_idx, score, kind) for every eval case the pool leaks."""
by_hash = {}
for i, ex in enumerate(fewshot_pool):
by_hash.setdefault(fingerprint(ex), i)
hits = []
for j, case in enumerate(eval_set):
i = by_hash.get(fingerprint(case))
if i is not None:
hits.append((j, i, 1.0, "exact"))
continue
scored = ((containment(case, ex, n), i) for i, ex in enumerate(fewshot_pool))
score, i = max(scored, default=(0.0, -1))
if score >= threshold:
hits.append((j, i, round(score, 3), f"{n}-gram"))
return hits
if __name__ == "__main__":
fewshot_pool = [
"Escalate to a human when the customer mentions legal action.",
"Refund the order automatically if it shipped more than 30 days ago.",
"When a customer asks to escalate to a human because they mention legal "
"action or a lawsuit, hand the ticket to the on-call agent immediately.",
]
eval_set = [
"Escalate to a human when the customer mentions legal action.", # verbatim
"escalate to a human when the customer mentions LEGAL ACTION!!", # cosmetic edit
"Escalate to a human because they mention legal action or a lawsuit.", # fragment
"Hand off to a person if the buyer threatens to sue.", # paraphrase
"Ask for the order number before checking shipment status.", # clean
]
hits = audit(eval_set, fewshot_pool)
leaked = {j for j, _, _, _ in hits}
for j, i, score, kind in hits:
print(f"LEAK eval[{j}] <- pool[{i}] score={score:<5} via {kind}")
print(f"\n{len(leaked)}/{len(eval_set)} eval cases leaked by the few-shot pool")
for j, case in enumerate(eval_set):
if j not in leaked:
print(f" clean: eval[{j}] {case!r}")
Output:
LEAK eval[0] <- pool[0] score=1.0 via exact
LEAK eval[1] <- pool[0] score=1.0 via exact
LEAK eval[2] <- pool[2] score=1.0 via 5-gram
3/5 eval cases leaked by the few-shot pool
clean: eval[3] 'Hand off to a person if the buyer threatens to sue.'
clean: eval[4] 'Ask for the order number before checking shipment status.'
Three of five caught. The cosmetic edit gets normalized into an exact hit, and the fragment gets caught by containment against the longer pool entry. Both of those would have slipped past a naive set(eval) & set(pool).
Now look at what the script calls clean.
The paraphrase the check calls clean
eval[3] is "Hand off to a person if the buyer threatens to sue." The pool contains "Escalate to a human when the customer mentions legal action." Same rule. Same decision. Zero shared 5-grams, so my check reports it as clean and moves on.
It is not clean. It is the same eval case wearing a different coat, and a model given the pool entry will get eval[3] right for reasons that have nothing to do with understanding your routing policy.
This is a known and documented hole, not something I discovered. Yang, Chiang, Zheng, Gonzalez and Stoica went at it directly in Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. Their finding is that "simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures," where the measures in question are exactly the string and n-gram matching my script does. They report that a 13B model can overfit a benchmark and reach performance on par with GPT-4 when rephrased test data is left in, and they found 8% to 18% overlap with HumanEval sitting in pre-training sets like RedPajama-Data-1T and StarCoder-Data. Their decontamination tool is public at lm-sys/llm-decontaminator, Apache-2.0, and it uses an LLM to catch what string matching cannot.
Again their setting is training corpora and mine is a prompt. The blind spot transfers cleanly, because n-grams do not know what a sentence means in either setting.
I have not run their tool against our pool yet, so I am not going to tell you what it would find. What I will say is that the cheap check is worth shipping anyway. It caught three of five in a toy example and it caught our real leak on the first run, because our leak was verbatim. Verbatim is the common case. Paraphrase contamination is real, and my honest position is that I do not currently know how much of it we have, which is a different sentence from "we don't have any."
Where else the same pool leaks
Once I had the shape of it I went looking elsewhere and found three more instances in our own repo within a day. Listing them because the retriever version is the flashiest and probably the least common.
Static few-shot, curated out of the labeled pool. Before the retriever we had eight hardcoded examples, and I had filed them as safe precisely because they were hardcoded. They came from the same file. Two of the eight were in the eval set. A fixed leak is smaller than a total leak. It is still a leak, and it sat in that prompt for months while I told myself static few-shot was the conservative option.
RAG evals where the corpus contains the eval documents. Same arithmetic, different index. If your eval questions were written from documents that live in the retrieval corpus, the retriever will hand the model the exact paragraph each question was written from. This one is arguably fine, because production does the same thing. It stops being fine the second you report the score as evidence about the model rather than about your retriever.
Synthetic eval cases generated from the seed examples. The easiest to walk into and the one I would bet is most widespread right now. You ask a model for 500 eval cases. You seed it with your best labeled examples so the output looks like your domain. Those same examples are in your few-shot block. Your eval set is now a paraphrase of your prompt. No file is shared, no hash collides, and contamination_check.py finds nothing, because there is nothing verbatim left to find. It is the paraphrase case from the Yang et al. paper, arriving through a door we built ourselves and held open.
The common factor is not retrieval. It is one pool of text doing two jobs: teaching the model and grading it. Whenever those roles drink from the same well, the score is compromised, and how badly is not knowable from the score.
Disjoint by construction, not by discipline
Detection is a smoke alarm. It tells you the kitchen is already on fire. The actual fix is to make the overlap impossible to express.
We stopped sampling the eval set and the few-shot index separately from one parent file. Instead, every record gets assigned to exactly one side by hashing its normalized content:
def assign(text, eval_frac=0.35, salt="fewshot-v3"):
h = hashlib.sha256((salt + normalize(text)).encode("utf-8")).hexdigest()
return "eval" if int(h[:8], 16) / 0xFFFFFFFF < eval_frac else "fewshot"
def split_pool(pool, eval_frac=0.35, salt="fewshot-v3"):
buckets = {"eval": [], "fewshot": []}
for rec in pool:
buckets[assign(rec, eval_frac, salt)].append(rec)
return buckets["eval"], buckets["fewshot"]
Three properties earn this over random.sample, and the third is the one that actually mattered to us.
It is deterministic. No seed to forget, no ordering dependency. The same record lands in the same bucket on my laptop, in CI, and in the batch job that rebuilds the index at 4am.
It is stable as the pool grows. Adding 500 new tickets does not reshuffle the existing split, so last month's eval numbers stay comparable to this month's. A reshuffling split quietly moves your baseline and you get to spend a day proving the model did not regress.
It puts duplicates on the same side. This is the part random.sample cannot do. Because the hash runs over the normalized text, our 40-odd exact duplicates and their cosmetic variants all resolve to one bucket. Under random sampling, a duplicated ticket had a real chance of landing one copy in eval and one in few-shot, which is the leak reappearing through the back door after you thought you had closed it.
Run it over 2,000 synthetic records and the realized eval fraction lands within about a point of the 0.35 target, which is the usual hash-bucket wobble and does not matter at this size. The exact counts depend on your strings, so do not pattern-match mine. The part that does not wobble: feed it the escalation rule plus its shouty and double-spaced variants and all three land in the same bucket, every run, on every machine.
The salt is there so you can rotate the split deliberately. Bump fewshot-v3 to fewshot-v4 and you get a fresh partition, on purpose, in a diff, with a name someone has to review.
Then it goes in CI as a gate, not a dashboard. The audit runs on every PR that touches either artifact, and a nonzero hit count fails the build with the offending indices printed. It takes about a second on our pool. I have opinions about slow eval gates, but a hash join over a few thousand strings is not where your merge queue goes to die.
The number after the fix
Rebuilt the index from the few-shot side only. Re-ran the 600-case eval.
0.79.
That is roughly a 15 point drop, and it is the first number that suite ever produced that meant anything. Nobody enjoyed the meeting. The version I would defend now is that we did not lose 15 points, we found out we never had them, and we found out from a trace instead of from a customer.
The follow-on was more interesting than the drop. With a real number, the error slices were legible for the first time. Abuse reports were dragging well below the mean while billing sat above it, and that gap had been invisible at 0.94 because copying the label works equally well for every category. Contamination does not just inflate your score. It flattens it, and a flat score hides the shape of the problem you were trying to see. We spent the next sprint on abuse-report examples and got some of the 15 points back honestly, which took actual work.
What I'd check first
- Print one real prompt from a failing eval run and read it end to end. Not the template. The rendered string, with the retrieved examples in it. If the eval case appears in its own prompt, stop and go fix that before you interpret another score.
- Diff the provenance of the eval set against the few-shot pool. Not the contents, the source. If both trace back to the same file, table, or index, assume overlap until a hash join says otherwise, and treat any shared parent as a leak that has not been found yet.
- Suspect the believable number, not just the perfect one. 1.00 gets investigated. 0.94 gets a slide. Ask what score you would have accepted without checking, and go check that one first.
Top comments (9)
The observation that 0.94 survived because it wasn't 1.00 is the most useful thing in here, and it generalizes well past eval contamination. A suspiciously perfect number gets audited within the hour. A merely good one goes in a deck. Most of our worst metrics bugs have lived in that band — plausible enough to celebrate, not clean enough to interrogate.
Your reframing is the right one: any pool you draw prompt content from is part of the eval's input surface. That's broader than most contamination checklists, which stop at training data because that's the vendor's problem and therefore feels like someone else's.
The part I'd underline for anyone reading: you found this while chasing p99 latency, not while auditing evals. Reading the fully-rendered prompt — the actual bytes sent, not the template — catches a class of bug that no amount of staring at selector code will. We've made it a habit to log one full rendered prompt per deploy and actually read it. Feels remedial; catches real things.
The rendered-prompt habit is the one I would put on a poster. Every abstraction between your template and the bytes on the wire is a place for a bug like this to hide, and the only way to see it is to read the actual assembled prompt, not the code that builds it. We do the same now, one full rendered prompt per deploy, read by a human. It feels remedial right up until it catches the thing no code review would. Glad it is not just us.
The paraphrase hole you flag at the end has a fix from the tabular world, and it isn't a smarter matcher — it's grouping by provenance instead of by content.
Your hash split groups on normalised text, so exact and cosmetic duplicates land on one side. Paraphrases share no n-grams, so they slip straight through. But the thing they do share is an origin: the same customer, the same support macro, the same seed example you generated 500 cases from. Group on that key and the paraphrase leak closes without any semantic matching at all.
GroupKFoldhas done exactly this in sklearn for years, and it's the standard defence against your 40-duplicate-tickets case too.Concrete version of how bad it gets otherwise. I ran a dataset of repeat patient visits — 400 patients, five visits each, near-identical rows with different wording. Random split, out-of-bag score came back 1.0000. Hold out whole patients with GroupKFold, same model, same data: 0.8620. Fourteen points of pure fiction, and not one character of it was verbatim, so no hash or n-gram check would have flagged it. Two visits from one patient share an entity, not a string.
So the ordering I'd suggest: group by the strongest identifier you have (customer, macro, seed), fall back to content hash for rows that genuinely have no identifier, then keep your containment check as the smoke alarm for whatever both miss.
Your fourth case is the one I'd worry about most — synthetic eval cases generated from the seed examples. That's the paraphrase leak with a generator aimed at it, and the seed ID is usually sitting right there in the pipeline, unused.
no this is the most common failure in eval work and it's rarely deliberate..
the eval set leaks into the few shot examples so the model is being graded on what it already memorised.. not on generalisation. five weeks at 0.94 says nobody was checking the actual routing decisions by hand.. just watching the number.
how did you catch it in the end.. a real world miss or did someone actually audit the examples?
Neither, which is the uncomfortable part. No user reported it and nobody audited the examples, because 0.94 never looked wrong enough to audit. I found it by accident chasing a p99 latency regression on the routing endpoint. I pulled a slow trace to see what was bloating the context window, read the fully rendered prompt top to bottom, and the last few-shot example was the ticket I was about to grade with its gold label attached. If that endpoint had stayed fast I might still be quoting 0.94. That is what pushed me to log and read one rendered prompt per deploy, so the next one does not need a latency bug to surface.
makes sense
It's frustrating but not uncommon—during a project I worked on with VoltageGPU, we had to discard some early accuracy metrics because the test data was too similar to training. The lesson? Always validate your splits, especially in confidential computing scenarios where data integrity is critical.
The "high enough to celebrate, low enough to trust" observation is exactly why contamination this shape survives so long - the number itself is camouflage. Genuine retrieval performance degrades measurably when you vary prompt structure or few-shot count, so a score that stays stable across those perturbations is a signal something else is doing the answering. Content-hash pool splitting is the right structural fix. I would pair it with a CI stress test: block 5% of eval cases from the few-shot index at build time and gate on the performance delta staying under a threshold, which makes the leak detectable before it ends up on a slide.
That stress test is a good idea, and it attacks the part my content-hash split does not: detection when someone reintroduces the overlap later. The hash split fixes it at the source, but nothing stops a future PR from rebuilding the index off the wrong parent file. A held-out slice kept deliberately out of the few-shot pool, with an alert on the score delta, would have caught our leak the day it shipped instead of five weeks later. I would tune the 5 percent to whatever keeps the delta above run-to-run noise on your set. Stealing this.