We trained a 360M parameter code model from scratch and published 26.8% pass@1 on HumanEval.
Two days later a reader asked whether we had checked our instruction data for benchmark overlap. We had not. So we did, and the honest number is 22.6%.
Here is what was in the data.
The scan
Two checks against each instruction set:
- Problem text. The first 80 characters of each HumanEval docstring, whitespace normalized.
- Canonical solutions. Half or more of a solution's 13-token n-grams, with comments and docstrings stripped.
| Dataset | Examples | Docstrings found | Solutions found |
|---|---|---|---|
| evol-codealpaca-v1 (we used this) | 111,272 | 60 problems, 156 examples | 8 problems, 67 examples |
| Magicoder-Evol-Instruct-110K (the "decontaminated" copy) | 111,183 | 52 problems, 116 examples | 8 problems, 66 examples |
| Magicoder-OSS-Instruct-75K (we used this) | 75,197 | 0 | 1 problem, 1 example |
The filtered redistribution differs from the original by 89 rows and still contains 52 of the 60 contaminated HumanEval problems. Switching to it would not have saved us.
Why the filter missed them
The matches are not copies. They are benchmark problems in practice-exam form: reworded, extended with an extra requirement, or ported to Java, each with a full solution.
HumanEval/43, pairs_sum_to_zero, appears with its original docstring, its doctests and a working solution, under an added instruction to make it linear time.
That is not an accident of collection. Evol-Instruct grows a dataset by repeatedly rewriting seed problems, and some of the seeds were benchmark problems. An exact-match filter cannot catch a rewrite, and rewriting is the entire method.
Was the model reciting answers?
No, and this is the part that surprised us.
We compared every chat-model pass against its canonical solution with a difflib ratio on whitespace-normalized lines. A recited answer scores above 0.9. Of 44 passes, the highest was 0.73, on a problem whose canonical solution is one line. Most were below 0.3.
So we removed the contaminated examples and retrained. We also retrained on the original data with nothing changed but the seed, so we could tell the effect of the data apart from the effect of a rerun.
| Checkpoint | HumanEval | HumanEval+ | MBPP |
|---|---|---|---|
| Base | 13.4% | 11.0% | 20.2% |
| Chat, as published | 26.8% | 25.6% | 38.9% |
| Same data, seed 2 | 26.2% | 25.6% | 34.6% |
| Decontaminated | 22.6% | 21.3% | 35.0% |
Decontaminated versus published on HumanEval: -4.3 points, 8 problems lost and 1 gained, McNemar p = 0.039. The two original-data runs agree to within one problem, so the drop is the data and not the rerun.
Where the lost points came from
If this were memorization, the losses would sit on the 63 problems that matched removed training data. They do not.
| Subset | Problems | Base | Published | Seed 2 | Decontaminated |
|---|---|---|---|---|---|
| In the removed data | 63 | 5 | 14 | 17 | 13 |
| Not in the removed data | 101 | 17 | 30 | 26 | 24 |
Against the mean of the two original-data runs the decontaminated model is about four points lower on both subsets.
The contamination was real and it was not memorization. Practice on HumanEval-style problems helped on HumanEval-style problems generally, including ones that were never in the data. We had reported that as ability.
What MBPP shows, which is a different lesson
Only 10 MBPP problems matched removed data, and the two original-data runs differ by 11 problems. The published 38.9% is best read as a lucky seed. Run-to-run variance on MBPP was about four points, which is the same size as the contamination effect on HumanEval.
A single run reports the model, the data, the harness and the seed as one number.
What to take from this
Scan your instruction data. Docstring prefixes plus solution n-grams. Ours runs in minutes and is in the repo.
Do not trust "decontaminated" as a label. Check what the filter actually matched on. An exact-match filter against a dataset built by rewriting is close to no filter at all.
Run the seed control. Without it we could not have separated a four point data effect from a four point variance effect, and on MBPP the variance was the larger of the two.
Ask vendors three things, including us: the decontamination method, the evaluation harness, and how many runs are behind the number.
Full report with the figures, confidence intervals and McNemar tests: trifero.ai/research/elroy
Code, nine chapters, every benchmark completion and failure record: github.com/strifero/elroy, Apache-2.0.
I run Strife Technologies and the trifero.ai lab. We built this model to be able to answer questions about AI with measurements instead of adjectives, and this is the measurement we liked least.
Top comments (2)
I pulled your per-problem records out of the repo (
results/*/*.jsonl) and recomputed the table independently — every cell reproduces exactly: 13.4 / 26.8 / 26.2 / 22.6 on HumanEval, 20.2 / 38.9 / 34.6 / 35.0 on MBPP, and the paired test on HumanEval is 8 lost against 1 gained, exact two-sided p = 0.0391. The 89-row difference between the two instruction sets reproduces from your logs too, as does the difflib result (max 0.73 on HumanEval/116, whose canonical solution is one line; 0 of 44 passes at ≥ 0.9).The part I'd add is the number that decides how to read the table: the resolution of each comparison, which your own records give you for free because every row is paired per problem. For a paired comparison the standard error of the difference is
sqrt(d - (b-c)^2/n)/nwithdthe discordant count, so:Three things fall out of that.
The seed control is not just "38.9% was a lucky seed" — it is the null distribution of the whole comparison, and on MBPP it is wider than the effect you are reporting. Two runs of the same data with a different seed differ by 4.3 points with 37 discordant problems. The decontamination gap on MBPP is 3.9 points with 36. So on MBPP the data effect sits inside the seed effect: "the drop is the data and not the rerun" is established on HumanEval — where the seed control moves only 7 problems — and not on MBPP. Your MBPP section already says the variance was the larger of the two; the number that makes that concrete is the one you have: at one run per side the MBPP contamination effect is not distinguishable from zero, which is a stronger sentence than "the same size" and the one I would print.
On HumanEval the effect is real and barely resolved: 1.2× its own noise floor. That is what p = 0.039 looks like on an n of 164 — I would write the −4.3 points as "just outside the comparison's resolution" and, if you have the compute, run one more seed on the decontaminated data, which would narrow it directly.
The subset split cannot carry the "not memorization" conclusion, and your seed control is the evidence. Against the published run the drop is not uniform: 1.6 pp on the flagged problems (a single discordant problem) against 6.0 pp on the unflagged 100 (8 discordant). The uniformity you report comes from comparing against the mean of the two original-data runs (−3.9 pp and −4.0 pp), which is a reasonable choice — but the two original runs disagree about where the drop sits, and by how much: the flagged-vs-unflagged interaction is −8.7 pp between published and seed 2, against −4.4 pp between published and decontaminated. The seed control's own interaction is twice the data's, and at n = 164 an interaction needs roughly 6 pp of differential to resolve at all. So the subset table is underpowered for the claim, while the difflib analysis is not: comparing each pass to the canonical solution is a direct test of recitation, and "max 0.73, most below 0.3, none above 0.9" survives it. That is the strand I would lead with, and the subset split goes in as a supporting remark with its interval.
One decomposition worth adding in the same table: 3 of the 8 problems the decontaminated model loses (143, 152, 50) are also lost by the same-data rerun. A third of the "losses" are inside the seed noise, so the data-attributable loss is 5 problems, about −3.0 points, with −4.3 as the top of the range rather than the estimate.
Two small reproducibility notes, offered as detail rather than as a defect.
scripts/check_contamination.pyaudits HumanEval only, andelroy/decontam.py— the matcher that does cover MBPP — has no reporting path, so the "10 MBPP problems matched removed data" figure is printed by nothing in the checkout; both logs underresults/are HumanEval. Your MBPP per-checkpoint numbers re-run fine (the two original runs differ by 11 problems, as you say), but the MBPP side of the contamination argument is the half a reader cannot re-run. And the union line inresults/contamination.logprints 64 problems where the post says 63; it moves nothing (47 of the 64 pass in no checkpoint, so the partition is numerically identical either way), but a reader matching the log to the table will trip on it.None of this changes your headline — the honest number is lower than the published one, and the decision to publish the lower number is the best thing in the post. It changes what the subset split can be asked to prove.
Running a second seed on the uncleaned data is the control most teams skip because nobody wants to burn double the training compute just to measure their own noise floor. Seeing an 11-problem swing (38.9% down to 34.6%) between two identical runs on MBPP puts a concrete price on single-seed leaderboard claims. When run-to-run initialization variance spans four percentage points, any published margin smaller than that is indistinguishable from a lucky draw.
The uniform four-point drop across both the 63 leaked problems and the 101 held-out HumanEval items also shows how synthetic instruction rewriting leaks the evaluation distribution even when line-level difflib ratios stay below 0.3. Mutating seed docstrings into practice variants teaches the model the exact signature conventions and boundary traps of the test harness, lifting pass@1 across the unseen split while sailing past exact-match decontamination filters.