We build an automated security scanner for Solana programs. To know whether it works, we keep a small benchmark: 19 programs with a planted vulnerability each, 7 written to be deliberately clean, and a written description of every planted flaw so a judge can check whether the scanner's findings actually match. We ran it for weeks. We published detection and false-alarm rates from it.
Then, the day before we open-sourced the corpus, we read it the way an outsider would.
It had been handing the scanner the answers the entire time.
Four leaks, all inside the file the scanner reads
The scanner takes a source file and returns findings. So anything in that file is input. We knew that in the abstract. We had not actually looked.
Comments. Nineteen of nineteen vulnerable samples carried a comment describing the planted flaw, the kind you write for yourself while building a test case and never think about again. Three of the seven clean samples declared themselves correct, in words.
Program IDs. Solana programs declare their own address in the source. Ours said things like FeeByp111... on the fee-bypass sample, and Guard1111... or T22G0od... on the clean ones. Speaking names.
Module names. pub mod vulnerable_vault. trust_me_lending.
Log strings. One read, verbatim: "Estimated output: {} (no minimum check)". That is the bug, printed as a string, inside the sample meant to test whether the bug gets found.
The one that took longest to spot was the quietest. We had a matched pair, same program, one clean and one vulnerable, and the only difference in their log lines was the word "real" versus "measured". Two words. Enough to separate the two classes, if you are a model reading the file.
What it did to the numbers
Detection went up, because the flaw was being announced rather than found. False alarms went down, because several clean files said they were clean.
And nothing about the output looked wrong. That is the part I keep coming back to. A leaking benchmark does not produce absurd numbers. It produces plausible ones. We had no reason to look, so we didn't.
What we did
We withdrew the published numbers. Not adjusted. Withdrew. The scanner had been measured on different inputs from the ones we would now use, and no correction recovers a number produced under different conditions.
Then we cleaned the corpus, and the rule we settled on is deliberately dumb: strip every comment, mechanically, all of them. We considered going through and removing only the ones that leaked. We decided against it, because case-by-case judgement is exactly the process that let the leak through in the first place. "grep -rc '//' samples/ returns 0" is a property anyone can verify without trusting us. "We removed the suspicious ones" is not.
Program IDs became neutral strings of the same length. Modules were renamed for their domain rather than their verdict, so vulnerable_vault became user_vault. Log strings were rewritten to say what the code does, not what it fails to do.
We ran one scan on the cleaned corpus before publishing, on the subtlest sample. It still found the bug, without the comment telling it where to look. That is the only number I will give you from the new corpus, and I am giving it as an anecdote, not a rate. We have not rebuilt the rates yet, and I would rather publish none than publish a hurried one.
Why I think this generalises
Nothing about this failure is Solana-specific, or even security-specific. Any benchmark whose items carry human-written prose can leak its labels to a model that reads the item. Comments, identifiers, docstrings, test names, string literals: all of it is input, and all of it was written by someone who knew the answer.
The contamination literature I have been able to find is mostly about train/test overlap, whether the model saw this exact item during training. This is a different failure. The model may never have seen the item. The item is just telling it what to say. And it is invisible from the outside, because the numbers it produces keep looking reasonable.
I do not have tooling for this. I would genuinely like to know if someone does.
The corpus
It is public under MIT: github.com/AIOilShield/solana-scanner-benchmark. Twenty-six programs, written ground truth for each, and the README tells this story in more detail. If you want to poke holes in it, please do. That is what it is for.
Disclosure, since it matters: we sell a scanner. It is the tool this benchmark caught out.
Top comments (0)